browsertrix/backend
Ilya Kreymer 3cd52342a7
Remove Crawl Workflow Configmaps (#1894)
Fixes #1893 

- Removes crawl workflow-scoped configmaps, and replaces with operator-controlled
per-crawl configmaps that only contain the json config passed to Browsertrix
Crawler (as a volume).
- Other configmap settings replaced are replaced the custom CrawlJob options
(mostly already were, just added profile_filename and storage_filename)
- Cron jobs also updated to create CrawlJob without relying on configmaps,
querying the db for additional settings.
- The `userid` associated with cron jobs is set to the user that last modified
 the schedule of the crawl, rather than whomever last modified the workflow
- Various functions that deal with updating configmaps have been removed,
including in migrations.
- New migration 0029 added to remove all crawl workflow configmaps
2024-06-28 15:25:23 -07:00
..
btrixcloud Remove Crawl Workflow Configmaps (#1894) 2024-06-28 15:25:23 -07:00
test Add slug validation and test (#1891) 2024-06-26 15:04:54 -04:00
test_nightly Give test_crawl_timeout 10 mins to finish (#1627) 2024-03-26 18:33:30 -07:00
.pylintrc
Dockerfile Backend mem usage fix - use fixed MOTOR_MAX_WORKERS + switch to gunicorn (#1468) 2024-01-16 15:32:42 -08:00
mypy.ini Support multiple crawler versions (#1420) 2024-01-16 15:32:12 -08:00
requirements.txt Add endpoints to read pages from older crawl WACZs into database (#1562) 2024-03-19 14:14:21 -07:00
test-requirements.txt