browsertrix/backend/btrixcloud
Ilya Kreymer c9c39d47b7
Scheduled Crawl Refactor: Handle via Operator + Add Skipped Crawls on Quota Reached (#1162)
* use metacontroller's decoratorcontroller to create CrawlJob from Job
* scheduled job work:
- use existing job name for scheduled crawljob
- use suspended job, set startTime, completionTime and succeeded status on job when crawljob is done
- simplify cronjob template: remove job_image, cron_namespace, using same namespace as crawls,
placeholder job image for cronjobs

* move storage quota check to crawljob handler:
- add 'skipped_quota_reached' as new failed status type
- check for storage quota before checking if crawljob can be started, fail if not (check before any pods/pvcs created)

* frontend:
- show all crawls in crawl workflow, no need to filter by status
- add 'skipped_quota_reached' status, show as 'Skipped (Quota Reached)', render same as failed

* migration: make release namespace available as DEFAULT_NAMESPACE, delete old cronjobs in DEFAULT_NAMESPACE and recreate in crawlers namespace with new template
2023-09-12 13:05:43 -07:00
..
migrations Scheduled Crawl Refactor: Handle via Operator + Add Skipped Crawls on Quota Reached (#1162) 2023-09-12 13:05:43 -07:00
templates Scheduled Crawl Refactor: Handle via Operator + Add Skipped Crawls on Quota Reached (#1162) 2023-09-12 13:05:43 -07:00
__init__.py
basecrawls.py Scheduled Crawl Refactor: Handle via Operator + Add Skipped Crawls on Quota Reached (#1162) 2023-09-12 13:05:43 -07:00
colls.py Add event webhook notifications system to backend (#1061) 2023-08-31 19:52:37 -07:00
crawlconfigs.py Scheduled Crawl Refactor: Handle via Operator + Add Skipped Crawls on Quota Reached (#1162) 2023-09-12 13:05:43 -07:00
crawlmanager.py Scheduled Crawl Refactor: Handle via Operator + Add Skipped Crawls on Quota Reached (#1162) 2023-09-12 13:05:43 -07:00
crawls.py Add and enforce org storage quota (#1106) 2023-09-07 12:45:43 -04:00
db.py Scheduled Crawl Refactor: Handle via Operator + Add Skipped Crawls on Quota Reached (#1162) 2023-09-12 13:05:43 -07:00
emailsender.py feat: add SMTP {port, use_tls} config (#1142) 2023-09-08 08:18:36 -07:00
invites.py
k8sapi.py Scheduled Crawl Refactor: Handle via Operator + Add Skipped Crawls on Quota Reached (#1162) 2023-09-12 13:05:43 -07:00
main_op.py Add event webhook notifications system to backend (#1061) 2023-08-31 19:52:37 -07:00
main.py Add event webhook notifications system to backend (#1061) 2023-08-31 19:52:37 -07:00
models.py Issue all non-upload storage-quota-update events from LiteElement (#1151) 2023-09-11 18:17:48 -07:00
operator.py Scheduled Crawl Refactor: Handle via Operator + Add Skipped Crawls on Quota Reached (#1162) 2023-09-12 13:05:43 -07:00
orgs.py Add and enforce org storage quota (#1106) 2023-09-07 12:45:43 -04:00
pagination.py
profiles.py Operator refactor to control pods + pvcs directly instead of statefulsets (#1149) 2023-09-11 10:38:04 -07:00
storages.py
uploads.py Add and enforce org storage quota (#1106) 2023-09-07 12:45:43 -04:00
users.py
utils.py
version.py bump version to 1.7.0-beta.0 2023-08-23 12:03:45 -07:00
webhooks.py Add event webhook notifications system to backend (#1061) 2023-08-31 19:52:37 -07:00
zip.py