browsertrix/backend
Tessa Walsh d38abbca7f
Standardize handling of storage and execution time quotas (#1969)
Fixes #1968 

Changes:
- `stopped_quota_reached` and `skipped_quota_reached` migrated to new
values that indicate which quota was reached
- Before crawls are run, the operator checks if storage or exec mins
quotas are reached and if so fails the crawl with the appropriate state
of `skipped_storage_quota_reached` or `skipped_time_quota_reached`
- While crawls are running, the operator checks if the exec mins quota
is reached or if the size of all running crawls will mean the storage
quota is reached once uploaded; if so, the crawl is stopped gracefully
and given `stopped_storage_quota_needed` or `stopped_time_quota_reached`
state as appropriate
- Adds new nightly tests for enforcing storage quota
2024-07-25 12:49:11 -07:00
..
btrixcloud Standardize handling of storage and execution time quotas (#1969) 2024-07-25 12:49:11 -07:00
test Implement downloading archived item + QA runs as multi-WACZ (#1933) 2024-07-25 10:28:57 -07:00
test_nightly Standardize handling of storage and execution time quotas (#1969) 2024-07-25 12:49:11 -07:00
.pylintrc Pydantic 2.x update + type fixes + python 3.12 (#1947) 2024-07-22 17:23:03 -07:00
dev-requirements.txt quickfix: pin mypy version to avoid issues with latest release 2024-07-19 18:30:57 -07:00
Dockerfile Pydantic 2.x update + type fixes + python 3.12 (#1947) 2024-07-22 17:23:03 -07:00
mypy.ini Support multiple crawler versions (#1420) 2024-01-16 15:32:12 -08:00
requirements.txt Implement downloading archived item + QA runs as multi-WACZ (#1933) 2024-07-25 10:28:57 -07:00
test-requirements.txt