browsertrix/backend
Ilya Kreymer 94e985ae13
optimize org quota lookups (#1973)
- instead of looking up storage and exec min quotas from oid, and
loading an org each time, load org once and then check quotas on the org
object - many times the org was already available, and was looked up
again
- storage and exec quota checks become sync
- rename can_run_crawl() to more generic can_write_data(), optionally
also checks exec minutes
- typing: get_org_by_id() always returns org, or throws, adjust methods
accordingly (don't check for none, catch exception)
- typing: fix typo in BaseOperator, catch type errors in operator
'org_ops'
- operator quota check: use up-to-date 'status.size' for current job,
ignore current job in all jobs list to avoid double-counting
- follow up to #1969
2024-07-25 14:00:16 -07:00
..
btrixcloud optimize org quota lookups (#1973) 2024-07-25 14:00:16 -07:00
test Implement downloading archived item + QA runs as multi-WACZ (#1933) 2024-07-25 10:28:57 -07:00
test_nightly Standardize handling of storage and execution time quotas (#1969) 2024-07-25 12:49:11 -07:00
.pylintrc Pydantic 2.x update + type fixes + python 3.12 (#1947) 2024-07-22 17:23:03 -07:00
dev-requirements.txt quickfix: pin mypy version to avoid issues with latest release 2024-07-19 18:30:57 -07:00
Dockerfile Pydantic 2.x update + type fixes + python 3.12 (#1947) 2024-07-22 17:23:03 -07:00
mypy.ini Support multiple crawler versions (#1420) 2024-01-16 15:32:12 -08:00
requirements.txt Implement downloading archived item + QA runs as multi-WACZ (#1933) 2024-07-25 10:28:57 -07:00
test-requirements.txt Add slugs to org backend (#1250) 2023-10-10 18:30:09 -07:00