browsertrix/backend/btrixcloud
Tessa Walsh 094f27bcff
Track bytes stored per file type and include in org metrics (#1207)
* Add bytes stored per type to org and metrics

The org now tracks bytesStored by type of crawl, uploads, and browser profiles
in addition to the total, and returns these values in the org metrics endpoint.

A migration is added to precompute these values in existing deployments.

In addition, all /metrics storage values are now returned solely as bytes, as
the GB form wasn't being used in the frontend and is unnecessary.

* Improve deletion of multiple archived item types via `/all-crawls` delete endpoint

- Update `/all-crawls` delete test to check that org and workflow size values
are correct following deletion.
- Fix bug where it was always assumed only one crawl was deleted per cid
and size was not tracked per cid
- Add type check within delete_crawls
2023-09-22 12:55:21 -04:00
..
migrations Track bytes stored per file type and include in org metrics (#1207) 2023-09-22 12:55:21 -04:00
templates Scheduled Crawl Refactor: Handle via Operator + Add Skipped Crawls on Quota Reached (#1162) 2023-09-12 13:05:43 -07:00
__init__.py
basecrawls.py Track bytes stored per file type and include in org metrics (#1207) 2023-09-22 12:55:21 -04:00
colls.py Improved type checking for backend with mypy (#1174) 2023-09-13 19:40:26 -07:00
crawlconfigs.py Track bytes stored per file type and include in org metrics (#1207) 2023-09-22 12:55:21 -04:00
crawlmanager.py Scheduled Crawl Refactor: Handle via Operator + Add Skipped Crawls on Quota Reached (#1162) 2023-09-12 13:05:43 -07:00
crawls.py Track bytes stored per file type and include in org metrics (#1207) 2023-09-22 12:55:21 -04:00
db.py Track bytes stored per file type and include in org metrics (#1207) 2023-09-22 12:55:21 -04:00
emailsender.py feat: add SMTP {port, use_tls} config (#1142) 2023-09-08 08:18:36 -07:00
invites.py Improved type checking for backend with mypy (#1174) 2023-09-13 19:40:26 -07:00
k8sapi.py ensure max crawl size and max crawl timeout values are set to 0 when unused, instead of null (#1167) 2023-09-13 09:51:26 -07:00
main_op.py Refactor / Cleanup: move ops functions back into classes (#1171) 2023-09-13 11:56:09 -07:00
main.py Add event webhook notifications system to backend (#1061) 2023-08-31 19:52:37 -07:00
models.py Track bytes stored per file type and include in org metrics (#1207) 2023-09-22 12:55:21 -04:00
operator.py Track bytes stored per file type and include in org metrics (#1207) 2023-09-22 12:55:21 -04:00
orgs.py Track bytes stored per file type and include in org metrics (#1207) 2023-09-22 12:55:21 -04:00
pagination.py Move pydantic models to separate module + refactor crawl response endpoints to be consistent (#983) 2023-07-20 13:05:33 +02:00
profiles.py Track bytes stored per file type and include in org metrics (#1207) 2023-09-22 12:55:21 -04:00
storages.py Implement sync streaming for finished crawl logs (#1168) 2023-09-14 17:05:19 -07:00
uploads.py Track bytes stored per file type and include in org metrics (#1207) 2023-09-22 12:55:21 -04:00
users.py Improved type checking for backend with mypy (#1174) 2023-09-13 19:40:26 -07:00
utils.py stopping fix: backend fixes for #836 + prep for additional status fields (#837) 2023-05-08 14:02:20 -07:00
version.py bump version to 1.7.0-beta.1 2023-09-18 14:33:03 -07:00
webhooks.py Improved type checking for backend with mypy (#1174) 2023-09-13 19:40:26 -07:00
zip.py Implement sync streaming for finished crawl logs (#1168) 2023-09-14 17:05:19 -07:00