browsertrix/backend
Ilya Kreymer 104ea097c4
switch to simpler streaming download + multiwacz metadata improvements: (#1982)
- download via presigned URLs via requests instead of boto APIs, remove boto
- follow-up to #1933 for streaming download improvements
- fixes datapackage.json in multi-wacz to contain the same resources
objects with: `name`, `path`, `hash`, `bytes` to match single WACZ.
- Add additional metadata to multi-wacz datapackage.json, including `type`
(`crawl`, `upload`, `collection`, `qaRun`), `id` (unique id for the
object), `title` / `description` if available (for
crawl/upload/collection), and `crawlId` for `qaRun`
2024-10-03 16:13:31 -07:00
..
btrixcloud switch to simpler streaming download + multiwacz metadata improvements: (#1982) 2024-10-03 16:13:31 -07:00
test switch to simpler streaming download + multiwacz metadata improvements: (#1982) 2024-10-03 16:13:31 -07:00
test_nightly Standardize handling of storage and execution time quotas (#1969) 2024-07-25 12:49:11 -07:00
.pylintrc security: tweak get /invite endpoints / InviteOut to: (#2087) 2024-09-20 11:52:56 -07:00
dev-requirements.txt quickfix: pin mypy version to avoid issues with latest release 2024-07-19 18:30:57 -07:00
Dockerfile Pydantic 2.x update + type fixes + python 3.12 (#1947) 2024-07-22 17:23:03 -07:00
mypy.ini Support multiple crawler versions (#1420) 2024-01-16 15:32:12 -08:00
requirements.txt switch to simpler streaming download + multiwacz metadata improvements: (#1982) 2024-10-03 16:13:31 -07:00
test-requirements.txt Add slugs to org backend (#1250) 2023-10-10 18:30:09 -07:00