104ea097c4
- download via presigned URLs via requests instead of boto APIs, remove boto - follow-up to #1933 for streaming download improvements - fixes datapackage.json in multi-wacz to contain the same resources objects with: `name`, `path`, `hash`, `bytes` to match single WACZ. - Add additional metadata to multi-wacz datapackage.json, including `type` (`crawl`, `upload`, `collection`, `qaRun`), `id` (unique id for the object), `title` / `description` if available (for crawl/upload/collection), and `crawlId` for `qaRun`
31 lines
457 B
Plaintext
31 lines
457 B
Plaintext
gunicorn
|
|
uvicorn[standard]
|
|
fastapi==0.103.2
|
|
motor
|
|
passlib
|
|
PyJWT==2.8.0
|
|
pydantic==2.8.2
|
|
email-validator
|
|
loguru
|
|
aiofiles
|
|
kubernetes-asyncio==29.0.0
|
|
kubernetes
|
|
aiobotocore
|
|
requests
|
|
redis>=5.0.0
|
|
pyyaml
|
|
jinja2
|
|
humanize
|
|
python-multipart
|
|
pathvalidate
|
|
https://github.com/ikreymer/stream-zip/archive/refs/heads/crc32-optional.zip
|
|
backoff>=2.2.1
|
|
python-slugify>=8.0.1
|
|
types_aiobotocore_s3
|
|
types-redis
|
|
types-python-slugify
|
|
types-pyYAML
|
|
remotezip
|
|
json-stream
|
|
aiostream
|