6dc452ebad
- Refactors storage to support replicas + custom storages on the Org. - There is a default primary + replica storage, while an Org can also have primary and replica storages. - StorageRef object is used to store references to default and custom storage. - CrawlFile has been updated to contain a StorageRef instead of a def_storage_name, which references either a default storage (in StorageOps) or custom storage (in Organization) - There is also a 'replicas' Optional[List[StorageRef]] which contains replicas, if any. - CrawlFileOut contain a numReplicas for how many replicas exist for a given file. - Migration: migration 0020 added to migrate existing Orgs, CrawlFile and ProfileFile objects to new storage system (CrawlFile and ProfileFile now extend BaseFile) Part of #1262 --------- Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
27 lines
504 B
Plaintext
27 lines
504 B
Plaintext
uvicorn
|
|
fastapi==0.103.2
|
|
motor==3.3.1
|
|
passlib
|
|
PyJWT==2.8.0
|
|
pydantic==1.10.13
|
|
email-validator
|
|
#fastapi-users[mongodb]==9.2.2
|
|
loguru
|
|
aiofiles
|
|
kubernetes-asyncio==25.11.0
|
|
kubernetes
|
|
aiobotocore
|
|
redis>=5.0.0
|
|
pyyaml
|
|
jinja2
|
|
humanize
|
|
python-multipart
|
|
pathvalidate
|
|
#https://github.com/ikreymer/stream-zip/archive/refs/heads/stream-uncompress.zip
|
|
https://github.com/ikreymer/stream-zip/archive/refs/heads/stream-ignore-local-crc32.zip
|
|
boto3
|
|
backoff>=2.2.1
|
|
python-slugify>=8.0.1
|
|
mypy_boto3_s3
|
|
types_aiobotocore_s3
|