browsertrix/backend/btrixcloud
Tessa Walsh 21ae38362e
Add endpoints to read pages from older crawl WACZs into database (#1562)
Fixes #1597

New endpoints (replacing old migration) to re-add crawl pages to db from
WACZs.

After a few implementation attempts, we settled on using
[remotezip](https://github.com/gtsystem/python-remotezip) to handle
parsing of the zip files and streaming their contents line-by-line for
pages. I've also modified the sync log streaming to use remotezip as
well, which allows us to remove our own zip module and let remotezip
handle the complexity of parsing zip files.

Database inserts for pages from WACZs are batched 100 at a time to help
speed up the endpoint, and the task is kicked off using
asyncio.create_task so as not to block before giving a response.

StorageOps now contains a method for streaming the bytes of any file in
a remote WACZ, requiring only the presigned URL for the WACZ and the
name of the file to stream.
2024-03-19 14:14:21 -07:00
..
migrations Temporarily remove pages migration (#1572) 2024-03-04 10:30:04 -08:00
operator profile browser fixes: better resource usage + load retry (main) (#1604) 2024-03-16 15:07:04 -07:00
__init__.py
auth.py
background_jobs.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
basecrawls.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
colls.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
crawlconfigs.py More friendly WARC prefix inside WACZ based on Org slug + Crawl Name / First Seed URL. (#1537) 2024-02-22 23:54:23 -08:00
crawlmanager.py More friendly WARC prefix inside WACZ based on Org slug + Crawl Name / First Seed URL. (#1537) 2024-02-22 23:54:23 -08:00
crawls.py Add endpoints to read pages from older crawl WACZs into database (#1562) 2024-03-19 14:14:21 -07:00
db.py Temporarily remove pages migration (#1572) 2024-03-04 10:30:04 -08:00
emailsender.py
invites.py
k8sapi.py kubernetes api: avoid overriding content-type header in kubernetes-asyncio, pass in via arg instead (main) (#1605) 2024-03-18 11:17:54 -07:00
main_op.py Refactor operator class into module (#1564) 2024-02-29 14:40:12 -08:00
main.py Configurable Max Scale for frontend (#1557) 2024-03-11 16:21:20 -07:00
models.py Add updatable QA reviewStatus field to crawls (#1575) 2024-03-05 16:49:23 -08:00
orgs.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
pages.py Add endpoints to read pages from older crawl WACZs into database (#1562) 2024-03-19 14:14:21 -07:00
pagination.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
profiles.py
storages.py Add endpoints to read pages from older crawl WACZs into database (#1562) 2024-03-19 14:14:21 -07:00
uploads.py
users.py
utils.py Add endpoints to read pages from older crawl WACZs into database (#1562) 2024-03-19 14:14:21 -07:00
version.py version: bump to 1.10.0-beta.0 2024-02-20 00:22:29 -08:00
webhooks.py