browsertrix/backend/btrixcloud
Ilya Kreymer cb50c7c2c2
Pause / Resume Crawls Initial Implmentation. (#2572)
- add 'pause' crawl state (fixes #2567)
- gracefully shut down crawler pods, and then redis pod when paused
- crawler uploads WACZ before shutting down (dependent on
webrecorder/browsertrix-crawler#824, supported in 1.6.1+)
- add 'paused_at' on crawl spec to indicate when crawl is paused
- support max pause time limit, after which crawl becomes automatically
stopped.
- add 'stopped_pause_expired' when pause automatically expires and crawl
is stopped
- /crawl/<id>/{pause,resume} apis to toggle 'paused' on crawl spec
- ui: add pause/resume button, paused state (partially addresses #2568)
- ui: add pausing/resuming derivative states when crawl is running and
pausing, or paused and not pausing (partially addresses #2569)
- Designed to work with crawler 1.6.1+ which support pausing + uploading on pause

Work on #2566, Fixes #2576 

---------
Co-authored-by: sua yoo <sua@webrecorder.org>
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Co-authored-by: sua yoo <sua@suayoo.com>
2025-05-21 14:05:16 -07:00
..
migrations Add ISO-639-1 language code validation to backend (#2602) 2025-05-13 16:54:33 -04:00
operator Pause / Resume Crawls Initial Implmentation. (#2572) 2025-05-21 14:05:16 -07:00
__init__.py
auth.py fixes token lifetime bug / improve security (#2490) 2025-03-19 10:07:09 -07:00
background_jobs.py Rework crawl page migration + MongoDB Query Optimizations (#2412) 2025-02-20 15:26:11 -08:00
basecrawls.py Pause / Resume Crawls Initial Implmentation. (#2572) 2025-05-21 14:05:16 -07:00
colls.py remove deleted collections from crawlconfigs (#2615) 2025-05-20 18:38:40 -07:00
crawlconfigs.py Pause / Resume Crawls Initial Implmentation. (#2572) 2025-05-21 14:05:16 -07:00
crawlmanager.py Pause / Resume Crawls Initial Implmentation. (#2572) 2025-05-21 14:05:16 -07:00
crawls.py Pause / Resume Crawls Initial Implmentation. (#2572) 2025-05-21 14:05:16 -07:00
db.py Add ISO-639-1 language code validation to backend (#2602) 2025-05-13 16:54:33 -04:00
emailsender.py Rework crawl page migration + MongoDB Query Optimizations (#2412) 2025-02-20 15:26:11 -08:00
invites.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
k8sapi.py Fixes #2488 (#2493) 2025-03-19 10:06:25 -07:00
main_bg.py move db migrations to initContainers: (#2449) 2025-03-03 13:13:15 -08:00
main_migrations.py move db migrations to initContainers: (#2449) 2025-03-03 13:13:15 -08:00
main_op.py move db migrations to initContainers: (#2449) 2025-03-03 13:13:15 -08:00
main.py Pause / Resume Crawls Initial Implmentation. (#2572) 2025-05-21 14:05:16 -07:00
models.py Pause / Resume Crawls Initial Implmentation. (#2572) 2025-05-21 14:05:16 -07:00
ops.py move db migrations to initContainers: (#2449) 2025-03-03 13:13:15 -08:00
orgs.py Add ISO-639-1 language code validation to backend (#2602) 2025-05-13 16:54:33 -04:00
pages.py Add Org Check for Collection access (#2616) 2025-05-20 15:30:22 -07:00
pagination.py
profiles.py support overriding crawler image pull policy per channel (#2523) 2025-03-31 14:11:41 -07:00
storages.py Optimize presigning for replay.json (#2516) 2025-05-20 12:09:35 -07:00
subs.py Add API endpoint to check if subscription is activated (#2582) 2025-05-06 17:36:58 -07:00
uploads.py Add Org Check for Collection access (#2616) 2025-05-20 15:30:22 -07:00
users.py Fix user emails use userout (#2511) 2025-03-24 12:04:39 -07:00
utils.py Add ISO-639-1 language code validation to backend (#2602) 2025-05-13 16:54:33 -04:00
version.py version: bump to 1.16.2 2025-05-20 18:43:22 -07:00
webhooks.py Better cacheing of presigned URLs + support for thumbnails (#2446) 2025-03-03 12:05:23 -08:00