browsertrix/backend
Ilya Kreymer 8ea16393c5
Optimize single-page crawl workflows (#2656)
For single page crawls:
- Always force 1 browser to be used, ignoring browser windows/scale
setting
- Don't use custom PVC volumes in crawler / redis, just use emptyDir -
no chance of crawler being interrupted and restarted on different
machine for a single page.

Adds a 'is_single_page' check to CrawlConfig, checking for either limit
or scopeType / no extra hops.

Fixes #2655
2025-06-10 12:13:57 -07:00
..
btrixcloud Optimize single-page crawl workflows (#2656) 2025-06-10 12:13:57 -07:00
test Optimize single-page crawl workflows (#2656) 2025-06-10 12:13:57 -07:00
test_nightly Fix nightly tests (#2460) 2025-03-06 16:23:30 -08:00
.pylintrc security: tweak get /invite endpoints / InviteOut to: (#2087) 2024-09-20 11:52:56 -07:00
dev-requirements.txt
Dockerfile Add backend support for custom behaviors + validation endpoint (#2505) 2025-04-02 16:20:51 -07:00
mypy.ini
requirements.txt Add ISO-639-1 language code validation to backend (#2602) 2025-05-13 16:54:33 -04:00
test-requirements.txt Fix nightly tests: Add boto3 as test requirement (#2116) 2024-10-23 13:41:22 -07:00