browsertrix/chart/app-templates
Tessa Walsh f8fb2d2c8d
Rework crawl page migration + MongoDB Query Optimizations (#2412)
Fixes #2406 

Converts migration 0042 to launch a background job (parallelized across
several pods) to migrate all crawls by optimizing their pages and
setting `version: 2` on the crawl when complete.

Also Optimizes MongoDB queries for better performance.

Migration Improvements:

- Add `isMigrating` and `version` fields to `BaseCrawl`
- Add new background job type to use in migration with accompanying
`migration_job.yaml` template that allows for parallelization
- Add new API endpoint to launch this crawl migration job, and ensure
that we have list and retry endpoints for superusers that work with
background jobs that aren't tied to a specific org
- Rework background job models and methods now that not all background
jobs are tied to a single org
- Ensure new crawls and uploads have `version` set to `2`
- Modify crawl and collection replay.json endpoints to only include
fields for replay optimization (`initialPages`, `pageQueryUrl`,
`preloadResources`) if all relevant crawls/uploads have `version` set to
`2`
- Remove `distinct` calls from migration pathways
- Consolidate collection recompute stats

Query Optimizations:
- Remove all uses of $group and $facet
- Optimize /replay.json endpoints to precompute preload_resources, avoid
fetching crawl list twice
- Optimize /collections endpoint by not fetching resources 
- Rename /urls -> /pageUrlCounts and avoid $group, instead sort with
index, either by seed + ts or by url to get top matches.
- Use $gte instead of $regex to get prefix matches on URL
- Use $text instead of $regex to get text search on title
- Remove total from /pages and /pageUrlCounts queries by not using
$facet
- frontend: only call /pageUrlCounts when dialog is opened.


---------

Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
2025-02-20 15:26:11 -08:00
..
background_job.yaml Rework crawl page migration + MongoDB Query Optimizations (#2412) 2025-02-20 15:26:11 -08:00
crawl_configmap.yaml Remove Crawl Workflow Configmaps (#1894) 2024-06-28 15:25:23 -07:00
crawl_cron_job.yaml Remove Crawl Workflow Configmaps (#1894) 2024-06-28 15:25:23 -07:00
crawl_job.yaml Configure browsertrix proxies (#1847) 2024-10-02 18:35:45 -07:00
crawler.yaml ensure running crawl configmap is updated when exclusions are added/removed (#2409) 2025-02-19 11:42:19 -08:00
profile_job.yaml Configure browsertrix proxies (#1847) 2024-10-02 18:35:45 -07:00
profilebrowser.yaml Configure browsertrix proxies (#1847) 2024-10-02 18:35:45 -07:00
qa_configmap.yaml include workflow config in QA runs + different browser instances for QA (#1829) 2024-05-29 13:32:25 -07:00
redis.yaml Move backend app templates to be installed from configmap volume (#1331) 2023-11-06 09:37:48 -08:00
replica_deletion_cron_job.yaml Optionally delay replica deletion (#2252) 2024-12-19 18:50:28 -08:00
replica_job.yaml QA Runs Initial Backend Implementation (#1586) 2024-03-20 22:42:16 -07:00