f8fb2d2c8d
Fixes #2406 Converts migration 0042 to launch a background job (parallelized across several pods) to migrate all crawls by optimizing their pages and setting `version: 2` on the crawl when complete. Also Optimizes MongoDB queries for better performance. Migration Improvements: - Add `isMigrating` and `version` fields to `BaseCrawl` - Add new background job type to use in migration with accompanying `migration_job.yaml` template that allows for parallelization - Add new API endpoint to launch this crawl migration job, and ensure that we have list and retry endpoints for superusers that work with background jobs that aren't tied to a specific org - Rework background job models and methods now that not all background jobs are tied to a single org - Ensure new crawls and uploads have `version` set to `2` - Modify crawl and collection replay.json endpoints to only include fields for replay optimization (`initialPages`, `pageQueryUrl`, `preloadResources`) if all relevant crawls/uploads have `version` set to `2` - Remove `distinct` calls from migration pathways - Consolidate collection recompute stats Query Optimizations: - Remove all uses of $group and $facet - Optimize /replay.json endpoints to precompute preload_resources, avoid fetching crawl list twice - Optimize /collections endpoint by not fetching resources - Rename /urls -> /pageUrlCounts and avoid $group, instead sort with index, either by seed + ts or by url to get top matches. - Use $gte instead of $regex to get prefix matches on URL - Use $text instead of $regex to get text search on title - Remove total from /pages and /pageUrlCounts queries by not using $facet - frontend: only call /pageUrlCounts when dialog is opened. --------- Co-authored-by: Ilya Kreymer <ikreymer@gmail.com> Co-authored-by: Emma Segal-Grossman <hi@emma.cafe> Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
25 lines
649 B
YAML
25 lines
649 B
YAML
apiVersion: v2
|
|
name: browsertrix
|
|
description: A chart for running the Webrecorder Browsertrix System
|
|
type: application
|
|
icon: https://webrecorder.net/assets/icon.png
|
|
|
|
# Browsertrix and Chart Version
|
|
version: v1.14.0-beta.5
|
|
|
|
dependencies:
|
|
- name: btrix-admin-logging
|
|
version: 0.1.0
|
|
condition: addons.admin.logging
|
|
repository: file://./admin/logging
|
|
- name: btrix-crds
|
|
version: 0.1.1
|
|
repository: file://./btrix-crds
|
|
- name: metacontroller-helm
|
|
version: 4.11.11
|
|
repository: "oci://ghcr.io/metacontroller"
|
|
- name: btrix-proxies
|
|
version: 0.1.0
|
|
condition: btrix-proxies.enabled
|
|
repository: file://./proxies/
|