f8fb2d2c8d
Fixes #2406 Converts migration 0042 to launch a background job (parallelized across several pods) to migrate all crawls by optimizing their pages and setting `version: 2` on the crawl when complete. Also Optimizes MongoDB queries for better performance. Migration Improvements: - Add `isMigrating` and `version` fields to `BaseCrawl` - Add new background job type to use in migration with accompanying `migration_job.yaml` template that allows for parallelization - Add new API endpoint to launch this crawl migration job, and ensure that we have list and retry endpoints for superusers that work with background jobs that aren't tied to a specific org - Rework background job models and methods now that not all background jobs are tied to a single org - Ensure new crawls and uploads have `version` set to `2` - Modify crawl and collection replay.json endpoints to only include fields for replay optimization (`initialPages`, `pageQueryUrl`, `preloadResources`) if all relevant crawls/uploads have `version` set to `2` - Remove `distinct` calls from migration pathways - Consolidate collection recompute stats Query Optimizations: - Remove all uses of $group and $facet - Optimize /replay.json endpoints to precompute preload_resources, avoid fetching crawl list twice - Optimize /collections endpoint by not fetching resources - Rename /urls -> /pageUrlCounts and avoid $group, instead sort with index, either by seed + ts or by url to get top matches. - Use $gte instead of $regex to get prefix matches on URL - Use $text instead of $regex to get text search on title - Remove total from /pages and /pageUrlCounts queries by not using $facet - frontend: only call /pageUrlCounts when dialog is opened. --------- Co-authored-by: Ilya Kreymer <ikreymer@gmail.com> Co-authored-by: Emma Segal-Grossman <hi@emma.cafe> Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
29 lines
621 B
Plaintext
29 lines
621 B
Plaintext
Failed Background Job
|
|
~~~
|
|
Failed Background Job
|
|
---------------------
|
|
{% if org %}
|
|
Organization: {{ org.name }} ({{ job.oid }})
|
|
{% endif %}
|
|
Job type: {{ job.type }}
|
|
|
|
Job ID: {{ job.id }}
|
|
Started: {{ job.started.isoformat(sep=" ", timespec="seconds") }}Z
|
|
Finished: {{ finished.isoformat(sep=" ", timespec="seconds") }}Z
|
|
|
|
{% if job.object_type %}
|
|
Object type: {{ job.object_type }}
|
|
{% endif %}
|
|
|
|
{% if job.object_id %}
|
|
Object ID: {{ job.object_id }}
|
|
{% endif %}
|
|
|
|
{% if job.file_path %}
|
|
File path: {{ job.file_path }}
|
|
{% endif %}
|
|
|
|
{% if job.replica_storage %}
|
|
Replica storage name: {{ job.replica_storage.name }}
|
|
{% endif %}
|