browsertrix/backend/btrixcloud/migrations
Ilya Kreymer 4f676e4e82
QA Runs Initial Backend Implementation (#1586)
Supports running QA Runs via the QA API!

Builds on top of the `issue-1498-crawl-qa-backend-support` branch, fixes
#1498

Also requires the latest Browsertrix Crawler 1.1.0+ (from
webrecorder/browsertrix-crawler#469 branch)

Notable changes:
- QARun objects contain info about QA runs, which are crawls
performed on data loaded from existing crawls.

- Various crawl db operations can be performed on either the crawl or
`qa.` object, and core crawl fields have been moved to CoreCrawlable.

- While running,`QARun` data stored in a single `qa` object, while
finished qa runs are added to `qaFinished` dictionary on the Crawl. The
QA list API returns data from the finished list, sorted by most recent
first.

- Includes additional type fixes / type safety, especially around
BaseCrawl / Crawl / UploadedCrawl functionality, also creating specific
get_upload(), get_basecrawl(), get_crawl() getters for internal use and
get_crawl_out() for API

- Support filtering and sorting pages via `qaFilterBy` (screenshotMatch, textMatch) 
along with `gt`, `lt`, `gte`, `lte` params to return pages based on QA results.

---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
2024-03-20 22:42:16 -07:00
..
__init__.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
migration_0001_archives_to_orgs.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0002_crawlconfig_crawlstats.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0003_mutable_crawl_configs.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0004_config_seeds.py QA Runs Initial Backend Implementation (#1586) 2024-03-20 22:42:16 -07:00
migration_0005_operator_scheduled_jobs.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0006_precompute_crawl_stats.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0007_colls_and_config_update.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0008_precompute_crawl_file_stats.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0009_crawl_types.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0010_collection_total_size.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0011_crawl_timeout_configmap.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0012_notes_to_description.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0013_crawl_name.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0014_to_collection_ids.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0015_org_storage_usage.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0016_operator_scheduled_jobs_v2.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0017_storage_by_type.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0018_usernames.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0019_org_slug.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0020_org_storage_refs.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0021_profile_filenames.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0022_partial_complete.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0023_available_extra_exec_mins.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0024_crawlerchannel.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00
migration_0025_workflow_db_configmap_fixes.py Add crawl pages and related API endpoints (#1516) 2024-02-28 12:11:35 -05:00