4f676e4e82
Supports running QA Runs via the QA API! Builds on top of the `issue-1498-crawl-qa-backend-support` branch, fixes #1498 Also requires the latest Browsertrix Crawler 1.1.0+ (from webrecorder/browsertrix-crawler#469 branch) Notable changes: - QARun objects contain info about QA runs, which are crawls performed on data loaded from existing crawls. - Various crawl db operations can be performed on either the crawl or `qa.` object, and core crawl fields have been moved to CoreCrawlable. - While running,`QARun` data stored in a single `qa` object, while finished qa runs are added to `qaFinished` dictionary on the Crawl. The QA list API returns data from the finished list, sorted by most recent first. - Includes additional type fixes / type safety, especially around BaseCrawl / Crawl / UploadedCrawl functionality, also creating specific get_upload(), get_basecrawl(), get_crawl() getters for internal use and get_crawl_out() for API - Support filtering and sorting pages via `qaFilterBy` (screenshotMatch, textMatch) along with `gt`, `lt`, `gte`, `lte` params to return pages based on QA results. --------- Co-authored-by: Tessa Walsh <tessa@bitarchivist.net> |
||
|---|---|---|
| .github | ||
| .vscode | ||
| ansible | ||
| assets | ||
| backend | ||
| chart | ||
| configs | ||
| docs | ||
| frontend | ||
| scripts | ||
| test | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| btrix | ||
| CHANGES.md | ||
| LICENSE | ||
| mkdocs.yml | ||
| NOTICE | ||
| pylintrc | ||
| README.md | ||
| update-version.sh | ||
| version.txt | ||
| yarn.lock | ||
Browsertrix is an open-source cloud-native high-fidelity browser-based crawling service designed to make web archiving easier and more accessible for everyone.
The service provides an API and UI for scheduling crawls and viewing results, and managing all aspects of crawling process. This system provides the orchestration and management around crawling, while the actual crawling is performed using Browsertrix Crawler containers, which are launched for each crawl.
See browsertrix.com for a feature overview and information about Browsertrix hosting.
Documentation
The full docs for using, deploying, and developing Browsertrix are available at: https://docs.browsertrix.cloud
Deployment
The latest deployment documentation is available at: https://docs.browsertrix.cloud/deploy
The docs cover deploying Browsertrix in different environments using Kubernetes, from a single-node setup to scalable clusters in the cloud.
Previously, Browsertrix also supported Docker Compose and podman-based deployment. This has been deprecated due to the complexity of maintaining feature parity across different setups, and with various Kubernetes deployment options being available and easy to deploy, even on a single machine.
Making deployment of Browsertrix as easy as possible remains a key goal, and we welcome suggestions for how we can further improve our Kubernetes deployment options.
If you are looking to just try running a single crawl, you may want to try Browsertrix Crawler first to test out the crawling capabilities.
Development Status
Browsertrix is currently in a beta, though the system and backend API is fairly stable, we are working on many additional features.
Additional developer documentation is available at https://docs.browsertrix.cloud/develop
Please see the GitHub issues and this GitHub Project for our current project plan and tasks.
License
Browsertrix is made available under the AGPLv3 License.
Documentation is made available under the Creative Commons Attribution 4.0 International License