6797b41de0
Fixes #2257 This is a follow-up to the public collections work, which adds pages to the database for uploads. All crawls and uploads now have a `pageCount` field which is populated when the item is successfully added. A new migration is also added to populate the field for existing archived items that don't have it set yet. OrgMetrics have also been modified to include `crawlPageCount` and `uploadPageCount`, and to include the total of both in `pageCount`, and all three included in the frontend org dashboard. The frontend has been updated to use `pageCount` rather than `stats.done` wherever appropriate, meaning that in archived item lists and details we now have a consistent page count for both crawls and uploads. ### New functionality - Deploy this branch - Create new crawls and uploads and verify that page count appears correctly throughout the frontend for all new crawls and uploads ### Migration - Deploy from latest main - Create some crawls and uploads - Change to this branch and re-deploy - Verify migration ran without errors in backend logs - Verify that page count has been populated successfully by checking archived items lists, crawl and upload detail pages, and dashboard to ensure there are no longer any missing page counts. --------- Co-authored-by: emma <hi@emma.cafe> |
||
|---|---|---|
| .github | ||
| .vscode | ||
| ansible | ||
| assets | ||
| backend | ||
| chart | ||
| configs | ||
| frontend | ||
| scripts | ||
| test | ||
| .gitattributes | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| btrix | ||
| CHANGES.md | ||
| LICENSE | ||
| NOTICE | ||
| pylintrc | ||
| README.md | ||
| update-version.sh | ||
| version.txt | ||
Browsertrix is a cloud-native, high-fidelity, browser-based crawling service designed to make web archiving easier and more accessible for everyone.
The service provides an API and UI for scheduling crawls and viewing results, and managing all aspects of crawling process. This system provides the orchestration and management around crawling, while the actual crawling is performed using Browsertrix Crawler containers, which are launched for each crawl.
See webrecorder.net/browsertrix for a feature overview and information about how to sign up for Webrecorder's hosted Browsertrix service.
Documentation
The full docs for using, deploying, and developing Browsertrix are available at docs.browsertrix.com.
Our docs are created with Material for MKDocs.
Deployment
The latest deployment documentation is available at docs.browsertrix.com/deploy.
The docs cover deploying Browsertrix in different environments using Kubernetes, from a single-node setup to scalable clusters in the cloud.
Early on, Browsertrix also supported Docker Compose and podman-based deployment. This was deprecated due to the complexity of maintaining feature parity across different setups, and with various Kubernetes deployment options being available and easy to deploy, even on a single machine.
Making deployment of Browsertrix as easy as possible remains a key goal, and we welcome suggestions for how we can further improve our Kubernetes deployment options.
If you are looking to just try running a single crawl, you may want to try Browsertrix Crawler first to test out the crawling capabilities.
Contributing
Though the system and backend API is fairly stable, we are working on many additional features. Please see the GitHub issues and this GitHub Project for our current project plan and tasks.
Guides for getting started with local development are available at docs.browsertrix.com/develop.
Translation
We use Weblate to manage translation contributions.
License
Browsertrix is made available under the AGPLv3 License.
Documentation is made available under the Creative Commons Attribution 4.0 International License