Go to file
Ilya Kreymer 28d39d8c4d
Fix migration to avoid duplicate collection slugs and names (#2318)
Follow-up to #2301 

Updates the 0039 migration to ensure collection slugs and names are
unique by:
- Removing all indexes
- Setting `slug` to random value
- Adding unique index to `slug` field.
- Attempting to set slug from name using `slug_from_name()`
- If rejected due to duplicate, append `-<counter>` at end of slug. Also
update name with ` <counter>`.
- Now that names should also be unique, add unique index on name field.

---------

Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
2025-01-21 14:23:32 -08:00
.github Ensure PR workflows can run on all PRs (but still skip when they're not needed) (#2237) 2025-01-14 14:24:27 -05:00
.vscode feat: Show collection created date (#2302) 2025-01-14 11:22:00 -05:00
ansible Update ansible pipfile (#2088) 2024-09-20 11:41:21 -07:00
assets refactor: Implement brand colors (#2141) 2024-11-12 08:54:11 -08:00
backend Fix migration to avoid duplicate collection slugs and names (#2318) 2025-01-21 14:23:32 -08:00
chart Add support for autoclick (#2313) 2025-01-16 12:44:00 -08:00
configs Remove Code and Configs for Swarm/podman support (#407) 2022-12-08 18:19:58 -08:00
frontend Add pageCount to crawls and uploads and use in frontend for page counts (#2315) 2025-01-16 14:41:14 -08:00
scripts Configure browsertrix proxies (#1847) 2024-10-02 18:35:45 -07:00
test fix(build): use /usr/bin/env bash instead of /bin/bash (#1020) 2023-07-28 21:50:04 -07:00
.gitattributes Add linguist-generated attribute to generated files (#2221) 2024-12-07 01:27:50 -05:00
.gitignore Make btrix helper work with microk8s (#768) 2023-04-18 08:50:46 -04:00
.pre-commit-config.yaml Add Repository Index + Chart Rename + Docs Rename (#1708) 2024-04-21 09:42:25 -07:00
btrix Add Repository Index + Chart Rename + Docs Rename (#1708) 2024-04-21 09:42:25 -07:00
CHANGES.md version: bump to 1.3.0 2023-02-24 18:07:56 -08:00
LICENSE Add License, Logo and README updates for release (#157) 2022-02-23 12:10:46 -08:00
NOTICE Add Repository Index + Chart Rename + Docs Rename (#1708) 2024-04-21 09:42:25 -07:00
pylintrc misc tweaks: 2021-08-25 18:34:49 -07:00
README.md Update Readme to add Weblate information (#2109) 2024-10-31 16:36:12 -04:00
update-version.sh enable screenshots by default + fix py version formatting (#1518) 2024-02-07 17:07:28 -08:00
version.txt version: bump to 1.14.0-beta.0 for public collections! 2025-01-13 19:29:54 -08:00

Browsertrix

 

Browsertrix is a cloud-native, high-fidelity, browser-based crawling service designed to make web archiving easier and more accessible for everyone.

The service provides an API and UI for scheduling crawls and viewing results, and managing all aspects of crawling process. This system provides the orchestration and management around crawling, while the actual crawling is performed using Browsertrix Crawler containers, which are launched for each crawl.

See webrecorder.net/browsertrix for a feature overview and information about how to sign up for Webrecorder's hosted Browsertrix service.

Documentation

The full docs for using, deploying, and developing Browsertrix are available at docs.browsertrix.com.

Our docs are created with Material for MKDocs.

Deployment

The latest deployment documentation is available at docs.browsertrix.com/deploy.

The docs cover deploying Browsertrix in different environments using Kubernetes, from a single-node setup to scalable clusters in the cloud.

Early on, Browsertrix also supported Docker Compose and podman-based deployment. This was deprecated due to the complexity of maintaining feature parity across different setups, and with various Kubernetes deployment options being available and easy to deploy, even on a single machine.

Making deployment of Browsertrix as easy as possible remains a key goal, and we welcome suggestions for how we can further improve our Kubernetes deployment options.

If you are looking to just try running a single crawl, you may want to try Browsertrix Crawler first to test out the crawling capabilities.

Contributing

Though the system and backend API is fairly stable, we are working on many additional features. Please see the GitHub issues and this GitHub Project for our current project plan and tasks.

Guides for getting started with local development are available at docs.browsertrix.com/develop.

Translation

We use Weblate to manage translation contributions.

Translation status

License

Browsertrix is made available under the AGPLv3 License.

Documentation is made available under the Creative Commons Attribution 4.0 International License