4bea7565bc
Operator: Modified init behavior to only load redis when at least one crawler pod available: - waits for at least one crawler pod to be available before starting redis pod, to avoid situation where many crawler pods are in pending mode, but redis pods are still running. - redis statefulset starts at scale of 0 - once crawler pod becomes available, redis sts is scaled to 1 (via `initRedis==true` status) - crawl remains in 'starting' or 'waiting_capacity' state until pod becomes available without redis pod running - set to 'running' state only after redis and at least one crawler pod is available - if no crawler pods available after running, or, if stuck in starting for >60 seconds, switch to 'waiting_capacity' state - when switching to 'waiting_capacity', also scale down redis to 0, wait for crawler pod to become available, only then scale up redis to 1, and get back to 'running' other tweaks: - add new status field 'initRedis', default to false, not displayed - crawler pod: consider 'ContainerCreating' state as available, as container will not be blocked by resource limits - add a resync after 3 seconds when waiting for crawler pod or redis pod to become available, configurable via 'operator_fast_resync_secs' - set_state: if not updating state, ensure state reflects actual value in db |
||
|---|---|---|
| .github | ||
| ansible | ||
| backend | ||
| chart | ||
| configs | ||
| docs | ||
| frontend | ||
| scripts | ||
| test | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| btrix | ||
| CHANGES.md | ||
| LICENSE | ||
| mkdocs.yml | ||
| NOTICE | ||
| pylintrc | ||
| README.md | ||
| update-version.sh | ||
| version.txt | ||
| yarn.lock | ||
Browsertrix Cloud
Browsertrix Cloud is an open-source cloud-native high-fidelity browser-based crawling service designed to make web archiving easier and more accessible for everyone.
The service provides an API and UI for scheduling crawls and viewing results, and managing all aspects of crawling process. This system provides the orchestration and management around crawling, while the actual crawling is performed using Browsertrix Crawler containers, which are launched for each crawl.
See Features for a high-level list of planned features.
Documentation
The full docs for using, deploying and developing Browsertrix Cloud are available at: https://docs.browsertrix.cloud
Deployment
The latest deployment documentation is available at: https://docs.browsertrix.cloud/deploy
The docs cover deploying Browsertrix Cloud in different environments using Kubernetes, from a single-node setup to scalable clusters in the cloud.
Previously, Browsertrix Cloud also supported Docker Compose and podman-based deployment. This is now deprecated due to the complexity of maintaining feature parity across different setups, and with various Kubernetes deployment options being available and easy to deploy, even on a single machine.
Making deployment of Browsertrix Cloud as easy as possible remains a key goal, and we welcome suggestions for how we can further improve our Kubernetes deployment options.
If you are looking to just try running a single crawl, you may want to try Browsertrix Crawler first to test out the crawling capabilities.
Development Status
Browsertrix Cloud is currently in a beta, though the system and backend API is fairly stable, we are working on many additional features.
Additional developer documentation is available at https://docs.browsertrix.cloud/develop
Please see the GitHub issues and this GitHub Project for our current project plan and tasks.
License
Browsertrix Cloud is made available under the AGPLv3 License.
Documentation is made available under the Creative Commons Attribution 4.0 International License