Go to file
Ilya Kreymer 00fb8ac048
Concurrent Crawl Limit (#874)
concurrent crawl limits: (addresses #866)
- support limits on concurrent crawls that can be run within a single org
- change 'waiting' state to 'waiting_org_limit' for concurrent crawl limit and 'waiting_capacity' for capacity-based
limits

orgs:
- add 'maxConcurrentCrawl' to new 'quotas' object on orgs
- add /quotas endpoint for updating quotas object

operator:
- add all crawljobs as related, appear to be returned in creation order
- operator: if concurrent crawl limit set, ensures current job is in the first N set of crawljobs (as provided via 'related' list of crawljob objects) before it can proceed to 'starting', otherwise set to 'waiting_org_limit'
- api: add org /quotas endpoint for configuring quotas
- remove 'new' state, always start with 'starting'
- crawljob: add 'oid' to crawljob spec and label for easier querying
- more stringent state transitions: add allowed_from to set_state()
- ensure state transitions only happened from allowed states, while failed/canceled can happen from any state
- ensure finished and state synched from db if transition not allowed
- add crawl indices by oid and cid

frontend: 
- show different waiting states on frontend: 'Waiting (Crawl Limit) and 'Waiting (At Capacity)'
- add gear icon on orgs admin page
- and initial popup for setting org quotas, showing all properties from org 'quotas' object

tests:
- add concurrent crawl limit nightly tests
- fix state waiting -> waiting_capacity
- ci: add logging of operator output on test failure
2023-05-30 15:38:03 -07:00
.github Concurrent Crawl Limit (#874) 2023-05-30 15:38:03 -07:00
ansible add authsign block to microk8s playbook (#776) 2023-05-05 11:32:32 -07:00
backend Concurrent Crawl Limit (#874) 2023-05-30 15:38:03 -07:00
chart tests: fixes for crawl cancel + crawl stopped (#864) 2023-05-22 20:17:29 -07:00
configs Remove Code and Configs for Swarm/podman support (#407) 2022-12-08 18:19:58 -08:00
docs Update backend local development docs (#884) 2023-05-25 09:50:56 -07:00
frontend Concurrent Crawl Limit (#874) 2023-05-30 15:38:03 -07:00
scripts Fix doc to build a local image for microk8s (#594) 2023-02-14 16:10:04 -08:00
test
.gitignore Make btrix helper work with microk8s (#768) 2023-04-18 08:50:46 -04:00
.pre-commit-config.yaml Rename archives/teams -> orgs in codebase + add db migration (#486) 2023-01-18 14:51:04 -08:00
btrix Add current context and confirmation dialog to reset/bootstrap methods (#887) 2023-05-25 13:43:53 -04:00
CHANGES.md version: bump to 1.3.0 2023-02-24 18:07:56 -08:00
LICENSE
mkdocs.yml Update backend local development docs (#884) 2023-05-25 09:50:56 -07:00
NOTICE
pylintrc
README.md docs: fix link to dev docs 2023-05-24 10:59:41 -07:00
update-version.sh
version.txt version: bump to 1.6.0-beta.0 2023-05-19 11:29:31 -07:00

Browsertrix Cloud

Browsertrix Cloud is an open-source cloud-native high-fidelity browser-based crawling service designed to make web archiving easier and more accessible for everyone.

The service provides an API and UI for scheduling crawls and viewing results, and managing all aspects of crawling process. This system provides the orchestration and management around crawling, while the actual crawling is performed using Browsertrix Crawler containers, which are launched for each crawl.

See Features for a high-level list of planned features.

Documentation

The full docs for using, deploying and developing Browsertrix Cloud are available at: https://docs.browsertrix.cloud

Deployment

The latest deployment documentation is available at: https://docs.browsertrix.cloud/deploy

The docs cover deploying Browsertrix Cloud in different environments using Kubernetes, from a single-node setup to scalable clusters in the cloud.

Previously, Browsertrix Cloud also supported Docker Compose and podman-based deployment. This is now deprecated due to the complexity of maintaining feature parity across different setups, and with various Kubernetes deployment options being available and easy to deploy, even on a single machine.

Making deployment of Browsertrix Cloud as easy as possible remains a key goal, and we welcome suggestions for how we can further improve our Kubernetes deployment options.

If you are looking to just try running a single crawl, you may want to try Browsertrix Crawler first to test out the crawling capabilities.

Development Status

Browsertrix Cloud is currently in a beta, though the system and backend API is fairly stable, we are working on many additional features.

Additional developer documentation is available at https://docs.browsertrix.cloud/develop

Please see the GitHub issues and this GitHub Project for our current project plan and tasks.

License

Browsertrix Cloud is made available under the AGPLv3 License.

Documentation is made available under the Creative Commons Attribution 4.0 International License