Go to file
Ilya Kreymer 5b6aa3bc95
Affinity + Tolerations + Cleanup Crawl Job (#256)
* k8s: add tolerations for 'nodeType=crawling:NoSchedule' to allow scheduling crawling on designated nodes for crawler and profiles jobs and statefulsets
* add affinity for 'nodeType=crawling' on crawling and profile browser statefulsets
* refactor crawljob: combine crawl_updater logic into base crawl_job
* increment new 'crawlAttemptCount' counter crawlconfig when crawl is started, not necessarily finished, to avoid deleting configs that had attempted but not finished crawls.
* better external mongodb support: use MONGO_DB_URL to set custom url directly, otherwise build from username, password and mongo host
2022-06-10 19:21:37 -07:00
.github/workflows CI setup for new swarm mode (#248) 2022-06-06 09:34:02 -07:00
backend Affinity + Tolerations + Cleanup Crawl Job (#256) 2022-06-10 19:21:37 -07:00
chart Affinity + Tolerations + Cleanup Crawl Job (#256) 2022-06-10 19:21:37 -07:00
configs affinity: add affinity for k8s crawl deployments: 2022-06-07 21:52:04 -07:00
frontend CI setup for new swarm mode (#248) 2022-06-06 09:34:02 -07:00
test CI setup for new swarm mode (#248) 2022-06-06 09:34:02 -07:00
.gitignore refactor to use docker swarm for local alternative to k8s instead of docker compose (#247): 2022-06-05 10:37:17 -07:00
build-backend.sh Mongo auth fix (#190) 2022-03-04 15:04:33 -08:00
build-frontend.sh Mongo auth fix (#190) 2022-03-04 15:04:33 -08:00
Deployment.md Backend API prefix (#240) 2022-05-31 19:29:20 -07:00
docker-compose.yml CI setup for new swarm mode (#248) 2022-06-06 09:34:02 -07:00
docker-restart.sh
LICENSE Add License, Logo and README updates for release (#157) 2022-02-23 12:10:46 -08:00
NOTICE Add License, Logo and README updates for release (#157) 2022-02-23 12:10:46 -08:00
pylintrc
README.md Move deployment info to deployment.md (#159) 2022-02-24 00:16:36 -08:00

Browsertrix Cloud

Browsertrix Cloud is an open-source cloud-native high-fidelity browser-based crawling service designed to make web archiving easier and more accessible for everyone.

The service provides an API and UI for scheduling crawls and viewing results, and managing all aspects of crawling process. This system provides the orchestration and management around crawling, while the actual crawling is performed using Browsertrix Crawler containers, which are launched for each crawl.

The system is designed to run equally in Kubernetes and Docker.

See Features for a high-level list of planned features.

Deployment

See the Deployment page for information on how to deploy Browsertrix Cloud.

Development Status

Browsertrix Cloud is currently in pre-alpha stages and not ready for production. This is an ambitious project and there's a lot to be done!

If you would like to help in a particular way, please open an issue or reach out to us in other ways.

License

Browsertrix Cloud is made available under the AGPLv3 License.

If you would like to use it under a different license or have a question, please reach out as that may be a possibility.