tea/browsertrix
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
198 Commits 2 Branches 0 Tags 15 MiB
TypeScript 56.4%
Python 39%
JavaScript 1.8%
CSS 1.3%
MDX 0.8%
Other 0.4%
0c8a5a49b4
Go to file
Ilya Kreymer 0c8a5a49b4 refactor to use docker swarm for local alternative to k8s instead of docker compose (#247): 
- use python-on-whale to use docker cli api directly, creating docker stack for each crawl or profile browser
- configure storages via storages.yaml secret
- add crawl_job, profile_job, splitting into base and k8s/swarm implementations
- split manager into base crawlmanager and k8s/swarm implementations
- swarm: load initial scale from db to avoid modifying fixed configs, in k8s, load from configmap
- swarm: support scheduled jobs via swarm-cronjob service
- remove docker dependencies (aiodocker, apscheduler, scheduling)
- swarm: when using local minio, expose via /data/ route in nginx via extra include (in k8s, include dir is empty and routing handled via ingress)
- k8s: cleanup minio chart: move init containers to minio.yaml
- swarm: stateful set implementation to be consistent with k8s scaling:
  - don't use service replicas,
  - create a unique service with '-N' appended and allocate unique volume for each replica
  - allows crawl containers to be restarted w/o losing data
- add volume pruning background service, as volumes can be deleted only after service shuts down fully
- watch: fully simplify routing, route via replica index instead of ip for both k8s and swarm
- rename network btrix-cloud-net -> btrix-net to avoid conflict with compose network
2022-06-05 10:37:17 -07:00
backend refactor to use docker swarm for local alternative to k8s instead of docker compose (#247): 2022-06-05 10:37:17 -07:00
chart refactor to use docker swarm for local alternative to k8s instead of docker compose (#247): 2022-06-05 10:37:17 -07:00
configs refactor to use docker swarm for local alternative to k8s instead of docker compose (#247): 2022-06-05 10:37:17 -07:00
frontend refactor to use docker swarm for local alternative to k8s instead of docker compose (#247): 2022-06-05 10:37:17 -07:00
.gitignore refactor to use docker swarm for local alternative to k8s instead of docker compose (#247): 2022-06-05 10:37:17 -07:00
build-backend.sh Mongo auth fix (#190) 2022-03-04 15:04:33 -08:00
build-frontend.sh Mongo auth fix (#190) 2022-03-04 15:04:33 -08:00
Deployment.md Backend API prefix (#240) 2022-05-31 19:29:20 -07:00
docker-compose.yml refactor to use docker swarm for local alternative to k8s instead of docker compose (#247): 2022-06-05 10:37:17 -07:00
docker-restart.sh
LICENSE Add License, Logo and README updates for release (#157) 2022-02-23 12:10:46 -08:00
NOTICE Add License, Logo and README updates for release (#157) 2022-02-23 12:10:46 -08:00
pylintrc misc tweaks: 2021-08-25 18:34:49 -07:00
README.md Move deployment info to deployment.md (#159) 2022-02-24 00:16:36 -08:00

README.md

Browsertrix Cloud

Browsertrix Cloud is an open-source cloud-native high-fidelity browser-based crawling service designed to make web archiving easier and more accessible for everyone.

The service provides an API and UI for scheduling crawls and viewing results, and managing all aspects of crawling process. This system provides the orchestration and management around crawling, while the actual crawling is performed using Browsertrix Crawler containers, which are launched for each crawl.

The system is designed to run equally in Kubernetes and Docker.

See Features for a high-level list of planned features.

Deployment

See the Deployment page for information on how to deploy Browsertrix Cloud.

Development Status

Browsertrix Cloud is currently in pre-alpha stages and not ready for production. This is an ambitious project and there's a lot to be done!

If you would like to help in a particular way, please open an issue or reach out to us in other ways.

License

Browsertrix Cloud is made available under the AGPLv3 License.

If you would like to use it under a different license or have a question, please reach out as that may be a possibility.