tea/browsertrix
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
130 Commits 2 Branches 0 Tags 15 MiB
TypeScript 56.4%
Python 39%
JavaScript 1.8%
CSS 1.3%
MDX 0.8%
Other 0.4%
aa5207915c
Go to file
Ilya Kreymer aa5207915c
backend: fix crawl config revision links (#149) 
backed: crawlconfig:
- ensure newId is saved on old config being replaced
- if old config replaced is being deleted, ensure newId link is set on its old config (if any),
and the oldId points to the oldId of config being replaced (if any)
2022-02-21 16:51:27 -08:00
backend backend: fix crawl config revision links (#149) 2022-02-21 16:51:27 -08:00
chart k8s chart: add permissions for pod exec and logs 2022-02-20 09:39:11 -08:00
configs Config superuser (#59) 2021-12-05 14:12:42 -08:00
frontend Deactivate crawl templates in UI (#145) 2022-02-21 11:37:15 -08:00
.gitignore Remove unused files (#69) 2021-12-26 18:01:10 -08:00
docker-compose.yml Refactor backend data model to support UUID (fixes #118) (#119) 2022-01-29 19:00:11 -08:00
docker-restart.sh README + docker-restart.sh add 2021-08-25 16:27:22 -07:00
pylintrc misc tweaks: 2021-08-25 18:34:49 -07:00
README.md Storage + Data Model Refactor (fixes #3): 2021-10-09 18:58:40 -07:00

README.md

Browsertrix Cloud

Browsertrix Cloud is a cloud-native crawling system, which supports a multi-user, multi-archive crawling system to run natively in the cloud via Kubernetes or locally via Docker.

The system currently includes support for the following:

  • Fully API-driven, with OpenAPI specification for all APIs.
  • Multiple users, registered via email and/or invited to join Archives.
  • Crawling centered around Archives which are associated with an S3-compatible storage bucket.
  • Users may be part of multiple archives and have different roles in different archives
  • Archives contain crawler configs, which are passed to the crawler.
  • Crawls launched via a crontab-based schedule or manually on-demand
  • Crawls performed using Browsertrix Crawler.
  • Crawl config includes an optional timeout, after which crawl is stopped gracefully.
  • Crawl status is tracked in the DB (possible crawl states include: Completed, Partially-Complete (due to timeout or cancelation), Cancelation, Failure)

Deploying to Docker

To deploy via local Docker instance, copy the config.sample.env to config.env.

Docker Compose is required.

Then, run docker-compose build; docker-compose up -d to launch.

To update/relaunch, use ./docker-restart.sh.

The API should be available at: http://localhost:8000/docs

Note: When deployed in local Docker, failed crawls are not retried currently. Scheduling is handled by a subprocess, which stores active schedule in the DB.

Deploying to Kubernetes

To deploy to K8s, helm is required. Browsertrix Cloud comes with a helm chart, which can be installed as follows:

helm install -f ./chart/values.yaml btrix ./chart/

This will create a browsertrix-cloud service in the default namespace.

For a quick update, the following is recommended:

helm upgrade -f ./chart/values.yaml btrix ./chart/

Note: When deployed in Kubernetes, failed crawls are automatically retried. Scheduling is handled via Kubernetes Cronjobs, and crawl jobs are run in the crawlers namespace.

Browsertrix Cloud is currently in pre-alpha stages and not ready for production.