Go to file
Ilya Kreymer f7836c345d
Crawl Queue API (#342)
* crawl queue api work: (#329)
- add api to /crawls/{crawl_id}/queue api to get crawl queue, with offset, count, and optional regex. returns results and regex matches within the results, along with total urls in queue.
- add api to match entire crawl queue, /crawls/{crawl_id}/queueMatch with query 'regex' arg, which processes entire crawl queue on backend and returns a list of matches (more experimental)
- if crawl not yet started / redis not available, return empty queue
- only supported for k8s deployment at the moment
2022-10-12 19:56:13 -07:00
.github/workflows Local swarm + podman support (#261) 2022-06-14 00:13:49 -07:00
ansible Deployment: Add Ansible setup to deploy with microk8s (#296) 2022-08-19 12:49:21 -07:00
backend Crawl Queue API (#342) 2022-10-12 19:56:13 -07:00
chart k8s chart + values tweak: (#317) 2022-09-21 12:45:08 -07:00
configs config sample: switch back to browsertrix-crawler:latest for now 2022-06-17 13:39:45 -07:00
frontend Improve crawl elapsed time UX (#323) 2022-10-05 21:12:31 -04:00
scripts config/scripts: 2022-06-16 22:36:44 -07:00
test Single config and env vars (#267) 2022-06-16 21:50:03 -07:00
.gitignore Local swarm + podman support (#261) 2022-06-14 00:13:49 -07:00
Deployment.md Single config and env vars (#267) 2022-06-16 21:50:03 -07:00
docker-compose.yml Single config and env vars (#267) 2022-06-16 21:50:03 -07:00
LICENSE Add License, Logo and README updates for release (#157) 2022-02-23 12:10:46 -08:00
NOTICE Add License, Logo and README updates for release (#157) 2022-02-23 12:10:46 -08:00
pylintrc
README.md Local swarm + podman support (#261) 2022-06-14 00:13:49 -07:00

Browsertrix Cloud

Browsertrix Cloud is an open-source cloud-native high-fidelity browser-based crawling service designed to make web archiving easier and more accessible for everyone.

The service provides an API and UI for scheduling crawls and viewing results, and managing all aspects of crawling process. This system provides the orchestration and management around crawling, while the actual crawling is performed using Browsertrix Crawler containers, which are launched for each crawl.

The system is designed to run in both Kubernetes and Docker Swarm, as well as locally under Podman.

See Features for a high-level list of planned features.

Deployment

See the Deployment page for information on how to deploy Browsertrix Cloud.

Development Status

Browsertrix Cloud is currently in an alpha stage and not ready for production. This is an ambitious project and there's a lot to be done!

If you would like to help in a particular way, please open an issue or reach out to us in other ways.

License

Browsertrix Cloud is made available under the AGPLv3 License.

If you would like to use it under a different license or have a question, please reach out as that may be a possibility.