2f63c7dcf8
* add profile creation, list endpoints at /archives/<aid>/profiles
* add profile browser creation, get, ping, commit, delete endpoints at /archives/<aid>/profiles/browser
* support creation of profile browser using browsertrix-crawler 'create-login-profile' in docker and k8s
* ensure profile browser expires after set time, k8s job or docker container automatically deleted on exit
* profile browser creation returns temporary browser id, or `{"detail": "waiting_for_browser"}` while waiting for browser container init
* nginx frontend: proxy /loadbrowser/ to port 9223 in browsertrix-crawler, connecting directly to chrome devtools
* profile api auth: use redis for auth
- store browserid->archiveid and browserid->browser ip mapping in redis
- browser apis: ensure profile browser is associated with specified archive
- browser ws: pass arcchiveid and browserid to ws query args, browserid is part of archive, and browserid corresponds to specified ip
* store profiles in /profiles/ directory in default storage, include profileid in profile tar.gz filename
* support profile in crawlconfig:
- add profileid to CrawlConfig, and profileName to CrawlConfigOut
- support resolving profile path via profileid, setting '--profile @{path/to/profile.tar.gz}' for crawler (assuming same storage for profile as output for now) in both docker and k8s setups
- docker: support out_filename, custom wacz output filename missing functionality
|
||
|---|---|---|
| backend | ||
| chart | ||
| configs | ||
| frontend | ||
| .gitignore | ||
| build-backend.sh | ||
| build-frontend.sh | ||
| Deployment.md | ||
| docker-compose.yml | ||
| docker-restart.sh | ||
| LICENSE | ||
| NOTICE | ||
| pylintrc | ||
| README.md | ||
Browsertrix Cloud
Browsertrix Cloud is an open-source cloud-native high-fidelity browser-based crawling service designed to make web archiving easier and more accessible for everyone.
The service provides an API and UI for scheduling crawls and viewing results, and managing all aspects of crawling process. This system provides the orchestration and management around crawling, while the actual crawling is performed using Browsertrix Crawler containers, which are launched for each crawl.
The system is designed to run equally in Kubernetes and Docker.
See Features for a high-level list of planned features.
Deployment
See the Deployment page for information on how to deploy Browsertrix Cloud.
Development Status
Browsertrix Cloud is currently in pre-alpha stages and not ready for production. This is an ambitious project and there's a lot to be done!
If you would like to help in a particular way, please open an issue or reach out to us in other ways.
License
Browsertrix Cloud is made available under the AGPLv3 License.
If you would like to use it under a different license or have a question, please reach out as that may be a possibility.