mkdocs setup (deploy, dev, user-guide) (#375)
* Initial docs move * Setup mkdocs * Adds instructions for building docs * add new deployment docs, local and prod * set up three sections: deployment, dev and user guide * remove old deployment docs * ci: mkdocs gh-pages publish Co-authored-by: sua yoo <sua@suayoo.com> Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
This commit is contained in:
Henry Wilkinson
GitHub
parent
82ffc0dfbc
commit
a74d88dcda
19
.github/workflows/docs-publish.yaml
vendored
Normal file
19
.github/workflows/docs-publish.yaml
vendored
Normal file
@ -0,0 +1,19 @@
|
||||
name: docs-publish
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- docs!
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
jobs:
|
||||
deploy_docs:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: 3.x
|
||||
- run: pip install mkdocs-material
|
||||
- run: mkdocs gh-deploy --force
|
||||
2
.github/workflows/release.yaml
vendored
2
.github/workflows/release.yaml
vendored
@ -36,6 +36,7 @@ jobs:
|
||||
uses: docker/build-push-action@v3
|
||||
with:
|
||||
context: backend
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: true
|
||||
tags: ${{ env.REPO_PREFIX }}webrecorder/browsertrix-backend:${{ env.VERSION }}, webrecorder/browsertrix-backend:latest
|
||||
cache-from: type=gha,scope=backend
|
||||
@ -46,6 +47,7 @@ jobs:
|
||||
uses: docker/build-push-action@v3
|
||||
with:
|
||||
context: frontend
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: true
|
||||
build-args: |
|
||||
VERSION=${{ env.VERSION }}
|
||||
|
||||
112
Deployment.md
112
Deployment.md
@ -1,112 +0,0 @@
|
||||
# Deploying Browsertrix Cloud
|
||||
|
||||
Browsertrix Cloud can be deployed anywhere from single-node isolated environments, multi-machine setups and cloud-native Kubernetes!
|
||||
|
||||
Browsertrix Cloud currently supports three deployment methods:
|
||||
- Rootless deployment with podman on a single-machine (no Docker required)
|
||||
- Docker Swarm for single or multi-machine deployments
|
||||
- Kubernetes Cluster deployment.
|
||||
|
||||
Some basic instructions are provided below, we plan to expand this into more detail tutorial in the future.
|
||||
|
||||
(All shell scripts can be found in the `./scripts` directory)
|
||||
|
||||
## Deploying with Docker Swarm
|
||||
|
||||
For local deployments, using Docker Swarm is recommended. Docker Swarm can be used in a single-machine mode as well
|
||||
as with multi-machine setups. Docker Swarm is part of Docker, so if you have Docker installed, you can use this method.
|
||||
|
||||
1. Run the `init-configs.sh` which will copy the sample config `configs/config.sample.env` to `configs/config.env` if it doesn't already exist.
|
||||
|
||||
2. You can edit `configs/config.env` to set all settings for Browsertrix Cloud including, default passwords for superadmin, minio and mongodb.
|
||||
|
||||
2.5. Optionally: Run `build-swarm.sh` to build images locally, such as if you've made any changes, otherwise can skip.
|
||||
|
||||
3. Run `run-swarm.sh` to initialize the cluster.
|
||||
|
||||
4. Load `http://localhost:9871/` to see the Browsertrix Cloud login page. (The API is also available at: `http://localhost:9871/api/docs`).
|
||||
|
||||
You can stop the deployment with `stop-swarm.sh` and restart again with `run-swarm.sh`
|
||||
|
||||
|
||||
Note: Currently, unless email settings are configured, you will need to look at the logs to get the invite code for invites. You can do this by running:
|
||||
`docker service logs btrix_backend`
|
||||
|
||||
|
||||
## Deploying with Podman
|
||||
|
||||
Browsertrix Cloud can now also be used with Podman for environments that don't support Docker.
|
||||
|
||||
Podman allows Browsertrix Cloud to be deployed locally by a non-root user.
|
||||
|
||||
Podman deployment also requires either docker-compose or podman-compose.
|
||||
|
||||
|
||||
### Initial Installation
|
||||
|
||||
To run with Podman as a non-root user, there's a few initial installation
|
||||
|
||||
1. Ensure the podman service over a socket is running with: `systemctl --user start podman.socket`. Podman does not require a service, but Browsertrix Cloud requires access to the socket to worker.
|
||||
|
||||
2. Ensure podman [can set cpu limits](https://github.com/containers/podman/blob/main/troubleshooting.md#26-running-containers-with-cpu-limits-fails-with-a-permissions-error) as Browsertrix Cloud uses cpu and memory limits for each crawl. After following instructions above, also run `sudo systemctl daemon-reload` to reload the delegate settings.
|
||||
|
||||
3. Ensure podman-compose is installed via `pip install podman-compose`.
|
||||
|
||||
4. Run `build-podman.sh` to build the local images.
|
||||
|
||||
5. Run the `init-configs.sh` which will copy the sample config `configs/config.sample.env` to `configs/config.env` if it doesn't already exist.
|
||||
|
||||
6. You can edit `configs/config.env` to set all settings for Browsertrix Cloud including, default passwords for superadmin, minio and mongodb.
|
||||
|
||||
6.5. Optionally: Run `build-podman.sh` to build images locally, such as if you've made any changes, otherwise can skip.
|
||||
|
||||
7. Run `run-podman.sh` to run Browsertrix Cloud using podman.
|
||||
|
||||
8. Load `http://localhost:9871/` to see the Browsertrix Cloud login page. (The API is also available at: `http://localhost:9871/api/docs`).
|
||||
|
||||
|
||||
You can stop the deployment with `stop-podman.sh` and restart again with `run-podman.sh`
|
||||
|
||||
Note: Currently, unless email settings are configured, you will need to look at the logs to get the invite code for invites. You can do this by running:
|
||||
`podman logs -f browsertrix-cloud_backend_1`
|
||||
|
||||
It's also possible to use Docker Compose with podman by setting `export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock`. You can change the setting
|
||||
in `run-podman.sh` and `stop-podman.sh` to use docker-compose instead if desired.
|
||||
|
||||
|
||||
### Enabling Signing (for Swarm and Podman Deployments)
|
||||
|
||||
Browsertrix Cloud can optionally sign WACZ files with the same key used to generate an SSL cert.
|
||||
|
||||
To use this functionality, the machine running Browsertrix Cloud must be associated with a domain and must have port 80 available on that domain,
|
||||
or another port forwarding to port 80.
|
||||
|
||||
The `docker-compose.signing.yml` adds the capability for signing with the `authsign` module.
|
||||
|
||||
To enable signing in the Docker-based deployment:
|
||||
|
||||
1. Copy `configs/signing.sample.yaml` to `configs/signing.yaml` and set the domain and email fields in the config. Set `staging` to false to generate real certificates.
|
||||
|
||||
2. In `docker-compose.signing.yaml`, set an optional signing token.
|
||||
|
||||
3. In `run-swarm.sh`, uncomment the option for running with signing.
|
||||
|
||||
|
||||
|
||||
|
||||
## Deploying to Kubernetes
|
||||
|
||||
For deploying in the cloud, the Kubernetes (k8s) deployment is recommended.
|
||||
Browsertrix Cloud uses `helm` to deploy to K8s.
|
||||
|
||||
|
||||
1. Ensure `helm` is installed locally and `kubectl` is configured for your k8s cluster.
|
||||
|
||||
2. Edit `chart/values.yaml` to configure your deployment. The `ingress` section contains the domain the service will be deployed in, and `signing` can be used to enable WACZ signing.
|
||||
|
||||
3. Run: `helm upgrade --install -f ./chart/values.yaml btrix ./chart/` to deploy or upgrade an existing deployment.
|
||||
|
||||
|
||||
To stop, run `helm uninstall btrix`.
|
||||
|
||||
*Additional info coming soon*
|
||||
20
README.md
20
README.md
@ -14,21 +14,27 @@ The system is designed to run in both Kubernetes and Docker Swarm, as well as lo
|
||||
|
||||
See [Features](https://browsertrix.cloud/features) for a high-level list of planned features.
|
||||
|
||||
## Deployment
|
||||
|
||||
See the [Deployment](Deployment.md) page for information on how to deploy Browsertrix Cloud.
|
||||
|
||||
|
||||
## Development Status
|
||||
|
||||
Browsertrix Cloud is currently in an alpha stage and not ready for production. This is an ambitious project and there's a lot to be done!
|
||||
Browsertrix Cloud is currently in an early beta stage and not fully ready for production. This is an ambitious project and there's a lot to be done!
|
||||
|
||||
If you would like to help in a particular way, please open an issue or reach out to us in other ways.
|
||||
|
||||
## Documentation
|
||||
|
||||
Docs are available at: [https://docs.browsertrix.cloud/](https://docs.browsertrix.cloud/) created from the markdown in the [./docs](./docs) on the main branch.
|
||||
|
||||
To build the documentation locally, install Material for MkDocs with pip:
|
||||
|
||||
```shell
|
||||
pip install mkdocs-material
|
||||
```
|
||||
|
||||
In the project root directory run `mkdocs serve` to run a local version of the documentation site.
|
||||
|
||||
## License
|
||||
|
||||
Browsertrix Cloud is made available under the AGPLv3 License.
|
||||
|
||||
If you would like to use it under a different license or have a question, please reach out as that may be a possibility.
|
||||
|
||||
|
||||
|
||||
14
docs/deploy/index.md
Normal file
14
docs/deploy/index.md
Normal file
@ -0,0 +1,14 @@
|
||||
# Introduction
|
||||
|
||||
Browsertrix Cloud is designed to be a cloud-native application running in Kubernetes.
|
||||
|
||||
However, despite the name, it is perfectly reasonable (and easy!) to deploy Browsertrix Cloud locally using one of the many available local Kubernetes options. Here are a few recommendations for different scenarios.
|
||||
|
||||
The main requirements for Browsertrix Cloud are:
|
||||
|
||||
- A Kubernetes Cluster
|
||||
- Helm (package manager for Kubernetes)
|
||||
|
||||
We have prepared a [Local Deployment](./local) and [Production (Self-Hosted and Cloud) Deployment](./production) guides to help with
|
||||
setting up Browsertrix Cloud for different scenarios.
|
||||
|
||||
82
docs/deploy/local.md
Normal file
82
docs/deploy/local.md
Normal file
@ -0,0 +1,82 @@
|
||||
# Local Deployment
|
||||
|
||||
To just test out Browsertrix Cloud on your local machine, you can use many of the single-node Kubernetes tools.
|
||||
|
||||
Browsertrix Cloud has been tested with Docker Desktop (with Kubernetes enabled), microk8s, minikube and k3s.
|
||||
|
||||
Here are a few different ways to get started with Browsertrix Cloud. The instructions below assume you have cloned
|
||||
the [https://github.com/webrecorder/browsertrix-cloud](https://github.com/webrecorder-browsertrix-cloud) repository locally,
|
||||
and have local package managers for your platform (eg. `brew` for mac) already installed.
|
||||
|
||||
After following the environment-specific instructions below, simply run:
|
||||
|
||||
`helm upgrade --install -f ./chart/values.yaml -f ./chart/local-config.yaml btrix ./chart/`
|
||||
|
||||
The local setup includes the full Browsertrix Cloud system, with frontend, backend api, db (via MongoDB) and storage (via Minio)
|
||||
|
||||
An admin user with name `admin@example.com` and password `PASSW0RD!` will be automatically created.
|
||||
|
||||
These settings can be changed in `charts/examples/local-config.yaml`. Note that the admin user and password will not be reset after creation.
|
||||
|
||||
To access the server load: [http://localhost:30870/](http://localhost:30870/) in your browser.
|
||||
|
||||
Here are some additional environment-specific deployment specific instructions:
|
||||
|
||||
### Docker Desktop -- Mac
|
||||
|
||||
For Mac, we recommend testing out Browsertrix Cloud on Docker Desktop as that will be one of the simplest options.
|
||||
|
||||
To run Browsertrix Cloud on Docker Desktop:
|
||||
|
||||
1. Ensure `Enable Kubernetes` is checked from the Preferences screen.
|
||||
|
||||
2. Install [Helm](https://helm.sh/), which can be installed with `brew install helm` or [other options](https://helm.sh/docs/intro/install/)
|
||||
|
||||
3. Run the Helm command as described above.
|
||||
|
||||
### MicroK8S
|
||||
|
||||
For Ubuntu and other linux distros, we recommend using MicroK8S for both local deployment and production.
|
||||
|
||||
1. Install MicroK8s, by running `sudo snap install microk8s --classic` [see more detailed instructions here](https://microk8s.io/docs/getting-started) or [alternate installation instructions here](https://microk8s.io/docs/install-alternatives)
|
||||
|
||||
2. Install the following addons `microk8s enable dns hostpath-storage registry helm`. (For production, also add `ingress cert-manager`)
|
||||
|
||||
3. Wait for add-ons to finish installing with `microk8s status --wait-ready`
|
||||
|
||||
4. Run the Helm command as described above, prefixed with `microk8s`, eg. `microk8s helm ...`
|
||||
|
||||
### Minikube
|
||||
|
||||
1. Install Minikube [following installation instructions](https://minikube.sigs.k8s.io/docs/start/), eg. `brew install minikube`
|
||||
|
||||
2. Run the Helm command as described above.
|
||||
|
||||
|
||||
## Waiting for Cluster to Start
|
||||
|
||||
Especially on first run, it may take a few minutes for the Browsertrix Cloud cluster to start, as all images need to be loaded.
|
||||
|
||||
You can try running the command: `kubectl wait --for=condition=ready pod --all` to wait for all pods to be initialized.
|
||||
|
||||
If this command fails, you can also run `kubectl get pods` to see the status of each of the pods.
|
||||
|
||||
There should be 4 pods listed: backend, fronend, minio and mongodb. If any one is not ready for a while, something may be wrong.
|
||||
|
||||
### Debugging Pod Issues
|
||||
|
||||
To get more details about why a pod has not started, you can run `kubectl describe <podname>` and see the latest status at the bottom.
|
||||
|
||||
Often, the error may be obvious, such as failed to pull an image.
|
||||
|
||||
If the pod is running, or previously ran, you can also get the logs from the container by running `kubectl logs <podname>`
|
||||
|
||||
The outputs of these commands will be helpful if you'd like to report an issue [on GitHub](https://github.com/webrecorder/browsertrix-cloud/issues)
|
||||
|
||||
|
||||
## Uninstalling
|
||||
|
||||
To uninstall, run `helm uninstall btrix`.
|
||||
|
||||
By default, the database + storage volumes are not automatically deleted. To fully delete all persistent data created in the cluster, also run `kubectl delete pvc --all`.
|
||||
|
||||
70
docs/deploy/production.md
Normal file
70
docs/deploy/production.md
Normal file
@ -0,0 +1,70 @@
|
||||
# Production: Self-Hosted and Cloud
|
||||
|
||||
For production and hosted deployments (both on a single machine or in the cloud), the only requirement is to have a designed domain
|
||||
and (strongly recommended, but not required) second domain for signing web archives.
|
||||
|
||||
We are also experimenting with Ansible playbooks for cloud deployment setups.
|
||||
|
||||
The production deployments also allow using an external mongodb server, and/or external S3-compatible storage instead of the bundled minio.
|
||||
|
||||
|
||||
## Single Machine Deployment with MicroK8S
|
||||
|
||||
For a single-machine production deployment, we recommend using microk8s.
|
||||
|
||||
1. Install MicroK8S, as suggested in [the local deployment guide](./local) and ensure the `ingress` and `cert-manager` addons are also enabled.
|
||||
|
||||
2. Set the `ingress.host`, `ingress.cert_email` and `signing.host` fields in `./examples/microk8s-hosted.yaml` to your host and domain
|
||||
|
||||
3. Set the supeadmin username and password, and mongodb username and password, also in `./examples/microk8s-hosted.yaml`
|
||||
|
||||
4. Run with `helm upgrade --install -f ./chart/values.yaml -f ./chart/microk8s-hosted.yaml btrix ./chart/`
|
||||
|
||||
|
||||
### Using Custom Storage
|
||||
|
||||
If you would like to use existing external storage, such an existing S3-compatible storage, also set the default storage, for example:
|
||||
|
||||
```
|
||||
minio_local: false
|
||||
|
||||
storages:
|
||||
- name: "default"
|
||||
access_key: <access key>
|
||||
secret_key: <secret key>
|
||||
|
||||
endpoint_url: "https://s3.<region>.amazonaws.com/bucket/path/"
|
||||
```
|
||||
|
||||
Note that this setup is not limited to Amazon S3, but should work with any S3-compatible storage service.
|
||||
|
||||
|
||||
### Using Custom MongoDB
|
||||
|
||||
If you would like to use an externally hosted MongoDB, you can add the following config to point to a custom MongoDB instance.
|
||||
|
||||
The `db_url` should follow the [MongoDB Connection String Format](https://www.mongodb.com/docs/manual/reference/connection-string/)
|
||||
which should include the username and password of the remote instance.
|
||||
|
||||
|
||||
```
|
||||
mongo_local: false
|
||||
|
||||
mongo_auth:
|
||||
db_url: mongodb+srv://...
|
||||
|
||||
|
||||
## Cloud Deployment
|
||||
|
||||
There are also many ways to deploy Browsertrix Cloud on various cloud providers.
|
||||
|
||||
To simplify this process, we are working on Ansible playbooks for setting up Browsertrix Cloud on commonly used infrastructure.
|
||||
|
||||
Thus, far we have the following Ansible playbooks available:
|
||||
|
||||
### Digital Ocean
|
||||
|
||||
*TODO: Add more details*
|
||||
|
||||
|
||||
|
||||
@ -1,7 +1,12 @@
|
||||
# Browsertrix Cloud frontend
|
||||
# Running the Frontend
|
||||
|
||||
This guide explains how to deploy an instance of the Browsertrix Cloud frontend for development.
|
||||
The frontend can connect to a Browsertrix Cloud API backend running locally or remotely.
|
||||
|
||||
## Quickstart
|
||||
|
||||
Ensure the current working directory is set to the `/frontend` folder.
|
||||
|
||||
Install dependencies:
|
||||
|
||||
```sh
|
||||
@ -20,6 +25,12 @@ Update `API_BASE_URL` in `.env.local` to point to your dev backend API. For exam
|
||||
API_BASE_URL=http://dev.example.com/api
|
||||
```
|
||||
|
||||
If connecting to a local deployment cluster, set API_BASE_URL to:
|
||||
|
||||
```
|
||||
API_BASE_URL=http://localhost:30870/api
|
||||
```
|
||||
|
||||
Start the dev server:
|
||||
|
||||
```sh
|
||||
21
docs/dev/index.md
Normal file
21
docs/dev/index.md
Normal file
@ -0,0 +1,21 @@
|
||||
# Developer Docs
|
||||
|
||||
Browsertrix Cloud consists of a Python-based backend and TypeScript-based frontend.
|
||||
|
||||
### Backend
|
||||
|
||||
The backend is an API-only system, using the FastAPI framework. The latest API reference is available
|
||||
under ./api of a running cluster.
|
||||
|
||||
*TODO Add additional info here*
|
||||
|
||||
### Frontend
|
||||
|
||||
The frontend UI is implemented in TypeScript, using the Lit framework and Shoelace component library.
|
||||
|
||||
The static build of the frontend is bundled with nginx, but the frontend can be deployed locally in dev mode against an existing backend.
|
||||
|
||||
See [Running Frontend](./frontend-dev) for more details.
|
||||
|
||||
*TODO Add additional info here*
|
||||
|
||||
12
docs/index.md
Normal file
12
docs/index.md
Normal file
@ -0,0 +1,12 @@
|
||||
# Welcome
|
||||
|
||||
Welcome to the Browsertrix Cloud official user guide and developer docs. These docs will contain the following sections.
|
||||
|
||||
- [Deployment Guide](./deploy) - Info on how to install and deploy Browsertrix Cloud locally or in the cloud.
|
||||
- [Developer Docs](./dev) - for info on how to developing Browsertrix Cloud itself.
|
||||
- [User Guide](./user-guide) - for info on how to use Browsertrix Cloud system.
|
||||
|
||||
If you are unfamiliar with Browsertrix Cloud, please check out [our website](https://browsertrix.cloud) or the main repository at [https://github.com/webrecorder/browsertrix-cloud](https://github.com/webrecorder/browsertrix-cloud)
|
||||
|
||||
Our docs are still under construction. If you find something missing, chances are we haven't gotten around to writing that part yet. If you find typoes or something isn't clear or seems incorrect, please open an [issue](https://github.com/webrecorder/browsertrix-cloud/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc) and we'll try to make sure that your questions get answered here in the future!
|
||||
|
||||
19
docs/user-guide/index.md
Normal file
19
docs/user-guide/index.md
Normal file
@ -0,0 +1,19 @@
|
||||
# Getting Started
|
||||
|
||||
Welcome to Browsertrix Cloud. This Guide will cover various aspects of using Browsertrix Cloud.
|
||||
|
||||
|
||||
## Creating an Account
|
||||
|
||||
- If you have been given an invite, you can create an account for that email address. Choose a password and name to create a new account.
|
||||
|
||||
- If you have been given a registration link, you can enter your email address, password and name to create a new account.
|
||||
|
||||
|
||||
## Quick Start
|
||||
|
||||
Once you've registrated, your account will have an `[Your Name's Archive]` where you can create crawl configs and run crawls!
|
||||
|
||||
|
||||
## Creating a Crawl Config
|
||||
|
||||
19
mkdocs.yml
Normal file
19
mkdocs.yml
Normal file
@ -0,0 +1,19 @@
|
||||
site_name: Browsertrix Cloud
|
||||
theme:
|
||||
name: material
|
||||
features:
|
||||
- navigation.sections
|
||||
- navigation.tabs
|
||||
- navigation.tabs.sticky
|
||||
- navigation.instant
|
||||
font:
|
||||
text: Inter
|
||||
code: Recursive
|
||||
markdown_extensions:
|
||||
- toc:
|
||||
toc_depth: 3
|
||||
- pymdownx.highlight:
|
||||
anchor_linenums: true
|
||||
- pymdownx.inlinehilite
|
||||
- pymdownx.superfences
|
||||
|
||||
Loading…
Reference in New Issue
Block a user