Update backend local development docs (#884)

* docs refactor:
- add local deployment guide local-dev-setup.md
- deploy/local.md focuses only on deployment with latest release, links to local-dev-setup.md
for local image deployment
- add nav to mkdocs.yml to ensure correct order of pages
- update microk8s specific info
- update minikube specific info

---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>

docs/deploy/local.md

@@ -1,6 +1,6 @@
 # Local Deployment
 
-To just test out Browsertrix Cloud on your local machine, you'll first need to have a working Kubernetes cluster.
+To try out the latest release of Browsertrix Cloud on your local machine, you'll first need to have a working Kubernetes cluster.
 
 ## Installing Kubernetes
 
@@ -24,7 +24,6 @@ Here are some environment specific instructions for setting up a local cluster f
 
     4. Install [Helm](https://helm.sh/), which can be installed with `brew install helm` (Mac) or `choco install kubernetes-helm` (Windows) or following some of the [other install options](https://helm.sh/docs/intro/install/)
 
-
 ??? info "MicroK8S (recommended for Ubuntu)"
 
     For Ubuntu and other linux distros, we recommend using MicroK8S for both local deployment and production.
@@ -39,13 +38,11 @@ Here are some environment specific instructions for setting up a local cluster f
 
 ??? info "Minikube (Windows, Mac or Linux)"
 
-    1. Install Minikube [following installation instructions](https://minikube.sigs.k8s.io/docs/start/), eg. `brew install minikube`
+    1. Install Minikube [following installation instructions](https://minikube.sigs.k8s.io/docs/start/), eg. `brew install minikube`.
+       Note that Minikube also requires Docker or another container management system to be installed as well.
 
     2. Install [Helm](https://helm.sh/), which can be installed with `brew install helm` (Mac) or `choco install kubernetes-helm` (Windows) or following some of the [other install options](https://helm.sh/docs/intro/install/)
 
-    3. Run the Helm command as described above.
-
-    4. Mac Only: To access Browsertrix Cloud running in minikube on a mac, run `minikube service browsertrix-cloud-frontend --url` and then access Browsertrix Cloud via the provided URL. This is needed as Browsertrix Cloud is running in a VM.
 
 ??? info "K3S (recommended for non-Ubuntu Linux)"
 
@@ -55,15 +52,27 @@ Here are some environment specific instructions for setting up a local cluster f
 
     3. Set `KUBECONFIG` to point to the config for K3S: `export KUBECONFIG=/etc/rancher/k3s/k3s.yaml` to ensure Helm will use the correct version.
 
-
 ## Launching Browsertrix Cloud with Helm
 
 Once you have a running Kubernetes cluster with one of the options above, and Helm 3 installed, you can then run from the Browsertrix Cloud repo directory:
 
 ```shell
-helm upgrade --install -f ./chart/values.yaml -f ./chart/examples/local-config.yaml btrix ./chart/
+helm upgrade --install -f ./chart/values.yaml \
+-f ./chart/examples/local-config.yaml btrix ./chart/
 ```
 
+??? info "MicroK8S"
+
+    If using microk8s, the command will be:
+
+    ```sh
+    microk8s helm3 upgrade --install -f ./chart/values.yaml \
+    -f ./chart/examples/local-config.yaml btrix ./chart/
+    ```
+
+    Subsequent commands will also use `microk8s helm3` instead of `helm`.
+
+
 The local setup includes the full Browsertrix Cloud system, with frontend, backend api, db (via MongoDB) and storage (via Minio)
 
 An admin user with name `admin@example.com` and password `PASSW0RD!` will be automatically created.
@@ -72,8 +81,7 @@ This config uses the standard config (`./chart/values.yaml`) with a couple addit
 
 These settings can be changed in [charts/examples/local-config.yaml](https://github.com/webrecorder/browsertrix-cloud/blob/main/chart/examples/local-config.yaml).
 
-Note that the admin user and password will not be reset after creation.
+For example, to change the default superadmin, uncomment the `superadmin` block in `local-config.yaml`, and then change the username (`admin@example.com`) and password (`PASSW0RD!`) to different values. (The admin username and password will be updated with each deployment)
-
 
 ## Waiting for Cluster to Start
 
@@ -97,6 +105,13 @@ The command will exit when all pods have been loaded, or if there is an error an
 
 If the command succeeds, you should be able to access Browsertrix Cloud by loading: **[http://localhost:30870/](http://localhost:30870/)** in your browser.
 
+??? info "Minikube (on Mac)"
+
+    When using Minikube on a Mac, the port will not be 30870. Instead, Minikube opens a tunnel to a random port,
+    obtained by running `minikube service browsertrix-cloud-frontend --url` in a separate terminal.
+    Use the provided URL (in the format `http://127.0.0.1:<TUNNEL_PORT>`) instead.
+
+
 ### Debugging Pod Issues
 
 If this command fails, you can also run `kubectl get pods` to see the status of each of the pods.
@@ -123,77 +138,17 @@ helm upgrade --install -f ./chart/values.yaml -f ./chart/examples/local-config.y
 
 To uninstall, run `helm uninstall btrix`.
 
-By default, the database + storage volumes are not automatically deleted, so you can run `helm upgrade...` again to restart the cluster in its current state.
+By default, the database + storage volumes are not automatically deleted, so you can run `helm upgrade ...` again to restart the cluster in its current state.
 
-To fully delete all persistent data created in the cluster, also run `kubectl delete pvc --all` after uninstalling.
+If you are upgrading from a previous version, and run into issues with `helm upgrade ...`, we recommend
+uninstalling and then re-running upgrade.
 
+## Deleting all Data
 
-## Running With Local Images
+To fully delete all persistent data (db + archives) created in the cluster, also run `kubectl delete pvc --all` after uninstalling.
 
-By default, this setup will pull the latest release of Browsertrix Cloud. However, if you are developing locally, you may want to use your local images instead.
+## Deploying for Local Development
 
-First, open `./chart/examples/local-config.yaml` and add the following, which will ensure only local images are used:
+These instructions are intended for deploying the cluster from the latest release.
-
+See [setting up cluster for local development](../develop/local-dev-setup.md) for additional customizations related to
-```yaml
+developing Browsertrix Cloud and deploying from local images.
-backend_pull_policy: "Never"
-frontend_pull_policy: "Never"
-```
-
-Now, rebuild either the backend and/or frontend images locally. The exact process depends on the Kubernetes deployment in use:
-
-
-??? info "Docker Desktop"
-
-    Rebuild the local images by running `./scripts/build-backend.sh` and/or `./scripts/build-frontend.sh` scripts to build the images in the local Docker.
-
-
-??? info "MicroK8S"
-
-    MicroK8s uses its own container registry, running on port 32000. 
-
-    1. Set `export REGISTRY=localhost:32000/` and then run `./scripts/build-backend.sh` and/or `./scripts/build-frontend.sh` to rebuild the images into the MicroK8S registry. 
-
-    2. In `./chart/examples/local-config.yaml`, uncomment out one or both of the following lines to use the local images:
-
-    ```yaml
-    backend_image: "localhost:32000/webrecorder/browsertrix-backend:latest"
-    frontend_image: "localhost:32000/webrecorder/browsertrix-frontend:latest"
-    ```
-
-??? info "Minikube" 
-
-    Minikube comes with its own image builder to update the images used in Minikube.
-
-    To build the backend image, run:
-
-    ```shell
-    minikube image build -t webrecorder/browsertrix-backend:latest ./backend
-    ```
-
-    To build a local frontend image, run:
-    
-    ```shell
-    minikube image build -t webrecorder/browsertrix-frontend:latest ./frontend
-    ```
-
-??? info "K3S"
-
-    K3S uses `containerd` by default. To use local images, they need to be imported after rebuilding.
-
-    1. Rebuild the images with Docker by running by running `./scripts/build-backend.sh` and/or `./scripts/build-frontend.sh` scripts. (Requires Docker to be installed as well).
-
-    2. Serializer the images to .tar:
-
-    ```shell
-    docker save webrecorder/browsertrix-backend:latest > ./backend.tar
-    docker save webrecorder/browsertrix-frontend:latest > ./frontend.tar
-    ```
-
-    3. Import images into k3s containerd:
-
-    ```shell
-    k3s ctr images import --base-name webrecorder/browsertrix-backend:latest ./backend.tar
-    k3s ctr images import --base-name webrecorder/browsertrix-frontend:latest ./frontend.tar
-    ```
-
-Once the images have been built and any other config changes made per the above instructions, simply run the `helm upgrade...` command again to restart with local images.

docs/develop/frontend-dev.md

@@ -1,7 +1,15 @@
-# Running the Frontend
+# Running the Frontend for Development
 
 This guide explains how to deploy an instance of the Browsertrix Cloud frontend for development.
-The frontend can connect to a Browsertrix Cloud API backend running locally or remotely.
+
+This setup may be ideal for rapid development on the frontend, as it will allow running the frontend
+via `yarn` instead of the frontend image deployed in Kubernetes.
+
+## Requirements
+
+This setup does require a Browsertrix Cloud API backend already in a Kubernetes cluster.
+
+Refer to [setup for local deployment](./local-dev-setup.md) for additional details on this setup.
 
 ## Quickstart
 
@@ -31,6 +39,15 @@ If connecting to a local deployment cluster, set API_BASE_URL to:
 API_BASE_URL=http://localhost:30870/api
 ```
 
+??? info "Minikube (on Mac)"
+
+    When using Minikube on a Mac, the port will not be 30870. Instead, Minikube opens a tunnel to a random port,
+    obtained by running `minikube service browsertrix-cloud-frontend --url` in a separate terminal.
+
+    Set API_BASE_URL to provided URL instead, eg. `API_BASE_URL=http://127.0.0.1:<TUNNEL_PORT>`
+
+
+
 Start the dev server:
 
 ```sh

docs/develop/index.md

@@ -1,13 +1,24 @@
+---
+hide:
+    - toc
+---
+
 # Developing Browsertrix Cloud
 
 Browsertrix Cloud consists of a Python-based backend and TypeScript-based frontend.
 
+To develop Browsertrix Cloud, the system must [first be deployed locally](../deploy/local.md) in a Kubernetes cluster.
+
+The deployment can then be [further customized for local development](./local-dev-setup.md).
+
 ### Backend
 
 The backend is an API-only system, using the FastAPI framework. The latest API reference is available
 under ./api of a running cluster.
 
-*TODO Add additional info here*
+At this time, the backend must be deployed in the Kubernetes cluster.
+
+<!-- *TODO Add additional info here* -->
 
 ### Frontend
 
@@ -17,5 +28,4 @@ The static build of the frontend is bundled with nginx, but the frontend can be
 
 See [Running Frontend](./frontend-dev) for more details.
 
-*TODO Add additional info here*
+<!-- *TODO Add additional info here* -->
-

docs/develop/local-dev-setup.md

@@ -0,0 +1,136 @@
+# Setup for Local Development
+
+## Installation
+
+First, see our [Local Deployment guide](../deploy/local.md#installing-kubernetes) for instructions on how to install the latest release with [Kubernetes](https://kubernetes.io/) with [Helm 3](https://v3.helm.sh/).
+
+## Local Dev Configuration
+
+The local deployment guide explains how to deploy Browsertrix Cloud with latest published images.
+
+However, if you are developing locally, you will need to use your local images instead.
+
+We recommend the following setup:
+
+1. Copy the provided `./chart/examples/local-config.yaml` Helm configuration file to a separate file `local.yaml`,
+so that local changes to it will not be accidentally committed to git. From the root directory:
+```sh
+cp ./chart/examples/local-config.yaml ./chart/local.yaml
+```
+
+2. Uncomment pull policies in `./chart/local.yaml`, which will ensure the local images are used:
+```yaml
+backend_pull_policy: 'Never'
+frontend_pull_policy: 'Never'
+```
+
+    ??? info "MicroK8S"
+
+        For microk8s, the pull policies actually need to be set to `IfNotPresent` instead of `Never`:
+
+        ```yaml
+        backend_pull_policy: 'IfNotPresent'
+        frontend_pull_policy: 'IfNotPresent'
+        ```
+
+        This will ensure images are pulled from the MicroK8S registry (configured in next section).
+
+
+
+3. Build the local backend and frontend images. The exact process depends on the Kubernetes environment you've selected in your initial deployment. Environment specific build instructions are as follows:
+
+    ??? info "Docker Desktop"
+
+        Rebuild the local images by running `./scripts/build-backend.sh` and/or `./scripts/build-frontend.sh` scripts to build the images in the local Docker.
+
+    ??? info "MicroK8S"
+
+        MicroK8s uses its own container registry, running on port 32000.
+
+        1. Ensure the registry add-on is enabled by running `microk8s enable registry`
+
+        2. Set `export REGISTRY=localhost:32000/` and then run `./scripts/build-backend.sh` and/or `./scripts/build-frontend.sh` to rebuild the images into the MicroK8S registry.
+
+        3. In `./chart/local.yaml`, also uncomment the following lines to use the local images:
+        ```yaml
+        backend_image: "localhost:32000/webrecorder/browsertrix-backend:latest"
+        frontend_image: "localhost:32000/webrecorder/browsertrix-frontend:latest"
+        ```
+
+    ??? info "Minikube"
+
+        Minikube comes with its own image builder to update the images used in Minikube.
+
+        To build the backend image, run:
+
+        ```shell
+        minikube image build -t webrecorder/browsertrix-backend:latest ./backend
+        ```
+
+        To build a local frontend image, run:
+
+        ```shell
+        minikube image build -t webrecorder/browsertrix-frontend:latest ./frontend
+        ```
+
+    ??? info "K3S"
+
+        K3S uses `containerd` by default. To use local images, they need to be imported after rebuilding.
+
+        1. Rebuild the images with Docker by running by running `./scripts/build-backend.sh` and/or `./scripts/build-frontend.sh` scripts. (Requires Docker to be installed as well).
+
+        2. Serializer the images to .tar:
+        ```shell
+        docker save webrecorder/browsertrix-backend:latest > ./backend.tar
+        docker save webrecorder/browsertrix-frontend:latest > ./frontend.tar
+        ```
+
+        3. Import images into k3s containerd:
+        ```shell
+        k3s ctr images import --base-name webrecorder/browsertrix-backend:latest ./backend.tar
+        k3s ctr images import --base-name webrecorder/browsertrix-frontend:latest ./frontend.tar
+        ```
+
+4. If you'd like to change other options, you can uncomment them as needed in `./chart/local.yaml`
+or add additional overrides from `./chart/values.yaml`
+
+    For example, to set a superuser email to `my_super_user_email@example.com` and password to `MySecretPassword!`, uncomment that block and set:
+```yaml
+superuser:
+  # set this to enable a superuser admin
+  email: my_super_user_email@example.com
+
+  # optional: if not set, automatically generated
+  # change or remove this
+  password: MySecretPassword!
+```
+
+5. Once the images have been built and config changes made in `./chart/local.yaml`, 
+
+    the cluster can be re-deployed by running:
+```sh
+helm upgrade --install -f ./chart/values.yaml \
+-f ./chart/local.yaml btrix ./chart/
+```
+
+    ??? info "MicroK8S"
+
+        If using microk8s, the commend will be:
+
+        ```sh
+        microk8s helm3 upgrade --install -f ./chart/values.yaml -f ./chart/local.yaml btrix ./chart/
+        ```
+
+Refer back to the [Local Development guide](../deploy/local.md#waiting-for-cluster-to-start) for additional information on running and debugging your local cluster.
+
+## Update the Images
+
+After making any changes to backend code (in `./backend`) or frontend code (in `./frontend`),
+you'll need to rebuild the images as specified above, before running `helm upgrade ...` to re-deploy.
+
+Changes to settings in `./chart/local.yaml` can be deployed with `helm upgrade ...` directly.
+
+## Deploying Frontend Only
+
+If you are just making changes to the frontend, you can also [deploy the frontend separately](frontend-dev.md)
+using a dev server for quicker iteration.

mkdocs.yml

@@ -38,7 +38,22 @@ theme:
     edit: bootstrap/pencil
     view: bootstrap/eye
 
-
+nav:
+  - index.md
+  - Deploy:
+      - deploy/index.md
+      - deploy/local.md
+      - deploy/production.md
+      - Ansible:
+        - deploy/ansible/digitalocean.md
+        - deploy/ansible/microk8s.md
+  - Develop:
+      - develop/index.md
+      - develop/local-dev-setup.md
+      - develop/frontend-dev.md
+      - develop/docs.md
+  - User Guide:
+      - user-guide/index.md
 
 markdown_extensions:
   - toc: