tea/browsertrix
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Docs: Proxy docs copy editing (#2102)

Browse Source 
### Changes
- Minor copy edits and sentence restructures
- Markdown formatting

---------

Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
This commit is contained in:
Henry Wilkinson 2024-10-03 12:18:05 -04:00 committed by GitHub
parent 00efb3615b
commit 2429bb620c
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 48 additions and 55 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

103
docs/deploy/proxies.md
View File

@ -1,10 +1,10 @@
# Configuring Proxies
Browsertrix can be configured to direct crawling traffic through dedicated proxy servers, so that websites can be crawled from a specific geographic location regardless of where Browsertrix itself is deployed.
Browsertrix can be configured to direct crawling traffic through dedicated proxy servers, allowing websites to be crawled from a specific geographic location regardless of where Browsertrix itself is deployed.
The Browsertrix superadmin can configure which proxy servers are available to which organizations (or if they are shared for all organizations) and users can choose from one of the available proxies in each crawl workflow. Users can also configure the default crawling proxy that will be used for the organization in organization-wide [Crawling Defaults](/user-guide/org-settings/#crawling-defaults).
The Browsertrix superadmin can configure which proxy servers are available to which organizations or if they are shared across all organizations, and users can [choose from one of the available proxies in each crawl workflow](../user-guide/workflow-setup.md#proxy). Users can also configure the default crawling proxy that will be used for the organization in organization-wide [Crawling Defaults](../user-guide/org-settings.md#crawling-defaults).
This guide covers how to set up proxy servers for use with Browsertrix, as well as how to configure Browsertrix to make those proxies available.
This guide covers how to set up proxy servers for use with Browsertrix, as well as how to configure Browsertrix to make those proxies available to organizations.
## Proxy Configuration
@ -13,31 +13,30 @@ Browsertrix supports crawling through HTTP and SOCKS5 proxies, including through
### Obtain an SSH Key-pair
To set up a proxy server to use with Browsertrix as SOCKS5 over SSH, you will need an SSH public key-pair and:
- The SSH public key configured on the remote machine
- The SSH private key configured in Browsertrix
- The public host key of the remote machine configured in Browsertrix (optional)
We recommend creating a dedicated SSH key-pair (we recommend an ECDSA key-pair) for use with Browsertrix, as well as a dedicated user, eg. `proxy-user`, and not reusing existing keys or users.
We recommend creating a dedicated SSH key-pair for use with Browsertrix, as well as a dedicated user, e.g. `proxy-user`, and not reusing existing keys or users.
For basic information on how to create a key-pair using `ssh-keygen`, see existing guides such as [this one from DigitalOcean](https://www.digitalocean.com/community/tutorials/how-to-configure-ssh-key-based-authentication-on-a-linux-server) or [this one from ssh.com](https://www.ssh.com/academy/ssh/keygen)
We recommend securing the SSH connection for the proxy user to contain the following settings. This can be done by adding a file
such as `/etc/ssh/sshd_config.d/99-ssh-proxy.conf` where `proxy-user` is the user connecting to the machine.
For basic information on how to create a key-pair using `ssh-keygen`, see existing guides such as [this one from DigitalOcean](https://www.digitalocean.com/community/tutorials/how-to-configure-ssh-key-based-authentication-on-a-linux-server) or [this one from ssh.com](https://www.ssh.com/academy/ssh/keygen). We recommend an ECDSA key-pair.
We recommend securing the SSH connection for the proxy user to contain the following settings. This can be done by adding a file such as `/etc/ssh/sshd_config.d/99-ssh-proxy.conf` where `proxy-user` is the user connecting to the machine.
```
Match User proxy-user
AllowTcpForwarding yes
X11Forwarding no
AllowAgentForwarding no
ForceCommand /bin/false
PubkeyAuthentication yes
PasswordAuthentication no
AllowTcpForwarding yes
X11Forwarding no
AllowAgentForwarding no
ForceCommand /bin/false
PubkeyAuthentication yes
PasswordAuthentication no
```
## Browsertrix Configuration
Proxies are configured in Browsertrix through a separate subchart, and can be configured in the `btrix-proxies` section of the main Helm chart (or local override file) for the Browsertrix deployment. Alternatively, they can be [configured as a separate subchart](#deploying-with-proxies-via-subchart)
Proxies are configured in Browsertrix through a separate subchart, and can be configured in the `btrix-proxies` section of the main Helm chart (or local override file) for the Browsertrix deployment. Alternatively, they can be [configured as a separate subchart](#deploying-with-proxies-via-subchart).
The proxy configuration will look like this, containing one or more proxy declarations.
@ -55,19 +54,18 @@ btrix-proxies:
url: ssh://proxy-user@ssh-proxy-host
ssh_host_public_key: <host public key>
ssh_private_key: <private key>
- id: proxy-id-2
shared: false
label: My SOCKS5 proxy
country_code: DE
url: socks5://username:password@proxy-host
...
- id: proxy-id-2
shared: false
label: My SOCKS5 proxy
country_code: DE
url: socks5://username:password@proxy-host
...
```
First, set `enabled` to `true` to enable proxies.
First, set `enabled` to `true`, which will enable this proxies in Browsertrix.
Next, provide the details of each proxy server that you want available within Browsertrix in the `proxies` list. Minimally, the `id`, `url` connection string, `label` name, and `country_code` two-letter country code must be set for each proxy.
Next, provide the details of each proxy server that you want to make available in the `proxies` list. Minimally, the `id`, `url` connection string, `label` name, and two-letter `country_code` must be set for each proxy.
### SSH Proxies
@ -87,63 +85,59 @@ This method is to be used with dedicated SOCKS5 proxies (not over SSH), such as
### Shared Proxies
The `shared` field on each proxy object defines if this proxy should be accessible to all organizations in a Browsertrix deployment
that are allowed to access shared proxy. If false, the proxy must be added directly to each organization that will have access to the proxy.
The `shared` field on each proxy object defines if this proxy should be accessible to all organizations in a Browsertrix deployment that are allowed to access shared proxy. If false, the proxy must be added directly to each organization that will have access to the proxy.
The proxy settings can be be configured in the super-admin UI by clicking on the 'Edit Proxies...' next to each organization.
The proxy settings can be be configured in the super-admin UI by clicking the _Edit Proxies_ dropdown option in the actions menu for each organization.
### Default Proxy
The `default_proxy` field in the root of the Helm values file can optionally be set to the id for one of the available proxies list. If set, the default proxy will be used for all crawls that do not have an alternate proxy set in the workflow configuration. This can be useful if Browsertrix is deployed on a private network and requires a proxy to access the outside world.
The `default_proxy` field in the root of the Helm values file can optionally be set to the `id` for one of the available proxies list. If set, the default proxy will be used for all crawls that do not have an alternate proxy set in the workflow configuration. This can be useful if Browsertrix is deployed on a private network and requires a proxy to access the outside world.
This is a deployment-wide setting and is not shown to users, and is designed for admins to route all traffic through a designated proxy. Browsertrix will fail to start if the default proxy is not listed in the available proxies.
## Deployment
If `btrix-proxies` have been set in the main Helm chart or a local override file for your Browsertrix deployment, proxies will be enabled on next deploy of the Browsertrix helm chart. For instance, if the proxy configuration is located in a local override file `local.yaml`, you can use the following Helm command to redeploy Browsertrix with the proxy configuration:
If `btrix-proxies` have been set in the main Helm chart or a local override file for your Browsertrix deployment, proxies will be enabled on next deploy of the Browsertrix helm chart. For instance, if the proxy configuration is located in a local override file `local.yaml`, you can use the following Helm command to redeploy Browsertrix with the proxy configuration:
```sh
helm upgrade --install -f ./chart/values.yaml -f ./chart/local.yaml btrix ./chart/
```
### Deploying with Proxies via Subchart
### Deploying with Proxies via subchart
Alternatively, the proxies can also be configured with a separate proxies sub-chart.
Proxies can alternatively be configured with a separate proxies subchart.
This allows for updating proxies without having to redeploy all of Browsertrix.
This allows proxies to be updated without having to redeploy all of Browsertrix.
A separate proxies YAML file should contain just the `proxies` key:
```yaml
proxies:
- id: proxy-id-1
shared: true
label: My Proxy
description: Proxy hosted in for Browsertrix
country_code: US
url: ssh://proxy-user@ssh-proxy-host
ssh_host_public_key: <host public key>
ssh_private_key: <private key>
- id: proxy-id-2
shared: false
label: My SOCKS5 proxy
country_code: DE
url: socks5://username:password@proxy-host
- id: proxy-id-1
shared: true
label: My Proxy
description: Proxy hosted in for Browsertrix
country_code: US
url: ssh://proxy-user@ssh-proxy-host
ssh_host_public_key: <host public key>
ssh_private_key: <private key>
- id: proxy-id-2
shared: false
label: My SOCKS5 proxy
country_code: DE
url: socks5://username:password@proxy-host
```
If the above YAML is placed in `proxies.yaml`, the subchart can be deployed with
If the above YAML is placed in `proxies.yaml`, the subchart can be deployed with the following command and Browsertrix will pick up the updated proxies:
```sh
helm upgrade --install -f ./chart/proxies.yaml proxies ./chart/proxies/
```
The proxies can be updated without redeploying all of Browsertrix, and Browsertrix will pick up the updated proxies.
### GitHub Release for subchart
### GitHub Release for Subchart
The above layout assumes a local copy of Browsertrix repo.
The above layout assumes a local copy of the Browsertrix repo.
The proxies subchart can also be deployed from the latest GitHub release via:
@ -151,7 +145,6 @@ The proxies subchart can also be deployed from the latest GitHub release via:
helm upgrade --install proxies https://github.com/webrecorder/browsertrix/releases/download/RELEASE/btrix-proxies-VERSION.tgz
```
where `RELEASE` are the Browsertrix release and the `VERSION` is the version of the proxies chart.
where `RELEASE` is the Browsertrix release and `VERSION` is the version of the proxies chart.
See the [Browsertrix releases page](https://github.com/webrecorder/browsertrix/releases) for the latest available versions.