browsertrix/chart
Ilya Kreymer fb3d88291f
Background Jobs Work (#1321)
Fixes #1252 

Supports a generic background job system, with two background jobs,
CreateReplicaJob and DeleteReplicaJob.
- CreateReplicaJob runs on new crawls, uploads, profiles and updates the
`replicas` array with the info about the replica after the job succeeds.
- DeleteReplicaJob deletes the replica.
- Both jobs are created from the new `replica_job.yaml` template. The
CreateReplicaJob sets secrets for primary storage + replica storage,
while DeleteReplicaJob only needs the replica storage.
- The job is processed in the operator when the job is finalized
(deleted), which should happen immediately when the job is done, either
because it succeeds or because the backoffLimit is reached (currently
set to 3).
- /jobs/ api lists all jobs using a paginated response, including filtering and sorting
- /jobs/<job id> returns details for a particular job
- tests: nightly tests updated to check create + delete replica jobs for crawls as well as uploads, job api endpoints
- tests: also fixes to timeouts in nightly tests to avoid crawls finishing too quickly.

---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
2023-11-02 13:02:17 -07:00
..
admin/logging Add lightweight logging mode (#668) 2023-03-10 14:34:37 -08:00
btrix-crds Add max crawl size option to backend and frontend (#1045) 2023-08-26 22:00:37 -07:00
charts Operator refactor to control pods + pvcs directly instead of statefulsets (#1149) 2023-09-11 10:38:04 -07:00
examples Rename "Production" to "Remote" (#1105) 2023-09-14 15:20:02 -07:00
templates Background Jobs Work (#1321) 2023-11-02 13:02:17 -07:00
test Background Jobs Work (#1321) 2023-11-02 13:02:17 -07:00
.helmignore
Chart.lock Operator refactor to control pods + pvcs directly instead of statefulsets (#1149) 2023-09-11 10:38:04 -07:00
Chart.yaml version: bump to 1.8.0-beta.1 2023-10-27 14:35:24 -07:00
README.md Refactor to use new operator on backend (#789) 2023-04-24 18:30:52 -07:00
values.yaml Background Jobs Work (#1321) 2023-11-02 13:02:17 -07:00

Update Helm dependencies

  • It needs to update Helm charts after changing its dependencies (e.g. logging)
$ helm dependency update .

Update metacontroller

#!/bin/bash

# intall metacontroller
git clone --depth=1 https://github.com/metacontroller/metacontroller.git
cd metacontroller
helm package deploy/helm/metacontroller --destination deploy/helm
cd ..

# update dependency
helm dependency update
  • Bump up the metacontroller version in Chart.yaml