a031fab313
Fixes #2182 This rather large PR adds the rest of what should be needed for public collections work in the frontend. New API endpoints include: - Public collections endpoints: GET, streaming download - Paginated list of URLs in collection with snapshot (page) info for each - Collection endpoint to set home URL - Collection endpoint to upload thumbnail as stream - DELETE endpoint to remove collection thumbnail Changes to existing API endpoints include: - Paginating public collection list results - Several `pages` endpoints that previously only supported `/crawls/` in their path, e.g. `/orgs/{oid}/crawls/all/pages/reAdd`, now support `/uploads/` and `/all-crawls/` namespaces as well. This is necessitated by adding pages for uploads to the database (see below). For `/orgs/{oid}/namespace/all/pages/reAdd`, `crawls` or `uploads` will serve as a filter to only affect crawls of that given type. Other endpoints are more liberal at this point, and will perform the same action regardless of the namespace used in the route (we'll likely want to change this in a follow-up to be more consistent). - `/orgs/{oid}/namespace/all/pages/reAdd` now kicks off a background job rather than doing all of the computation in an asyncio task in the backend container. The background job additionally updates collection date ranges, page/size counts, and tags for each collection in the org after pages have been (re)added. Other big changes: - New uploads will now have their pages read into the database! Collection page counts now also include uploads - A migration was added to start a background job for each org that will add the pages for previously-uploaded WACZ files to the database and update collections accordingly - Adds a new `ImageFile` subclass of `BaseFile` for thumbnails that we can use for other user-uploaded image files moving forward, with separate output models for authenticated and public endpoints
63 lines
1.3 KiB
YAML
63 lines
1.3 KiB
YAML
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: "{{ id }}"
|
|
labels:
|
|
role: "background-job"
|
|
job_type: {{ job_type }}
|
|
btrix.org: {{ oid }}
|
|
|
|
spec:
|
|
ttlSecondsAfterFinished: 90
|
|
backoffLimit: 3
|
|
template:
|
|
spec:
|
|
restartPolicy: Never
|
|
priorityClassName: bg-job
|
|
podFailurePolicy:
|
|
rules:
|
|
- action: FailJob
|
|
onExitCodes:
|
|
containerName: btrixbgjob
|
|
operator: NotIn
|
|
values: [0]
|
|
|
|
volumes:
|
|
- name: ops-configs
|
|
secret:
|
|
secretName: ops-configs
|
|
|
|
containers:
|
|
- name: btrixbgjob
|
|
image: {{ backend_image }}
|
|
imagePullPolicy: {{ pull_policy }}
|
|
env:
|
|
- name: BG_JOB_TYPE
|
|
value: {{ job_type }}
|
|
|
|
- name: OID
|
|
value: {{ oid }}
|
|
|
|
- name: CRAWL_TYPE
|
|
value: {{ crawl_type }}
|
|
|
|
envFrom:
|
|
- configMapRef:
|
|
name: backend-env-config
|
|
- secretRef:
|
|
name: mongo-auth
|
|
|
|
volumeMounts:
|
|
- name: ops-configs
|
|
mountPath: /ops-configs/
|
|
|
|
command: ["python3", "-m", "btrixcloud.main_bg"]
|
|
|
|
resources:
|
|
limits:
|
|
memory: "200Mi"
|
|
|
|
requests:
|
|
memory: "200Mi"
|
|
cpu: "50m"
|