browsertrix/chart/templates
Ilya Kreymer 887cb16146
Allow configurable max pages per crawl in deployment settings (#717)
* backend: max pages per crawl limit, part of fix for #716:
- set 'max_pages_crawl_limit' in values.yaml, default to 100,000
- if set/non-0, automatically set limit if none provided
- if set/non-0, return 400 if adding config with limit exceeding max limit
- return limit as 'maxPagesPerCrawl' in /api/settings
- api: /all/crawls - add runningOnly=0 to show all crawls, default to 1/true (for more reliable testing)

tests: add test for 'max_pages_per_crawl' setting
- ensure 'limit' can not be set higher than max_pages_per_crawl
- ensure pages crawled is at the limit
- set test limit to max 2 pages
- add settings test
- check for pages.jsonl and extraPages.jsonl when crawling 2 pages
2023-03-28 16:26:29 -07:00
..
backend.yaml Rename api / nginx settings -> backend / frontend, set pull policy job images (#504) 2023-01-18 20:21:36 -08:00
configmap.yaml Allow configurable max pages per crawl in deployment settings (#717) 2023-03-28 16:26:29 -07:00
frontend.yaml health readiness check: add /healthz endpoint for nginx readiness check, set failure threshold to 3 (similar to ingress) (#562) 2023-02-06 15:08:05 -08:00
ingress.yaml have ingress for signer only when it is enabled 2023-01-06 14:06:33 -08:00
minio.yaml chart / deployment fixes to run on microk8s: (fixes #385) (#387) 2022-11-30 09:21:58 -08:00
mongo.yaml Rename api / nginx settings -> backend / frontend, set pull policy job images (#504) 2023-01-18 20:21:36 -08:00
namespaces.yaml refactoring to use statefulsets + job (#245) 2022-06-05 10:37:17 -07:00
role.yaml refactoring to use statefulsets + job (#245) 2022-06-05 10:37:17 -07:00
secrets.yaml Rename api / nginx settings -> backend / frontend, set pull policy job images (#504) 2023-01-18 20:21:36 -08:00
signer.yaml Rename api / nginx settings -> backend / frontend, set pull policy job images (#504) 2023-01-18 20:21:36 -08:00