browsertrix/backend/btrixcloud
Ilya Kreymer 88a9f3baf7
ensure running crawl configmap is updated when exclusions are added/removed (#2409)
exclusions are already updated dynamically if crawler pod is running,
but when crawler pod is restarted, this ensures new exclusions are also
picked up:
- mount configmap in separate path, avoiding subPath, to allow dynamic
updates of mounted volume
- adds a lastConfigUpdate timestamp to CrawlJob - if lastConfigUpdate in
spec is different from current, the configmap is recreated by operator
- operator: also update image from channel avoid any issues with
updating crawler in channel
- only updates for exclusion add/remove so far, can later be expanded to
other crawler settings (see: #2355 for broader running crawl config
updates)
- fixes #2408
2025-02-19 11:42:19 -08:00
..
migrations Modify page upload migration (#2400) 2025-02-17 16:47:58 -08:00
operator ensure running crawl configmap is updated when exclusions are added/removed (#2409) 2025-02-19 11:42:19 -08:00
__init__.py
auth.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
background_jobs.py Add missing os import 2025-01-13 15:15:48 -08:00
basecrawls.py Add initial pages + pagesQuery endpoint to /replay.json APIs (#2380) 2025-02-13 16:53:47 -08:00
colls.py Add initial pages + pagesQuery endpoint to /replay.json APIs (#2380) 2025-02-13 16:53:47 -08:00
crawlconfigs.py Fix max pages quota setting and display (#2370) 2025-02-10 16:15:21 -08:00
crawlmanager.py ensure running crawl configmap is updated when exclusions are added/removed (#2409) 2025-02-19 11:42:19 -08:00
crawls.py ensure running crawl configmap is updated when exclusions are added/removed (#2409) 2025-02-19 11:42:19 -08:00
db.py Add WACZ filename, depth, favIconUrl, isSeed to pages (#2352) 2025-02-05 15:50:04 -05:00
emailsender.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
invites.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
k8sapi.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
main_bg.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
main_op.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
main.py Add collection page list/search endpoint (#2354) 2025-02-10 16:44:37 -08:00
models.py Modify page upload migration (#2400) 2025-02-17 16:47:58 -08:00
ops.py Modify page upload migration (#2400) 2025-02-17 16:47:58 -08:00
orgs.py feat: Update collection sorting, metadata, stats (#2327) 2025-01-23 13:32:23 -05:00
pages.py Modify page upload migration (#2400) 2025-02-17 16:47:58 -08:00
pagination.py
profiles.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
storages.py Mark all pages from pages.jsonl as seeds (#2390) 2025-02-13 16:54:30 -08:00
subs.py Send subscription cancelation email (#2234) 2024-12-12 11:52:38 -08:00
uploads.py quickfix: fix typo (missing self) that did not make it into #2351 2025-01-30 13:11:42 -08:00
users.py Add superuser endpoint to get user emails with org info (#2211) 2024-12-09 16:38:01 -08:00
utils.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
version.py style change: remove spaces from python version docstring 2025-02-17 16:52:49 -08:00
webhooks.py