browsertrix/backend/btrixcloud
Ilya Kreymer 0653657637
better handling of failed redis connection + exec time updates (#1520)
This PR addresses a possible failure when Redis pod was inaccessible
from Crawler pod.
- Ensure crawl is set to 'waiting_for_capacity' if either no crawler
pods are available or no redis pod. previously, missing/inaccessible
redis would not result in 'waiting_for_capacity' if crawler pods are
available
- Rework logic: if no crawler and redis after >60 seconds, shutdown
redis. if crawler and no redis, init (or reinit) redis
- track 'lastUpdatedTime' in db when incrementing exec time to avoid
double counting if lastUpdatedTime has not changed, eg. if operator sync
fails.
- add redis timeout of 20 seconds to avoid timing out operator responses
if redis conn takes too long, assume unavailable
2024-02-09 16:14:29 -08:00
..
migrations Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
__init__.py
auth.py
background_jobs.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
basecrawls.py Add crawl, upload, and collection delete webhook event notifications (#1363) 2023-11-09 18:19:08 -08:00
colls.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
crawlconfigs.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
crawlmanager.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
crawls.py better handling of failed redis connection + exec time updates (#1520) 2024-02-09 16:14:29 -08:00
db.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
emailsender.py Email Templates (#1375) 2023-11-15 15:22:12 -08:00
invites.py Email Templates (#1375) 2023-11-15 15:22:12 -08:00
k8sapi.py better handling of failed redis connection + exec time updates (#1520) 2024-02-09 16:14:29 -08:00
main_op.py
main.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
models.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
operator.py better handling of failed redis connection + exec time updates (#1520) 2024-02-09 16:14:29 -08:00
orgs.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
pagination.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
profiles.py Support multiple crawler versions (#1420) 2024-01-16 15:32:12 -08:00
storages.py storages: use asynccontextmanager instead of sync to close client (#1521) 2024-02-08 08:28:53 -08:00
uploads.py
users.py Add API endpoints for crawl statistics (#1461) 2024-01-10 13:30:47 -08:00
utils.py Add API endpoints for crawl statistics (#1461) 2024-01-10 13:30:47 -08:00
version.py enable screenshots by default + fix py version formatting (#1518) 2024-02-07 17:07:28 -08:00
webhooks.py Add crawl, upload, and collection delete webhook event notifications (#1363) 2023-11-09 18:19:08 -08:00
zip.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00