fa86555eed
* keep track of per pod status on crawljob: - crashes time, and reason - 'used' vs 'allocated' resources - 'percent' used / allocated * crawl log errors: log error when crawler crashes via OOM, either via redis error log or to console * add initial autoscaling support! - detect if metrics server is available via K8SApi.is_pod_metrics_available() - if available, use metrics for 'used' fields - if no metrics, set memory used for redis only (using redis apis) - allow overriding memory and cpu via newMemory and newCpu settings on pod status - scale memory / cpu based on newMemory and newCpu setting - templates: update jinja templates to allow restarting crawler and redis with new resources - ci: enable metrics-server on k3d, microk8s and nightly k3d ci runs * roles: cleanup unused roles, add permissions for listing metrics * stats for running crawls: - update in db via operator - avoids losing stats if redis pod happens to be done - tradeoff is more db access in operator, but less extra connections to redis + already loading from db in backend - size stat: ensure size of previous files is added to the stats * crawler deployment tweaks: - adjust cpu/mem per browser - add --headless flag to configmap to use new headless mode by default! |
||
|---|---|---|
| .. | ||
| migrations | ||
| templates | ||
| __init__.py | ||
| basecrawls.py | ||
| colls.py | ||
| crawlconfigs.py | ||
| crawlmanager.py | ||
| crawls.py | ||
| db.py | ||
| emailsender.py | ||
| invites.py | ||
| k8sapi.py | ||
| main_op.py | ||
| main.py | ||
| models.py | ||
| operator.py | ||
| orgs.py | ||
| pagination.py | ||
| profiles.py | ||
| storages.py | ||
| uploads.py | ||
| users.py | ||
| utils.py | ||
| version.py | ||
| webhooks.py | ||
| zip.py | ||