* keep track of per pod status on crawljob: - crashes time, and reason - 'used' vs 'allocated' resources - 'percent' used / allocated * crawl log errors: log error when crawler crashes via OOM, either via redis error log or to console * add initial autoscaling support! - detect if metrics server is available via K8SApi.is_pod_metrics_available() - if available, use metrics for 'used' fields - if no metrics, set memory used for redis only (using redis apis) - allow overriding memory and cpu via newMemory and newCpu settings on pod status - scale memory / cpu based on newMemory and newCpu setting - templates: update jinja templates to allow restarting crawler and redis with new resources - ci: enable metrics-server on k3d, microk8s and nightly k3d ci runs * roles: cleanup unused roles, add permissions for listing metrics * stats for running crawls: - update in db via operator - avoids losing stats if redis pod happens to be done - tradeoff is more db access in operator, but less extra connections to redis + already loading from db in backend - size stat: ensure size of previous files is added to the stats * crawler deployment tweaks: - adjust cpu/mem per browser - add --headless flag to configmap to use new headless mode by default! |
||
---|---|---|
.. | ||
ansible-lint.yaml | ||
deploy-dev.yaml | ||
docs-publish.yaml | ||
frontend-build-check.yaml | ||
k3d-ci.yaml | ||
k3d-log-ci.yaml | ||
k3d-nightly-ci.yaml | ||
lint.yaml | ||
microk8s-ci.yaml | ||
password-check.yaml | ||
project-assign-issue.yml | ||
publish-helm-chart.yaml | ||
release.yaml | ||
ui-tests-playwright.yml |