Ilya Kreymer 5b6aa3bc95 Affinity + Tolerations + Cleanup Crawl Job (#256) ... * k8s: add tolerations for 'nodeType=crawling:NoSchedule' to allow scheduling crawling on designated nodes for crawler and profiles jobs and statefulsets * add affinity for 'nodeType=crawling' on crawling and profile browser statefulsets * refactor crawljob: combine crawl_updater logic into base crawl_job * increment new 'crawlAttemptCount' counter crawlconfig when crawl is started, not necessarily finished, to avoid deleting configs that had attempted but not finished crawls. * better external mongodb support: use MONGO_DB_URL to set custom url directly, otherwise build from username, password and mongo host		2022-06-10 19:21:37 -07:00
..
templates	Affinity + Tolerations + Cleanup Crawl Job (#256)	2022-06-10 19:21:37 -07:00
__init__.py
base_job.py	refactor to use docker swarm for local alternative to k8s instead of docker compose (#247):	2022-06-05 10:37:17 -07:00
crawl_job.py	Affinity + Tolerations + Cleanup Crawl Job (#256)	2022-06-10 19:21:37 -07:00
k8sapi.py	refactor to use docker swarm for local alternative to k8s instead of docker compose (#247):	2022-06-05 10:37:17 -07:00
k8sman.py	refactor to use docker swarm for local alternative to k8s instead of docker compose (#247):	2022-06-05 10:37:17 -07:00
profile_job.py	refactor to use docker swarm for local alternative to k8s instead of docker compose (#247):	2022-06-05 10:37:17 -07:00
utils.py