5b6aa3bc95
* k8s: add tolerations for 'nodeType=crawling:NoSchedule' to allow scheduling crawling on designated nodes for crawler and profiles jobs and statefulsets * add affinity for 'nodeType=crawling' on crawling and profile browser statefulsets * refactor crawljob: combine crawl_updater logic into base crawl_job * increment new 'crawlAttemptCount' counter crawlconfig when crawl is started, not necessarily finished, to avoid deleting configs that had attempted but not finished crawls. * better external mongodb support: use MONGO_DB_URL to set custom url directly, otherwise build from username, password and mongo host |
||
|---|---|---|
| .. | ||
| k8s | ||
| swarm | ||
| __init__.py | ||
| archives.py | ||
| colls.py | ||
| crawl_job.py | ||
| crawlconfigs.py | ||
| crawlmanager.py | ||
| crawls.py | ||
| db.py | ||
| emailsender.py | ||
| invites.py | ||
| main.py | ||
| profile_job.py | ||
| profiles.py | ||
| storages.py | ||
| users.py | ||
| utils.py | ||