7 Commits

Author	SHA1	Message	Date
Ilya Kreymer	60b48ee8a6	dockermanager + scheduler: ... - run as child process using aioprocessing - cleanup: support cleanup of orphaned containers - timeout: support crawlTimeout via check in cleanup loop - support crawl listing + crawl stopping	2021-08-25 15:28:57 -07:00
Ilya Kreymer	b417d7c185	docker manager: support scheduling with apscheduler and separate 'scheduler' process	2021-08-25 12:21:03 -07:00
Ilya Kreymer	91e9fc8699	dockerman: initial pass ... - support for creating, deleting crawlconfigs, running crawls on-demand - config stored in volume - list to docker events and clean up containers when they exit	2021-08-24 22:49:06 -07:00
Ilya Kreymer	627e9a6f14	cleanup crawl config, add separate 'runNow' field ... crawler: add cpu/memory limits minio: auto-create bucket for local minio	2021-08-19 14:15:21 -07:00
Ilya Kreymer	a111bacfb5	add k8s support ... - working apis for adding crawls, removing crawls in mongo, mapped to k8s cronjobs - more complete crawl spec - option to start on-demand job from cronjobs - optional minio in separate deployment/service	2021-06-30 21:48:44 -07:00
Ilya Kreymer	c3143df0a2	rename archives -> storages ... add crawlconfig apis run lint pass, prep for k8s / docker crawl manager support	2021-06-29 20:30:33 -07:00
Ilya Kreymer	b08a188fea	initial commit!	2021-06-28 15:48:59 -07:00