6 Commits

Author	SHA1	Message	Date
Ilya Kreymer	b417d7c185	docker manager: support scheduling with apscheduler and separate 'scheduler' process	2021-08-25 12:21:03 -07:00
Ilya Kreymer	91e9fc8699	dockerman: initial pass ... - support for creating, deleting crawlconfigs, running crawls on-demand - config stored in volume - list to docker events and clean up containers when they exit	2021-08-24 22:49:06 -07:00
Ilya Kreymer	627e9a6f14	cleanup crawl config, add separate 'runNow' field ... crawler: add cpu/memory limits minio: auto-create bucket for local minio	2021-08-19 14:15:21 -07:00
Ilya Kreymer	a111bacfb5	add k8s support ... - working apis for adding crawls, removing crawls in mongo, mapped to k8s cronjobs - more complete crawl spec - option to start on-demand job from cronjobs - optional minio in separate deployment/service	2021-06-30 21:48:44 -07:00
Ilya Kreymer	c3143df0a2	rename archives -> storages ... add crawlconfig apis run lint pass, prep for k8s / docker crawl manager support	2021-06-29 20:30:33 -07:00
Ilya Kreymer	b08a188fea	initial commit!	2021-06-28 15:48:59 -07:00