7 Commits

Author	SHA1	Message	Date
Ilya Kreymer	60b48ee8a6	dockermanager + scheduler: ... - run as child process using aioprocessing - cleanup: support cleanup of orphaned containers - timeout: support crawlTimeout via check in cleanup loop - support crawl listing + crawl stopping	2021-08-25 15:28:57 -07:00
Ilya Kreymer	b417d7c185	docker manager: support scheduling with apscheduler and separate 'scheduler' process	2021-08-25 12:21:03 -07:00
Ilya Kreymer	91e9fc8699	dockerman: initial pass ... - support for creating, deleting crawlconfigs, running crawls on-demand - config stored in volume - list to docker events and clean up containers when they exit	2021-08-24 22:49:06 -07:00
Ilya Kreymer	20b19f932f	make crawlTimeout a per-crawconfig property ... allow crawl complete/partial complete to update existing crawl state, eg. timeout enable handling backofflimitexceeded / deadlineexceeded failure, with possible success able to override the failure state filter out only active jobs in running crawls listing	2021-08-24 11:29:15 -07:00
Ilya Kreymer	66c4e618eb	crawls work (#1), support for: ... - canceling a crawl (via sigterm) - stopping a crawl gracefully (via custom exec sigint)	2021-08-23 12:25:04 -07:00
Ilya Kreymer	a8255a76b2	crawljob: ... - support run once on existing crawl job - support updating/patching existing crawl job with new crawl config, new schedule and run once	2021-08-21 22:10:31 -07:00
Ilya Kreymer	170958be37	rename crawls -> crawlconfigs.py ... add crawls for crawl api management	2021-08-20 15:15:51 -07:00