6df10d5fb0
Fixes #1888 Refactors scale handling: - Ensures number of scaled instances does not exceed number of pages, but is also at minimum 1 - Checks for finish condition to be numFailed + numDone >= desired scale - If at least one instance succeeds, crawl considers successful / done. - If all instances fail, crawl considered failed - Ensures that pod done count >= redis done count --------- Co-authored-by: Tessa Walsh <tessa@bitarchivist.net> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| baseoperator.py | ||
| bgjobs.py | ||
| crawls.py | ||
| cronjobs.py | ||
| models.py | ||
| profiles.py | ||