6df10d5fb0
Fixes #1888 Refactors scale handling: - Ensures number of scaled instances does not exceed number of pages, but is also at minimum 1 - Checks for finish condition to be numFailed + numDone >= desired scale - If at least one instance succeeds, crawl considers successful / done. - If all instances fail, crawl considered failed - Ensures that pod done count >= redis done count --------- Co-authored-by: Tessa Walsh <tessa@bitarchivist.net> |
||
|---|---|---|
| .. | ||
| btrixcloud | ||
| test | ||
| test_nightly | ||
| .pylintrc | ||
| Dockerfile | ||
| mypy.ini | ||
| requirements.txt | ||
| test-requirements.txt | ||