tea/browsertrix
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
b6d1e492d7
browsertrix/backend
History
Ilya Kreymer b6d1e492d7 add redis for storing crawl state data! 
- supported in both docker and k8s
- additional pods with same job id automatically use same crawl state in redis
- support dynamic scaling (#2) via /scale endpoint - k8s job parallelism adjusted dynamically for running job (only supported in k8s so far)
2021-09-17 15:02:11 -07:00
..
archives.py crawls work (#1): 2021-08-23 18:01:29 -07:00
crawlconfigs.py add redis for storing crawl state data! 2021-09-17 15:02:11 -07:00
crawls.py add redis for storing crawl state data! 2021-09-17 15:02:11 -07:00
db.py
Dockerfile
dockerman.py add redis for storing crawl state data! 2021-09-17 15:02:11 -07:00
emailsender.py new features: 2021-08-20 11:02:29 -07:00
k8sman.py add redis for storing crawl state data! 2021-09-17 15:02:11 -07:00
main.py add redis for storing crawl state data! 2021-09-17 15:02:11 -07:00
requirements.txt add redis for storing crawl state data! 2021-09-17 15:02:11 -07:00
scheduler.py add missing scheduler! 2021-08-25 16:18:53 -07:00
users.py support usage counters per archive, per user -- handle crawl completion 2021-08-20 23:05:42 -07:00