browsertrix/backend/btrixcloud
Ilya Kreymer 793611e5bb
add exclusion api, fixes #311 (#349)
* add exclusion api, fixes #311
add new apis: `POST crawls/{crawl_id}/exclusion?regex=...` and `DELETE crawls/{crawl_id}/exclusion?regex=...` which will:
- create new config with add 'regex' as exclusion (deleting or making inactive previous config) OR remove as exclusion.
- update crawl to point to new config
- update statefulset to point to new config, causing crawler pods to restart
- filter out urls matching 'regex' from both queue and seen list (currently a bit slow) (when adding only)
- return 400 if exclusion already existing when adding, or doesn't exist when removing
- api reads redis list in reverse to match how exclusion queue is used
2022-11-12 17:24:30 -08:00
..
k8s add exclusion api, fixes #311 (#349) 2022-11-12 17:24:30 -08:00
swarm add exclusion api, fixes #311 (#349) 2022-11-12 17:24:30 -08:00
__init__.py refactoring to use statefulsets + job (#245) 2022-06-05 10:37:17 -07:00
archives.py style pass: normalize docstring spacing 2022-10-19 21:47:34 -07:00
colls.py style pass: normalize docstring spacing 2022-10-19 21:47:34 -07:00
crawl_job.py add exclusion api, fixes #311 (#349) 2022-11-12 17:24:30 -08:00
crawlconfigs.py add exclusion api, fixes #311 (#349) 2022-11-12 17:24:30 -08:00
crawlmanager.py add exclusion api, fixes #311 (#349) 2022-11-12 17:24:30 -08:00
crawls.py add exclusion api, fixes #311 (#349) 2022-11-12 17:24:30 -08:00
db.py style pass: normalize docstring spacing 2022-10-19 21:47:34 -07:00
emailsender.py style pass: normalize docstring spacing 2022-10-19 21:47:34 -07:00
invites.py style pass: normalize docstring spacing 2022-10-19 21:47:34 -07:00
main.py style pass: normalize docstring spacing 2022-10-19 21:47:34 -07:00
profile_job.py style pass: normalize docstring spacing 2022-10-19 21:47:34 -07:00
profiles.py style pass: normalize docstring spacing 2022-10-19 21:47:34 -07:00
storages.py style pass: normalize docstring spacing 2022-10-19 21:47:34 -07:00
users.py style pass: normalize docstring spacing 2022-10-19 21:47:34 -07:00
utils.py style pass: normalize docstring spacing 2022-10-19 21:47:34 -07:00