browsertrix/.github/workflows
Ilya Kreymer 00fb8ac048
Concurrent Crawl Limit (#874)
concurrent crawl limits: (addresses #866)
- support limits on concurrent crawls that can be run within a single org
- change 'waiting' state to 'waiting_org_limit' for concurrent crawl limit and 'waiting_capacity' for capacity-based
limits

orgs:
- add 'maxConcurrentCrawl' to new 'quotas' object on orgs
- add /quotas endpoint for updating quotas object

operator:
- add all crawljobs as related, appear to be returned in creation order
- operator: if concurrent crawl limit set, ensures current job is in the first N set of crawljobs (as provided via 'related' list of crawljob objects) before it can proceed to 'starting', otherwise set to 'waiting_org_limit'
- api: add org /quotas endpoint for configuring quotas
- remove 'new' state, always start with 'starting'
- crawljob: add 'oid' to crawljob spec and label for easier querying
- more stringent state transitions: add allowed_from to set_state()
- ensure state transitions only happened from allowed states, while failed/canceled can happen from any state
- ensure finished and state synched from db if transition not allowed
- add crawl indices by oid and cid

frontend: 
- show different waiting states on frontend: 'Waiting (Crawl Limit) and 'Waiting (At Capacity)'
- add gear icon on orgs admin page
- and initial popup for setting org quotas, showing all properties from org 'quotas' object

tests:
- add concurrent crawl limit nightly tests
- fix state waiting -> waiting_capacity
- ci: add logging of operator output on test failure
2023-05-30 15:38:03 -07:00
..
deploy-dev.yaml ci: set env vars for deploy script 2023-02-03 10:54:03 -08:00
docs-publish.yaml
frontend-build-check.yaml Run unit tests in frontend PR check (#569) 2023-02-06 17:47:15 -08:00
k3d-ci.yaml Concurrent Crawl Limit (#874) 2023-05-30 15:38:03 -07:00
k3d-log-ci.yaml ci: 2023-02-08 11:24:54 -08:00
k3d-nightly-ci.yaml tests: fixes for crawl cancel + crawl stopped (#864) 2023-05-22 20:17:29 -07:00
lint.yaml ci: 2023-02-08 11:24:54 -08:00
microk8s-ci.yaml Make pending invites expire via TTL index (#568) 2023-02-14 16:07:14 -05:00
project-assign-issue.yml chore: switch actions for issue assign automation 2023-03-08 10:01:00 -08:00
release.yaml
ui-tests-playwright.yml ci: make playwright integration tests run only on PRs involving frontend 2023-04-05 09:57:34 -07:00