browsertrix/backend/btrixcloud
Ilya Kreymer 989ed2a8da
Use Shared Services for Crawling, Redis, Profile Browsers (#1088)
* refactor to use shared role-based service shared across pods:
- 'crawler' service for all crawler screencasting, scales 0 .. N with crawler-<ID>-N.crawl
- 'redis' service for all redis access, redis-<ID>-0.redis
- 'browser' service for all browser access (profile browsers), browser-<ID>-0.browser
- don't create a new service per crawl/profile at all
- enable 'publishNotReadyAddresses' for potentially faster resolving, esp for redis
- remove service as type managed by operator as no longer creating services dynamically
- remove frontend var CRAWLER_SVC_SUFFIX, suffix always '.crawler' to match crawler service name
2023-08-24 20:08:53 -07:00
..
migrations Backend: standardize search values, filters, and sorting for archived items (#1039) 2023-08-04 09:56:52 -07:00
templates Use Shared Services for Crawling, Redis, Profile Browsers (#1088) 2023-08-24 20:08:53 -07:00
__init__.py
basecrawls.py fix redis connection leaks + exclusions error: (fixes #1065) (#1066) 2023-08-14 18:29:28 -07:00
colls.py fix public collections endpoint returning 404 when not public (#1052) 2023-08-04 13:29:13 -04:00
crawlconfigs.py Add and enforce org maxPagesPerCrawl quota (#1044) 2023-08-23 10:38:36 -04:00
crawlmanager.py Move pydantic models to separate module + refactor crawl response endpoints to be consistent (#983) 2023-07-20 13:05:33 +02:00
crawls.py fix redis connection leaks + exclusions error: (fixes #1065) (#1066) 2023-08-14 18:29:28 -07:00
db.py Backend: standardize search values, filters, and sorting for archived items (#1039) 2023-08-04 09:56:52 -07:00
emailsender.py
invites.py Move pydantic models to separate module + refactor crawl response endpoints to be consistent (#983) 2023-07-20 13:05:33 +02:00
k8sapi.py Use Shared Services for Crawling, Redis, Profile Browsers (#1088) 2023-08-24 20:08:53 -07:00
main_op.py
main_scheduled_job.py fix issues that caused cronjob container to crash: (#987) 2023-07-18 18:08:53 +02:00
main.py Move pydantic models to separate module + refactor crawl response endpoints to be consistent (#983) 2023-07-20 13:05:33 +02:00
models.py Add and enforce org maxPagesPerCrawl quota (#1044) 2023-08-23 10:38:36 -04:00
operator.py Use Shared Services for Crawling, Redis, Profile Browsers (#1088) 2023-08-24 20:08:53 -07:00
orgs.py Add and enforce org maxPagesPerCrawl quota (#1044) 2023-08-23 10:38:36 -04:00
pagination.py Move pydantic models to separate module + refactor crawl response endpoints to be consistent (#983) 2023-07-20 13:05:33 +02:00
profiles.py Use Shared Services for Crawling, Redis, Profile Browsers (#1088) 2023-08-24 20:08:53 -07:00
storages.py Streaming Download for Collections (#1012) 2023-07-26 15:42:17 -07:00
uploads.py Backend: standardize search values, filters, and sorting for archived items (#1039) 2023-08-04 09:56:52 -07:00
users.py Support for Public / Shareable Collections (#1038) 2023-08-03 19:11:01 -07:00
utils.py
version.py bump version to 1.7.0-beta.0 2023-08-23 12:03:45 -07:00
zip.py