browsertrix/backend/btrixcloud
Ilya Kreymer 8ae032ff88 More friendly WARC prefix inside WACZ based on Org slug + Crawl Name / First Seed URL. (#1537)
Supports setting WARC prefix for WARCs inside WACZ to `<org slug>-<slug
[crawl name | first seed host]>`.
- Prefix set via WARC_PREFIX env var, supported in browsertrix-crawler
1.0.0-beta.4 or higher
If crawl name is provided, uses crawl name, other hostname of first
seed. The name is 'sluggified', using lowercase alphanum characters
separated by dashes.

Ex: in an organization called `Default Org`, a crawl of
`https://specs.webrecorder.net/` and no name will have WARCs named:
`default-org-specs-webrecorder-net-....warc.gz`
If the crawl is given the name `SPECS`, the WARCs will be named
`default-org-specs-manual-....warc.gz`

Fixes #412 in a default way.
2024-02-22 23:54:23 -08:00
..
migrations Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
__init__.py
auth.py
background_jobs.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
basecrawls.py
colls.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
crawlconfigs.py More friendly WARC prefix inside WACZ based on Org slug + Crawl Name / First Seed URL. (#1537) 2024-02-22 23:54:23 -08:00
crawlmanager.py More friendly WARC prefix inside WACZ based on Org slug + Crawl Name / First Seed URL. (#1537) 2024-02-22 23:54:23 -08:00
crawls.py better handling of failed redis connection + exec time updates (#1520) 2024-02-09 16:14:29 -08:00
db.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
emailsender.py
invites.py
k8sapi.py More friendly WARC prefix inside WACZ based on Org slug + Crawl Name / First Seed URL. (#1537) 2024-02-22 23:54:23 -08:00
main_op.py
main.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
models.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
operator.py More friendly WARC prefix inside WACZ based on Org slug + Crawl Name / First Seed URL. (#1537) 2024-02-22 23:54:23 -08:00
orgs.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
pagination.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
profiles.py
storages.py storages: use asynccontextmanager instead of sync to close client (#1521) 2024-02-08 08:28:53 -08:00
uploads.py
users.py
utils.py
version.py version: bump to 1.10.0-beta.0 2024-02-20 00:22:29 -08:00
webhooks.py
zip.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00