Supports setting WARC prefix for WARCs inside WACZ to `<org slug>-<slug
[crawl name | first seed host]>`.
- Prefix set via WARC_PREFIX env var, supported in browsertrix-crawler
1.0.0-beta.4 or higher
If crawl name is provided, uses crawl name, other hostname of first
seed. The name is 'sluggified', using lowercase alphanum characters
separated by dashes.
Ex: in an organization called `Default Org`, a crawl of
`https://specs.webrecorder.net/` and no name will have WARCs named:
`default-org-specs-webrecorder-net-....warc.gz`
If the crawl is given the name `SPECS`, the WARCs will be named
`default-org-specs-manual-....warc.gz`
Fixes#412 in a default way.
fix call from when(...) to call function directly, avoid implicit true,
which results in page scope being shown for url list.
fixes#1535
To test:
1) Create new workflow of type URL List
2) Ensure the Crawl Scope drop down is not shown.
---------
Co-authored-by: sua yoo <sua@suayoo.com>
Resolves https://github.com/webrecorder/browsertrix-cloud/issues/1493
<!-- Fixes #issue_number -->
### Changes
Adds WIP QA page with basic grid layout sections and navigation.
### Manual testing
Page can be access by adding `/review/screenshots` or `/review/replay`
to a crawl detail page URL. For example:
```
/orgs/suas-dev-sandbox-2/items/crawl/manual-20240124023524-422e41d6-97d/review/screenshots
```
---------
Co-authored-by: emma <hi@emma.cafe>
Closes#1527
Improves front-end types & ensures the data being accessed matches the
data sent by the back-end.
Tested by hand by using the returned data from the `/orgs/${orgId}`
endpoint in prod where this is happening in dev
Fixes#1522
## Changes
- Adds further security recommendations to change the password to
accounts you care about after crawling
Adds more details about the capabilities afforded with browser profiles.
This is now split into the following sections:
- Logging into Websites
- Accepting Popups
- Changing Browser Settings
- More in the future??? Extensions???
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
This PR addresses a possible failure when Redis pod was inaccessible
from Crawler pod.
- Ensure crawl is set to 'waiting_for_capacity' if either no crawler
pods are available or no redis pod. previously, missing/inaccessible
redis would not result in 'waiting_for_capacity' if crawler pods are
available
- Rework logic: if no crawler and redis after >60 seconds, shutdown
redis. if crawler and no redis, init (or reinit) redis
- track 'lastUpdatedTime' in db when incrementing exec time to avoid
double counting if lastUpdatedTime has not changed, eg. if operator sync
fails.
- add redis timeout of 20 seconds to avoid timing out operator responses
if redis conn takes too long, assume unavailable
Fixes#1525
### Changes
- Changes one of the table cell component usages in the crawl list page
to correctly use the `rowClickTarget` prop, rather than setting the
class to `rowClickTarget`.
- Updates the `rowClickTarget` styling to only apply _within_ a
`<btrix-table-cell>`
configmap: add --screenshot thumbnail,view as default screenshots
version: update update-version.sh to add newline in version.py to match
new black formatting (from changes in #1507)
Fixes#1519
Fixes#1387
### Context
While checking some other keyboard navigation issues, I found that I was
unable to create a crawl workflow using only keyboard navigation. This
PR fixes that!
### Changes
- Changes from `<div>`s to `<button>`s so that these can be selected
with tab and enter.
- Adds tabindex for correct selection of items
- Removes the H3 & combines with window title
- Adds width and height to image and width to its container, should make
for a more stable layout while loading (#1387)
Updates Prettier to major version 3, and also updates a couple
prettier-related other things.
Prelude to #1511 so that that PR doesn't include a bunch of unrelated
changes
Fixes#1463
### Changes
- Explains execution time
- Adds style guide section about adding a badge for paid features
- Updates config for mkdocs-material 9.5, materialx emoji support is
being removed.
- Adds better tooltips, a cool feature that also got released with
mkdocs-material 9.5
- Adds search suggestions
### Caveats
- [mkdocs 1.5 has improved the way they handle link
validation](https://www.mkdocs.org/about/release-notes/#expanded-validation-of-links).
Looks like way I've gone about linking things could be improved, and it
will give a bunch of warnings as a result. The site still builds fine,
but I'm going to fix this in a different PR so this one doesn't take as
much effort to review :)
EDIT: Here's that PR
https://github.com/webrecorder/browsertrix-cloud/pull/1476
### Testing
- Make sure you are up to date with `pip install --upgrade
mkdocs-material`
### Screenshot
**Badge!**
<img width="884" alt="Screenshot 2024-01-17 at 11 59 00 PM"
src="https://github.com/webrecorder/browsertrix-cloud/assets/5672810/62a51cf6-24bd-49f1-a6d0-d335f730bfbe">
### Future
- Should mkdocs-material be versioned in our deployment script? We risk
things breaking if I don't get to them fast enough! 🙃
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
## Overview
Adds a bunch of ESLint rules, mostly from `typescript-eslint`, and fixes
the issues turning on these rules raises.
Also updates Typescript & typescript-eslint.
## Rationale
Most of these new rules are auto-fixable, so I've tackled a bunch of the
little fixes that do need manual intervention now with the intention
that this shouldn't add much of any additional friction in future
development work, and also give us a good bump in overall code quality.
A lot of the rules here are also great for catching potential bugs!
## Changes
- Adds `void` to most un-awaited and unhandled promises (i.e. places
where async functions are called but nothing is done with the promise)
- Converts properties that are only ever read to `readonly`
- Adds a new `isApiError` function that informs Typescript of when an
error is an `APIError`
- Adds types to a bunch of places that were previously untyped
- Changes instances of `Map<string, any>` in lit property update methods
to `PropertyValues<this>`, or sometimes `PropertyValues<this> &
Map<string, unknown>` where private or protected members are used
(`keyof` doesn't include private and protected members, unfortunately)
- Adds types to a bunch of custom events
- Cleans up a regex by removing unnecessary escape characters
- Makes a number of implied type conversions explicit (by wrapping with
`Boolean(...)` or calling `.toString()`)
- More consistently applies type coercions when necessary, and removes
them when unnecessary
- Converts a couple const strings to an enum
- Removes the need to type debounced functions as `any` by doing type
coercions to the underlying function type at where the method is bound
to the event in the `html` block
- Updates browser profile list styles to match other data table styles
- Makes entire collection item clickable
- Refactors row click area to fix text overflow
New features & enhancements:
- New UI for collection item selection dialog
- Consistent data table styles for collection list and collection item
list
Refactors:
- Adds `btrix-table` as low-level table component
- Adds `btrix-archived-item-list`, removes `checkbox-list` and
deprecates `crawl-list`
- Upgrades Shoelace for `sl-tree` fixes
- Fixes `ArchivedItem` typing
Fixes#1255
### Changes
- Fixes incorrect time zone conversion when generating UTC schedule in
workflow.
- Fixes minute input display not prefixing single digits with `0`
Co-authored-by: emma <hi@emma.cafe>
Fixes#1341
Adds "User Agent" field to workflow editor under the Browser Settings
tab. If not set, the crawler will use the browser's default user agent.
Also added to docs and to the workflow details page (if set).
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
Refactors backend deployment to:
- Use MOTOR_MAX_WORKERS (defaulting to 1) to reduce threads used by
mongodb connections
- Also sets backend workers to 1 by default to reduce default memory
usage
- Switches to gunicorn with uvloop worker for production use instead of
uvicorn (as recommended by uvicorn)
Lower thread count should address memory leak/increased usage, which
resulted in 5x thread x cpus x workers, eg. potentially 20 or 40 threads
just for mongodb connections. Lower default number of workers should
make it easier to scale backend with HPA if additional capacity.
Fixes#1467
Fixes#1385
## Changes
Supports multiple crawler 'channels' which can be configured to
different browsertrix-crawler versions
- Replaces `crawler_image` in helm chart with `crawler_channels` array
similar to how storages are handled
- The `default` crawler channel must always be provided and specifies
the default crawler image
- Adds backend `/orgs/{oid}/crawlconfigs/crawler-channels` API endpoint
to fetch information about available crawler versions (name, image, and
label) and test
- Adds crawler channel select to workflow creation/edit screens and
profile creation dialog, and updates related API endpoints and
configmaps accordingly. The select dropdown is shown only if more than
one channel is configured.
- Adds `crawlerChannel` to workflow and crawl details.
- Add `image` to crawler image, used to display actual image used as
part of the crawl.
- Modifies `crawler_crawl_id` backend test fixture to use `test` crawler
version to ensure crawler versions other than latest work
- Adds migration to add `crawlerChannel` set to `default` to existing
workflow and profile objects and workflow configmaps
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Closes#1434
### Changes
#### Developer
- Adds the K3S playbook guide to the navigation
- Adds note about restarting MKDocs when adding new icons
- Adds note about concise language to the styleguide ([see previous
discussion](https://github.com/webrecorder/browsertrix-cloud/pull/1394#discussion_r1402666872))
- Adds a note about noun usage to the styleguide
#### User guide
- Adds tables for archived item and workflow statuses
- Adds custom styles for displaying statuses with their icons like we do
in the app
- Fixes capitalization issues
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Co-authored-by: sua yoo <sua@webrecorder.org>
Fixes#1157
Adds `./btrix setup` command to `btrix` helper which copies the example
local config to `./chart/local.yaml` where `btrix` expects it.
If another command is run when the local config file doesn't yet exist,
the helper will stop and suggest to the user to run `./btrix setup` and
edit the resulting file.
Fixes#1158
Introduces two new API endpoints that stream crawling statistics CSVs
(with a suggested attachment filename header):
- `GET /api/orgs/all/crawls/stats` - crawls from all orgs (superuser
only)
- `GET /api/orgs/{oid}/crawls/stats` - crawls from just one org
(available to org crawler/admin users as well as superusers)
Also includes tests for both endpoints.
Fixes#1454
## Motivation
We've had a number of cases recently where a build dependency is added
to `devDependencies`, the PR passes the frontend build check
(`frontend-build-check.yaml`) in the branch, and then fails the cluster
run (`k3d-ci.yaml`) in `main` because the frontend build check installs
all dependencies, whereas the cluster run uses the frontend Dockerfile,
which skips everything but prod dependencies.
## Changes
This runs an additional step in the frontend build check, after running
unit tests and the l10n build but before doing the build, that re-runs
`yarn` with the same arguments as are in the frontend Dockerfile,
installing just prod dependencies.
This results in slightly longer frontend build check runtimes, but
should save us some wasted time fixing broken `main`.
