Fixes#1158
Introduces two new API endpoints that stream crawling statistics CSVs
(with a suggested attachment filename header):
- `GET /api/orgs/all/crawls/stats` - crawls from all orgs (superuser
only)
- `GET /api/orgs/{oid}/crawls/stats` - crawls from just one org
(available to org crawler/admin users as well as superusers)
Also includes tests for both endpoints.
Fixes#1454
## Motivation
We've had a number of cases recently where a build dependency is added
to `devDependencies`, the PR passes the frontend build check
(`frontend-build-check.yaml`) in the branch, and then fails the cluster
run (`k3d-ci.yaml`) in `main` because the frontend build check installs
all dependencies, whereas the cluster run uses the frontend Dockerfile,
which skips everything but prod dependencies.
## Changes
This runs an additional step in the frontend build check, after running
unit tests and the l10n build but before doing the build, that re-runs
`yarn` with the same arguments as are in the frontend Dockerfile,
installing just prod dependencies.
This results in slightly longer frontend build check runtimes, but
should save us some wasted time fixing broken `main`.
Resolves https://github.com/webrecorder/browsertrix-cloud/issues/1333
- Moves "Select Crawls" / "Select Uploads" steps into a single "Select
Archived Items" dialog
- Refactors new collection metadata dialog to accept editing existing
collection
- Prevents RWP component from rendering if there are no archived items
(@Shrinks99 made a comment about this figma, but this prevents
unnecessary requests when there isn't an archive to replay)
- Shows collection description at bottom of detail page at all times
(@Shrinks99 seems useful to see even on archived items view?)
- Switches collection detail primary action to "Add Archived Items" if
none are included (cc @Shrinks99)
- Displays friendlier "name taken" error
- Removes unused Collection edit route
- Upgrades markdown dependencies for fixes/improvements to description
editing
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
- Adds notify, navigate, and log in events to global event map, handle
in `btrix-app`
- Adds console debugs, which are stripped in prod
- Replaces TODO redundant `navTo`s with controller implementation
- Refactors rest of `LitElement` helpers into arrow functions
More-or-less cherry-picked from #1433
## Changes
- Updated the data table to use `formatExecutionSeconds` rather than
`formatSeconds`
- Fixed an issue in `formatExecutionSeconds` where the time in minutes
would sometimes be displayed twice when `options.displaySeconds` was
false or unset
## Testing
Tested locally with orgs with and without execution limits of various
kinds set
If configmap is missing (eg. was accidentally deleted from k8s) recreate
the configmap when updating the crawl workflow or running a crawl.
Previously, this would result in an error, but now the configmap should
be correctly recreated.
Fixes#1430
Uses a date squarely in the middle of the month to avoid timezone
differences from displaying the previous month's name for monthly usage
and execution time data
Fixes#1426
- Update expected org field for execMinutesQuotaReached
- Move event handlers from child components to org index connectedCallback
- Add composed to events
- Improve typing
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Fixes#1358
- Adds `extraExecMinutes` and `giftedExecMinutes` org quotas, which are
not reset monthly but are updateable amounts that carry across months
- Adds `quotaUpdate` field to `Organization` to track when quotas were
updated with timestamp
- Adds `extraExecMinutesAvailable` and `giftedExecMinutesAvailable`
fields to `Organization` to help with tracking available time left
(includes tested migration to initialize these to 0)
- Modifies org backend to track time across multiple categories, using
monthlyExecSeconds, then giftedExecSeconds, then extraExecSeconds.
All time is also written into crawlExecSeconds, which is now the monthly
total and also contains any overage time above the quotas
- Updates Dashboard crawling meter to include all types of execution
time if `extraExecMinutes` and/or `giftedExecMinutes` are set above 0
- Updates Dashboard Usage History table to include all types of
execution time (only displaying columns that have data)
- Adds backend nightly test to check handling of quotas and execution
time
- Includes migration to add new fields and copy crawlExecSeconds to
monthlyExecSeconds for previous months
Co-authored-by: emma <hi@emma.cafe>
- Copies navigation and notification utility methods into separate
controllers
- Adds deprecation notice to `LitElement` methods
- Default type import start to inline
- Adds a link to the new ReplayWeb.page issue type selection list where
users can file replay specific bugs.
- Updates security email URL now that we have one so this actually gets
shown!
Fixes#1395
- Adds new `POST /orgs/<orgid>/jobs/retryFailed` API endpoint to retry all failed
background jobs for a specific org.
- Also adds `POST /orgs/all/jobs/retryFailed` for superadmin to retry all failed background jobs for all orgs
### Context
Components currently can't access `LiteElement` utility methods without
being rendered into the light DOM. This is an initial step towards
breaking out parts of `LiteElement` into composable units. (see
https://github.com/webrecorder/browsertrix-cloud/issues/1380)
### Changes
Moves `apiFetch` from `LiteElement` into a reactive controller. New
components should use `APIController` directly instead of extending
`LiteElement`. We'll also work to move existing uses of `LiteElement`
off of it with time.
### Manual testing
No visible changes, skim through the app to verify that that backend API
fetches work as expected.
Because tests aren't meant to be run on the CI builder, testing
dependencies aren't installed. `fork-ts-checker-webpack-plugin` still
was trying to type-check the test files though, and wasn't finding the
types for them. This changes the `fork-ts-checker-webpack-plugin` config
to ignore test files entirely.
Tested by manually in Docker with `scripts/build-frontend.sh`.
Thanks for your help @tw4l!
Fixes build regression introduced in
https://github.com/webrecorder/browsertrix-cloud/pull/1407
Closes#1405
- Properly uses `typescript-eslint`: we were missing the preset from it,
so some of the default `eslint` rules (that don't properly work with
typescript) were being applied and causing false positives
- I also moved the `eslint` config into its own file, and enabled
`typescript-eslint`'s type-awareness, so that we can enable more
type-aware rules in the future if we like
- Adds `ts-lit-plugin` to the typescript config, which _hopefully_ will
allow us to catch issues during build (in CI)
- It looks like `ts-lit-plugin` is sort of abandonware at the moment,
and unfortunately _doesn't_ actually work for this purpose right now,
but the lit team is working on a replacement here:
https://www.npmjs.com/package/@lit-labs/analyzer
- Adds `fork-ts-checker-webpack-plugin`, which allows the typescript
checking process to be run on a separate forked thread in Webpack, which
can help speed up builds & checking
- Enables incremental type checking for better speed
- Fixes a whole bunch of `eslint`-auto-fixable issues (unused imports
and variables, some type issues, etc)
- Fixes a bunch of `lit-analyzer` issues (mostly attribute naming, some
type issues as well)
- Fixes various other type issues:
- Improves type safety in a bunch of places, notably anywhere `apiFetch`
and `APIPaginatedList` are used
- Removes some `any`s
- Adds confirmation dialog for workflow crawls
- Changes archived item confirmation from default browser dialog to
shoelace dialog
- Increase dialog title size
- Out of scope: Localizes other workflow detail confirmation buttons
- Out of scope: Reword missed "Archive" reference in file uploader
Closes#1294
### Changes
- `crawl-list` component
- Adds a check if there are any items in the actions menu. If not, skip
rendering the actions menu.
- This allows us to give the component no actions! Currently required to
remove them for viewers!
- Collection Details
- Hides "Remove from Collection" option for viewers
- Crawls List
- Removes the single "View Crawl Details" option from archived items for
viewers
- All the other actions were already set up correctly to be used by all
roles!
- Dashboard
- Hides org settings gear icon button unless the user is an admin
- Hides "Create New" dropdown for viewers
- Workflow Details
- Hides workflow edit icon button for viewers
- Hides the "Delete Crawl" option in archived items for viewers
- Hides the "Run Crawl" option for viewers
- Workflow List
- Hides all edit-related options for viewers, the only option now is
copying tags
- Removes the deactivate / delete options (were only visible when
running a crawl) in the workflow list actions
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: sua yoo <sua@suayoo.com>
Closes#1369
### Changes
- Adds improved getting started steps and intro contact information to
the User Guide homepage
- Adds a small section about the execution minutes graph for orgs with a
quota set
- Moves existing signup content to a dedicated signup page
- Changes admonitions from using em dashes to using colons.
- Em dashes are great and I love em.... But sometimes I love them a
little _too_ much and they were a bad fit here.
- Fixes user guide homepage link
- Fixes `ReplayWeb.page` and `ArchiveWeb.page` names
- Fixes broken links (would be good to have a CI system for this I
think)
---------
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
- Emails are now processed from Jinja2 templates found in
`charts/email-templates`, to support easier updates via helm chart in
the future.
- The available templates are: `invite`, `password_reset`, `validate` and
`failed_bg_job`.
- Each template can be text only or also include HTML. The format of the
template is:
```
subject
~~~
<html content>
~~~
text
```
- A new `support_email` field is also added to the email block in
values.yaml
Invite Template:
- Currently, only the invite template includes an HTML version, other
templates are text only.
- The same template is used for new and existing users, with slightly
different text if adding user to an existing org.
- If user is invited by the superadmin, the invited by field is not
included, otherwise it also includes 'You have been invited by X to join Y'
