Fixes#1525
### Changes
- Changes one of the table cell component usages in the crawl list page
to correctly use the `rowClickTarget` prop, rather than setting the
class to `rowClickTarget`.
- Updates the `rowClickTarget` styling to only apply _within_ a
`<btrix-table-cell>`
Fixes#1387
### Context
While checking some other keyboard navigation issues, I found that I was
unable to create a crawl workflow using only keyboard navigation. This
PR fixes that!
### Changes
- Changes from `<div>`s to `<button>`s so that these can be selected
with tab and enter.
- Adds tabindex for correct selection of items
- Removes the H3 & combines with window title
- Adds width and height to image and width to its container, should make
for a more stable layout while loading (#1387)
Updates Prettier to major version 3, and also updates a couple
prettier-related other things.
Prelude to #1511 so that that PR doesn't include a bunch of unrelated
changes
Fixes#1463
### Changes
- Explains execution time
- Adds style guide section about adding a badge for paid features
- Updates config for mkdocs-material 9.5, materialx emoji support is
being removed.
- Adds better tooltips, a cool feature that also got released with
mkdocs-material 9.5
- Adds search suggestions
### Caveats
- [mkdocs 1.5 has improved the way they handle link
validation](https://www.mkdocs.org/about/release-notes/#expanded-validation-of-links).
Looks like way I've gone about linking things could be improved, and it
will give a bunch of warnings as a result. The site still builds fine,
but I'm going to fix this in a different PR so this one doesn't take as
much effort to review :)
EDIT: Here's that PR
https://github.com/webrecorder/browsertrix-cloud/pull/1476
### Testing
- Make sure you are up to date with `pip install --upgrade
mkdocs-material`
### Screenshot
**Badge!**
<img width="884" alt="Screenshot 2024-01-17 at 11 59 00 PM"
src="https://github.com/webrecorder/browsertrix-cloud/assets/5672810/62a51cf6-24bd-49f1-a6d0-d335f730bfbe">
### Future
- Should mkdocs-material be versioned in our deployment script? We risk
things breaking if I don't get to them fast enough! 🙃
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
## Overview
Adds a bunch of ESLint rules, mostly from `typescript-eslint`, and fixes
the issues turning on these rules raises.
Also updates Typescript & typescript-eslint.
## Rationale
Most of these new rules are auto-fixable, so I've tackled a bunch of the
little fixes that do need manual intervention now with the intention
that this shouldn't add much of any additional friction in future
development work, and also give us a good bump in overall code quality.
A lot of the rules here are also great for catching potential bugs!
## Changes
- Adds `void` to most un-awaited and unhandled promises (i.e. places
where async functions are called but nothing is done with the promise)
- Converts properties that are only ever read to `readonly`
- Adds a new `isApiError` function that informs Typescript of when an
error is an `APIError`
- Adds types to a bunch of places that were previously untyped
- Changes instances of `Map<string, any>` in lit property update methods
to `PropertyValues<this>`, or sometimes `PropertyValues<this> &
Map<string, unknown>` where private or protected members are used
(`keyof` doesn't include private and protected members, unfortunately)
- Adds types to a bunch of custom events
- Cleans up a regex by removing unnecessary escape characters
- Makes a number of implied type conversions explicit (by wrapping with
`Boolean(...)` or calling `.toString()`)
- More consistently applies type coercions when necessary, and removes
them when unnecessary
- Converts a couple const strings to an enum
- Removes the need to type debounced functions as `any` by doing type
coercions to the underlying function type at where the method is bound
to the event in the `html` block
- Updates browser profile list styles to match other data table styles
- Makes entire collection item clickable
- Refactors row click area to fix text overflow
New features & enhancements:
- New UI for collection item selection dialog
- Consistent data table styles for collection list and collection item
list
Refactors:
- Adds `btrix-table` as low-level table component
- Adds `btrix-archived-item-list`, removes `checkbox-list` and
deprecates `crawl-list`
- Upgrades Shoelace for `sl-tree` fixes
- Fixes `ArchivedItem` typing
Fixes#1255
### Changes
- Fixes incorrect time zone conversion when generating UTC schedule in
workflow.
- Fixes minute input display not prefixing single digits with `0`
Co-authored-by: emma <hi@emma.cafe>
Fixes#1341
Adds "User Agent" field to workflow editor under the Browser Settings
tab. If not set, the crawler will use the browser's default user agent.
Also added to docs and to the workflow details page (if set).
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
Fixes#1385
## Changes
Supports multiple crawler 'channels' which can be configured to
different browsertrix-crawler versions
- Replaces `crawler_image` in helm chart with `crawler_channels` array
similar to how storages are handled
- The `default` crawler channel must always be provided and specifies
the default crawler image
- Adds backend `/orgs/{oid}/crawlconfigs/crawler-channels` API endpoint
to fetch information about available crawler versions (name, image, and
label) and test
- Adds crawler channel select to workflow creation/edit screens and
profile creation dialog, and updates related API endpoints and
configmaps accordingly. The select dropdown is shown only if more than
one channel is configured.
- Adds `crawlerChannel` to workflow and crawl details.
- Add `image` to crawler image, used to display actual image used as
part of the crawl.
- Modifies `crawler_crawl_id` backend test fixture to use `test` crawler
version to ensure crawler versions other than latest work
- Adds migration to add `crawlerChannel` set to `default` to existing
workflow and profile objects and workflow configmaps
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Resolves https://github.com/webrecorder/browsertrix-cloud/issues/1333
- Moves "Select Crawls" / "Select Uploads" steps into a single "Select
Archived Items" dialog
- Refactors new collection metadata dialog to accept editing existing
collection
- Prevents RWP component from rendering if there are no archived items
(@Shrinks99 made a comment about this figma, but this prevents
unnecessary requests when there isn't an archive to replay)
- Shows collection description at bottom of detail page at all times
(@Shrinks99 seems useful to see even on archived items view?)
- Switches collection detail primary action to "Add Archived Items" if
none are included (cc @Shrinks99)
- Displays friendlier "name taken" error
- Removes unused Collection edit route
- Upgrades markdown dependencies for fixes/improvements to description
editing
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
- Adds notify, navigate, and log in events to global event map, handle
in `btrix-app`
- Adds console debugs, which are stripped in prod
- Replaces TODO redundant `navTo`s with controller implementation
- Refactors rest of `LitElement` helpers into arrow functions
More-or-less cherry-picked from #1433
## Changes
- Updated the data table to use `formatExecutionSeconds` rather than
`formatSeconds`
- Fixed an issue in `formatExecutionSeconds` where the time in minutes
would sometimes be displayed twice when `options.displaySeconds` was
false or unset
## Testing
Tested locally with orgs with and without execution limits of various
kinds set
Fixes#1430
Uses a date squarely in the middle of the month to avoid timezone
differences from displaying the previous month's name for monthly usage
and execution time data
Fixes#1426
- Update expected org field for execMinutesQuotaReached
- Move event handlers from child components to org index connectedCallback
- Add composed to events
- Improve typing
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Fixes#1358
- Adds `extraExecMinutes` and `giftedExecMinutes` org quotas, which are
not reset monthly but are updateable amounts that carry across months
- Adds `quotaUpdate` field to `Organization` to track when quotas were
updated with timestamp
- Adds `extraExecMinutesAvailable` and `giftedExecMinutesAvailable`
fields to `Organization` to help with tracking available time left
(includes tested migration to initialize these to 0)
- Modifies org backend to track time across multiple categories, using
monthlyExecSeconds, then giftedExecSeconds, then extraExecSeconds.
All time is also written into crawlExecSeconds, which is now the monthly
total and also contains any overage time above the quotas
- Updates Dashboard crawling meter to include all types of execution
time if `extraExecMinutes` and/or `giftedExecMinutes` are set above 0
- Updates Dashboard Usage History table to include all types of
execution time (only displaying columns that have data)
- Adds backend nightly test to check handling of quotas and execution
time
- Includes migration to add new fields and copy crawlExecSeconds to
monthlyExecSeconds for previous months
Co-authored-by: emma <hi@emma.cafe>
- Copies navigation and notification utility methods into separate
controllers
- Adds deprecation notice to `LitElement` methods
- Default type import start to inline
### Context
Components currently can't access `LiteElement` utility methods without
being rendered into the light DOM. This is an initial step towards
breaking out parts of `LiteElement` into composable units. (see
https://github.com/webrecorder/browsertrix-cloud/issues/1380)
### Changes
Moves `apiFetch` from `LiteElement` into a reactive controller. New
components should use `APIController` directly instead of extending
`LiteElement`. We'll also work to move existing uses of `LiteElement`
off of it with time.
### Manual testing
No visible changes, skim through the app to verify that that backend API
fetches work as expected.
Because tests aren't meant to be run on the CI builder, testing
dependencies aren't installed. `fork-ts-checker-webpack-plugin` still
was trying to type-check the test files though, and wasn't finding the
types for them. This changes the `fork-ts-checker-webpack-plugin` config
to ignore test files entirely.
Tested by manually in Docker with `scripts/build-frontend.sh`.
Thanks for your help @tw4l!
Fixes build regression introduced in
https://github.com/webrecorder/browsertrix-cloud/pull/1407
Closes#1405
- Properly uses `typescript-eslint`: we were missing the preset from it,
so some of the default `eslint` rules (that don't properly work with
typescript) were being applied and causing false positives
- I also moved the `eslint` config into its own file, and enabled
`typescript-eslint`'s type-awareness, so that we can enable more
type-aware rules in the future if we like
- Adds `ts-lit-plugin` to the typescript config, which _hopefully_ will
allow us to catch issues during build (in CI)
- It looks like `ts-lit-plugin` is sort of abandonware at the moment,
and unfortunately _doesn't_ actually work for this purpose right now,
but the lit team is working on a replacement here:
https://www.npmjs.com/package/@lit-labs/analyzer
- Adds `fork-ts-checker-webpack-plugin`, which allows the typescript
checking process to be run on a separate forked thread in Webpack, which
can help speed up builds & checking
- Enables incremental type checking for better speed
- Fixes a whole bunch of `eslint`-auto-fixable issues (unused imports
and variables, some type issues, etc)
- Fixes a bunch of `lit-analyzer` issues (mostly attribute naming, some
type issues as well)
- Fixes various other type issues:
- Improves type safety in a bunch of places, notably anywhere `apiFetch`
and `APIPaginatedList` are used
- Removes some `any`s
- Adds confirmation dialog for workflow crawls
- Changes archived item confirmation from default browser dialog to
shoelace dialog
- Increase dialog title size
- Out of scope: Localizes other workflow detail confirmation buttons
- Out of scope: Reword missed "Archive" reference in file uploader
Closes#1294
### Changes
- `crawl-list` component
- Adds a check if there are any items in the actions menu. If not, skip
rendering the actions menu.
- This allows us to give the component no actions! Currently required to
remove them for viewers!
- Collection Details
- Hides "Remove from Collection" option for viewers
- Crawls List
- Removes the single "View Crawl Details" option from archived items for
viewers
- All the other actions were already set up correctly to be used by all
roles!
- Dashboard
- Hides org settings gear icon button unless the user is an admin
- Hides "Create New" dropdown for viewers
- Workflow Details
- Hides workflow edit icon button for viewers
- Hides the "Delete Crawl" option in archived items for viewers
- Hides the "Run Crawl" option for viewers
- Workflow List
- Hides all edit-related options for viewers, the only option now is
copying tags
- Removes the deactivate / delete options (were only visible when
running a crawl) in the workflow list actions
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: sua yoo <sua@suayoo.com>
Closes#1351
### Changes
- Removes the collection share access icon's header icon which was just
an icon. This is now mostly in line with how we display status icons in
the archived items list (there is a spacing difference between the two
lists regarding the placement of the icon vs the label and its alignment
with the text (here) vs the icon (archived items list).
- Adds two new crawl finished state, stopped_by_user and
stopped_quota_reached
- Tracking other possible 'stop reasons' in operator, though not making
them distinct states for now.
- Updated frontend with 'Stopped by User' and 'Stopped: Time Quota
Reached', shown with same icon as current partial_complete
- Added migration of partial_complete to either stopped_by_user or
complete (no historical quota data available)
- Addresses edge case in scaling: if crawl never scaled (no redis entry,
no pod), automatically scale down
- Edge case in status: if crawl is somehow 'canceled' but not deleted,
immediately delete crawl object and begin finalizing.
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
- instead of restarting crawler when exclusion added/removed, add a
message to a redis list (per crawler instance)
- no longer filtering existing queue on backend, now handled via crawler (implemented in 0.12.0 via webrecorder/browsertrix-crawler#408)
- match response optimization: instead of returning first 1000 matches,
limits response to 500K and returns however many matches fit in that
response size (for optional pagination on frontend)
Fixes#1261Closes#1092
The quota for monthly execution minutes is treated as a hard cap. Once
it is exceeded, an alert indicating that an org has exceeded its monthly
execution minutes will display and the user will be unable to start new
crawls. Any running crawls will be stopped once the quota is exceeded.
An execution minutes meter bar is also added in the Org Dashboard and
displayed if a quota is set. More detail in #1305 which was
merged into this branch.
## Changes
- Enable setting 'maxExecMinutesPerMonth' in orgs list quotas by superadmin
- Enforce quota by stopping crawls in operator once quota is reached
- Show alert banner once execution time quota is hit:
- Once quota is hit, disable Run Crawl buttons in frontend, return 403
message with `exec_minutes_quota_reached` detail in backend from
crawl config `/run` endpoint, and don't run new workflows on creation
(similar to storage quota)
- Display execution time for crawls in the crawl details overview,
immediately below
- Show execution minutes meter on dashboard (from #1305)
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: sua yoo <sua@webrecorder.org>
Partially resolves#1223, fixes#1298
- Adds crawl usage table in dashboard under metrics
- Shows skeleton loading indicator when metrics are loading (@Shrinks99
feel free to adjust how this looks)
- Shows max number of concurrent crawls running if any are running ("`running` / `max` Crawls Running")
### Context
- Adds custom padding to each side based on if the tag is removable or not
- Improves hover state for the remove button when the tag is focused
- Adds padding to the remove button
Fixes#1270
After 5 consecutive failed logins from the same user, we now prevent the
user from logging in even with the correct password until they reset it
via their email, or wait an hour.
- After failure threshold is reached, all further login attempts are rejected
- Attempts for invalid email addresses are also tracked
- On 6th try, a reset password email is automatically sent, only once
- Failed login counter resets after an hour of no further logins after last attempted login.
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
### Changes
- URLs on the config review pages are now links that open in a new tab
- Does not do anything with the `Extra URLs in Scope` field (which we currently render as a regex so left that alone)
- Hides / removes the previously deep-linked but now broken config section rendering.
Fixes#1050
Major refactor of the user/auth system to remove fastapi_users
dependency. Refactors users.py to be standalone
and adds new auth.py module for handling auth. UserManager now works
similar to other ops classes.
The auth should be fully backwards compatible with fastapi_users auth,
including accepting previous JWT tokens w/o having to re-login. The User
data model in mongodb is also unchanged.
Additional fixes:
- allows updating fastapi to latest
- add webhook docs to openapi (follow up to #1041)
API changes:
- Removing the`GET, PATCH, DELETE /users/<id>` endpoints, which were not
in used before, as users are scoped to orgs. For deletion, probably
auto-delete when user is removed from last org (to be implemented).
- Rename `/users/me-with-orgs` is renamed to just `/users/me/`
- New `PUT /users/me/change-password` endpoint with password required to update password, fixes #1269, supersedes #1272
Frontend changes:
- Fixes from #1272 to support new change password endpoint.
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Co-authored-by: sua yoo <sua@suayoo.com>
- Replaces org UUID in URL/browser location bar with org slug.
- Refactor: Adds shared app state utility using https://sijakret.github.io/lit-shared-state/ to
access org data from deep descendants.
- Backwards compatible: org UUID URLs should auto-redirect to org slug URLs.
- Show the org UUID in org settings general tab for use with APIs
(Resolves#1258, Follows #1279)
Fast follower https://github.com/webrecorder/browsertrix-cloud/pull/1276
Updates label, info text, and preview text for org slug field to be more user-friendly
use 'Custom URL Identifier' and 'Custom your organization's web address for accessing Browsertrix Cloud'
---------
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Closes#1273
- Viewers can see the share button and the dialogue's sharing info if the collection is sharable
- Viewers can't see or change the share toggle
- Viewers can't see the share button if the collection is not sharable
- Allows editing of org slugs (actual URL updates will be handled in
https://github.com/webrecorder/browsertrix-cloud/issues/1258.)
- Converts user input to slug using slugify
- Adds help text to org name and slug
- Renames tab from "information" to "general" settings
- Adds `position: sticky` to the workflow creator / editor controls to
affix them to the bottom of the screen, they are now always visible!
- Renames "Extra URLs in Scope" to "Extra URL Prefixes in Scope"
- Updates documentation accordingly
- Adjusts casing for checkboxes
- Adds the multiplication sign to the crawler instances settings to
better communicate that they are increases in scale and not arbitrary
numbers.
