This PR adds Identical Files to the QA Page Match Analysis meter bars.
To do this, the backend calculates the number of non-HTML pages once and
includes it under the key `Files` in each of the `screenshotMatch` and
`textMatch` QA stats return arrays.
The backend additionally removes the file count from "No Data" to
prevent these from being counted twice.
---------
Co-authored-by: emma <hi@emma.cafe>
Resolves https://github.com/webrecorder/browsertrix/issues/1409
### Changes
- Enables clicking on Browser Profiles column header to sort the table, including by starting URL
- More consistent column widths throughout app
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Co-authored-by: emma <hi@emma.cafe>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Follow up to #1815
- Improved detection / responsiveness to unavailable browser
- Disable Save buttons when browser disconnected / unavailable
- Only add unload handler when browser is available
- Consistency between new and edit views: new profile view also has
cancel button with confirmation
- Don't show cancel confirmation if browser is already unavailable.
Various text tweaks for consistency / accuracy:
- Save New Profile... to indicate new profile will be saved after dialog
- Load new browser to indicate new browser will be created
- Mention connection to browser is lost in disconnected message.
Also: migrated to LiteElement -> TailwindElement for browser-profiles-new
-------
Co-authored-by: emma <hi@emma.cafe>
- Shows browser profile last modified or created by name, if available
- Moves backed-up status to browser profile subsection header
- Moves "Last Updated" column to last and displays user name on hover,
to match archived items list view
- Updates browser profile docs
- Paginates browser profile list view
- Moves back-up status icon to list view
- Display last updated date if available (fallback to created date)
- Adds max length validation to description
- Better handle description and crawl workflow list
Closes#1818
- Refactors image renderers, and changes the condition for which the
loading indicator is displayed from "`<data>.blobUrl` exists and has a
truthey value" to "`<data>` is not nullish"
- Adds an additional class config to the shared QA loading indicator,
and uses it in the image slider view to add a background and aspect
ratio to the loading indicator's container so that it takes up the space
it should and has an opaque background
| Before | After |
| --- | --- |
| <img width="813" alt="Screenshot 2024-05-22 at 5 17 14 PM"
src="https://github.com/webrecorder/browsertrix/assets/5727389/fb8a3a83-a0c8-46c1-b59b-a59adff2a175">
| <img width="813" alt="Screenshot 2024-05-22 at 5 17 31 PM"
src="https://github.com/webrecorder/browsertrix/assets/5727389/6fe5d04a-ffa9-4fc3-bc3b-bdb9b7b30948">
|
Resolves https://github.com/webrecorder/browsertrix/issues/438
<!-- Fixes #issue_number -->
### Changes
- Layout and design pass on profile detail view to better match other
detail views
- Ensures profile save buttons are always visible when creating or
editing a profile
- Handle temporarily disconnected browser pings
- Enable recreating browser after it expires
- Show confirmation dialog when attempting to navigate away from active
browser
- Tweak copy referencing profiles
- Refactors `profile-browser` into `TailwindComponent`
### Follow-ups
UX improvements to the browser profile detail layout and list view will
be handled in https://github.com/webrecorder/browsertrix/issues/1816
---------
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
- Fixes browser profile metadata form submission
- Fixes rendering of browser profile description in list view
- Changes label to "metadata" for consistency with other features
Resolves https://github.com/webrecorder/browsertrix/issues/1398
### Changes
- Updates org list in user info state after an org is created
- Minor refactor to add `btrix` prefix to custom `update-user-info`
event and replace unnecessary `defaultOrg` property
- Adds user role select to superadmin dashboard
- Requires org selection (unless there's only one org) to prevent
accidental crawler invites to default testing org
- Shows link to org members after invite, retaining form for quick
re-invite
- Refactor `invite-form` into `TailwindComponent`
Closes#1789
### Changes
- Converts stylesheet in js to css now that it doesn't use any runtime
values
- Removes fixed height on QA container, and instead sets a min height
- Applies `contain: size` to page list & tab content so they don't cause
the QA container to stretch
- Also updates instances of `h-{n} w-{n}` to `size-{n}`
- Removes leftover outlines on text and resource comparison areas from
early QA prototyping
Closes#1787
- Crawls with only one seed will load viewing that seed URL
- URL list crawls with >1 URLs will continue to load the pages list as no
ReplayWeb.page enbed URL has been defined
Co-authored-by: sua yoo <sua@webrecorder.org>
Fixes#1782
- Dash icons are now used to convey status exclusively
- Slash icons are now used to convey no data states
- Updates status icons to filled in the docs (also required for QA
docs!)
Resolves https://github.com/webrecorder/browsertrix/issues/1761
<!-- Fixes #issue_number -->
### Changes
- Switches QA review page chip tooltip colors to dark text on light
background
- Shows newest comment when hovering over comment cell in QA tab page
list to match tooltip in QA review page list
- Validates textarea max entry for description and comments
- Prevents misleading "no text" message shown when switching tabs
- Reset data and show loading indicator when switching pages
- Fix replay tab height and visually differentiates empty replay window
from empty tab
- Refactors console.logs to debugs
Fixes https://github.com/webrecorder/browsertrix/issues/1638
### Changes
- Fixes collection selection during upload and in workflow settings
- Refactors `btrix-collections-add` to `TailwindComponent` with lit
`Task` usage
### Manual testing
1. Log in as crawler
2. Go to "Archived Items"
3. Click "Upload WACZ"
4. Select wacz file to upload
5. Search for collection in "Add to Collection". Verify you're able to
select a search result
6. Save. Verify collection saves as expected.
7. Go to "Crawling"
8. Create a new workflow
9. Go to "Metadata"
10. Repeat 5-6.
Disable the 'Review Crawl' button only when there are no finished QA
Runs:
- track last finished qa run 'lastFinishedQARunId'
- track if qa run is running via 'isQAActive'
- only add tooltip / disabled state when there are no finished qa runs
Also, only add href when not disabled, to avoid clickable link with
disabled button
---------
Co-authored-by: emma <hi@emma.cafe>
Fixes https://github.com/webrecorder/browsertrix/issues/1743
On the backend, support adding user to new org and improved error
messaging:
- if user exists is not part of the org to be added to, add user to the
registration org, return 201
- if user is already part of the org, return 400 with
'user_already_is_org_member' error
- if user is not being added to a new org but already exists, return
'user_already_exists'
frontend:
- if user user same password, they will just be logged in and added to
registration org (same as before)
- if user uses wrong password, show alert message
- handle both user_already_is_org_member and user_already_registered
with alert message and link to log-in page.
note: this means existing user is added to the registration org even if
they provide wrong password, but they won't be able
to login until they use correct password/reset/etc...
Now that we're using a bunch of tooltips, I found when there are very
few items listed the tooltips sometimes get squished in weird ways —
this fixes that.
Related to https://github.com/webrecorder/browsertrix/issues/1477, minor
UI tweaks as fast follow:
- Makes archived item status icons all the same size
- Increases hover hit target of archived item cells
- Reduces top padding of QA review page
- Adds "Back" button to QA review page to match all other pages
- Updates percentage formatting of tab labels to match pages
- Refreshes archived items page every 5 seconds to get in progress QA
runs.
- Refactors archived items to use `TailwindComponent` and lit tasks
- Adds back all columns to upload list to prevent layout shift (see
follow-ups)
- Fixes list not updating after search value is selected.
- Resolves https://github.com/webrecorder/browsertrix/issues/1705
- Resolves https://github.com/webrecorder/browsertrix/issues/1477
Changes:
- Takes user back to archived item QA tab after submitting review.
- Prevents clicking on page in QA tab when there's no analysis runs.
- Disables analysis-related sort options in QA tab when there's no
analysis runs.
- Handles pages without a title in QA tab.
- Adds initial loading state to QA review page list.
Sort of relevant to #1696
- Improves a number of layout elements at smaller viewport sizes
(specifically, letting button groups wrap onto next lines & ensure
titles can't shrink to 0 width)
- Adds "No page title" to places where there'd normally be a page title
but isn't
- Applies grid styling to page list area to fix overflow issues
- When selecting or loading with a page selected that's farther down on
the page list, the top of the page header could be scrolled away from,
and there'd be no way to scroll back because the area had `overflow:
hidden` applied
- Adds default width and height to image comparer element, so that it
displays correctly when images haven't loaded and doesn't change layout
when images load
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
- Shows QA review status, latest QA state, and QA run count in archived
item list. "Created by" column replaced for space and moved to "Date
Created" tooltip
- Enables sort archived items by QA columns
- Adds tooltip to all columns except name
- Hides non-applicable columns in when archived items are filtered to
uploads only
- Fixes issue where tooltips weren't activated on hover when using
`btrix-table`
- Minor refactor to reuse status labels/colors, renames page review ->
page approval to clarify
Repository Index: Generate an index.yaml in ./docx/helm-repo/index.yaml
to allow for browsertrix to be a helm repository.
docs: rename docs.browsertrix.cloud -> docs.browsertrix.com
docs: update deployment doc to mention helm repo as preferred way to
install
docs build action: generate repository index in GH action
publish action: update auto-generated message to mention installing from
the repo.
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Follow-up to #1689
Instead of just using resource types, using a combination of mime type, resource type and extension analysis.
Current logic is:
- if resourceType is script, stylesheet, image, font, use that definitely
- if resourceType is fetch/xhr, also check for extension, specifically for images
- categorize favicon.ico separately as 'favicon'
- also check for extension for json
- also check for pdf by type + extension
- categorize 'ping' as 'other'
- by default, use the first part of mime as the type
### Motivation
While using the browser default locale is often good enough, we probably
want more full control over locales, especially when we allow users to
choose their own locale — it's a poor experience to not have dates and
numbers formatted consistently with your chosen locale.
### Changes
Ensures (almost) all instances of `<sl-format-date>` as well as `Intl.*`
have the correct locale passed into them from our Lit localization
system.
The one exception here is in `frontend/src/utils/number.ts` where
ordinal suffixes aren't localized, so the locale is hardcoded to `en` —
I'll revisit this in the future.
Fixes#1696
- Inverts the toggle button state for screenshot comparison, is now
untoggled by default
- Changes the Resources tab to a puzzle piece
- Parity with the metaphor used in ReplayWeb.page. Would like to leave
databases to represent actual databases... Which given the app could
very well make their way in here!
- Fixes the dashed boarder radius being applied to the text comparison
tab
- Sets the data table header row to `position: sticky;` so it's always
visible if the table scrolls!
- This applies to all data tables, tested the one in QA review, the
overview page, and the org settings members list
- Casing changes for parity with other dropdown sorting controls
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Closes#1655
### Changes
Removes the separate element inside the `<sl-select>` and instead
doesn't show localized language unless it's different from the language
in the current locale
Closes#424
### Changes
- Adds Browsertrix logo to the nav bar!!
- Will license the typeface (Konsole) as soon as this gets merged.
- Changes the theme colour to Webrecorder's turquoise brand colour
- Adjusts the usage of `primary` vs `blue` for elements where it
shouldn't be the same
- Changes the _Cancel & Discard_ button of the cancel crawl dialogue to
`danger` button variant
- Replaces the Replay icon with the new ReplayWebpage icon!
- Fixes the favicon SVG linking (wrong name, oops!)
- Fills the microscope lens in the microscope icon
- Removes the old btrix-cloud.svg logo
Co-authored-by: sua yoo <sua@webrecorder.org>
### Changes
- improves overflow issues at smaller screen sizes
- adds icons to buttons
- updates text & layout to match mocks
- changes primary button & button options depending on if there's a
qa run available
- adds a loading state for qa run status & buttons
- updates `<btrix-crawl-status>` with a `type` param allowing for
crawls, uploads, and QA runs
- Updates `<btrix-alert>` to match `<sl-tag>` styling
- Improves overflow issues at smaller viewport sizes by making tab
lists overflow when necessary
### Features
- Ability to start/stop/cancel QA runs
https://github.com/webrecorder/browsertrix-cloud/pull/1666 @SuaYoo
- Ability to see progress of current QA run @emma-sg
- Ability to delete QA runs @emma-sg
- Ability to download QA run files
https://github.com/webrecorder/browsertrix-cloud/pull/1666 @SuaYoo
- Only able to start review if a QA Run is finished (for now,
initial pass). @SuaYoo
- Only most recent running or successful QA run is displayed in
header
---------
Co-authored-by: sua yoo <sua@webrecorder.org>
Co-authored-by: sua yoo <sua@suayoo.com>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
- fixes#1684
- can be used to optionally restrict QA to only some crawls (eg. with
browsertrix-crawler>=1.0.0)
- enforce error on backend (return 400) and handle special error on the
frontend
- Includes eslint-related files in Dockerfile build step
- Switches to using a cache mount in Dockerfile build step for yarn
cache, rather than deleting yarn cache, which should speed up most
builds
Following [discussion on
Discord](https://discord.com/channels/895426029194207262/910966759165657161/1227299497546223707),
this enables ESLint errors showing up during dev within Webpack, and
also enforces that all issues (both warnings and errors) will cause
builds to fail during CI.
Other changes:
- Updates some dependencies (primarily to versions that are
type-checked)
QA Details page:
- Enables QA tab with ability to start automated analysis QA Run + view a and manual review status
- Pages listed with review status + overall crawl review status shown on QA details (relates to #1508)
- Initial placeholder for QA run analytics (part of #1589)
- Addresses a good deal of #1477
Automated Analysis QA in Review Mode:
- Ability to select from multiple analysis QA runs / view QA runs in QA details
- Shows analysis screenshot, text and resources compare and replay tabs (fixes#1496)
- Sorting by worst screenshot / worst text score for each QA run
- Includes pages sidebar with screenshot/text/resource compare results (fixes#1497)
Manual Review QA in Review Mode:
- Per-page replay available as separate tab (fixes#1499)
- Supports thumbs up, thumbs down, notes for each page
- Supports entering review status approval (good/acceptable/bad can be entered when finishing review
---------
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Closes#1591
### Changes
- Converts one instance of a button with an icon in it to an `icon-button`
- Makes all the trashcan icon buttons have a red hover state
- Adds localization function & placeholder to upload dialog "Name" field
- Adds localization functions to some missing icon-button label
instances
- Adds a few missing icon button labels
Co-authored-by: sua yoo <sua@suayoo.com>
Co-authored-by: sua yoo <sua@webrecorder.org>
Adds a `browserlist` field to `package.json`, which Webpack picks up so
it doesn't convert nullish coalescing operators into obfuscated messes
like `key !== null && key !== void 0 ? key : null`.
This improves output size a little & improves the debugging experience
as well.
Tested in Chrome, FF, & Safari locally and didn't encounter any issues.
See title.
The only place this changes behaviour is in the placeholder page list,
which will be replaced by the real one shortly, so I'm going to just
merge this.
Addresses failing test in
https://github.com/webrecorder/browsertrix-cloud/pull/1592 by fixing
asset imports in unit tests. Unit tests now import an empty string for
all assets--note: if we want to test actual asset content, will need to
update this config.
Follow-up to #1608 — quick fix for an issue I encountered after merging
main into #1497
Just going to directly merge once this completes (cc @SuaYoo for
visibility)
Closes#328
## Changes
The app has favicons now!
Added:
- SVG
- Changes to slightly brighter colours in dark mode for better contrast!
- Fallback ICO
- `apple-touch-icon` (some browsers also use this, not just iOS)
- Web manifest with app description
- Two web manifest icon sizes should users add the app to their local
launcher (Windows' Start or macOS' Dock / Launchpad
- Lighting & render by @emma-sg, thanks!
The manifest and icons are copied to the root directory at build time by
webpack. All of the dedicated ways of doing this seemed more complicated
than this?
---------
Co-authored-by: emma <hi@emma.cafe>
- Backend: Use separate resource constraints for profiles: default
profile browser resources to either 'profile_browser_cpu' /
'profile_browser_memory' or single browser 'crawler_memory_base' /
'crawler_cpu_base', instead of scaled to the number of browser workers
- Frontend: check that profile html page is loading, keep retrying if
still getting nginx error instead of loading an iframe with the error.
Fixes#1598 (Copy of #1599 from 1.9.4)
Closes#1568
## Changes
- Status icons are now filled!
- Uses Bootstrap Icons' new `copy` icon for all actions involving
copying to clipboard!
- Finally! A real copy icon! 🎉
- Removes `copy-code.svg` as it is no longer used
- Actions involving duplicating objects still use `files`... Which is
good! Now they have distinct symbols!
- Adds orange to the tailwind colour palette
---------
Co-authored-by: sua yoo <sua@webrecorder.org>
- Automatically update view to first page if page ID isn't specified
- Show current page URL in location bar (resolves
https://github.com/webrecorder/browsertrix-cloud/issues/1495)
- Approve, reject, or leave notes on a page
- Display temporary list of links to pages in the sidebar
Part of #1241
### Changes
- Renames all instances of "Browsertrix Cloud" to "Browsertrix" on the
front end, emails, and documentation
---------
Co-authored-by: emma <hi@emma.cafe>
Allow maximum scale option to be fully configurable via
`max_crawl_scale`. Already configurable on the backend, and now exposed
to the frontend via API `/api/settings` `maxCrawlScale` value.
The workflow editor and workflow details are updated to allow selecting
the scale up to the maxCrawlScale setting (which defaults to 3 if not
set).
Based on #1534
Figured this should be in place so we can work on other front-end things
with these, rather than dealing with refactoring later
<!-- Fixes #issue_number -->
### Changes
- Adds `ArchivedItemPage` and `ArchivedItemPageComment` types from #1534
(thank you @SuaYoo!)
- Adds typedefs for match and resource count properties
- sets properties optional in the db schema to optional in the type as
well
### Manual testing
1.
### Screenshots
| Page | Image/video |
| ---- | ----------- |
| | |
<!-- ### Follow-ups -->
fix call from when(...) to call function directly, avoid implicit true,
which results in page scope being shown for url list.
fixes#1535
To test:
1) Create new workflow of type URL List
2) Ensure the Crawl Scope drop down is not shown.
---------
Co-authored-by: sua yoo <sua@suayoo.com>
Resolves https://github.com/webrecorder/browsertrix-cloud/issues/1493
<!-- Fixes #issue_number -->
### Changes
Adds WIP QA page with basic grid layout sections and navigation.
### Manual testing
Page can be access by adding `/review/screenshots` or `/review/replay`
to a crawl detail page URL. For example:
```
/orgs/suas-dev-sandbox-2/items/crawl/manual-20240124023524-422e41d6-97d/review/screenshots
```
---------
Co-authored-by: emma <hi@emma.cafe>
Closes#1527
Improves front-end types & ensures the data being accessed matches the
data sent by the back-end.
Tested by hand by using the returned data from the `/orgs/${orgId}`
endpoint in prod where this is happening in dev
Fixes#1525
### Changes
- Changes one of the table cell component usages in the crawl list page
to correctly use the `rowClickTarget` prop, rather than setting the
class to `rowClickTarget`.
- Updates the `rowClickTarget` styling to only apply _within_ a
`<btrix-table-cell>`
Fixes#1387
### Context
While checking some other keyboard navigation issues, I found that I was
unable to create a crawl workflow using only keyboard navigation. This
PR fixes that!
### Changes
- Changes from `<div>`s to `<button>`s so that these can be selected
with tab and enter.
- Adds tabindex for correct selection of items
- Removes the H3 & combines with window title
- Adds width and height to image and width to its container, should make
for a more stable layout while loading (#1387)
Updates Prettier to major version 3, and also updates a couple
prettier-related other things.
Prelude to #1511 so that that PR doesn't include a bunch of unrelated
changes
Fixes#1463
### Changes
- Explains execution time
- Adds style guide section about adding a badge for paid features
- Updates config for mkdocs-material 9.5, materialx emoji support is
being removed.
- Adds better tooltips, a cool feature that also got released with
mkdocs-material 9.5
- Adds search suggestions
### Caveats
- [mkdocs 1.5 has improved the way they handle link
validation](https://www.mkdocs.org/about/release-notes/#expanded-validation-of-links).
Looks like way I've gone about linking things could be improved, and it
will give a bunch of warnings as a result. The site still builds fine,
but I'm going to fix this in a different PR so this one doesn't take as
much effort to review :)
EDIT: Here's that PR
https://github.com/webrecorder/browsertrix-cloud/pull/1476
### Testing
- Make sure you are up to date with `pip install --upgrade
mkdocs-material`
### Screenshot
**Badge!**
<img width="884" alt="Screenshot 2024-01-17 at 11 59 00 PM"
src="https://github.com/webrecorder/browsertrix-cloud/assets/5672810/62a51cf6-24bd-49f1-a6d0-d335f730bfbe">
### Future
- Should mkdocs-material be versioned in our deployment script? We risk
things breaking if I don't get to them fast enough! 🙃
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
## Overview
Adds a bunch of ESLint rules, mostly from `typescript-eslint`, and fixes
the issues turning on these rules raises.
Also updates Typescript & typescript-eslint.
## Rationale
Most of these new rules are auto-fixable, so I've tackled a bunch of the
little fixes that do need manual intervention now with the intention
that this shouldn't add much of any additional friction in future
development work, and also give us a good bump in overall code quality.
A lot of the rules here are also great for catching potential bugs!
## Changes
- Adds `void` to most un-awaited and unhandled promises (i.e. places
where async functions are called but nothing is done with the promise)
- Converts properties that are only ever read to `readonly`
- Adds a new `isApiError` function that informs Typescript of when an
error is an `APIError`
- Adds types to a bunch of places that were previously untyped
- Changes instances of `Map<string, any>` in lit property update methods
to `PropertyValues<this>`, or sometimes `PropertyValues<this> &
Map<string, unknown>` where private or protected members are used
(`keyof` doesn't include private and protected members, unfortunately)
- Adds types to a bunch of custom events
- Cleans up a regex by removing unnecessary escape characters
- Makes a number of implied type conversions explicit (by wrapping with
`Boolean(...)` or calling `.toString()`)
- More consistently applies type coercions when necessary, and removes
them when unnecessary
- Converts a couple const strings to an enum
- Removes the need to type debounced functions as `any` by doing type
coercions to the underlying function type at where the method is bound
to the event in the `html` block
- Updates browser profile list styles to match other data table styles
- Makes entire collection item clickable
- Refactors row click area to fix text overflow
New features & enhancements:
- New UI for collection item selection dialog
- Consistent data table styles for collection list and collection item
list
Refactors:
- Adds `btrix-table` as low-level table component
- Adds `btrix-archived-item-list`, removes `checkbox-list` and
deprecates `crawl-list`
- Upgrades Shoelace for `sl-tree` fixes
- Fixes `ArchivedItem` typing
Fixes#1255
### Changes
- Fixes incorrect time zone conversion when generating UTC schedule in
workflow.
- Fixes minute input display not prefixing single digits with `0`
Co-authored-by: emma <hi@emma.cafe>
Fixes#1341
Adds "User Agent" field to workflow editor under the Browser Settings
tab. If not set, the crawler will use the browser's default user agent.
Also added to docs and to the workflow details page (if set).
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
Fixes#1385
## Changes
Supports multiple crawler 'channels' which can be configured to
different browsertrix-crawler versions
- Replaces `crawler_image` in helm chart with `crawler_channels` array
similar to how storages are handled
- The `default` crawler channel must always be provided and specifies
the default crawler image
- Adds backend `/orgs/{oid}/crawlconfigs/crawler-channels` API endpoint
to fetch information about available crawler versions (name, image, and
label) and test
- Adds crawler channel select to workflow creation/edit screens and
profile creation dialog, and updates related API endpoints and
configmaps accordingly. The select dropdown is shown only if more than
one channel is configured.
- Adds `crawlerChannel` to workflow and crawl details.
- Add `image` to crawler image, used to display actual image used as
part of the crawl.
- Modifies `crawler_crawl_id` backend test fixture to use `test` crawler
version to ensure crawler versions other than latest work
- Adds migration to add `crawlerChannel` set to `default` to existing
workflow and profile objects and workflow configmaps
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Resolves https://github.com/webrecorder/browsertrix-cloud/issues/1333
- Moves "Select Crawls" / "Select Uploads" steps into a single "Select
Archived Items" dialog
- Refactors new collection metadata dialog to accept editing existing
collection
- Prevents RWP component from rendering if there are no archived items
(@Shrinks99 made a comment about this figma, but this prevents
unnecessary requests when there isn't an archive to replay)
- Shows collection description at bottom of detail page at all times
(@Shrinks99 seems useful to see even on archived items view?)
- Switches collection detail primary action to "Add Archived Items" if
none are included (cc @Shrinks99)
- Displays friendlier "name taken" error
- Removes unused Collection edit route
- Upgrades markdown dependencies for fixes/improvements to description
editing
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
- Adds notify, navigate, and log in events to global event map, handle
in `btrix-app`
- Adds console debugs, which are stripped in prod
- Replaces TODO redundant `navTo`s with controller implementation
- Refactors rest of `LitElement` helpers into arrow functions
More-or-less cherry-picked from #1433
## Changes
- Updated the data table to use `formatExecutionSeconds` rather than
`formatSeconds`
- Fixed an issue in `formatExecutionSeconds` where the time in minutes
would sometimes be displayed twice when `options.displaySeconds` was
false or unset
## Testing
Tested locally with orgs with and without execution limits of various
kinds set
Fixes#1430
Uses a date squarely in the middle of the month to avoid timezone
differences from displaying the previous month's name for monthly usage
and execution time data
Fixes#1426
- Update expected org field for execMinutesQuotaReached
- Move event handlers from child components to org index connectedCallback
- Add composed to events
- Improve typing
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Fixes#1358
- Adds `extraExecMinutes` and `giftedExecMinutes` org quotas, which are
not reset monthly but are updateable amounts that carry across months
- Adds `quotaUpdate` field to `Organization` to track when quotas were
updated with timestamp
- Adds `extraExecMinutesAvailable` and `giftedExecMinutesAvailable`
fields to `Organization` to help with tracking available time left
(includes tested migration to initialize these to 0)
- Modifies org backend to track time across multiple categories, using
monthlyExecSeconds, then giftedExecSeconds, then extraExecSeconds.
All time is also written into crawlExecSeconds, which is now the monthly
total and also contains any overage time above the quotas
- Updates Dashboard crawling meter to include all types of execution
time if `extraExecMinutes` and/or `giftedExecMinutes` are set above 0
- Updates Dashboard Usage History table to include all types of
execution time (only displaying columns that have data)
- Adds backend nightly test to check handling of quotas and execution
time
- Includes migration to add new fields and copy crawlExecSeconds to
monthlyExecSeconds for previous months
Co-authored-by: emma <hi@emma.cafe>
- Copies navigation and notification utility methods into separate
controllers
- Adds deprecation notice to `LitElement` methods
- Default type import start to inline
### Context
Components currently can't access `LiteElement` utility methods without
being rendered into the light DOM. This is an initial step towards
breaking out parts of `LiteElement` into composable units. (see
https://github.com/webrecorder/browsertrix-cloud/issues/1380)
### Changes
Moves `apiFetch` from `LiteElement` into a reactive controller. New
components should use `APIController` directly instead of extending
`LiteElement`. We'll also work to move existing uses of `LiteElement`
off of it with time.
### Manual testing
No visible changes, skim through the app to verify that that backend API
fetches work as expected.
Because tests aren't meant to be run on the CI builder, testing
dependencies aren't installed. `fork-ts-checker-webpack-plugin` still
was trying to type-check the test files though, and wasn't finding the
types for them. This changes the `fork-ts-checker-webpack-plugin` config
to ignore test files entirely.
Tested by manually in Docker with `scripts/build-frontend.sh`.
Thanks for your help @tw4l!
Fixes build regression introduced in
https://github.com/webrecorder/browsertrix-cloud/pull/1407
Closes#1405
- Properly uses `typescript-eslint`: we were missing the preset from it,
so some of the default `eslint` rules (that don't properly work with
typescript) were being applied and causing false positives
- I also moved the `eslint` config into its own file, and enabled
`typescript-eslint`'s type-awareness, so that we can enable more
type-aware rules in the future if we like
- Adds `ts-lit-plugin` to the typescript config, which _hopefully_ will
allow us to catch issues during build (in CI)
- It looks like `ts-lit-plugin` is sort of abandonware at the moment,
and unfortunately _doesn't_ actually work for this purpose right now,
but the lit team is working on a replacement here:
https://www.npmjs.com/package/@lit-labs/analyzer
- Adds `fork-ts-checker-webpack-plugin`, which allows the typescript
checking process to be run on a separate forked thread in Webpack, which
can help speed up builds & checking
- Enables incremental type checking for better speed
- Fixes a whole bunch of `eslint`-auto-fixable issues (unused imports
and variables, some type issues, etc)
- Fixes a bunch of `lit-analyzer` issues (mostly attribute naming, some
type issues as well)
- Fixes various other type issues:
- Improves type safety in a bunch of places, notably anywhere `apiFetch`
and `APIPaginatedList` are used
- Removes some `any`s
- Adds confirmation dialog for workflow crawls
- Changes archived item confirmation from default browser dialog to
shoelace dialog
- Increase dialog title size
- Out of scope: Localizes other workflow detail confirmation buttons
- Out of scope: Reword missed "Archive" reference in file uploader
Closes#1294
### Changes
- `crawl-list` component
- Adds a check if there are any items in the actions menu. If not, skip
rendering the actions menu.
- This allows us to give the component no actions! Currently required to
remove them for viewers!
- Collection Details
- Hides "Remove from Collection" option for viewers
- Crawls List
- Removes the single "View Crawl Details" option from archived items for
viewers
- All the other actions were already set up correctly to be used by all
roles!
- Dashboard
- Hides org settings gear icon button unless the user is an admin
- Hides "Create New" dropdown for viewers
- Workflow Details
- Hides workflow edit icon button for viewers
- Hides the "Delete Crawl" option in archived items for viewers
- Hides the "Run Crawl" option for viewers
- Workflow List
- Hides all edit-related options for viewers, the only option now is
copying tags
- Removes the deactivate / delete options (were only visible when
running a crawl) in the workflow list actions
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: sua yoo <sua@suayoo.com>
Closes#1351
### Changes
- Removes the collection share access icon's header icon which was just
an icon. This is now mostly in line with how we display status icons in
the archived items list (there is a spacing difference between the two
lists regarding the placement of the icon vs the label and its alignment
with the text (here) vs the icon (archived items list).
- Adds two new crawl finished state, stopped_by_user and
stopped_quota_reached
- Tracking other possible 'stop reasons' in operator, though not making
them distinct states for now.
- Updated frontend with 'Stopped by User' and 'Stopped: Time Quota
Reached', shown with same icon as current partial_complete
- Added migration of partial_complete to either stopped_by_user or
complete (no historical quota data available)
- Addresses edge case in scaling: if crawl never scaled (no redis entry,
no pod), automatically scale down
- Edge case in status: if crawl is somehow 'canceled' but not deleted,
immediately delete crawl object and begin finalizing.
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
- instead of restarting crawler when exclusion added/removed, add a
message to a redis list (per crawler instance)
- no longer filtering existing queue on backend, now handled via crawler (implemented in 0.12.0 via webrecorder/browsertrix-crawler#408)
- match response optimization: instead of returning first 1000 matches,
limits response to 500K and returns however many matches fit in that
response size (for optional pagination on frontend)
Fixes#1261Closes#1092
The quota for monthly execution minutes is treated as a hard cap. Once
it is exceeded, an alert indicating that an org has exceeded its monthly
execution minutes will display and the user will be unable to start new
crawls. Any running crawls will be stopped once the quota is exceeded.
An execution minutes meter bar is also added in the Org Dashboard and
displayed if a quota is set. More detail in #1305 which was
merged into this branch.
## Changes
- Enable setting 'maxExecMinutesPerMonth' in orgs list quotas by superadmin
- Enforce quota by stopping crawls in operator once quota is reached
- Show alert banner once execution time quota is hit:
- Once quota is hit, disable Run Crawl buttons in frontend, return 403
message with `exec_minutes_quota_reached` detail in backend from
crawl config `/run` endpoint, and don't run new workflows on creation
(similar to storage quota)
- Display execution time for crawls in the crawl details overview,
immediately below
- Show execution minutes meter on dashboard (from #1305)
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: sua yoo <sua@webrecorder.org>
Partially resolves#1223, fixes#1298
- Adds crawl usage table in dashboard under metrics
- Shows skeleton loading indicator when metrics are loading (@Shrinks99
feel free to adjust how this looks)
- Shows max number of concurrent crawls running if any are running ("`running` / `max` Crawls Running")
### Context
- Adds custom padding to each side based on if the tag is removable or not
- Improves hover state for the remove button when the tag is focused
- Adds padding to the remove button
Fixes#1270
After 5 consecutive failed logins from the same user, we now prevent the
user from logging in even with the correct password until they reset it
via their email, or wait an hour.
- After failure threshold is reached, all further login attempts are rejected
- Attempts for invalid email addresses are also tracked
- On 6th try, a reset password email is automatically sent, only once
- Failed login counter resets after an hour of no further logins after last attempted login.
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
### Changes
- URLs on the config review pages are now links that open in a new tab
- Does not do anything with the `Extra URLs in Scope` field (which we currently render as a regex so left that alone)
- Hides / removes the previously deep-linked but now broken config section rendering.
Fixes#1050
Major refactor of the user/auth system to remove fastapi_users
dependency. Refactors users.py to be standalone
and adds new auth.py module for handling auth. UserManager now works
similar to other ops classes.
The auth should be fully backwards compatible with fastapi_users auth,
including accepting previous JWT tokens w/o having to re-login. The User
data model in mongodb is also unchanged.
Additional fixes:
- allows updating fastapi to latest
- add webhook docs to openapi (follow up to #1041)
API changes:
- Removing the`GET, PATCH, DELETE /users/<id>` endpoints, which were not
in used before, as users are scoped to orgs. For deletion, probably
auto-delete when user is removed from last org (to be implemented).
- Rename `/users/me-with-orgs` is renamed to just `/users/me/`
- New `PUT /users/me/change-password` endpoint with password required to update password, fixes #1269, supersedes #1272
Frontend changes:
- Fixes from #1272 to support new change password endpoint.
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Co-authored-by: sua yoo <sua@suayoo.com>
- Replaces org UUID in URL/browser location bar with org slug.
- Refactor: Adds shared app state utility using https://sijakret.github.io/lit-shared-state/ to
access org data from deep descendants.
- Backwards compatible: org UUID URLs should auto-redirect to org slug URLs.
- Show the org UUID in org settings general tab for use with APIs
(Resolves#1258, Follows #1279)
Fast follower https://github.com/webrecorder/browsertrix-cloud/pull/1276
Updates label, info text, and preview text for org slug field to be more user-friendly
use 'Custom URL Identifier' and 'Custom your organization's web address for accessing Browsertrix Cloud'
---------
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Closes#1273
- Viewers can see the share button and the dialogue's sharing info if the collection is sharable
- Viewers can't see or change the share toggle
- Viewers can't see the share button if the collection is not sharable
- Allows editing of org slugs (actual URL updates will be handled in
https://github.com/webrecorder/browsertrix-cloud/issues/1258.)
- Converts user input to slug using slugify
- Adds help text to org name and slug
- Renames tab from "information" to "general" settings
- Adds `position: sticky` to the workflow creator / editor controls to
affix them to the bottom of the screen, they are now always visible!
- Renames "Extra URLs in Scope" to "Extra URL Prefixes in Scope"
- Updates documentation accordingly
- Adjusts casing for checkboxes
- Adds the multiplication sign to the crawler instances settings to
better communicate that they are increases in scale and not arbitrary
numbers.
- Ensures need-login event bubbles until handled
- Redirects on 401 from /refresh endpoint
- Go to previous URL upon login, rather than always to home page
- Shows accurate login notification (rather than less precise "couldn't retrieve org" or similar message)
- Limits URL list entry to 1,000 URLs
- Limits additional URL list entry to 100 URLs
- Shows first invalid URL in list in error message
- Quick and dirty fix for long URLs wrapping: Show URLs in list on one line, with entire container scrolling
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
* Give protocol selection box smaller max-width
* Add warning and docs link to browser profile creation
- Updates dialog styling to btrix dialog
- Updates button sizes
- Updates button placement in dialog
- Updates button labels for consistency with other buttons in app
- Updates docs page with new button labels
* Update browser profile edit metadata dialog. Matches updated dialog shown on profile creation
* Open docs page in new tab
- Require that all passwords are between 8 and 64 characters
- Fixes account settings password reset form to only trigger
logged-in event after successful password change.
- Password validation can be extended within the UserManager's
validate_password method to add or modify requirements.
- Add tests for password validation
- If set, and any of the seeds fails, the entire crawl is marked as a failure.
- Add checkbox which adds --failOnFailedSeed checkbox to URL list workflows
- Add 'Fail Crawl On Failed URL' to crawl workflow setup docs
- Applies user permissions check before deleting anything in all /delete endpoints
- Shuts down running crawls before deleting anything in /all-crawls/delete as well as /crawls/delete
- Splits delete_list.crawl_ids into crawls and upload lists at same time as checks in /all-crawls/delete
- Updates frontend notification message to Only org owners can delete other users' archived items. when a crawler user attempts to delete another users' archived items
- Adds "Logs" tab to workflow detail
- Shows error logs in expandable section in "Watch" tab
- Show corresponding message (no logs yet or logs temporarily unavailable) when `/errors` returns 503 based on crawl state
- text tweaks: use error logs instead of logs, change 'crawl start' -> 'crawl continue' in log message
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
- Remove config.seeds from workflow and crawl detail endpoints
- Add new paginated GET /crawls/{crawl_id}/seeds and /crawlconfigs/{cid}/seeds endpoints to retrieve seeds for a crawl or workflow
- Include firstSeed in GET /crawlconfigs/{cid} endpoint (was missing before)
- Modify frontend to fetch seeds from new /seeds endpoints with loading indicator
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
- Replaces individual "New" buttons in home page with dropdown button in header (includes Crawl Workflow, Upload Collection, Browser Profile)
- Refactors required step of new workflow and new collection into dialog
* Reorders actions, adds tooltip
- All copy buttons on the collection share dialog are now on the right side
- Adds a tooltip to tell the user the button opens the link in a new tab
* Make vertical `dec-list` items fill 100% width of their parent container
- Allows for better placement of items within the container
- Adds horizontal padding to info bars
* Right align copy button in item details page
* Remove config from list endpoints
- Remove config field from workflow and crawl list endpoints
- Add seedCount to CrawlConfigOut on backend and Workflow on frontend
- Refactor CrawlConfig and CrawlConfigOut to extend CrawlConfigCore + CrawlConfigAdditional
- Refactor workflow list in frontend to use firstSeed and seedCount
- Frontend uses ListWorkflow type which is Omit<Workflow, "config">
- Moves metadata tab to first position
- Adds save button to each section, stays in edit view on saving
- Validates name exists before moving to next section or saving
- Changes save button text to "Create Collection without Items" if crawl/uploads aren't selected in new collection
- Fix server error not showing in UI
* update text
* remove trailing slash removal
* make scope help text responsive as user types
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
* use metacontroller's decoratorcontroller to create CrawlJob from Job
* scheduled job work:
- use existing job name for scheduled crawljob
- use suspended job, set startTime, completionTime and succeeded status on job when crawljob is done
- simplify cronjob template: remove job_image, cron_namespace, using same namespace as crawls,
placeholder job image for cronjobs
* move storage quota check to crawljob handler:
- add 'skipped_quota_reached' as new failed status type
- check for storage quota before checking if crawljob can be started, fail if not (check before any pods/pvcs created)
* frontend:
- show all crawls in crawl workflow, no need to filter by status
- add 'skipped_quota_reached' status, show as 'Skipped (Quota Reached)', render same as failed
* migration: make release namespace available as DEFAULT_NAMESPACE, delete old cronjobs in DEFAULT_NAMESPACE and recreate in crawlers namespace with new template
- More specific toast notification error messages to the action being attempted
- Single dismissable global banner shown when org storage is reached
- Removed check for storage quota reached in `runNow`, since buttons are disabled in UI, and errors handled if request fails.
- Allow creating new workflow when storage quota reached
- More responsive storage quota updates: add storageQuotaReached to archived item replay.json, updates w/o reload when crawl pushes quota over limit
- Modify LiteElement to check for storageQuotaReached on GET requests
---------
Co-authored-by: sua yoo <sua@suayoo.com>
- Ability for pod to be Completed, unlike in Statefulset - eg. if 3 pods are running and first one finishes, all 3 must be running until all 3 are done. With this setup, the first finished pod can remain in Completed state.
- Fixed shutdown order - crawler pods now correctly shutdown first before redis pods, by switching to background deletion.
- Pod priority decreases with scale: 1st instance of a new crawl can preempt 3rd or 2nd instance of another crawl
- Create priority classes upto 'max_crawl_scale, configured in values.yaml
- Improved scale change reconciliation: if increasing scale, immediately scale up. If decreasing scale,
graceful stop scaled-down instance to complete via redis 'stopone' key, wait until they exit with Completed state
before adjust status.scale / removing scaled down pods. Ensures unaccepted interrupts don't cause scaled down data to be deleted.
- Redis pod remains inactive until crawler is first active, or after no crawl pods are active for 60 seconds
- Configurable Redis storage with 'redis_storage' value, set to 3Gi by default
- CrawlJob deletion starts as soon as post-finish crawl operations are run
- Post-crawl operations get their own redis instance, since one during response is being cleaned up in finalizer
- Finalizer ignores request with incorrect state (returns 400 if reported as not finished while crawl is finished)
- Current resource usage added to status
- Profile browser: also manage single pod directly without statefulset for consistency.
- Restart pods via restartTime value: if spec.restartTime != status.restartTime, clear out pods and update status.restartTime (using OnDelete policy to avoid recreate loops in edge cases).
- Update to latest metacontroller (v4.11.0)
- Add --restartOnError flag for crawler (for browsertrix-crawler 0.11.0)
- Failed crawl logging: dd 'fail_crawl()' to be used for failing a crawl, which prints logs for default container (if enabled) as well as pod status
- tests: check other finished states to avoid stuck in infinite loop if crawl fails
- tests: disable disk utilization check, which adds unpredictability to crawl testing!
fixes#1147
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
* Implement in backend
- Track bytesStored in org
- Add migration to pre-calculate based on size of crawlfiles and profilefiles
- Add methods to increase or decrease org storage when crawl or profile files
are added or deleted
- Include storageQuotaReached boolean in API responses that alter storage
- Don't start new crawls and fail uploads if storage quota reached
* Implement in frontend
- Add to orgs-list quotas
- Update org's storageQuotaReached based on backend endpoint responses
- Disable buttons when storage quota is met
- Show toast notification when attempting to run a crawl when org
storage quota is met
* supports overriding the replayweb.page version without having to be rebuild frontend image:
- ensures 'rwp_base_url' from helm chart is passed to nginx
- ensures both ui.js and sw.js are loaded based on nginx environment variable, not hard-coded
- ui.js loaded via redirect from new /replay/ui.js path
- pin RWP to known working release in default values.yaml
- remove RWP_BASE_URL from Dockerfile, no longer needed, set via chart env var
- set default RWP_BASE_URL for devserver to use CDN
- set RWP version to 1.8.11
- Lists collections that an archived item belongs to in item detail view
- Improves performance of collection add component
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Backend:
- add 'maxCrawlSize' to models and crawljob spec
- add 'MAX_CRAWL_SIZE' to configmap
- add maxCrawlSize to new crawlconfig + update APIs
- operator: gracefully stop crawl if current size (from stats) exceeds maxCrawlSize
- tests: add max crawl size tests
Frontend:
- Add Max Crawl Size text box Limits tab
- Users enter max crawl size in GB, convert to bytes
- Add BYTES_PER_GB as constant for converting to bytes
- docs: Crawl Size Limit to user guide workflow setup section
Operator Refactor:
- use 'status.stopping' instead of 'crawl.stopping' to indicate crawl is being stopped, as changing later has no effect in operator
- add is_crawl_stopping() to return if crawl is being stopped, based on crawl.stopping or size or time limit being reached
- crawlerjob status: store byte size under 'size', human readable size under 'sizeHuman' for clarity
- size stat always exists so remove unneeded conditional (defaults to 0)
- store raw byte size in 'size', human readable size in 'sizeHuman'
Charts:
- subchart: update crawlerjob crd in btrix-crds to show status.stopping instead of spec.stopping
- subchart: show 'sizeHuman' property instead of 'size'
- bump subchart version to 0.1.1
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
* fix latest crawl (lastRun) sort:
- don't cast 'started' value to string when setting as starting crawl time (regression from #937)
- caused incorrect sorting as finished crawl time was a datetime, while starting crawl time was a string
- move updated config crawl info in one place, simplify to avoid returning started time altogether, just set directly
- pass mdb crawlconfigs and crawls collections directly to add_new_crawl() function
- fixes#1108
* Add dropdown menu containing 'Remove from Collection' to archived items in collection view (#1110)
- Enables users to remove an item from a collection from the collection detail view - menu was previously missing
- Fixes: #1102 (missing dropdown menu) by making use of the inactive menu trigger button.
- Updates collection items page size to match "Archived Items" page size (20 items per page)
---------
Co-authored-by: sua yoo <sua@webrecorder.org>
- rename 'collections' -> 'collectionIds', adding migration 0014
- only populate 'collections' array with {name, id} pair for get_crawl() / single archived item
path, but not for aggregate/list methods
- remove Crawl.get_crawl(), redundant with BaseCrawl.get_crawl() version
- ensure _files_to_resources returns an empty [] instead of none if empty (matching BaseCrawl.get_crawl() behavior to Crawl.get_crawl())
- tests: update tests to use collectionIds for id list, add 'collections' for {name, id} test
- frontend: change Crawl object to have collectionIds instead of collections
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
* refactor to use shared role-based service shared across pods:
- 'crawler' service for all crawler screencasting, scales 0 .. N with crawler-<ID>-N.crawl
- 'redis' service for all redis access, redis-<ID>-0.redis
- 'browser' service for all browser access (profile browsers), browser-<ID>-0.browser
- don't create a new service per crawl/profile at all
- enable 'publishNotReadyAddresses' for potentially faster resolving, esp for redis
- remove service as type managed by operator as no longer creating services dynamically
- remove frontend var CRAWLER_SVC_SUFFIX, suffix always '.crawler' to match crawler service name
