- Refactors instances of `btrix-tab-list` except in workflow editor in
preparation for https://github.com/webrecorder/browsertrix/issues/2169
- Removes the visual space above navigation item since many tab headings
describe the first section in the tab, rather than the entire tab itself
Also fixes a shoelace menu-item bug where checkbox menu items would have
their checked state flipped on click, regardless of `checked` value.
~~Deploying to dev to test if this fixes language switching...~~ Yep!
Seems to fix the issues.
- Uses crawl count to determine whether workflow can be deleted instead
of last crawl ID
- Display delete confirmation dialog when trying to delete a workflow
Closes#2223
- [x] Adds `localesAvailable` to `/api/settings` endpoint, and uses that
list if available, rather than the full list of translated locales, to
determine which options to display to users
- [x] ~~Uses the user's browser locales, filtered to the current
language setting, for formatting numbers, dates, and durations~~
- [x] Adds & persists checkbox for "use same language for formatting
dates and numbers" in user settings
- [x] Replaces uses of `sl-format-bytes` with `localize.bytes(...)`, and
`sl-format-date` with replacement `btrix-format-date` that properly
handles fallback locales
- [x] Caches all number/duration/datetime formatters by a combined key
consisting of app language, browser language, browser setting, and
formatter options so that all formatters can be reused if needed
(previously any formatter with non-default options would be recreated
every render)
- [x] Splits out ordinal formatting from number formatter, as it didn't
make much sense in some non-English locales
- [x] Adds a little demo of date/time/duration/number formatting so you
can see what effect your language settings have
https://github.com/user-attachments/assets/724858cb-b140-4d72-a38d-83f602c71bc7
---------
Signed-off-by: emma <hi@emma.cafe>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
This has been bugging me for a while — any type of repetitive action
that shows a toast will quickly cause a big stack of toasts to build up.
This fixes that by allowing different categories of toasts to exist,
where each category can only have one active toast. _Different_
categories of toast can coexist just fine, but toasts with the same `id`
will replace any existing toast with that same `id`.
It's been a pet peeve for a while, and it should make any sort of "power
user" use of Browsertrix a lot nicer :)
I've gone over all the existing toasts and applied what feel like decent
differentiations, but I'm not set on any of them — open to suggestions!
https://github.com/user-attachments/assets/dbe39141-9e56-427d-b702-124c45ef2b9a
---------
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
This PR syncs the weblate branch back to main, and adds:
- New Spanish translations contributed in weblate!
- New GH action which reformats weblate commits with `localize:extract`,
ensuring strings are not changed, and also runs `localize:build` to
extract templates (as tested in PR: #2207)
- Reformats existing XLIFF files to match `localize:extract`, including
adding self-closing tags.
- Update on generated templates
- Prettier: disable alpha sorting XML attributes to hopefully minimize
conflcits
- Strings: change some strings that use `&` to be wrapped in html`` to
ensure proper unencoding, or just switch to 'and' where that's not
possible.
---------
Co-authored-by: Weblate (bot) <hosted@weblate.org>
Co-authored-by: Webrecorder Dev <dev@webrecorder.org>
Co-authored-by: Kamborio <Kamborio15@users.noreply.hosted.weblate.org>
Co-authored-by: Clara Itzel <missclaraitzel@gmail.com>
Co-authored-by: weblate <weblate@users.noreply.github.com>
Co-authored-by: emma <hi@emma.cafe>
Co-authored-by: sua yoo <sua@webrecorder.org>
Closes#2213
This ended up being a bit of a variety PR, but primarily this fixes
issues around archived item pages for crawls at URL locations we no
longer support — specifically, crawls, which now live at
`/orgs/{oid}/workflows/{workflow_id}/crawl/{crawl_id}`, rather than
`/orgs/{oid}/items/crawl/{crawl_id}`, including the QA pages at
sub-paths of these. Attempting to view a crawl at the old (unsupported)
URL will now redirect you to the correct URL!
Other changes include:
- Updated links in various parts of the app — nothing should link to the
old URL pattern anymore for crawls!
- Improved `APIRouter` and `ViewState` typing
- A couple extra `APIRouter` tests
- An addition to the route definition for crawls at
`/orgs/{oid}/items/crawl/{crawl_id}` that allows for redirections from
QA review page URLs, rather than 404 errors — this was the actual
reported problem!
- Caching in CI for playwright tests
- `needsLogin` conversion from the deprecated `LiteElement` to
`BtrixElement`, along with all the components that use it
---------
Co-authored-by: emma-sg <emma-sg@users.noreply.github.com>
Partially addresses
https://github.com/webrecorder/browsertrix/issues/2171
- [x] Reimplement `pretty-ms`'s rounding logic to convert to a
`Intl.DurationFormat`-compatible object
- [x] Add `localize.duration` and `localize.humanizeDuration` methods
- [x] Replace `pretty-ms` usage with localized formatting
- [x] Add `Intl.DurationFormat` polyfill
- [x] Update localize tests
- [x] Fix broken tests (thank you @SuaYoo!)
- This looks to be possibly a bug somewhere in the web test runner's
code? Have yet to figure this one out
- Might be related to these:
https://github.com/web-dev-server/web-dev-server/issues/1https://github.com/guybedford/es-module-lexer/issues/82 — both closed
with no comments or resolution 🙃 — `es-module-lexer` seems to be most
likely, given it's a native module & might be handling unicode sequences
incorrectly or not at all?
- [x] Clean up messy `index.d.ts` — probably split this out? The
definitions haven't yet made their way into `esnext`
https://github.com/microsoft/TypeScript/issues/60608
---------
Co-authored-by: emma-sg <emma-sg@users.noreply.github.com>
Co-authored-by: sua yoo <sua@suayoo.com>
- Defaults to user's browser language preference when displaying dates,
bytes, and numbers (with caveats.) This fixes an issue where numbers
were always formatted in English.
- Shows user's browser language preferences in language dropdown
- Fixes timezone not displayed in archived item detail start and finish
times
- Standardizes number formatting
- Sets timezone that unit tests run in
---------
Co-authored-by: SuaYoo <SuaYoo@users.noreply.github.com>
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Co-authored-by: emma-sg <emma-sg@users.noreply.github.com>
Resolves#1354
Supports crawling through pre-configured proxy servers, allowing users to select which proxy servers to use (requires browsertrix crawler 1.3+)
Config:
- proxies defined in btrix-proxies subchart
- can be configured via btrix-proxies key or separate proxies.yaml file via separate subchart
- proxies list refreshed automatically if crawler_proxies.json changes if subchart is deployed
- support for ssh and socks5 proxies
- proxy keys added to secrets in subchart
- support for default proxy to be always used if no other proxy configured, prevent starting cluster if default proxy not available
- prevent starting manual crawl if previously configured proxy is no longer available, return error
- force 'btrix' username and group name on browsertrix-crawler non-root user to support ssh
Operator:
- support crawling through proxies, pass proxyId in CrawlJob
- support running profile browsers which designated proxy, pass proxyId to ProfileJob
- prevent starting scheduled crawl if previously configured proxy is no longer available
API / Access:
- /api/orgs/all/crawlconfigs/crawler-proxies - get all proxies (superadmin only)
- /api/orgs/{oid}/crawlconfigs/crawler-proxies - get proxies available to particular org
- /api/orgs/{oid}/proxies - update allowed proxies for particular org (superadmin only)
- superadmin can configure which orgs can use which proxies, stored on the org
- superadmin can also allow an org to access all 'shared' proxies, to avoid having to allow a shared proxy on each org.
UI:
- Superadmin has 'Edit Proxies' dialog to configure for each org if it has: dedicated proxies, has access to shared proxies.
- User can select a proxy in Crawl Workflow browser settings
- Users can choose to launch a browser profile with a particular proxy
- Display which proxy is used to create profile in profile selector
- Users can choose with default proxy to use for new workflows in Crawling Defaults
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Resolves https://github.com/webrecorder/browsertrix/issues/2073
### Changes
- Removes "URL List" and "Seeded Crawl" job type distinction and adds as
additional crawl scope types instead.
- 'New Workflow' button defaults to Single Page
- 'New Workflow' dropdown includes Page Crawl (Single Page, Page List, In-Page Links) and Site Crawl (Page in Same Directory, Page on Same Domain, + Subdomains and Custom Page Prefix)
- Enables specifying `DOCS_URL` in `.env`
- Additional follow-ups in #2090, #2091
Use timezone aware datetimes instead of timezone naive datetimes:
- Update mongodb client to use tz-aware conversion
- Convert dt_now() to return timezone aware UTC date
- Rename to_k8s_date -> date_to_str, just returns ISO UTC date with 'Z'
(instead of '+00:00' suffix)
- Rename from_k8s_date -> str_to_date, returns timezone aware date from
str
- Standardize all string<->date conversion to use either date_to_str or
str_to_date
- Update frontend to assume iso date, not append 'Z' directly
- Update tests to check for 'Z' suffix on some dates
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
- Fixes empty watch tab during some active states
- Disables exclusion and browser window editing while a crawl is
stopping
- Refactors frontend crawl states to match backend
- Replaces QA Review breadcrumbs with "Back" button that takes you back
to the archived item
- Crawl breadcrumbs always point to workflow
- Shows archived item icon next to item name to differentiate from
workflow
- Adds "Edit Workflow" button to "Crawl Settings"
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
- Adds breadcrumb navigation to all detail views, returning to correct
origin for workflow and collection items
- Refactors list page headers into layout utility
- Refactors crawl tab labels and renames "Files" tab to "WACZ Files"
- Adds new org settings tab for updating crawl details
- Refactors `workflow-editor` to move out utility functions
- Updates user guide on org settings
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Fixes#1782
- Dash icons are now used to convey status exclusively
- Slash icons are now used to convey no data states
- Updates status icons to filled in the docs (also required for QA
docs!)
### Motivation
While using the browser default locale is often good enough, we probably
want more full control over locales, especially when we allow users to
choose their own locale — it's a poor experience to not have dates and
numbers formatted consistently with your chosen locale.
### Changes
Ensures (almost) all instances of `<sl-format-date>` as well as `Intl.*`
have the correct locale passed into them from our Lit localization
system.
The one exception here is in `frontend/src/utils/number.ts` where
ordinal suffixes aren't localized, so the locale is hardcoded to `en` —
I'll revisit this in the future.
Closes#424
### Changes
- Adds Browsertrix logo to the nav bar!!
- Will license the typeface (Konsole) as soon as this gets merged.
- Changes the theme colour to Webrecorder's turquoise brand colour
- Adjusts the usage of `primary` vs `blue` for elements where it
shouldn't be the same
- Changes the _Cancel & Discard_ button of the cancel crawl dialogue to
`danger` button variant
- Replaces the Replay icon with the new ReplayWebpage icon!
- Fixes the favicon SVG linking (wrong name, oops!)
- Fills the microscope lens in the microscope icon
- Removes the old btrix-cloud.svg logo
Co-authored-by: sua yoo <sua@webrecorder.org>
QA Details page:
- Enables QA tab with ability to start automated analysis QA Run + view a and manual review status
- Pages listed with review status + overall crawl review status shown on QA details (relates to #1508)
- Initial placeholder for QA run analytics (part of #1589)
- Addresses a good deal of #1477
Automated Analysis QA in Review Mode:
- Ability to select from multiple analysis QA runs / view QA runs in QA details
- Shows analysis screenshot, text and resources compare and replay tabs (fixes#1496)
- Sorting by worst screenshot / worst text score for each QA run
- Includes pages sidebar with screenshot/text/resource compare results (fixes#1497)
Manual Review QA in Review Mode:
- Per-page replay available as separate tab (fixes#1499)
- Supports thumbs up, thumbs down, notes for each page
- Supports entering review status approval (good/acceptable/bad can be entered when finishing review
---------
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Closes#1591
### Changes
- Converts one instance of a button with an icon in it to an `icon-button`
- Makes all the trashcan icon buttons have a red hover state
- Adds localization function & placeholder to upload dialog "Name" field
- Adds localization functions to some missing icon-button label
instances
- Adds a few missing icon button labels
Co-authored-by: sua yoo <sua@suayoo.com>
Co-authored-by: sua yoo <sua@webrecorder.org>
Closes#1568
## Changes
- Status icons are now filled!
- Uses Bootstrap Icons' new `copy` icon for all actions involving
copying to clipboard!
- Finally! A real copy icon! 🎉
- Removes `copy-code.svg` as it is no longer used
- Actions involving duplicating objects still use `files`... Which is
good! Now they have distinct symbols!
- Adds orange to the tailwind colour palette
---------
Co-authored-by: sua yoo <sua@webrecorder.org>
Allow maximum scale option to be fully configurable via
`max_crawl_scale`. Already configurable on the backend, and now exposed
to the frontend via API `/api/settings` `maxCrawlScale` value.
The workflow editor and workflow details are updated to allow selecting
the scale up to the maxCrawlScale setting (which defaults to 3 if not
set).
Updates Prettier to major version 3, and also updates a couple
prettier-related other things.
Prelude to #1511 so that that PR doesn't include a bunch of unrelated
changes
## Overview
Adds a bunch of ESLint rules, mostly from `typescript-eslint`, and fixes
the issues turning on these rules raises.
Also updates Typescript & typescript-eslint.
## Rationale
Most of these new rules are auto-fixable, so I've tackled a bunch of the
little fixes that do need manual intervention now with the intention
that this shouldn't add much of any additional friction in future
development work, and also give us a good bump in overall code quality.
A lot of the rules here are also great for catching potential bugs!
## Changes
- Adds `void` to most un-awaited and unhandled promises (i.e. places
where async functions are called but nothing is done with the promise)
- Converts properties that are only ever read to `readonly`
- Adds a new `isApiError` function that informs Typescript of when an
error is an `APIError`
- Adds types to a bunch of places that were previously untyped
- Changes instances of `Map<string, any>` in lit property update methods
to `PropertyValues<this>`, or sometimes `PropertyValues<this> &
Map<string, unknown>` where private or protected members are used
(`keyof` doesn't include private and protected members, unfortunately)
- Adds types to a bunch of custom events
- Cleans up a regex by removing unnecessary escape characters
- Makes a number of implied type conversions explicit (by wrapping with
`Boolean(...)` or calling `.toString()`)
- More consistently applies type coercions when necessary, and removes
them when unnecessary
- Converts a couple const strings to an enum
- Removes the need to type debounced functions as `any` by doing type
coercions to the underlying function type at where the method is bound
to the event in the `html` block
Resolves https://github.com/webrecorder/browsertrix-cloud/issues/1333
- Moves "Select Crawls" / "Select Uploads" steps into a single "Select
Archived Items" dialog
- Refactors new collection metadata dialog to accept editing existing
collection
- Prevents RWP component from rendering if there are no archived items
(@Shrinks99 made a comment about this figma, but this prevents
unnecessary requests when there isn't an archive to replay)
- Shows collection description at bottom of detail page at all times
(@Shrinks99 seems useful to see even on archived items view?)
- Switches collection detail primary action to "Add Archived Items" if
none are included (cc @Shrinks99)
- Displays friendlier "name taken" error
- Removes unused Collection edit route
- Upgrades markdown dependencies for fixes/improvements to description
editing
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
