- Updates status icons & colors in several places in the app
- Moves "Action Menus" and updated "Status Indicators" design docs from
public docs to Storybook
- [Storybook] Adds `remark-gfm` to enable tables in MDX
- [Storybook] Adds a custom `ColorSwatch` block
- [Browsertrix Docs] Swaps out custom colors and fonts included with
docs for color variables from Hickory and Webrecorder CDN's hosted font
files, respectively
---------
Co-authored-by: sua yoo <sua@suayoo.com>
- Adds new `<btrix-data-grid>` component
- Refactors `<btrix-usage-history-table>` to data grid
- Refactors Refactors `<btrix-syntax-input>` and
`<btrix-link-selector-table>` to be form-associated controls.
---------
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
- Upgrades typescript-eslint to a more performant version and related
dependencies. Note that these dependencies were not upgraded to the
latest version to avoid upgrading to eslint 9 at this time.
- Upgrades Lit one minor version
Fixes regression introduced by
7c6bae8d61
## Changes
Handles orgs without any crawl defaults correctly. Areas that use
crawling defaults are also more strongly typed now to prevent similar
issues.
- consolidate list_pages() and list_replay_query_pages() into
list_pages()
- to keep backwards compatibility, add <crawl>/pagesSearch that does not
include page totals, keep <crawl>/pages with page total (slower)
- qa frontend: add default 'Crawl Order' sort order, to better show
pages in QA view
- bgjob: account for parallelism in bgjobs, add logging if succeeded
mismatches parallelism
- QA sorting: default to 'crawl order' by default to get better results.
- Optimize pages job: also cover crawls that may not have any pages but have pages listed in done stats
- Bgjobs: give custom op jobs more memory
Fixes#2406
Converts migration 0042 to launch a background job (parallelized across
several pods) to migrate all crawls by optimizing their pages and
setting `version: 2` on the crawl when complete.
Also Optimizes MongoDB queries for better performance.
Migration Improvements:
- Add `isMigrating` and `version` fields to `BaseCrawl`
- Add new background job type to use in migration with accompanying
`migration_job.yaml` template that allows for parallelization
- Add new API endpoint to launch this crawl migration job, and ensure
that we have list and retry endpoints for superusers that work with
background jobs that aren't tied to a specific org
- Rework background job models and methods now that not all background
jobs are tied to a single org
- Ensure new crawls and uploads have `version` set to `2`
- Modify crawl and collection replay.json endpoints to only include
fields for replay optimization (`initialPages`, `pageQueryUrl`,
`preloadResources`) if all relevant crawls/uploads have `version` set to
`2`
- Remove `distinct` calls from migration pathways
- Consolidate collection recompute stats
Query Optimizations:
- Remove all uses of $group and $facet
- Optimize /replay.json endpoints to precompute preload_resources, avoid
fetching crawl list twice
- Optimize /collections endpoint by not fetching resources
- Rename /urls -> /pageUrlCounts and avoid $group, instead sort with
index, either by seed + ts or by url to get top matches.
- Use $gte instead of $regex to get prefix matches on URL
- Use $text instead of $regex to get text search on title
- Remove total from /pages and /pageUrlCounts queries by not using
$facet
- frontend: only call /pageUrlCounts when dialog is opened.
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
- Upgrades webpack and webpack tool versions
- Updates dev source map to webpack recommendation
- Implements `webpack.DllPlugin` in dev for faster rebuilds
- Implements `thread-loader` to run `ts-loader` in a worker pool
Closes#2223
- [x] Adds `localesAvailable` to `/api/settings` endpoint, and uses that
list if available, rather than the full list of translated locales, to
determine which options to display to users
- [x] ~~Uses the user's browser locales, filtered to the current
language setting, for formatting numbers, dates, and durations~~
- [x] Adds & persists checkbox for "use same language for formatting
dates and numbers" in user settings
- [x] Replaces uses of `sl-format-bytes` with `localize.bytes(...)`, and
`sl-format-date` with replacement `btrix-format-date` that properly
handles fallback locales
- [x] Caches all number/duration/datetime formatters by a combined key
consisting of app language, browser language, browser setting, and
formatter options so that all formatters can be reused if needed
(previously any formatter with non-default options would be recreated
every render)
- [x] Splits out ordinal formatting from number formatter, as it didn't
make much sense in some non-English locales
- [x] Adds a little demo of date/time/duration/number formatting so you
can see what effect your language settings have
https://github.com/user-attachments/assets/724858cb-b140-4d72-a38d-83f602c71bc7
---------
Signed-off-by: emma <hi@emma.cafe>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
Partially addresses
https://github.com/webrecorder/browsertrix/issues/2171
- [x] Reimplement `pretty-ms`'s rounding logic to convert to a
`Intl.DurationFormat`-compatible object
- [x] Add `localize.duration` and `localize.humanizeDuration` methods
- [x] Replace `pretty-ms` usage with localized formatting
- [x] Add `Intl.DurationFormat` polyfill
- [x] Update localize tests
- [x] Fix broken tests (thank you @SuaYoo!)
- This looks to be possibly a bug somewhere in the web test runner's
code? Have yet to figure this one out
- Might be related to these:
https://github.com/web-dev-server/web-dev-server/issues/1https://github.com/guybedford/es-module-lexer/issues/82 — both closed
with no comments or resolution 🙃 — `es-module-lexer` seems to be most
likely, given it's a native module & might be handling unicode sequences
incorrectly or not at all?
- [x] Clean up messy `index.d.ts` — probably split this out? The
definitions haven't yet made their way into `esnext`
https://github.com/microsoft/TypeScript/issues/60608
---------
Co-authored-by: emma-sg <emma-sg@users.noreply.github.com>
Co-authored-by: sua yoo <sua@suayoo.com>
- Defaults to user's browser language preference when displaying dates,
bytes, and numbers (with caveats.) This fixes an issue where numbers
were always formatted in English.
- Shows user's browser language preferences in language dropdown
- Fixes timezone not displayed in archived item detail start and finish
times
- Standardizes number formatting
- Sets timezone that unit tests run in
---------
Co-authored-by: SuaYoo <SuaYoo@users.noreply.github.com>
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Co-authored-by: emma-sg <emma-sg@users.noreply.github.com>
- Ensures extracted strings get formatted before checking against index
- Fixes index check by switching from `git diff-index` to `git diff`,
and ensures the proper `--exit-code` flag is present (implicitly turned
on by `--quiet`)
- Adds actionable error message when the check fails
- Updates Github's actions versions from v3 to v4 (major version bump is
primarily just for default node version updates, but this way we'll get
future updates)
- Adds formatting step to npm script for extracting messages
- Runs a string extraction & format against current main
