- Removes share link from the dialogue footer
- Removes stickied collection navigation, replaces with improved
viewport-based scaling!
- Adds a max-width for the collection description in the logged in view.
- Moves the markdown editor buttons to below the editor
- Controls are now In-line with how we handle dialogue options
elsewhere, fixes a minor responsive design issue.
- Minor copy changes
---------
Co-authored-by: emma <hi@emma.cafe>
Co-authored-by: sua yoo <sua@webrecorder.org>
- Renames `inject_analytics` to `inject_extra` and updates docs
- Manually tracks page views to enable passing custom props
- Tracks copying collection share link and downloading a public
collection
---------
Co-authored-by: emma <hi@emma.cafe>
- Allows user to choose collection replay home page and collection
thumbnail (resolves
https://github.com/webrecorder/browsertrix/issues/2182)
- Displays collection thumbnails on org dashboard and public page
- Enables downloading public collection (resolves
https://github.com/webrecorder/browsertrix/issues/2233)
- Adds caption as "Summary" to metadata dialog
- Moves description editor to "About" tab
---------
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
- Enables creating a public org profile page with description and
website at `/profile/<org slug>`
- Updates current "Overview" page to be "Dashboard", found under
`/dashboard`
- Organizes org "General" settings tab by "General", "Profile", and
"Developer Tools"
- Adds sign up banner to log in page for consistent CTA banners
- Updates copy and docs to support changes
- Allows user to set collection to private, public, or unlisted
- Adds route for public collection page with basic page layout
- Refactors copy button to abstract clipboard functionality
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: emma <hi@emma.cafe>
Fixes#2158
- Adds `Organization.listPublicCollections` field and API endpoint to
update it
- Replaces `Collection.isPublic` boolean with `Collection.access`
(values: `private`, `unlisted`, `public`) and add database migration
- Update frontend to use `Collection.access` instead of `isPublic`,
otherwise not changing current behavior
---------
Co-authored-by: sua yoo <sua@suayoo.com>
Also fixes a shoelace menu-item bug where checkbox menu items would have
their checked state flipped on click, regardless of `checked` value.
~~Deploying to dev to test if this fixes language switching...~~ Yep!
Seems to fix the issues.
- Uses crawl count to determine whether workflow can be deleted instead
of last crawl ID
- Display delete confirmation dialog when trying to delete a workflow
Fixes#2170
The number of days to delay file replication deletion by is configurable
in the Helm chart with `replica_deletion_delay_days` (set by default to
7 days in `values.yaml` to encourage good practice, though we could
change this).
When `replica_deletion_delay_days` is set to an int above 0, when a
delete replica job would otherwise be started as a Kubernetes Job,
a CronJob is created instead with a cron schedule set to run yearly,
starting x days from the current moment. This cronjob is then deleted by
the operator after the job successfully completes. If a failed
background job is retried, it is re-run immediately as a Job rather
than being scheduled out into the future again.
---------
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
Merge more Spanish translations updates to main for 1.13 release
Co-authored-by: Weblate (bot) <hosted@weblate.org>
Co-authored-by: Clara Itzel <missclaraitzel@gmail.com>
Co-authored-by: Lorena Ramírez-López <lorena@webrecorder.org>
Co-authored-by: Webrecorder Dev <dev@webrecorder.org>
Co-authored-by: weblate <weblate@users.noreply.github.com>
- org.created may be null (for backwards compatibility before it was
set)
- fix frontend type to match backend
- update format.date() to accept null, return empty string
- By default, all locales are enabled to make it easy for local deployments to test new locales
- Adds DE, FR, PT locales to make way for translation in Weblate
Closes#2223
- [x] Adds `localesAvailable` to `/api/settings` endpoint, and uses that
list if available, rather than the full list of translated locales, to
determine which options to display to users
- [x] ~~Uses the user's browser locales, filtered to the current
language setting, for formatting numbers, dates, and durations~~
- [x] Adds & persists checkbox for "use same language for formatting
dates and numbers" in user settings
- [x] Replaces uses of `sl-format-bytes` with `localize.bytes(...)`, and
`sl-format-date` with replacement `btrix-format-date` that properly
handles fallback locales
- [x] Caches all number/duration/datetime formatters by a combined key
consisting of app language, browser language, browser setting, and
formatter options so that all formatters can be reused if needed
(previously any formatter with non-default options would be recreated
every render)
- [x] Splits out ordinal formatting from number formatter, as it didn't
make much sense in some non-English locales
- [x] Adds a little demo of date/time/duration/number formatting so you
can see what effect your language settings have
https://github.com/user-attachments/assets/724858cb-b140-4d72-a38d-83f602c71bc7
---------
Signed-off-by: emma <hi@emma.cafe>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
This has been bugging me for a while — any type of repetitive action
that shows a toast will quickly cause a big stack of toasts to build up.
This fixes that by allowing different categories of toasts to exist,
where each category can only have one active toast. _Different_
categories of toast can coexist just fine, but toasts with the same `id`
will replace any existing toast with that same `id`.
It's been a pet peeve for a while, and it should make any sort of "power
user" use of Browsertrix a lot nicer :)
I've gone over all the existing toasts and applied what feel like decent
differentiations, but I'm not set on any of them — open to suggestions!
https://github.com/user-attachments/assets/dbe39141-9e56-427d-b702-124c45ef2b9a
---------
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
quick follow-up to #2226:
- rename script _plausible.js -> analytics.js to be more generic
- devserver: serve empty /analytics.js from webpack dev server
---------
Co-authored-by: emma <hi@emma.cafe>
Adds sending a cancellation email when a subscription is cancelled.
- The email may also include an option survey optional survey URL, if
configured in helm chart `survey_url` setting.
- Cancellation e-mail configured in `sub_cancel` e-mail template
- E-mails are sent to all org admins.
- Also adds `trialing_canceled` subscription state to differentiate from
a default `trialing` which will automatically rollover into `active`.
- The email is sent when: a new cancellation date is added for an
`active` subscription, or a `trialing` subscription is changed to to
`trialing_canceled`. (A subscription can be canceled/uncanceled several
times before actual date, and e-mail is sent every time it is canceled.)
- The 'You have X days left of your trial' is also always displayed when
state is in trialing_canceled.
Fixes#2229
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Closes#2222
Adds a runtime script that gets set to either inject the plausible
script tags, or do nothing, that runs at initialization of the frontend
container.
This PR syncs the weblate branch back to main, and adds:
- New Spanish translations contributed in weblate!
- New GH action which reformats weblate commits with `localize:extract`,
ensuring strings are not changed, and also runs `localize:build` to
extract templates (as tested in PR: #2207)
- Reformats existing XLIFF files to match `localize:extract`, including
adding self-closing tags.
- Update on generated templates
- Prettier: disable alpha sorting XML attributes to hopefully minimize
conflcits
- Strings: change some strings that use `&` to be wrapped in html`` to
ensure proper unencoding, or just switch to 'and' where that's not
possible.
---------
Co-authored-by: Weblate (bot) <hosted@weblate.org>
Co-authored-by: Webrecorder Dev <dev@webrecorder.org>
Co-authored-by: Kamborio <Kamborio15@users.noreply.hosted.weblate.org>
Co-authored-by: Clara Itzel <missclaraitzel@gmail.com>
Co-authored-by: weblate <weblate@users.noreply.github.com>
Co-authored-by: emma <hi@emma.cafe>
Co-authored-by: sua yoo <sua@webrecorder.org>
Closes#2213
This ended up being a bit of a variety PR, but primarily this fixes
issues around archived item pages for crawls at URL locations we no
longer support — specifically, crawls, which now live at
`/orgs/{oid}/workflows/{workflow_id}/crawl/{crawl_id}`, rather than
`/orgs/{oid}/items/crawl/{crawl_id}`, including the QA pages at
sub-paths of these. Attempting to view a crawl at the old (unsupported)
URL will now redirect you to the correct URL!
Other changes include:
- Updated links in various parts of the app — nothing should link to the
old URL pattern anymore for crawls!
- Improved `APIRouter` and `ViewState` typing
- A couple extra `APIRouter` tests
- An addition to the route definition for crawls at
`/orgs/{oid}/items/crawl/{crawl_id}` that allows for redirections from
QA review page URLs, rather than 404 errors — this was the actual
reported problem!
- Caching in CI for playwright tests
- `needsLogin` conversion from the deprecated `LiteElement` to
`BtrixElement`, along with all the components that use it
---------
Co-authored-by: emma-sg <emma-sg@users.noreply.github.com>
Resolves https://github.com/webrecorder/browsertrix/issues/2192
## Changes
- Displays more accurate subscription/trial end message when
subscription ends between 1-2 days
- Shows timezone for end date
---------
Co-authored-by: SuaYoo <SuaYoo@users.noreply.github.com>
Some minor UX improvements, mostly suggested by @dcgomes
### Changes
- Updates in-app help text for crawl workflow limits.
- Adds a prompt to view error logs on the QA screen if failed pages > 1
- The error logs will give some insight as to why the pages failed every
time! Good thing to make the user do next!
### Screenshots
<img width="324" alt="Screenshot 2024-11-29 at 6 44 28 PM"
src="https://github.com/user-attachments/assets/fc481576-6f7f-461b-b8f2-5cbf7fcec0bc">
---------
Co-authored-by: Shrinks99 <Shrinks99@users.noreply.github.com>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
Partially addresses
https://github.com/webrecorder/browsertrix/issues/2171
- [x] Reimplement `pretty-ms`'s rounding logic to convert to a
`Intl.DurationFormat`-compatible object
- [x] Add `localize.duration` and `localize.humanizeDuration` methods
- [x] Replace `pretty-ms` usage with localized formatting
- [x] Add `Intl.DurationFormat` polyfill
- [x] Update localize tests
- [x] Fix broken tests (thank you @SuaYoo!)
- This looks to be possibly a bug somewhere in the web test runner's
code? Have yet to figure this one out
- Might be related to these:
https://github.com/web-dev-server/web-dev-server/issues/1https://github.com/guybedford/es-module-lexer/issues/82 — both closed
with no comments or resolution 🙃 — `es-module-lexer` seems to be most
likely, given it's a native module & might be handling unicode sequences
incorrectly or not at all?
- [x] Clean up messy `index.d.ts` — probably split this out? The
definitions haven't yet made their way into `esnext`
https://github.com/microsoft/TypeScript/issues/60608
---------
Co-authored-by: emma-sg <emma-sg@users.noreply.github.com>
Co-authored-by: sua yoo <sua@suayoo.com>
Fixes https://github.com/webrecorder/browsertrix/issues/2190
## Changes
- Fixes not being able to add live exclusion
- Fixes not being able to delete last exclusion
- Fixes collection list search
- Refactors exclusion form event name
can either link to redoc hosted elsewhere or make a local copy:
- for local frontend build, just redirect to
http://localhost:30870/api/redoc
- for deployment, make local copy: run copy-api-docs.sh, copy locally
from prod and serve at /api/
- copy-api-docs.sh copies openapi.json, redoc and logo to /api/ dir
- if analytics enabled, also injects analytics scripts
- for local testing, run copy-api-docs.sh and then run mkdocs serve
- ci: copy from prod server
- fixes#1582
- Separates translatable html strings into parts to make them easier to
translate
- Updates copy regarding subscription
- Updates plan feature list to match website
- Defaults to user's browser language preference when displaying dates,
bytes, and numbers (with caveats.) This fixes an issue where numbers
were always formatted in English.
- Shows user's browser language preferences in language dropdown
- Fixes timezone not displayed in archived item detail start and finish
times
- Standardizes number formatting
- Sets timezone that unit tests run in
---------
Co-authored-by: SuaYoo <SuaYoo@users.noreply.github.com>
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Co-authored-by: emma-sg <emma-sg@users.noreply.github.com>
Fixes password strength not being checked when new passwords are
auto-filled, resulting in submit buttons remaining disabled.
---------
Co-authored-by: SuaYoo <SuaYoo@users.noreply.github.com>
Co-authored-by: emma <hi@emma.cafe>
- Adds a missing `/docs` to the docs edit urls
- Adds instructions for installing `mkdocs-material` with pipx & uvx,
and describes correct place to run `mkdocs serve` from
- Fixes edit url, opens in a new tab
- Adds Plausible
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
languages specified in `translatedLocales`
- Updates user language preference beta badge text
- Adds link to translation contribution docs
- Refactors component name
---------
Co-authored-by: SuaYoo <SuaYoo@users.noreply.github.com>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: emma <hi@emma.cafe>
- Migrates the following `LiteElement`s to `BtrixElement`:
- `App` (`<browsetrix-home>`)
- `Admin` (`<btrix-home`)
- `Dashboard` (`<btrix-dashboard>`)
- `LogInPage` (`<btrix-log-in>`)
- Replaces custom light DOM `btrix-input` with `sl-input` since password
managers should work now.
- Shows language selector on log in page
- Adds tabs to account settings
- Adds language preference selector on account settings
- Adds issue template for new language
- Updates dev docs for frontend localization
---------
Co-authored-by: SuaYoo <SuaYoo@users.noreply.github.com>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Fixes#2106
Docs are now hosted as part of the frontend at `/docs` by default.
- If `docs_url` is set in the helm chart, the `/docs` endpoint will
redirect to that endpoint instead
- Use multi-stage python image to build mkdocs as part of frontend, then
copy static output
- Dir layout: mkdocs.yml and docs into frontend/docs
- CI: Update docs build GH action to use new path
- Update all frontend paths to use `/docs/` instead of
`https://docs.browsertrix.com/`
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
- ensure user-guide iframe is always visible if not fading in when
'Setup Guide' button is clicked
- Clicking 'Setup Guide' scrolls to relevant section of guide (scope,
limits, browser settings, scheduling, metadata) based on current
workflow editor tab via hashtag
---------
Co-authored-by: sua yoo <sua@webrecorder.org>
- crawler isActive() should only look at running states, not 'stopping',
since stopping may not be reset to false after crawl is stopped
(probably should reset, but also a record that the crawl ended as being
stopped). crawl is guaranteed to remain in one of the active states
while stopping on the backend
- fixes#2100
Resolves the big red
`[DEP_WEBPACK_DEV_SERVER_ON_BEFORE_SETUP_MIDDLEWARE] DeprecationWarning:
'onBeforeSetupMiddleware' option is deprecated. Please use the
'setupMiddlewares' option.` warnings when starting Webpack.
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Resolves https://github.com/webrecorder/browsertrix/issues/2091
<!-- Fixes #issue_number -->
### Changes
Saves scope type chosen from "+ New Workflow" dropdown to local storage, as well as from within workflow editor when creating a new workflow (but not editing an existing one).
Resolves#1354
Supports crawling through pre-configured proxy servers, allowing users to select which proxy servers to use (requires browsertrix crawler 1.3+)
Config:
- proxies defined in btrix-proxies subchart
- can be configured via btrix-proxies key or separate proxies.yaml file via separate subchart
- proxies list refreshed automatically if crawler_proxies.json changes if subchart is deployed
- support for ssh and socks5 proxies
- proxy keys added to secrets in subchart
- support for default proxy to be always used if no other proxy configured, prevent starting cluster if default proxy not available
- prevent starting manual crawl if previously configured proxy is no longer available, return error
- force 'btrix' username and group name on browsertrix-crawler non-root user to support ssh
Operator:
- support crawling through proxies, pass proxyId in CrawlJob
- support running profile browsers which designated proxy, pass proxyId to ProfileJob
- prevent starting scheduled crawl if previously configured proxy is no longer available
API / Access:
- /api/orgs/all/crawlconfigs/crawler-proxies - get all proxies (superadmin only)
- /api/orgs/{oid}/crawlconfigs/crawler-proxies - get proxies available to particular org
- /api/orgs/{oid}/proxies - update allowed proxies for particular org (superadmin only)
- superadmin can configure which orgs can use which proxies, stored on the org
- superadmin can also allow an org to access all 'shared' proxies, to avoid having to allow a shared proxy on each org.
UI:
- Superadmin has 'Edit Proxies' dialog to configure for each org if it has: dedicated proxies, has access to shared proxies.
- User can select a proxy in Crawl Workflow browser settings
- Users can choose to launch a browser profile with a particular proxy
- Display which proxy is used to create profile in profile selector
- Users can choose with default proxy to use for new workflows in Crawling Defaults
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Resolves https://github.com/webrecorder/browsertrix/issues/2073
### Changes
- Removes "URL List" and "Seeded Crawl" job type distinction and adds as
additional crawl scope types instead.
- 'New Workflow' button defaults to Single Page
- 'New Workflow' dropdown includes Page Crawl (Single Page, Page List, In-Page Links) and Site Crawl (Page in Same Directory, Page on Same Domain, + Subdomains and Custom Page Prefix)
- Enables specifying `DOCS_URL` in `.env`
- Additional follow-ups in #2090, #2091
- Ensures extracted strings get formatted before checking against index
- Fixes index check by switching from `git diff-index` to `git diff`,
and ensures the proper `--exit-code` flag is present (implicitly turned
on by `--quiet`)
- Adds actionable error message when the check fails
- Updates Github's actions versions from v3 to v4 (major version bump is
primarily just for default node version updates, but this way we'll get
future updates)
- Adds formatting step to npm script for extracting messages
- Runs a string extraction & format against current main
Use timezone aware datetimes instead of timezone naive datetimes:
- Update mongodb client to use tz-aware conversion
- Convert dt_now() to return timezone aware UTC date
- Rename to_k8s_date -> date_to_str, just returns ISO UTC date with 'Z'
(instead of '+00:00' suffix)
- Rename from_k8s_date -> str_to_date, returns timezone aware date from
str
- Standardize all string<->date conversion to use either date_to_str or
str_to_date
- Update frontend to assume iso date, not append 'Z' directly
- Update tests to check for 'Z' suffix on some dates
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
- Fixes empty watch tab during some active states
- Disables exclusion and browser window editing while a crawl is
stopping
- Refactors frontend crawl states to match backend
- Replaces QA Review breadcrumbs with "Back" button that takes you back
to the archived item
- Crawl breadcrumbs always point to workflow
- Shows archived item icon next to item name to differentiate from
workflow
- Adds "Edit Workflow" button to "Crawl Settings"
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Fixes https://github.com/webrecorder/browsertrix/issues/2064
### Changes
- Switches MDE library to one that supports shadow DOM
- Refactors collection components to btrix components
- Fixes collection detail not expanding and contracting correctly
- Adds breadcrumb navigation to all detail views, returning to correct
origin for workflow and collection items
- Refactors list page headers into layout utility
- Refactors crawl tab labels and renames "Files" tab to "WACZ Files"
- Adds new org settings tab for updating crawl details
- Refactors `workflow-editor` to move out utility functions
- Updates user guide on org settings
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Improves time to first load an org with the following:
- Users user info from login response to set org slug and route user on
log in
- Stores user info in session storage so that it's available on reload
- Stores app settings in local storage until user logs out
- Loads critical org components synchronously
WIP for https://github.com/webrecorder/browsertrix/issues/2041
<!-- Fixes #issue_number -->
### Changes
- Adds button to open embedded support guide
- Adds link to help forum
- Refactors app bar to look nicer on smaller screens
- Updates workflow job type copy and adds additional clarifying text
- Changes "List of URLs" label to "Crawl URL(s)"
- Refactors `NewWorkflowDialog` into tailwind element
- Renames "Org Settings" -> "Settings"
- Reduces gap between settings panel heading and panel
- Always show "Pending Invites" section and update heading styles to
match panel heading
- Update "Current Plan" and "Usage History" sections to be on the same
hierarchical level under "Billing"
- Refactors `<btrix-org>` to move `isAdmin` and `isCrawler` helpers to
app state
Also improves the slug editing experience by partially-slugifying the
value as it's entered.
Previously, submitting a org slug value of ".." or similar would cause
the frontend to redirect to a "page not found" page, with all accessible
links leading to only `/account/settings`. This also causes the backend
to reset the org slug to one generated from the org name on a reload.
---------
Co-authored-by: sua yoo <sua@webrecorder.org>
* Fixes issue in FailedLogin model:
- fix data-model to remove nested 'attempted.attempted'
- migrate existing data to remove nested field
* Also, avoid setting dt_now() in model as that results in fixed date for
all objects:
- update FailedLogin to update 'attempted' date on every attempt
- also update PageNote object to set date in constructor
* Update text for too many logins to make it clear it is set only if its a
valid email
* fixes#2001
Following https://github.com/webrecorder/browsertrix/pull/1995, we want
to keep the usage history table on the dashboard for now for video
demos.
### Changes
- Adds usage history table back to org dashboard
- Makes "No usage history" message more apparent
Tweaks to how execution time is tracked for more accuracy + excluding
waiting states:
- don't update if crawl state is in a 'waiting state' (waiting for
capacity or waiting for org limit)
- rename start states -> waiting states for clarity
- reset lastUpdatedTime if two consecutive updates of non-running state,
to ensure non-running states don't count, but also account for
occasional hiccups -- if only one update detects non-running state,
don't reset
- webhooks: move start webhook to when crawl actually starts for first
time (db lastUpdatedTime is not yet + crawl is running)
- don't set lastUpdatedTime until pods actually running
- set crawljob update interval to every 10 seconds for more accurate
execution time tracking
- frontend: show seconds in 'Execution Time' display
- Follow-up to #1914, allows SubscriptionUpdate event to also update
quotas.
- Passes current usage info + current billing page URL to portalUrl
request for external app to be able to respond with best portalUrl
- get_origin() moved to utils to be available more generally.
- Updates billing tab to show current plans, switches order of quotas to
list execution time, storage first
an org is made read-only while crawls are running:
- treat similar to other stopped_* states, do a graceful stop
- update UI to display "Stopped: Crawling Disabled" for this status
- don't add corresponding skipped status - just skip running crawls if org is read-only
Resolves https://github.com/webrecorder/browsertrix/issues/1951
### Changes
- Shows date org was created in superadmin org list
- Visually differentiates unnamed org ID
- Adds "Admin" badge to app header to make current login more apparent
- Fixes logic to show "create org" dialog if there are no orgs in an
instance
- Refactors `btrix-home` to remove unused references to non-superadmin
org list
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Resolves https://github.com/webrecorder/browsertrix/issues/1912
### Changes
- Show support email, if available, in invalid invite error message
- Separate error message for invite email that doesn't match current
user's
Fixes#1968
Changes:
- `stopped_quota_reached` and `skipped_quota_reached` migrated to new
values that indicate which quota was reached
- Before crawls are run, the operator checks if storage or exec mins
quotas are reached and if so fails the crawl with the appropriate state
of `skipped_storage_quota_reached` or `skipped_time_quota_reached`
- While crawls are running, the operator checks if the exec mins quota
is reached or if the size of all running crawls will mean the storage
quota is reached once uploaded; if so, the crawl is stopped gracefully
and given `stopped_storage_quota_needed` or `stopped_time_quota_reached`
state as appropriate
- Adds new nightly tests for enforcing storage quota
Fixes#1412
## Changes
### Backend
- Adds `all-crawls`, `crawls`, and `uploads` API endpoints to download
archived item as multi-WACZ
- Download QA runs as multi-WACZ
- Adds backend tests for new endpoints
- Update to new version of stream-zip library which does not require crc-32 to be present for ZIP members,
computes after streaming, fixing invalid crc-32 issues as previously computed crc-32s from crawler may be invalid.
### Frontend
Adds ability to download archived item from:
- Button in archived item detail Files tab
- Archived item details actions menu
- Archived items list menu
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: sua yoo <sua@webrecorder.org>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
The monthly execution minutes meter was using the wrong metric,
`crawlExecSeconds` instead of `monthlyExecSeconds`.
Since the meter is showing minutes used out of the monthly quota, it
should be using `monthlyExecSeconds`.
This is a quick fix, however, this system may need another pass at some
point.
---------
Co-authored-by: sua yoo <sua@suayoo.com>
Fixes https://github.com/webrecorder/browsertrix/issues/1954
### Changes
- Refactors app state to include org data
- Fixes banner not showing if storage or execution minutes is exceeded
after page load
- Disables closing banners
- Refreshes org when tab changes
---------
Co-authored-by: emma <hi@emma.cafe>
Directs user that doesn't belong to any orgs to account settings page,
with banner.
Also contains some minor out-of-scope changes:
- Refactors `isAdmin` key to `isSuperAdmin` for more legibility on
whether current user is superadmin or regular user without orgs
- Adds "cancel" button to change password form
Fixes#1955
Orgs list endpoint sorting now works as follows:
- Default org is always sorted first
- Name sorting now works on a lowercased version of the org names to
ensure lexical sorting
The lodash `sortBy` resorting of orgs in the "All Organizations"
dropdown list in the nav bar has also been removed so that the backend
sorting is applied instead.
Tests have been updated accordingly.
### Changes
- Updates customer portal link label
- Opens billing portal in the same tab
- Shows separate cancel date
- Makes payment failed appear as error
- Fixes crawl time quota
Fixes a few things that have been bugging me:
- Overflow buttons in list view now (mostly) take up the their full cell
area, instead of there being a couple pixels around the button where
clicking would do nothing or cause navigation
- | before | after |
| --- | --- |
| <img width="238" alt="Screenshot 2024-07-16 at 3 35 25 PM"
src="https://github.com/user-attachments/assets/afbda6d6-703b-4ed8-96be-a9c37660430d">
| <img width="236" alt="Screenshot 2024-07-16 at 3 35 02 PM"
src="https://github.com/user-attachments/assets/417a326a-08d2-42b2-85c3-fa007ea3bff8">
|
- Changes the class that `tab-list` uses internally so that it doesn't
conflict with Tailwind's `container` class, which prevents the tab
content from being limited at the default Tailwind container width
- Adds a couple of Tailwind plugins for styling CSS parts
(`part-[...]:`) and for arbitrary attributes (`attr-[...]:`)
Initial implementation of #1892
- Modifies the backend to return `duplicate_org_name` or
`duplicate_org_slug` as appropriate on a pymongo `DuplicateKeyError`
- Updates frontend to handle `duplicate_org_name`, `duplicate_org_slug`,
and `invalid_slug` error details
- Update errors to be more consistent, also return `duplicate_org_subscription.subId` for duplicate subscription instead of the more generic `already_exists`
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
No issue created, but noticed issue here
ed0d489cda
### Changes
- Removes unused node module mocks and use commonjs plugin to import
modules in tests
- Fixes org form test after removing temporary stub
The default org will always be sorted first, regardless of sort options.
Orgs after the first will be sorted by name ascending by default.
Sorting currently supported on name, slug, and readOnly.
Fixes#890
This PR introduces new streaming superuser-only API endpoints to export
and import database information for an organization. New Adminstrator
deployment documentation on how to manage the process and copy files
between S3 buckets as needed is also included.
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Resolves https://github.com/webrecorder/browsertrix/issues/1874
Support for new two-part sign up flow if first admin user is added to org
- If new user, user registers first, then is able to change the org name / slug on following screen
- If existing user, user accepts invite, then is able to change the org name / slug on following screen
- After confirming org slug name, user is taken to dashboard, or error is shown if org name or slug already taken.
- If org name == org id, org name and slug is automatically set to `{Your Name}'s Archive` when first user is registered / accepts invite
- Email templates updated to better reflect new / existing users and not show org name if it is 'unset' (org name == org id internally)
- tests: frontend unit testing for accept + invite screens.
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: sua yoo <sua@suayoo.com>
Co-authored-by: sua yoo <sua@webrecorder.org>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Closes#1851
This changes how we use the `<btrix-meter>` component: previously, we
didn't set a `max` value on it, so it was drawing its internal (rounded)
bar at 100% of the meter width, and then drawing the
`<btrix-meter-bar>`s inside that 100% width bar. This changes that, and
instead it now does the following:
- Draws the internal rounded bar to the width of the proportion of their
counts to the total page count;
- Adjusts the `<btrix-meter-bar>`s to be proportional to the completed
page count rather than the total page count
- Uses the `"available"` slot in `<btrix-meter>` to draw the remaining
"No data" bar & tooltip in the remaining space outside of the internal
rounded bar
This also includes a couple unrelated but adjacent changes:
- Changes the page count text to say "Analysis starting" when the
analysis run is starting, because there's no meaningful page count at
that point.
- Uses the pending state when the total page count is falsy (`undefined`
or `0`), rather than just `undefined`
- Switches from spans with classes to `ins` and `del` tags for text
comparison
- Adds `aria-label` attributes for screen readers. Text comparison is
accessible now! 🎉
- I kept the strikethrough text on the `del` tag but can remove? Only
visible on hover.
- Removes autofocus on the delete analysis run cancel button
- Adds `aria-label` attribute for the heuristic nav bar to describe the
context of the content users will be seeing on each page.
Fixes#1846
- Ensure meter auto-updates as new stats are ready
- Switch meter to new QA run when new analysis run is started
- Remove Files from QA meter (files and errors will be reported separately)
Co-authored-by: emma <hi@emma.cafe>
Co-authored-by: sua yoo <sua@webrecorder.org>
Switches `sl-progress-ring` and `sl-icon` to inherit font size from
table cell for consistent sizing as base size, while also fixing
`var(--font-size-base)` not being available when progress ring size is
first calculated.
Fixes#1835
Fixes#1833
- Add firstSeed and seedCount to workflow information in profile detail
API endpoint (tests updated accordingly), update name of model used for
limited workflow information to be more accurate
- Fix name display in Crawl Workflows list at bottom of Profile detail
page to be consistent with rest of application
---------
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
This PR adds Identical Files to the QA Page Match Analysis meter bars.
To do this, the backend calculates the number of non-HTML pages once and
includes it under the key `Files` in each of the `screenshotMatch` and
`textMatch` QA stats return arrays.
The backend additionally removes the file count from "No Data" to
prevent these from being counted twice.
---------
Co-authored-by: emma <hi@emma.cafe>
Resolves https://github.com/webrecorder/browsertrix/issues/1409
### Changes
- Enables clicking on Browser Profiles column header to sort the table, including by starting URL
- More consistent column widths throughout app
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Co-authored-by: emma <hi@emma.cafe>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Follow up to #1815
- Improved detection / responsiveness to unavailable browser
- Disable Save buttons when browser disconnected / unavailable
- Only add unload handler when browser is available
- Consistency between new and edit views: new profile view also has
cancel button with confirmation
- Don't show cancel confirmation if browser is already unavailable.
Various text tweaks for consistency / accuracy:
- Save New Profile... to indicate new profile will be saved after dialog
- Load new browser to indicate new browser will be created
- Mention connection to browser is lost in disconnected message.
Also: migrated to LiteElement -> TailwindElement for browser-profiles-new
-------
Co-authored-by: emma <hi@emma.cafe>
- Shows browser profile last modified or created by name, if available
- Moves backed-up status to browser profile subsection header
- Moves "Last Updated" column to last and displays user name on hover,
to match archived items list view
- Updates browser profile docs
- Paginates browser profile list view
- Moves back-up status icon to list view
- Display last updated date if available (fallback to created date)
- Adds max length validation to description
- Better handle description and crawl workflow list
Closes#1818
- Refactors image renderers, and changes the condition for which the
loading indicator is displayed from "`<data>.blobUrl` exists and has a
truthey value" to "`<data>` is not nullish"
- Adds an additional class config to the shared QA loading indicator,
and uses it in the image slider view to add a background and aspect
ratio to the loading indicator's container so that it takes up the space
it should and has an opaque background
| Before | After |
| --- | --- |
| <img width="813" alt="Screenshot 2024-05-22 at 5 17 14 PM"
src="https://github.com/webrecorder/browsertrix/assets/5727389/fb8a3a83-a0c8-46c1-b59b-a59adff2a175">
| <img width="813" alt="Screenshot 2024-05-22 at 5 17 31 PM"
src="https://github.com/webrecorder/browsertrix/assets/5727389/6fe5d04a-ffa9-4fc3-bc3b-bdb9b7b30948">
|
Resolves https://github.com/webrecorder/browsertrix/issues/438
<!-- Fixes #issue_number -->
### Changes
- Layout and design pass on profile detail view to better match other
detail views
- Ensures profile save buttons are always visible when creating or
editing a profile
- Handle temporarily disconnected browser pings
- Enable recreating browser after it expires
- Show confirmation dialog when attempting to navigate away from active
browser
- Tweak copy referencing profiles
- Refactors `profile-browser` into `TailwindComponent`
### Follow-ups
UX improvements to the browser profile detail layout and list view will
be handled in https://github.com/webrecorder/browsertrix/issues/1816
---------
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
- Fixes browser profile metadata form submission
- Fixes rendering of browser profile description in list view
- Changes label to "metadata" for consistency with other features
Resolves https://github.com/webrecorder/browsertrix/issues/1398
### Changes
- Updates org list in user info state after an org is created
- Minor refactor to add `btrix` prefix to custom `update-user-info`
event and replace unnecessary `defaultOrg` property
- Adds user role select to superadmin dashboard
- Requires org selection (unless there's only one org) to prevent
accidental crawler invites to default testing org
- Shows link to org members after invite, retaining form for quick
re-invite
- Refactor `invite-form` into `TailwindComponent`
Closes#1789
### Changes
- Converts stylesheet in js to css now that it doesn't use any runtime
values
- Removes fixed height on QA container, and instead sets a min height
- Applies `contain: size` to page list & tab content so they don't cause
the QA container to stretch
- Also updates instances of `h-{n} w-{n}` to `size-{n}`
- Removes leftover outlines on text and resource comparison areas from
early QA prototyping
Closes#1787
- Crawls with only one seed will load viewing that seed URL
- URL list crawls with >1 URLs will continue to load the pages list as no
ReplayWeb.page enbed URL has been defined
Co-authored-by: sua yoo <sua@webrecorder.org>
Fixes#1782
- Dash icons are now used to convey status exclusively
- Slash icons are now used to convey no data states
- Updates status icons to filled in the docs (also required for QA
docs!)
Resolves https://github.com/webrecorder/browsertrix/issues/1761
<!-- Fixes #issue_number -->
### Changes
- Switches QA review page chip tooltip colors to dark text on light
background
- Shows newest comment when hovering over comment cell in QA tab page
list to match tooltip in QA review page list
- Validates textarea max entry for description and comments
- Prevents misleading "no text" message shown when switching tabs
- Reset data and show loading indicator when switching pages
- Fix replay tab height and visually differentiates empty replay window
from empty tab
- Refactors console.logs to debugs
Fixes https://github.com/webrecorder/browsertrix/issues/1638
### Changes
- Fixes collection selection during upload and in workflow settings
- Refactors `btrix-collections-add` to `TailwindComponent` with lit
`Task` usage
### Manual testing
1. Log in as crawler
2. Go to "Archived Items"
3. Click "Upload WACZ"
4. Select wacz file to upload
5. Search for collection in "Add to Collection". Verify you're able to
select a search result
6. Save. Verify collection saves as expected.
7. Go to "Crawling"
8. Create a new workflow
9. Go to "Metadata"
10. Repeat 5-6.
Disable the 'Review Crawl' button only when there are no finished QA
Runs:
- track last finished qa run 'lastFinishedQARunId'
- track if qa run is running via 'isQAActive'
- only add tooltip / disabled state when there are no finished qa runs
Also, only add href when not disabled, to avoid clickable link with
disabled button
---------
Co-authored-by: emma <hi@emma.cafe>
Fixes https://github.com/webrecorder/browsertrix/issues/1743
On the backend, support adding user to new org and improved error
messaging:
- if user exists is not part of the org to be added to, add user to the
registration org, return 201
- if user is already part of the org, return 400 with
'user_already_is_org_member' error
- if user is not being added to a new org but already exists, return
'user_already_exists'
frontend:
- if user user same password, they will just be logged in and added to
registration org (same as before)
- if user uses wrong password, show alert message
- handle both user_already_is_org_member and user_already_registered
with alert message and link to log-in page.
note: this means existing user is added to the registration org even if
they provide wrong password, but they won't be able
to login until they use correct password/reset/etc...
Now that we're using a bunch of tooltips, I found when there are very
few items listed the tooltips sometimes get squished in weird ways —
this fixes that.
Related to https://github.com/webrecorder/browsertrix/issues/1477, minor
UI tweaks as fast follow:
- Makes archived item status icons all the same size
- Increases hover hit target of archived item cells
- Reduces top padding of QA review page
- Adds "Back" button to QA review page to match all other pages
- Updates percentage formatting of tab labels to match pages
- Refreshes archived items page every 5 seconds to get in progress QA
runs.
- Refactors archived items to use `TailwindComponent` and lit tasks
- Adds back all columns to upload list to prevent layout shift (see
follow-ups)
- Fixes list not updating after search value is selected.
- Resolves https://github.com/webrecorder/browsertrix/issues/1705
- Resolves https://github.com/webrecorder/browsertrix/issues/1477
Changes:
- Takes user back to archived item QA tab after submitting review.
- Prevents clicking on page in QA tab when there's no analysis runs.
- Disables analysis-related sort options in QA tab when there's no
analysis runs.
- Handles pages without a title in QA tab.
- Adds initial loading state to QA review page list.
Sort of relevant to #1696
- Improves a number of layout elements at smaller viewport sizes
(specifically, letting button groups wrap onto next lines & ensure
titles can't shrink to 0 width)
- Adds "No page title" to places where there'd normally be a page title
but isn't
- Applies grid styling to page list area to fix overflow issues
- When selecting or loading with a page selected that's farther down on
the page list, the top of the page header could be scrolled away from,
and there'd be no way to scroll back because the area had `overflow:
hidden` applied
- Adds default width and height to image comparer element, so that it
displays correctly when images haven't loaded and doesn't change layout
when images load
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
- Shows QA review status, latest QA state, and QA run count in archived
item list. "Created by" column replaced for space and moved to "Date
Created" tooltip
- Enables sort archived items by QA columns
- Adds tooltip to all columns except name
- Hides non-applicable columns in when archived items are filtered to
uploads only
- Fixes issue where tooltips weren't activated on hover when using
`btrix-table`
- Minor refactor to reuse status labels/colors, renames page review ->
page approval to clarify
Repository Index: Generate an index.yaml in ./docx/helm-repo/index.yaml
to allow for browsertrix to be a helm repository.
docs: rename docs.browsertrix.cloud -> docs.browsertrix.com
docs: update deployment doc to mention helm repo as preferred way to
install
docs build action: generate repository index in GH action
publish action: update auto-generated message to mention installing from
the repo.
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Follow-up to #1689
Instead of just using resource types, using a combination of mime type, resource type and extension analysis.
Current logic is:
- if resourceType is script, stylesheet, image, font, use that definitely
- if resourceType is fetch/xhr, also check for extension, specifically for images
- categorize favicon.ico separately as 'favicon'
- also check for extension for json
- also check for pdf by type + extension
- categorize 'ping' as 'other'
- by default, use the first part of mime as the type
### Motivation
While using the browser default locale is often good enough, we probably
want more full control over locales, especially when we allow users to
choose their own locale — it's a poor experience to not have dates and
numbers formatted consistently with your chosen locale.
### Changes
Ensures (almost) all instances of `<sl-format-date>` as well as `Intl.*`
have the correct locale passed into them from our Lit localization
system.
The one exception here is in `frontend/src/utils/number.ts` where
ordinal suffixes aren't localized, so the locale is hardcoded to `en` —
I'll revisit this in the future.
Fixes#1696
- Inverts the toggle button state for screenshot comparison, is now
untoggled by default
- Changes the Resources tab to a puzzle piece
- Parity with the metaphor used in ReplayWeb.page. Would like to leave
databases to represent actual databases... Which given the app could
very well make their way in here!
- Fixes the dashed boarder radius being applied to the text comparison
tab
- Sets the data table header row to `position: sticky;` so it's always
visible if the table scrolls!
- This applies to all data tables, tested the one in QA review, the
overview page, and the org settings members list
- Casing changes for parity with other dropdown sorting controls
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Closes#1655
### Changes
Removes the separate element inside the `<sl-select>` and instead
doesn't show localized language unless it's different from the language
in the current locale
Closes#424
### Changes
- Adds Browsertrix logo to the nav bar!!
- Will license the typeface (Konsole) as soon as this gets merged.
- Changes the theme colour to Webrecorder's turquoise brand colour
- Adjusts the usage of `primary` vs `blue` for elements where it
shouldn't be the same
- Changes the _Cancel & Discard_ button of the cancel crawl dialogue to
`danger` button variant
- Replaces the Replay icon with the new ReplayWebpage icon!
- Fixes the favicon SVG linking (wrong name, oops!)
- Fills the microscope lens in the microscope icon
- Removes the old btrix-cloud.svg logo
Co-authored-by: sua yoo <sua@webrecorder.org>
### Changes
- improves overflow issues at smaller screen sizes
- adds icons to buttons
- updates text & layout to match mocks
- changes primary button & button options depending on if there's a
qa run available
- adds a loading state for qa run status & buttons
- updates `<btrix-crawl-status>` with a `type` param allowing for
crawls, uploads, and QA runs
- Updates `<btrix-alert>` to match `<sl-tag>` styling
- Improves overflow issues at smaller viewport sizes by making tab
lists overflow when necessary
### Features
- Ability to start/stop/cancel QA runs
https://github.com/webrecorder/browsertrix-cloud/pull/1666 @SuaYoo
- Ability to see progress of current QA run @emma-sg
- Ability to delete QA runs @emma-sg
- Ability to download QA run files
https://github.com/webrecorder/browsertrix-cloud/pull/1666 @SuaYoo
- Only able to start review if a QA Run is finished (for now,
initial pass). @SuaYoo
- Only most recent running or successful QA run is displayed in
header
---------
Co-authored-by: sua yoo <sua@webrecorder.org>
Co-authored-by: sua yoo <sua@suayoo.com>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
