- Moves page count out from under "Size" label in archived item detail
- Renames "Pages Crawled" to "Pages" in archived item leading heading
and detail overview
- Renames "Crawl ID" to "Archived Item ID"
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
- add ensure_page_limit_quotas() which sets the config limit to the max
pages quota, if any
- set the page limit on the config when: creating new crawl, creating
configmap
- don't set the quota page limit on new or existing crawl workflows
(remove setting it on new workflows) to allow updated quotas to take
affect for next crawl
- frontend: correctly display page limit on workflow settings page from
org quotas, if any.
- operator: get org on each sync in one place
- fixes#2369
---------
Co-authored-by: sua yoo <sua@webrecorder.org>
- Displays workflow form as collapsible sections
- Combines run now toggle into submit
- Fixes exclusion field errors not preventing form submission
- Refactors `<btrix-observable>` into new `Observable` controller
---------
Co-authored-by: emma <hi@emma.cafe>
- Refactors dashboard and org profile preview to use private API
endpoint, to fix public collections not showing when the org
visibility is hidden
- Adds additional sorting options for collections
- Adds unique page url counts for archived items, collections, and
organizations to backend and exposes this in collections
- Shows collection period (i.e. `dateEarliest` to `dateLatest`) in
collections list
- Shows same collection metadata in private and public views, updates
private view info bar
- Fixes "Update Org Profile" action item showing for crawler roles
---------
Co-authored-by: sua yoo <sua@webrecorder.org>
Co-authored-by: sua yoo <sua@suayoo.com>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Fixes issue where collection thumbnail is always the screenshot, even if
a Browsertrix provided default thumbnail is selected after choosing the
screenshot.
Fixes#2257
This is a follow-up to the public collections work, which adds pages to
the database for uploads. All crawls and uploads now have a `pageCount`
field which is populated when the item is successfully added. A new
migration is also added to populate the field for existing archived
items that don't have it set yet.
OrgMetrics have also been modified to include `crawlPageCount` and
`uploadPageCount`, and to include the total of both in `pageCount`, and
all three included in the frontend org dashboard.
The frontend has been updated to use `pageCount` rather than
`stats.done` wherever appropriate, meaning that in archived item lists
and details we now have a consistent page count for both crawls and
uploads.
### New functionality
- Deploy this branch
- Create new crawls and uploads and verify that page count appears
correctly throughout the frontend for all new crawls and uploads
### Migration
- Deploy from latest main
- Create some crawls and uploads
- Change to this branch and re-deploy
- Verify migration ran without errors in backend logs
- Verify that page count has been populated successfully by checking
archived items lists, crawl and upload detail pages, and dashboard to
ensure there are no longer any missing page counts.
---------
Co-authored-by: emma <hi@emma.cafe>
Fixes#2259
This PR brings backend and frontend support for the new autoclick
behavior in Browsertrix, introduces in Browsertrix 1.5.0+
On the backend, we introduce `min_autoclick_crawler_image` to
`values.yaml`, with a default value of
`"docker.io/webrecorder/browsertrix-crawler:1.5.0"`. If this is set and
the crawler version for a new crawl is less than this value, the
autoclick behavior is removed from the behaviors list in the configmap
created for the crawl.
The one caveat for this is that a crawler image tag like "latest" will
always be parsed as greater than `min_autoclick_crawler_image`, so there
is the potential for the crawler to run into issues if using a
non-numeric image tag with an older version of the crawler. For
production we use hardcoded specific versions of the crawler except for
the dev channel, which from here on out will including autoclick
support, so I think this should be okay (and is also true of the
existing implementation for checking `min_qa_crawler_image`).
On the frontend, I've added a checkbox (unchecked by default) in the
"Limits" section just below the current checkbox for autoscroll. We
might want to move these to a different section eventually - I'm not
sure Limits is the right place for them - but I wanted to be consistent
with things as they are.
---------
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
Fixes https://github.com/webrecorder/browsertrix/issues/2306
## Changes
Refactors collection view configuration to wait for thumbnail preview
image (using `URL.createObjectURL`, like in QA screenshots) to be fully
loaded from `replay-web-page` before saving.
Resolves https://github.com/webrecorder/browsertrix/issues/2298
## Changes
- Slugs added to collections, can be specified separately when creating
or updating collections or else is based off of supplied collection name
- Migration added to backfill slugs for existing collections
- Redirect collection to newest slug if changed
- Adds option to copy public profile link to "Public Collections" action
menu
- Show "Back to <Org>" link instead of breadcrumbs
---------
Co-authored-by: sua yoo <sua@suayoo.com>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
- Disables saving collection start page if valid snapshot is not
selected
- Shows full URL in page URL status check mark
- Shows error in page URL status exclamation mark
- Fixes pasting in URL
Fixes#2260
- Adds `lastCrawlFinished` to Organization model, updated after crawls
are added/deleted and with an idempotent migration to backfill existing
orgs
- Adds Last Crawl column to end of admin orgs list table
- Adds subscription icon next to existing status icon in orgs list
- Adds "lastCrawlFinished", "subscriptionStatus", and "subscriptionPlan"
sort options to orgs list backend endpoint in anticipation of future
sorting/filtering of orgs list
---------
Co-authored-by: emma <hi@emma.cafe>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
- Removes share link from the dialogue footer
- Removes stickied collection navigation, replaces with improved
viewport-based scaling!
- Adds a max-width for the collection description in the logged in view.
- Moves the markdown editor buttons to below the editor
- Controls are now In-line with how we handle dialogue options
elsewhere, fixes a minor responsive design issue.
- Minor copy changes
---------
Co-authored-by: emma <hi@emma.cafe>
Co-authored-by: sua yoo <sua@webrecorder.org>
- Renames `inject_analytics` to `inject_extra` and updates docs
- Manually tracks page views to enable passing custom props
- Tracks copying collection share link and downloading a public
collection
---------
Co-authored-by: emma <hi@emma.cafe>
- Allows user to choose collection replay home page and collection
thumbnail (resolves
https://github.com/webrecorder/browsertrix/issues/2182)
- Displays collection thumbnails on org dashboard and public page
- Enables downloading public collection (resolves
https://github.com/webrecorder/browsertrix/issues/2233)
- Adds caption as "Summary" to metadata dialog
- Moves description editor to "About" tab
---------
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
- Enables creating a public org profile page with description and
website at `/profile/<org slug>`
- Updates current "Overview" page to be "Dashboard", found under
`/dashboard`
- Organizes org "General" settings tab by "General", "Profile", and
"Developer Tools"
- Adds sign up banner to log in page for consistent CTA banners
- Updates copy and docs to support changes
- Allows user to set collection to private, public, or unlisted
- Adds route for public collection page with basic page layout
- Refactors copy button to abstract clipboard functionality
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: emma <hi@emma.cafe>
Fixes#2158
- Adds `Organization.listPublicCollections` field and API endpoint to
update it
- Replaces `Collection.isPublic` boolean with `Collection.access`
(values: `private`, `unlisted`, `public`) and add database migration
- Update frontend to use `Collection.access` instead of `isPublic`,
otherwise not changing current behavior
---------
Co-authored-by: sua yoo <sua@suayoo.com>
Also fixes a shoelace menu-item bug where checkbox menu items would have
their checked state flipped on click, regardless of `checked` value.
~~Deploying to dev to test if this fixes language switching...~~ Yep!
Seems to fix the issues.
Closes#2223
- [x] Adds `localesAvailable` to `/api/settings` endpoint, and uses that
list if available, rather than the full list of translated locales, to
determine which options to display to users
- [x] ~~Uses the user's browser locales, filtered to the current
language setting, for formatting numbers, dates, and durations~~
- [x] Adds & persists checkbox for "use same language for formatting
dates and numbers" in user settings
- [x] Replaces uses of `sl-format-bytes` with `localize.bytes(...)`, and
`sl-format-date` with replacement `btrix-format-date` that properly
handles fallback locales
- [x] Caches all number/duration/datetime formatters by a combined key
consisting of app language, browser language, browser setting, and
formatter options so that all formatters can be reused if needed
(previously any formatter with non-default options would be recreated
every render)
- [x] Splits out ordinal formatting from number formatter, as it didn't
make much sense in some non-English locales
- [x] Adds a little demo of date/time/duration/number formatting so you
can see what effect your language settings have
https://github.com/user-attachments/assets/724858cb-b140-4d72-a38d-83f602c71bc7
---------
Signed-off-by: emma <hi@emma.cafe>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
This has been bugging me for a while — any type of repetitive action
that shows a toast will quickly cause a big stack of toasts to build up.
This fixes that by allowing different categories of toasts to exist,
where each category can only have one active toast. _Different_
categories of toast can coexist just fine, but toasts with the same `id`
will replace any existing toast with that same `id`.
It's been a pet peeve for a while, and it should make any sort of "power
user" use of Browsertrix a lot nicer :)
I've gone over all the existing toasts and applied what feel like decent
differentiations, but I'm not set on any of them — open to suggestions!
https://github.com/user-attachments/assets/dbe39141-9e56-427d-b702-124c45ef2b9a
---------
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
Adds sending a cancellation email when a subscription is cancelled.
- The email may also include an option survey optional survey URL, if
configured in helm chart `survey_url` setting.
- Cancellation e-mail configured in `sub_cancel` e-mail template
- E-mails are sent to all org admins.
- Also adds `trialing_canceled` subscription state to differentiate from
a default `trialing` which will automatically rollover into `active`.
- The email is sent when: a new cancellation date is added for an
`active` subscription, or a `trialing` subscription is changed to to
`trialing_canceled`. (A subscription can be canceled/uncanceled several
times before actual date, and e-mail is sent every time it is canceled.)
- The 'You have X days left of your trial' is also always displayed when
state is in trialing_canceled.
Fixes#2229
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
This PR syncs the weblate branch back to main, and adds:
- New Spanish translations contributed in weblate!
- New GH action which reformats weblate commits with `localize:extract`,
ensuring strings are not changed, and also runs `localize:build` to
extract templates (as tested in PR: #2207)
- Reformats existing XLIFF files to match `localize:extract`, including
adding self-closing tags.
- Update on generated templates
- Prettier: disable alpha sorting XML attributes to hopefully minimize
conflcits
- Strings: change some strings that use `&` to be wrapped in html`` to
ensure proper unencoding, or just switch to 'and' where that's not
possible.
---------
Co-authored-by: Weblate (bot) <hosted@weblate.org>
Co-authored-by: Webrecorder Dev <dev@webrecorder.org>
Co-authored-by: Kamborio <Kamborio15@users.noreply.hosted.weblate.org>
Co-authored-by: Clara Itzel <missclaraitzel@gmail.com>
Co-authored-by: weblate <weblate@users.noreply.github.com>
Co-authored-by: emma <hi@emma.cafe>
Co-authored-by: sua yoo <sua@webrecorder.org>
Closes#2213
This ended up being a bit of a variety PR, but primarily this fixes
issues around archived item pages for crawls at URL locations we no
longer support — specifically, crawls, which now live at
`/orgs/{oid}/workflows/{workflow_id}/crawl/{crawl_id}`, rather than
`/orgs/{oid}/items/crawl/{crawl_id}`, including the QA pages at
sub-paths of these. Attempting to view a crawl at the old (unsupported)
URL will now redirect you to the correct URL!
Other changes include:
- Updated links in various parts of the app — nothing should link to the
old URL pattern anymore for crawls!
- Improved `APIRouter` and `ViewState` typing
- A couple extra `APIRouter` tests
- An addition to the route definition for crawls at
`/orgs/{oid}/items/crawl/{crawl_id}` that allows for redirections from
QA review page URLs, rather than 404 errors — this was the actual
reported problem!
- Caching in CI for playwright tests
- `needsLogin` conversion from the deprecated `LiteElement` to
`BtrixElement`, along with all the components that use it
---------
Co-authored-by: emma-sg <emma-sg@users.noreply.github.com>
Resolves https://github.com/webrecorder/browsertrix/issues/2192
## Changes
- Displays more accurate subscription/trial end message when
subscription ends between 1-2 days
- Shows timezone for end date
---------
Co-authored-by: SuaYoo <SuaYoo@users.noreply.github.com>
Partially addresses
https://github.com/webrecorder/browsertrix/issues/2171
- [x] Reimplement `pretty-ms`'s rounding logic to convert to a
`Intl.DurationFormat`-compatible object
- [x] Add `localize.duration` and `localize.humanizeDuration` methods
- [x] Replace `pretty-ms` usage with localized formatting
- [x] Add `Intl.DurationFormat` polyfill
- [x] Update localize tests
- [x] Fix broken tests (thank you @SuaYoo!)
- This looks to be possibly a bug somewhere in the web test runner's
code? Have yet to figure this one out
- Might be related to these:
https://github.com/web-dev-server/web-dev-server/issues/1https://github.com/guybedford/es-module-lexer/issues/82 — both closed
with no comments or resolution 🙃 — `es-module-lexer` seems to be most
likely, given it's a native module & might be handling unicode sequences
incorrectly or not at all?
- [x] Clean up messy `index.d.ts` — probably split this out? The
definitions haven't yet made their way into `esnext`
https://github.com/microsoft/TypeScript/issues/60608
---------
Co-authored-by: emma-sg <emma-sg@users.noreply.github.com>
Co-authored-by: sua yoo <sua@suayoo.com>
Fixes https://github.com/webrecorder/browsertrix/issues/2190
## Changes
- Fixes not being able to add live exclusion
- Fixes not being able to delete last exclusion
- Fixes collection list search
- Refactors exclusion form event name
- Separates translatable html strings into parts to make them easier to
translate
- Updates copy regarding subscription
- Updates plan feature list to match website
- Defaults to user's browser language preference when displaying dates,
bytes, and numbers (with caveats.) This fixes an issue where numbers
were always formatted in English.
- Shows user's browser language preferences in language dropdown
- Fixes timezone not displayed in archived item detail start and finish
times
- Standardizes number formatting
- Sets timezone that unit tests run in
---------
Co-authored-by: SuaYoo <SuaYoo@users.noreply.github.com>
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Co-authored-by: emma-sg <emma-sg@users.noreply.github.com>
Fixes password strength not being checked when new passwords are
auto-filled, resulting in submit buttons remaining disabled.
---------
Co-authored-by: SuaYoo <SuaYoo@users.noreply.github.com>
Co-authored-by: emma <hi@emma.cafe>
languages specified in `translatedLocales`
- Updates user language preference beta badge text
- Adds link to translation contribution docs
- Refactors component name
---------
Co-authored-by: SuaYoo <SuaYoo@users.noreply.github.com>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: emma <hi@emma.cafe>
- Migrates the following `LiteElement`s to `BtrixElement`:
- `App` (`<browsetrix-home>`)
- `Admin` (`<btrix-home`)
- `Dashboard` (`<btrix-dashboard>`)
- `LogInPage` (`<btrix-log-in>`)
- Replaces custom light DOM `btrix-input` with `sl-input` since password
managers should work now.
Fixes#2106
Docs are now hosted as part of the frontend at `/docs` by default.
- If `docs_url` is set in the helm chart, the `/docs` endpoint will
redirect to that endpoint instead
- Use multi-stage python image to build mkdocs as part of frontend, then
copy static output
- Dir layout: mkdocs.yml and docs into frontend/docs
- CI: Update docs build GH action to use new path
- Update all frontend paths to use `/docs/` instead of
`https://docs.browsertrix.com/`
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Resolves https://github.com/webrecorder/browsertrix/issues/2091
<!-- Fixes #issue_number -->
### Changes
Saves scope type chosen from "+ New Workflow" dropdown to local storage, as well as from within workflow editor when creating a new workflow (but not editing an existing one).
Resolves#1354
Supports crawling through pre-configured proxy servers, allowing users to select which proxy servers to use (requires browsertrix crawler 1.3+)
Config:
- proxies defined in btrix-proxies subchart
- can be configured via btrix-proxies key or separate proxies.yaml file via separate subchart
- proxies list refreshed automatically if crawler_proxies.json changes if subchart is deployed
- support for ssh and socks5 proxies
- proxy keys added to secrets in subchart
- support for default proxy to be always used if no other proxy configured, prevent starting cluster if default proxy not available
- prevent starting manual crawl if previously configured proxy is no longer available, return error
- force 'btrix' username and group name on browsertrix-crawler non-root user to support ssh
Operator:
- support crawling through proxies, pass proxyId in CrawlJob
- support running profile browsers which designated proxy, pass proxyId to ProfileJob
- prevent starting scheduled crawl if previously configured proxy is no longer available
API / Access:
- /api/orgs/all/crawlconfigs/crawler-proxies - get all proxies (superadmin only)
- /api/orgs/{oid}/crawlconfigs/crawler-proxies - get proxies available to particular org
- /api/orgs/{oid}/proxies - update allowed proxies for particular org (superadmin only)
- superadmin can configure which orgs can use which proxies, stored on the org
- superadmin can also allow an org to access all 'shared' proxies, to avoid having to allow a shared proxy on each org.
UI:
- Superadmin has 'Edit Proxies' dialog to configure for each org if it has: dedicated proxies, has access to shared proxies.
- User can select a proxy in Crawl Workflow browser settings
- Users can choose to launch a browser profile with a particular proxy
- Display which proxy is used to create profile in profile selector
- Users can choose with default proxy to use for new workflows in Crawling Defaults
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Resolves https://github.com/webrecorder/browsertrix/issues/2073
### Changes
- Removes "URL List" and "Seeded Crawl" job type distinction and adds as
additional crawl scope types instead.
- 'New Workflow' button defaults to Single Page
- 'New Workflow' dropdown includes Page Crawl (Single Page, Page List, In-Page Links) and Site Crawl (Page in Same Directory, Page on Same Domain, + Subdomains and Custom Page Prefix)
- Enables specifying `DOCS_URL` in `.env`
- Additional follow-ups in #2090, #2091
Use timezone aware datetimes instead of timezone naive datetimes:
- Update mongodb client to use tz-aware conversion
- Convert dt_now() to return timezone aware UTC date
- Rename to_k8s_date -> date_to_str, just returns ISO UTC date with 'Z'
(instead of '+00:00' suffix)
- Rename from_k8s_date -> str_to_date, returns timezone aware date from
str
- Standardize all string<->date conversion to use either date_to_str or
str_to_date
- Update frontend to assume iso date, not append 'Z' directly
- Update tests to check for 'Z' suffix on some dates
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
- Fixes empty watch tab during some active states
- Disables exclusion and browser window editing while a crawl is
stopping
- Refactors frontend crawl states to match backend
- Replaces QA Review breadcrumbs with "Back" button that takes you back
to the archived item
- Crawl breadcrumbs always point to workflow
- Shows archived item icon next to item name to differentiate from
workflow
- Adds "Edit Workflow" button to "Crawl Settings"
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
