Fixes#2259
This PR brings backend and frontend support for the new autoclick
behavior in Browsertrix, introduces in Browsertrix 1.5.0+
On the backend, we introduce `min_autoclick_crawler_image` to
`values.yaml`, with a default value of
`"docker.io/webrecorder/browsertrix-crawler:1.5.0"`. If this is set and
the crawler version for a new crawl is less than this value, the
autoclick behavior is removed from the behaviors list in the configmap
created for the crawl.
The one caveat for this is that a crawler image tag like "latest" will
always be parsed as greater than `min_autoclick_crawler_image`, so there
is the potential for the crawler to run into issues if using a
non-numeric image tag with an older version of the crawler. For
production we use hardcoded specific versions of the crawler except for
the dev channel, which from here on out will including autoclick
support, so I think this should be okay (and is also true of the
existing implementation for checking `min_qa_crawler_image`).
On the frontend, I've added a checkbox (unchecked by default) in the
"Limits" section just below the current checkbox for autoscroll. We
might want to move these to a different section eventually - I'm not
sure Limits is the right place for them - but I wanted to be consistent
with things as they are.
---------
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
- By default, use only `ingressClassName` for ingress class name and
corresponding field in cert-manager
- Only use old 'kubernetes.io/ingress.class' if
ingress.useOldClassAnnotation is set
- Allow for using old annotation only for backwards compatibility, eg.
for GCP
- Closes#2267 and #1570
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Fixes https://github.com/webrecorder/browsertrix/issues/2306
## Changes
Refactors collection view configuration to wait for thumbnail preview
image (using `URL.createObjectURL`, like in QA screenshots) to be fully
loaded from `replay-web-page` before saving.
Resolves https://github.com/webrecorder/browsertrix/issues/2298
## Changes
- Slugs added to collections, can be specified separately when creating
or updating collections or else is based off of supplied collection name
- Migration added to backfill slugs for existing collections
- Redirect collection to newest slug if changed
- Adds option to copy public profile link to "Public Collections" action
menu
- Show "Back to <Org>" link instead of breadcrumbs
---------
Co-authored-by: sua yoo <sua@suayoo.com>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
- Upgrades webpack and webpack tool versions
- Updates dev source map to webpack recommendation
- Implements `webpack.DllPlugin` in dev for faster rebuilds
- Implements `thread-loader` to run `ts-loader` in a worker pool
- Disables saving collection start page if valid snapshot is not
selected
- Shows full URL in page URL status check mark
- Shows error in page URL status exclamation mark
- Fixes pasting in URL
- Refactors instances of `btrix-tab-list` except in workflow editor in
preparation for https://github.com/webrecorder/browsertrix/issues/2169
- Removes the visual space above navigation item since many tab headings
describe the first section in the tab, rather than the entire tab itself
Fixes#2260
- Adds `lastCrawlFinished` to Organization model, updated after crawls
are added/deleted and with an idempotent migration to backfill existing
orgs
- Adds Last Crawl column to end of admin orgs list table
- Adds subscription icon next to existing status icon in orgs list
- Adds "lastCrawlFinished", "subscriptionStatus", and "subscriptionPlan"
sort options to orgs list backend endpoint in anticipation of future
sorting/filtering of orgs list
---------
Co-authored-by: emma <hi@emma.cafe>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
- Removes share link from the dialogue footer
- Removes stickied collection navigation, replaces with improved
viewport-based scaling!
- Adds a max-width for the collection description in the logged in view.
- Moves the markdown editor buttons to below the editor
- Controls are now In-line with how we handle dialogue options
elsewhere, fixes a minor responsive design issue.
- Minor copy changes
---------
Co-authored-by: emma <hi@emma.cafe>
Co-authored-by: sua yoo <sua@webrecorder.org>
- Renames `inject_analytics` to `inject_extra` and updates docs
- Manually tracks page views to enable passing custom props
- Tracks copying collection share link and downloading a public
collection
---------
Co-authored-by: emma <hi@emma.cafe>
- Allows user to choose collection replay home page and collection
thumbnail (resolves
https://github.com/webrecorder/browsertrix/issues/2182)
- Displays collection thumbnails on org dashboard and public page
- Enables downloading public collection (resolves
https://github.com/webrecorder/browsertrix/issues/2233)
- Adds caption as "Summary" to metadata dialog
- Moves description editor to "About" tab
---------
Co-authored-by: Emma Segal-Grossman <hi@emma.cafe>
Fixes#2182
This rather large PR adds the rest of what should be needed for public
collections work in the frontend.
New API endpoints include:
- Public collections endpoints: GET, streaming download
- Paginated list of URLs in collection with snapshot (page) info for
each
- Collection endpoint to set home URL
- Collection endpoint to upload thumbnail as stream
- DELETE endpoint to remove collection thumbnail
Changes to existing API endpoints include:
- Paginating public collection list results
- Several `pages` endpoints that previously only supported `/crawls/` in
their path, e.g. `/orgs/{oid}/crawls/all/pages/reAdd`, now support
`/uploads/` and `/all-crawls/` namespaces as well. This is necessitated
by adding pages for uploads to the database (see below). For
`/orgs/{oid}/namespace/all/pages/reAdd`, `crawls` or `uploads` will
serve as a filter to only affect crawls of that given type. Other
endpoints are more liberal at this point, and will perform the same
action regardless of the namespace used in the route (we'll likely want
to change this in a follow-up to be more consistent).
- `/orgs/{oid}/namespace/all/pages/reAdd` now kicks off a background job
rather than doing all of the computation in an asyncio task in the
backend container. The background job additionally updates collection
date ranges, page/size counts, and tags for each collection in the org
after pages have been (re)added.
Other big changes:
- New uploads will now have their pages read into the database!
Collection page counts now also include uploads
- A migration was added to start a background job for each org that will
add the pages for previously-uploaded WACZ files to the database and
update collections accordingly
- Adds a new `ImageFile` subclass of `BaseFile` for thumbnails that we
can use for other user-uploaded image files moving forward, with
separate output models for authenticated and public endpoints
- Enables creating a public org profile page with description and
website at `/profile/<org slug>`
- Updates current "Overview" page to be "Dashboard", found under
`/dashboard`
- Organizes org "General" settings tab by "General", "Profile", and
"Developer Tools"
- Adds sign up banner to log in page for consistent CTA banners
- Updates copy and docs to support changes
- Allows user to set collection to private, public, or unlisted
- Adds route for public collection page with basic page layout
- Refactors copy button to abstract clipboard functionality
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: emma <hi@emma.cafe>
Fixes#1051
If org with provided slug doesn't exist or no public collections exist
for that org, return same 404 response with a detail of
"public_profile_not_found" to prevent people from using public endpoint
to determine whether an org exists.
Endpoint is `GET /api/public-collections/<org-slug>` (no auth needed) to
avoid collisions with existing org and collection endpoints.
Fixes#2158
- Adds `Organization.listPublicCollections` field and API endpoint to
update it
- Replaces `Collection.isPublic` boolean with `Collection.access`
(values: `private`, `unlisted`, `public`) and add database migration
- Update frontend to use `Collection.access` instead of `isPublic`,
otherwise not changing current behavior
---------
Co-authored-by: sua yoo <sua@suayoo.com>
Also fixes a shoelace menu-item bug where checkbox menu items would have
their checked state flipped on click, regardless of `checked` value.
~~Deploying to dev to test if this fixes language switching...~~ Yep!
Seems to fix the issues.
- Uses crawl count to determine whether workflow can be deleted instead
of last crawl ID
- Display delete confirmation dialog when trying to delete a workflow
Fixes#2170
The number of days to delay file replication deletion by is configurable
in the Helm chart with `replica_deletion_delay_days` (set by default to
7 days in `values.yaml` to encourage good practice, though we could
change this).
When `replica_deletion_delay_days` is set to an int above 0, when a
delete replica job would otherwise be started as a Kubernetes Job,
a CronJob is created instead with a cron schedule set to run yearly,
starting x days from the current moment. This cronjob is then deleted by
the operator after the job successfully completes. If a failed
background job is retried, it is re-run immediately as a Job rather
than being scheduled out into the future again.
---------
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
add filtering by role to related crawljobs query:
- for regular crawls (role 'job'), only count other regular crawls
- for qa runs (role 'qa-job') only count other qa jobs
- ensures that concurrent crawl limits apply separately to regular
crawls and qa runs
- fixes#2261
Merge more Spanish translations updates to main for 1.13 release
Co-authored-by: Weblate (bot) <hosted@weblate.org>
Co-authored-by: Clara Itzel <missclaraitzel@gmail.com>
Co-authored-by: Lorena Ramírez-López <lorena@webrecorder.org>
Co-authored-by: Webrecorder Dev <dev@webrecorder.org>
Co-authored-by: weblate <weblate@users.noreply.github.com>
- org.created may be null (for backwards compatibility before it was
set)
- fix frontend type to match backend
- update format.date() to accept null, return empty string
- By default, all locales are enabled to make it easy for local deployments to test new locales
- Adds DE, FR, PT locales to make way for translation in Weblate
