Follow-up to #1608 — quick fix for an issue I encountered after merging
main into #1497
Just going to directly merge once this completes (cc @SuaYoo for
visibility)
- instead of overriding the content-type header globally, pass
'application/merge-patch+json' to
self.custom_api.patch_namespaced_custom_object() directly
- bump kubernetes-asyncio to 29.0.0
- fixes potential issues with global override of the header in
kubernetes-asyncio
- copy of #1602 for main
Closes#328
## Changes
The app has favicons now!
Added:
- SVG
- Changes to slightly brighter colours in dark mode for better contrast!
- Fallback ICO
- `apple-touch-icon` (some browsers also use this, not just iOS)
- Web manifest with app description
- Two web manifest icon sizes should users add the app to their local
launcher (Windows' Start or macOS' Dock / Launchpad
- Lighting & render by @emma-sg, thanks!
The manifest and icons are copied to the root directory at build time by
webpack. All of the dedicated ways of doing this seemed more complicated
than this?
---------
Co-authored-by: emma <hi@emma.cafe>
Partially addresses #1241
### Changes
- Adds Browsertrix logo to readme
- It detects if you're in light or dark mode and adjusts the text color
accordingly! _The future is now!_
- Minor readme updates
- Updates icon and adds favicon SVGs to the docs
- This does not yet use Konsole for the docs site title. Will have to
sort this out later along with private hosting for that font.
- Updates docs theme to use new brand colours — picked the green for
this one, will probably be consistent across all of Webrecorder's MKDocs
sites.
- Backend: Use separate resource constraints for profiles: default
profile browser resources to either 'profile_browser_cpu' /
'profile_browser_memory' or single browser 'crawler_memory_base' /
'crawler_cpu_base', instead of scaled to the number of browser workers
- Frontend: check that profile html page is loading, keep retrying if
still getting nginx error instead of loading an iframe with the error.
Fixes#1598 (Copy of #1599 from 1.9.4)
Changes:
- Edits templates for succinctness and precision
- Separate section for screenshots and OS/browser for bugs
- Removes requirements and TODO section of features to simplify
interface for external-facing requests
I came across [this
problem](https://forum.webrecorder.net/t/deleting-crawl-failure/512) and
noticed that the access URL is used when deleting files, causing my file
deletions to fail on OpenStack SWIFT S3 (relates to #1090). This trivial
change makes it work there.
Closes#1568
## Changes
- Status icons are now filled!
- Uses Bootstrap Icons' new `copy` icon for all actions involving
copying to clipboard!
- Finally! A real copy icon! 🎉
- Removes `copy-code.svg` as it is no longer used
- Actions involving duplicating objects still use `files`... Which is
good! Now they have distinct symbols!
- Adds orange to the tailwind colour palette
---------
Co-authored-by: sua yoo <sua@webrecorder.org>
- Automatically update view to first page if page ID isn't specified
- Show current page URL in location bar (resolves
https://github.com/webrecorder/browsertrix-cloud/issues/1495)
- Approve, reject, or leave notes on a page
- Display temporary list of links to pages in the sidebar
Part of #1241
### Changes
- Renames all instances of "Browsertrix Cloud" to "Browsertrix" on the
front end, emails, and documentation
---------
Co-authored-by: emma <hi@emma.cafe>
Allow maximum scale option to be fully configurable via
`max_crawl_scale`. Already configurable on the backend, and now exposed
to the frontend via API `/api/settings` `maxCrawlScale` value.
The workflow editor and workflow details are updated to allow selecting
the scale up to the maxCrawlScale setting (which defaults to 3 if not
set).
Fixes#1539
Adds `reviewStatus` field to `BaseCrawl` model, updatable via the crawl
update API endpoint. Acceptable values are "good", "acceptable" or
"failure", enforced by an Enum.
Added to `BaseCrawl` so that we can extend support to uploads more
easily later on, but for now we'll only display this for crawls in the
frontend.
Based on #1534
Figured this should be in place so we can work on other front-end things
with these, rather than dealing with refactoring later
<!-- Fixes #issue_number -->
### Changes
- Adds `ArchivedItemPage` and `ArchivedItemPageComment` types from #1534
(thank you @SuaYoo!)
- Adds typedefs for match and resource count properties
- sets properties optional in the db schema to optional in the type as
well
### Manual testing
1.
### Screenshots
| Page | Image/video |
| ---- | ----------- |
| | |
<!-- ### Follow-ups -->
Fixes#1555
This is a first pass at some of the configuration options within the
Helm chart that might be most applicable to users. Emphasis is placed on
configuration that's particular to our application, such as storage and
crawler channels.
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
The operator class has gotten fairly large, this is a first pass in
refactoring operator.py into a submodule instead, with multiple operator
instances which handle different types of objects.
- The main k8s interface has been split into K8sOpApi which extends K8sApi
and is shared across all operators.
- Each operator extends BaseOperator which also has an instance of K8sOpApi
- The CrawlOperator is still the bulk of the functionality, but will likely be further refactored
to support QA jobs
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Fixes#1558
- Adds crawl errors to database incrementally during crawl rather than
after crawl completes
- Simplifies crawl /errors API endpoint to always return errors from
database
- increases the failureThreshold for startupProbe for the api backend
container to account for long running migrations, upto 300 seconds
- add `/healthzStartup` which checks if db is ready
- bump
- keeps `/healthz` to always return 200 when running
- increases livenessProbe failureThreshold to be higher than readiness
probe, following recommended best practice of liveness probe > readiness
probe
- fixes#1559
Fixes#1502
- Adds pages to database as they get added to Redis during crawl
- Adds migration to add pages to database for older crawls from
pages.jsonl and extraPages.jsonl files in WACZ
- Adds GET, list GET, and PATCH update endpoints for pages
- Adds POST (add), PATCH, and POST (delete) endpoints for page notes,
each with their own id, timestamp, and user info in addition to text
- Adds page_ops methods for 1. adding resources/urls to page, and 2.
adding automated heuristics and supplemental info (mime, type, etc.) to
page (for use in crawl QA job)
- Modifies `Migration` class to accept kwargs so that we can pass in ops
classes as needed for migrations
- Deletes WACZ files and pages from database for failed crawls during
crawl_finished process
- Deletes crawl pages when a crawl is deleted
Note: Requires a crawler version 1.0.0 beta3 or later, with support for
`--writePagesToRedis` to populate pages at crawl completion. Beta 4 is
configured in the test chart, which should be upgraded to stable 1.0.0
when it's released.
Connected to https://github.com/webrecorder/browsertrix-crawler/pull/464
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Supports setting WARC prefix for WARCs inside WACZ to `<org slug>-<slug
[crawl name | first seed host]>`.
- Prefix set via WARC_PREFIX env var, supported in browsertrix-crawler
1.0.0-beta.4 or higher
If crawl name is provided, uses crawl name, other hostname of first
seed. The name is 'sluggified', using lowercase alphanum characters
separated by dashes.
Ex: in an organization called `Default Org`, a crawl of
`https://specs.webrecorder.net/` and no name will have WARCs named:
`default-org-specs-webrecorder-net-....warc.gz`
If the crawl is given the name `SPECS`, the WARCs will be named
`default-org-specs-manual-....warc.gz`
Fixes#412 in a default way.
fix call from when(...) to call function directly, avoid implicit true,
which results in page scope being shown for url list.
fixes#1535
To test:
1) Create new workflow of type URL List
2) Ensure the Crawl Scope drop down is not shown.
---------
Co-authored-by: sua yoo <sua@suayoo.com>
Resolves https://github.com/webrecorder/browsertrix-cloud/issues/1493
<!-- Fixes #issue_number -->
### Changes
Adds WIP QA page with basic grid layout sections and navigation.
### Manual testing
Page can be access by adding `/review/screenshots` or `/review/replay`
to a crawl detail page URL. For example:
```
/orgs/suas-dev-sandbox-2/items/crawl/manual-20240124023524-422e41d6-97d/review/screenshots
```
---------
Co-authored-by: emma <hi@emma.cafe>
Closes#1527
Improves front-end types & ensures the data being accessed matches the
data sent by the back-end.
Tested by hand by using the returned data from the `/orgs/${orgId}`
endpoint in prod where this is happening in dev
Fixes#1522
## Changes
- Adds further security recommendations to change the password to
accounts you care about after crawling
Adds more details about the capabilities afforded with browser profiles.
This is now split into the following sections:
- Logging into Websites
- Accepting Popups
- Changing Browser Settings
- More in the future??? Extensions???
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
This PR addresses a possible failure when Redis pod was inaccessible
from Crawler pod.
- Ensure crawl is set to 'waiting_for_capacity' if either no crawler
pods are available or no redis pod. previously, missing/inaccessible
redis would not result in 'waiting_for_capacity' if crawler pods are
available
- Rework logic: if no crawler and redis after >60 seconds, shutdown
redis. if crawler and no redis, init (or reinit) redis
- track 'lastUpdatedTime' in db when incrementing exec time to avoid
double counting if lastUpdatedTime has not changed, eg. if operator sync
fails.
- add redis timeout of 20 seconds to avoid timing out operator responses
if redis conn takes too long, assume unavailable
Fixes#1525
### Changes
- Changes one of the table cell component usages in the crawl list page
to correctly use the `rowClickTarget` prop, rather than setting the
class to `rowClickTarget`.
- Updates the `rowClickTarget` styling to only apply _within_ a
`<btrix-table-cell>`
