Fixes#1261Closes#1092
The quota for monthly execution minutes is treated as a hard cap. Once
it is exceeded, an alert indicating that an org has exceeded its monthly
execution minutes will display and the user will be unable to start new
crawls. Any running crawls will be stopped once the quota is exceeded.
An execution minutes meter bar is also added in the Org Dashboard and
displayed if a quota is set. More detail in #1305 which was
merged into this branch.
## Changes
- Enable setting 'maxExecMinutesPerMonth' in orgs list quotas by superadmin
- Enforce quota by stopping crawls in operator once quota is reached
- Show alert banner once execution time quota is hit:
- Once quota is hit, disable Run Crawl buttons in frontend, return 403
message with `exec_minutes_quota_reached` detail in backend from
crawl config `/run` endpoint, and don't run new workflows on creation
(similar to storage quota)
- Display execution time for crawls in the crawl details overview,
immediately below
- Show execution minutes meter on dashboard (from #1305)
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: sua yoo <sua@webrecorder.org>
Closes#1215
- Adds account settings page
- Adds overview page
- Adds archived items page
- Adds note about browser profile metadata editing
- Adds note on editing the crawler instances scale while crawling
- Adds details on permission levels for the org settings
- Removes note about not being able to change your display name (follows
#1265)
- Adds `position: sticky` to the workflow creator / editor controls to
affix them to the bottom of the screen, they are now always visible!
- Renames "Extra URLs in Scope" to "Extra URL Prefixes in Scope"
- Updates documentation accordingly
- Adjusts casing for checkboxes
- Adds the multiplication sign to the crawler instances settings to
better communicate that they are increases in scale and not arbitrary
numbers.
* Refactor microk8s playbook to follow structure with shared roles
- Integrates with btrix/deploy role for deploying
- Seperated RedHat and Debian into seperate roles
- Created Common role
- allow running remotely by default
- use 'browsertrix_cloud_home' for charts path
- add additional customizable options to btrix_values.j2 (todo: unify all the templates)
- docs: update to new playbook path
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
* Give protocol selection box smaller max-width
* Add warning and docs link to browser profile creation
- Updates dialog styling to btrix dialog
- Updates button sizes
- Updates button placement in dialog
- Updates button labels for consistency with other buttons in app
- Updates docs page with new button labels
* Update browser profile edit metadata dialog. Matches updated dialog shown on profile creation
* Open docs page in new tab
- If set, and any of the seeds fails, the entire crawl is marked as a failure.
- Add checkbox which adds --failOnFailedSeed checkbox to URL list workflows
- Add 'Fail Crawl On Failed URL' to crawl workflow setup docs
* feat: move do_setup to new unified format at root of ansible/ dir to allow sharing roles, inventory with playbooks for other deployment types
* fix: pass ansible lint
* update do settings to current deployment:
- bump main node params
- add additional settings to helm values template
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
It changes the directory layout of the ansible playbook to a
more "best practices" friendly approach using ansible roles and
a real inventory file
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
- no longer using :latest by default in values.yaml, instead updating version with each release
- set chart version to match app version in Chart.yaml
- update version in helm chart and values.yaml as part of update-version.sh script
- update test.yaml and local-config.yaml to enable using :latest tag images
- ci: add ci script for packaging current helm chart
- docs: updates docs to indicate deploying directly from GitHub release
- docs: add script to fill in latest version for 'VERSION' using custom script
- chart: set local_service_port to 30870 by default, but use only if no ingress.
- default values.yaml set up for local deployment, local-config.yaml contains additional commented out examples
- ci draft: add deployment info to draft with helm install command for current version
- test: fix password check test
Backend:
- add 'maxCrawlSize' to models and crawljob spec
- add 'MAX_CRAWL_SIZE' to configmap
- add maxCrawlSize to new crawlconfig + update APIs
- operator: gracefully stop crawl if current size (from stats) exceeds maxCrawlSize
- tests: add max crawl size tests
Frontend:
- Add Max Crawl Size text box Limits tab
- Users enter max crawl size in GB, convert to bytes
- Add BYTES_PER_GB as constant for converting to bytes
- docs: Crawl Size Limit to user guide workflow setup section
Operator Refactor:
- use 'status.stopping' instead of 'crawl.stopping' to indicate crawl is being stopped, as changing later has no effect in operator
- add is_crawl_stopping() to return if crawl is being stopped, based on crawl.stopping or size or time limit being reached
- crawlerjob status: store byte size under 'size', human readable size under 'sizeHuman' for clarity
- size stat always exists so remove unneeded conditional (defaults to 0)
- store raw byte size in 'size', human readable size in 'sizeHuman'
Charts:
- subchart: update crawlerjob crd in btrix-crds to show status.stopping instead of spec.stopping
- subchart: show 'sizeHuman' property instead of 'size'
- bump subchart version to 0.1.1
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
* 1.6 docs update
### Changes
- Adds note in style guide about referencing actions in the app
- Adds page for Browser Profiles
- Adds callout for uploads in the context of combining items from multiple sources
- Adds page for Collections
- Adds page for Crawl Workflows
- Updates index to link to new dedicated Crawl Workflow page in addition to the Crawl Workflow Setup page
- Updates Org Settings page action styling in accordance with new rules
- Updates Crawl Workflow Setup page with links to the new pages and a hierarchy fix for the first item
- Updates user guide navigation with a new section for crawling related items
---------
Co-authored-by: sua yoo <sua@webrecorder.org>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
- Adds section about the admonitions we use and their meanings when writing documentation
- Heading hierarchy changes (fixed my past blunders!)
- Removes section about GitHub Flavored Markdown — it's not really relevant here anymore considering how much custom stuff we have.
- Clarifies use case for frontend development server
- Fixes incorrect sample API URLs
- Adds additional detail around requirements and quickstart
- Links back to docs from frontend README
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
* feat: ansible DO teardown
* fix(DO): idempotency issues in ansible teardown
* chore(DO): remove unused code
* docs(ansible): mention teardown in the docs
* fix: pass ansible-lint
* fix: point database backup upload to the correct location in DO space
- Adds collections search and list to workflow editor
- Adds collections to workflow details component
- Adds namePrefix filter to backend GET /orgs/{oid}/collections endpoint to support case-insensitive searching of collections
- Adds documentation for new setting
---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
* Adds initial version of the documentation style guide
* Adds a note about adding new pages
* Instructs users about where to edit the `nav:` for the section
* Adds acronym rule clarification
* docs refactor:
- add local deployment guide local-dev-setup.md
- deploy/local.md focuses only on deployment with latest release, links to local-dev-setup.md
for local image deployment
- add nav to mkdocs.yml to ensure correct order of pages
- update microk8s specific info
- update minikube specific info
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
* Revert docs body font to Inter
- Adds Inter V4 Beta variable font files
- Main docs title in the nav retains Recursive as its typeface
- Updates text styling to use app-aligned typography
- Updates code blocks, now with distinct borders!
- Cleans up code blocks, moves like parts together
* font-weight fix
- Aligns with app font weight for titles
- Nav bar text is now 20% higher opacity, hover state also differentiated with weight
- In-body links are now underlined
- Lightened BG colour and darkened link colour — now achieves an APCA score of 84!
* Add stylesheet & mkdocs features
- Adds a custom stylesheet & brand colours
- Adds Recursive as the code font
- Adds repo info to the nav bar
- Adds auto tracking ID links for deep linking to sections as users scroll the page
- Index pages are now a part of their section as determined by their H1
- Removes mkdocs info from future footer
* Reorganize content
- Renames "Dev" to "Develop" for improved navigation labels
- Adds links to tools the first time they're mentioned
- Rewords part of the homepage
- Hides section navigation on the homepage (now we don't have a blank section nav bar!
- Adds some syntax highlighting
- Removes some manual word wrapping — this was done very rarely / inconsistently
* Rename "Developer Docs" index page
- Better title for sidebar
* Update docs.md
- Adds links to tools
- Adds future docs style guide section
- Updates name and makes it an H1
- Replaces hyphens on the homepage with em dashes
* deployment index page: changed title, removed non-k8s section, cleaned up intro
* develop index page: changed title
fixed typo on main page
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
- Renames "Dev" to "Develop" for improved navigation labels
- Deployment docs are now located under a larger "Development" section (fewer nav bar choices & realistically I think anyone who wants to do one is going to be referring to the other)
- Adds links to tools the first time they're mentioned
- Rewords part of the homepage
- Hides section navigation on the homepage (now we don't have a blank section nav bar!
- Adds some syntax highlighting
- Removes some manual word wrapping — this was done very rarely / inconsistently
- Adds a custom stylesheet & brand colours
- Adds Recursive as the code font
- Adds repo info to the nav bar
- Adds auto tracking ID links for deep linking to sections as users scroll the page
- Index pages are now a part of their section as determined by their H1
- Removes mkdocs info from future footer
* modify the template file to highlight optional host that stores WAC
files
* numerically reorder the tcp ports - fix the 404's on the documentation
* add a configuration file - this allows automatic selection of inventory directory
* provide better examples on documentation
* add digital ocean documentation
* remove microk8s role
* use a single playbook to install microk8s
* use inventory hostname
* use a host_ip for endpoint so not to clash with domain
* add RedHat control loops
* add microk8s documentation
* enable permissive mode for redhat
* Minor style updates
- Uses the Material `teal` colour which seems to be a close match... This will be changed later once I do more branding things with a proper theme override
- Adds a monochrome version of the current Browsertrix Cloud logo.
- Fixes typo :P
- local deploy update: add collapsible details blocks for vendor-specific instructions, move installing section before running
- add missing docs on how to run docs
- fix doc typos
