- make crawl args a reusable template
- adds QA_ARGS to configmap, setting to same value as CRAWL_ARGS but
with --behaviors= prepended to disable behaviors for QA, to improve
performance of QA runs.
fixes#1962
- only enable if 'enable_auto_resize' is true, default to false
- if true, set memory limit to 1.2 of memory requests, resize when
hitting 'soft oom' of initial request, adjust by 1.2 (current behavior)
up to max_crawler_memory
- if false, set memory limit to max_crawler_memory and never adjust
memory requests or memory limits
- part of #1959
Directs user that doesn't belong to any orgs to account settings page,
with banner.
Also contains some minor out-of-scope changes:
- Refactors `isAdmin` key to `isSuperAdmin` for more legibility on
whether current user is superadmin or regular user without orgs
- Adds "cancel" button to change password form
Fixes#1955
Orgs list endpoint sorting now works as follows:
- Default org is always sorted first
- Name sorting now works on a lowercased version of the org names to
ensure lexical sorting
The lodash `sortBy` resorting of orgs in the "All Organizations"
dropdown list in the nav bar has also been removed so that the backend
sorting is applied instead.
Tests have been updated accordingly.
* updates pydantic to 2.x
* also update to python 3.12
* additional type fixes:
- all Optional[] types must have a default value
- update to constrained types
- URL types converted from str
- test updates
Fixes#1940
Follow-up to regressions from #1928, this PR:
- Fixes response models for queue endpoints, which had incorrect model
- Adds tests for queue get, queue match, and exclusions add / remove to
ensure regressions like this can be caught via tests. This involves
starting a new crawl in test_run_crawls() instead of relying on implicit
running via fixtures, make it easier to test crawl while it's running.
- Adds additional typing for crawls apis, including making
delete_crawls() have correct typing, consistent derived class override
- Adds check to ensure queue + exclusion operations can not be called
when crawl is not running
Fixes#1927
Also adds tests to ensure index is working as expected, and migration to
rename orgs that have names or slugs identical to other orgs except for
case before the new case-insensitive index is built.
### Changes
- Updates customer portal link label
- Opens billing portal in the same tab
- Shows separate cancel date
- Makes payment failed appear as error
- Fixes crawl time quota
- ensure crawlFilenameTemplate is part of the CrawlConfig model
- change CrawlConfig init to use type-safe construction
- add a run_now_internal() that is shared for starting crawl, either on
demand or from new config
- add OrgOps.can_run_crawls() to check against org quotas for crawling
- cleanup profile updates, remove _lookup_profile, only check for
EmptyStr in update
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Fixes#1926
- adds /subscriptions/import endpoint for importing an existing subscription to an existing org
- add SubscriptionImport object and log as 'import' event in subscription events collection
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Fixes a few things that have been bugging me:
- Overflow buttons in list view now (mostly) take up the their full cell
area, instead of there being a couple pixels around the button where
clicking would do nothing or cause navigation
- | before | after |
| --- | --- |
| <img width="238" alt="Screenshot 2024-07-16 at 3 35 25 PM"
src="https://github.com/user-attachments/assets/afbda6d6-703b-4ed8-96be-a9c37660430d">
| <img width="236" alt="Screenshot 2024-07-16 at 3 35 02 PM"
src="https://github.com/user-attachments/assets/417a326a-08d2-42b2-85c3-fa007ea3bff8">
|
- Changes the class that `tab-list` uses internally so that it doesn't
conflict with Tailwind's `container` class, which prevents the tab
content from being limited at the default Tailwind container width
- Adds a couple of Tailwind plugins for styling CSS parts
(`part-[...]:`) and for arbitrary attributes (`attr-[...]:`)
- Verifies org slug (name) availability when creating new org
- Show org max length error when signing up
- Highlights org error field when signing up
- Fixes org name max length discrepancy
- Standardizes org slug to lowercase
Fixes#1920
Adds response models to all API endpoints that were missing them,
documenting current behavior without making any changes at this stage to
standardize responses.
Follow-up work will involve adding generics to some of the response models
Fixes#1916
- Add `created` field to Organization and OrgOut, set on org creation
- Add migration to backfill `created` dates from first workflow
`created`
- Replace `datetime.now()` and `datetime.utcnow()` across app with
consistent timezone-aware `utils.dt_now` helper function, which now uses
`datetime.now(timezone.utc)`. This is in part to ensure consistency in
how we handle datetimes, and also to get ahead of timezone naive
datetime creation methods like `datetime.utcnow()` being deprecated in
Python 3.12. For more, see:
https://blog.miguelgrinberg.com/post/it-s-time-for-a-change-datetime-utcnow-is-now-deprecated
Initial implementation of #1892
- Modifies the backend to return `duplicate_org_name` or
`duplicate_org_slug` as appropriate on a pymongo `DuplicateKeyError`
- Updates frontend to handle `duplicate_org_name`, `duplicate_org_slug`,
and `invalid_slug` error details
- Update errors to be more consistent, also return `duplicate_org_subscription.subId` for duplicate subscription instead of the more generic `already_exists`
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Fixes https://github.com/webrecorder/browsertrix/issues/1905
- adds a new top-level `/api/subscriptions` endpoint and SubOps handler on
the backend.
- enable subscriptions API endpoints available only if `billing_enabled` is
set in helm chart
- new POST /subscriptions/create, /subscriptions/update,
/subscriptions/cancel API endpoints
- Subscriptions mongo collection storing timestamped /subscription
API events
- GET /subscriptions/events API to get subscription events, support for filtering and sorting
- Subscription data model
- Support for setting and handling readOnlyOnCancel on org
- /orgs/<id>/billing-portal to lookup portalUrl using external API
- subscription in org getter and list views
- mark org as readOnly for subscription status `paused_payment_failed`, clears it on status `active`
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
- Displays org status in superadmin view
- Enables superadmins to update org read-only status
- Sorts orgs alphabetically
- Refactors org list to `TailwindComponent`
No issue created, but noticed issue here
ed0d489cda
### Changes
- Removes unused node module mocks and use commonjs plugin to import
modules in tests
- Fixes org form test after removing temporary stub
Limit egress traffic from crawler/profilebrowser pods to the internet
and limited internal services like dns, redis, frontend, auth-signer on certain ports
---------
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
