Commit Graph

445 Commits

Author SHA1 Message Date
Ilya Kreymer
4901fc2fe9 version: bump to 1.3.0 2023-02-24 18:07:56 -08:00
Ilya Kreymer
0d2a2de66e
rename Information -> Metadata, rebuild localization strings list (#642) 2023-02-24 18:01:33 -08:00
Tessa Walsh
e2f359c352
CrawlConfig migration and crawl stats query optimization (#633)
* Drop crawl stats fields from CrawlConfig and add migration

* Remove migrate_down from BaseMigration

* Get crawl stats from optimized mongo query
2023-02-24 18:01:15 -08:00
sua yoo
1dea7ecdf9
Update crawls list styles (#630)
- Improves crawls list UI for UX and visual consistency
- Enables editing crawl metadata from the crawls list
- Upgraded Tailwind CSS
2023-02-24 17:36:34 -08:00
Francis Kayiwa
3ba77f0ed2
ansible: rocky firewall (#635)
* modify the template file to highlight optional host that stores WAC
files

* numerically reorder the tcp ports - fix the 404's on the documentation

* add a configuration file - this allows automatic selection of inventory directory

* provide better examples on documentation
2023-02-24 17:28:21 -08:00
Ilya Kreymer
413fd8d7ea
Chart: split Crawl args into separate variables (#639)
* chart crawl args cleanup:
- move configurable settings out of 'crawler_args'
- add 'crawler_session_size_limit_bytes' and 'crawler_session_time_limit_seconds' for --timeLimit and --sizeLimit option for crawler
- remove hard-coded 'timeout' to allow configuring via crawl config
- set liveness check port from existing config value
- add comments that requests hd must be at least double the size limit
- defaults: set crawler_requests_hd to 22GB, default crawl session size limit to 10GB
2023-02-24 17:24:04 -08:00
Henry Wilkinson
2e5db2b1f4
Merge pull request #638 from webrecorder/frontend-update-page-headers 2023-02-24 16:45:06 -05:00
Henry Wilkinson
d36d22fea3 Run prettier on crawl-configs-list.ts 2023-02-24 16:20:32 -05:00
Henry Wilkinson
5df0808c39 Merge branch 'main' into frontend-update-page-headers 2023-02-24 16:19:56 -05:00
sua yoo
5b2e101d20
chore: add editorconfig in frontend
Matches indent style and size to prettier formatter
2023-02-24 13:04:11 -08:00
Sara Tavares
8167d7da8d
fix typos (#640) 2023-02-24 11:10:49 -08:00
Henry Wilkinson
133d8b10ab Removes H1 from nav bar
Accessibility improvement, better for screen readers to have h1 be the content of the page as opposed to the application / brand name.  Rest of the nav bar to be dealt with at a later date.
2023-02-24 00:25:16 -05:00
Henry Wilkinson
d0abf2b324 Fix crawl config grid on sm
- Improves grid breakpoints on md
2023-02-24 00:10:16 -05:00
Henry Wilkinson
7f3cdad5b9 Adds page titles, edits heading hierarchy
### Main Pages + General
- Adds H1 page titles for all main pages
- Moves the New Crawl Config action into the title row from the search controls box
- Gives the Crawl Config search controls box the same style as the Crawls search controls box
- Adds +8px of padding to the search controls box to match mockups
- Search box: medium → small
- Title row control buttons: medium → small

### Details Pages
- h2 → h1 for crawl config and crawl detail pages
- h3 → h2 for crawl config and crawl detail pages
- Removes crawl title margin bottom at medium breakpoint on crawl details page
- Aligns crawl config details title row controls with end of flexbox on mobile to match crawl details controls in the same spot
2023-02-24 00:02:49 -05:00
Tessa Walsh
fff74ee754
Fix microk8s CI (#634) 2023-02-23 16:58:25 -05:00
Tessa Walsh
1b1bc10c60
Fix nightly tests (#632) 2023-02-23 13:57:22 -05:00
sua yoo
e8b835df34
Disable editing crawl config of running crawls (#620) 2023-02-22 21:46:45 -08:00
Tessa Walsh
567e851235
Dynamically calculate crawl stats for crawlconfig endpoints (#623) 2023-02-22 22:17:45 -05:00
Tessa Walsh
cbab425fec
Make nightly tests run nightly, not monthly (#624) 2023-02-22 17:54:16 -05:00
Ilya Kreymer
3df6e0f146
crawler arguments fixes: (#621)
- partial fix to #321, don't hard-code behavior limit into crawler args
- allow setting number of crawler browser instances via 'crawler_browser_instances' to avoid having to override the full crawler args
2023-02-22 13:23:19 -08:00
sua yoo
974aeb5e93
Update crawls list control bar UI (#611) 2023-02-22 11:14:44 -06:00
Tessa Walsh
ed94dde7e6
Include firstSeed and seedCount in crawl endpoints (#618) 2023-02-22 10:27:31 -05:00
sua yoo
c309b809da
Edit crawl notes from crawl detail view (#595) 2023-02-21 12:26:38 -06:00
Ilya Kreymer
0fd18ed3dd version: bump to 1.3.0-beta.0
CHANGES: add upcoming release, link to release changelist for 1.2.0
2023-02-21 10:14:08 -08:00
D. Lee
362dc67532
fix the admin logging doc (#612)
* fix the admin logging doc

* Update chart/admin/logging/README.md

---------

Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
2023-02-21 09:52:46 -08:00
Tessa Walsh
4234f89d25
Rename crawlconfig name from file suffixes (#610) 2023-02-21 12:52:22 -05:00
Francis Kayiwa
25e680e80f
rocky firewall (#604)
* enable firewalld ports

* open up http/s and the microk8s ports

* point to current documentation
2023-02-21 09:41:20 -08:00
sua yoo
dae98e1865
Allow user to delete individual crawls (#609) 2023-02-21 10:52:29 -06:00
Tessa Walsh
30f1930519
Add back GET /users/invite/{token} used by frontend (#607) 2023-02-16 13:02:38 -05:00
Francis Kayiwa
1ebffd8b05
ansible: enable firewalld ports (#602)
open up http/s and the microk8s ports
2023-02-15 20:50:14 -08:00
Tessa Walsh
bd4fba7af7
Fix POST /orgs/{oid}/crawls/delete (#591)
* Fix POST /orgs/{oid}/crawls/delete

- Add permissions check to ensure crawler users can only delete
their own crawls
- Fix broken delete_crawls endpoint
- Delete files from storage as well as deleting crawl from db
- Add tests, including nightly test that ensures crawl files are
no longer accessible after the crawl is deleted
2023-02-15 21:06:12 -05:00
sua yoo
9532f48515
Fix app not rendering with bad auth storage states (#597)
* render even if session store throws

* handle after timeout

* remove localstorage key

* update tests
2023-02-14 18:35:21 -08:00
D. Lee
d15e6c8ad8
Fix doc to build a local image for microk8s (#594)
- add push to registry cmd
- update to frontend_* and backend_*
2023-02-14 16:10:04 -08:00
Tessa Walsh
14b349443f
Make pending invites expire via TTL index (#568)
* Make invites expire after configurable window

The value can be set in EXPIRE_AFTER_SECONDS env var and via
helm chart values, and defaults to 7 days.

* Create nightly test CI and add invite expiration test to it

* Update 404 error message for missing or expired invite

---------

Co-authored-by: sua yoo <sua@suayoo.com>
2023-02-14 16:07:14 -05:00
sua yoo
baa2214c9f
Make all config form help text localizable (#593) 2023-02-13 16:53:33 -08:00
Henry Wilkinson
fea30d23ee
Merge pull request #589 from webrecorder/crawl-scale-to-instances 2023-02-13 15:08:14 -05:00
Henry Wilkinson
7da732331b
Update frontend/src/pages/org/crawl-detail.ts
Co-authored-by: sua yoo <sua@webrecorder.org>
2023-02-12 13:26:27 -05:00
sua yoo
a180b92f4a
Improve superadmin invite UI (#581) 2023-02-12 10:12:53 -08:00
Henry Wilkinson
b84a70b394 Adds help text
Matches crawl config help text
2023-02-09 01:00:14 -05:00
Henry Wilkinson
b7a9d811a0 "Crawl Scale" → "Crawler Instances"
- Changes name to match crawl config label
- Makes the buttons small
2023-02-09 00:41:34 -05:00
Tessa Walsh
103d91556f
Remove non-org-scoped invites from backend (#585)
* Remove non-org-scoped invites
- remove POST /users/invite and related tests
- remove GET /users/invite-delete/{token}
2023-02-08 18:56:28 -08:00
Tessa Walsh
b642c53c59
Make crawlconfig name optional (#588) 2023-02-08 18:38:15 -08:00
Tessa Walsh
ce8f426978
Add notes to crawl and crawl updates (#587) 2023-02-08 18:36:22 -08:00
sua yoo
7463becdff
Manage org member roles and invites (#558)
- View and delete pending invites
- Update user roles for members
- Remove members
2023-02-08 18:32:40 -08:00
Ilya Kreymer
40fb04b385
backend: /orgs/<id>/remove: return 404 if org user doesn't exist, fix… (#561)
* backend: /orgs/<id>/remove: return 404 if org user doesn't exist, fixes issue in #535

Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
2023-02-08 16:22:36 -05:00
Tessa Walsh
a7a18b9db0
Add org-specific delete invite endpoint (#575)
Adds POST /orgs/{oid}/invites/delete, which expects the invited
email address in the POST body.

This endpoint will also delete duplicate invites with the same
email/oid combination if env var ALLOW_DUPE_INVITES allows dupes.
2023-02-08 16:10:09 -05:00
Ilya Kreymer
3261e7d666 ci:
- run k3d-ci on all changes to charts or backend, including PRs (faster)
- run microk8s and k3d-log-ci only on commits to main that change charts or backend (slower)
- run lint on all changes to backend, including PRs

fix
2023-02-08 11:24:54 -08:00
Tessa Walsh
95155e6fbf
Invite token improvements (#564)
- URL decode email address in invites.invite_user
- Add tests for accepting invites
2023-02-07 20:40:28 -08:00
sua yoo
a7a5b7fd63
test: add shoelace form utility to import map
Temporarily fixes test build error, see FIXME
2023-02-07 13:58:14 -08:00
sua yoo
ac947421c0
Allow URL list to have URLs containing commas (#572) 2023-02-07 10:52:34 -08:00