Commit Graph

52 Commits

Author SHA1 Message Date
Ilya Kreymer
06cf9c7cc3
add crawl ending states: 'generate-wacz', 'uploading-wacz', 'pending-wait' that occur after a crawl is finished or is being stopped (#1022)
operator: ensure transitions from each of these states is supported, including to 'waiting_capacity'
add extra check on stopping to avoid transitioning back to a running state after crawl is finished
ui: add states to UI display, localization, add as active states
fixes #263
2023-08-01 00:15:59 -07:00
sua yoo
75b011f951
Upload WACZ via UI (#992)
- Users can now upload .WACZ archives from the "Archived Data" page.
- Can specify name, description, tags and collection(s) to add upload to
- Show progress of upload
- Support canceling upload
2023-07-21 16:45:52 +02:00
sua yoo
66b3befef9
Frontend collections beta UI (#886)
- Support for creating new collections and editing existing collections
- Can select crawling workflows which adds entire workflow, and then deselect individual crawls
- Can edit existing collections and add more crawls
- Can view, create and delete collections via new Collections top-level nav entry
2023-06-06 17:52:01 -07:00
Ilya Kreymer
00fb8ac048
Concurrent Crawl Limit (#874)
concurrent crawl limits: (addresses #866)
- support limits on concurrent crawls that can be run within a single org
- change 'waiting' state to 'waiting_org_limit' for concurrent crawl limit and 'waiting_capacity' for capacity-based
limits

orgs:
- add 'maxConcurrentCrawl' to new 'quotas' object on orgs
- add /quotas endpoint for updating quotas object

operator:
- add all crawljobs as related, appear to be returned in creation order
- operator: if concurrent crawl limit set, ensures current job is in the first N set of crawljobs (as provided via 'related' list of crawljob objects) before it can proceed to 'starting', otherwise set to 'waiting_org_limit'
- api: add org /quotas endpoint for configuring quotas
- remove 'new' state, always start with 'starting'
- crawljob: add 'oid' to crawljob spec and label for easier querying
- more stringent state transitions: add allowed_from to set_state()
- ensure state transitions only happened from allowed states, while failed/canceled can happen from any state
- ensure finished and state synched from db if transition not allowed
- add crawl indices by oid and cid

frontend: 
- show different waiting states on frontend: 'Waiting (Crawl Limit) and 'Waiting (At Capacity)'
- add gear icon on orgs admin page
- and initial popup for setting org quotas, showing all properties from org 'quotas' object

tests:
- add concurrent crawl limit nightly tests
- fix state waiting -> waiting_capacity
- ci: add logging of operator output on test failure
2023-05-30 15:38:03 -07:00
Ilya Kreymer
2cae065c46
Add Waiting state on the backend and frontend (#839)
* operator: add waiting state
- add pods as related objects
- inspect pod status, set crawl status to 'waiting' if no pods are running

frontend:
- frontend support for 'waiting' state
- show waiting icon from mocks

---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
2023-05-08 17:05:01 -07:00
sua yoo
85c96de883
Show critical errors in Crawl detail logs (#811) 2023-05-05 11:30:38 -07:00
sua yoo
7888c4fde3
Frontend crawl workflows rework (#775) 2023-04-25 14:16:07 -07:00
sua yoo
c60dc5d086
Crawls list backend pagination (#735) 2023-04-05 10:55:42 -07:00
sua yoo
bca67c74e2
chore: format frontend files with prettier 2023-03-27 11:05:19 -07:00
sua yoo
f2b7946960
Improve crawl list rendering (#645)
* add load more button

* adjust height

* refactor to improve performance

* remove unused observable component

* contain status

* update dropdown animation
2023-02-28 18:36:23 -08:00
sua yoo
23795ec5fd
Compute name from seed URLs in UI (#644) 2023-02-28 15:51:43 -08:00
sua yoo
1dea7ecdf9
Update crawls list styles (#630)
- Improves crawls list UI for UX and visual consistency
- Enables editing crawl metadata from the crawls list
- Upgraded Tailwind CSS
2023-02-24 17:36:34 -08:00
sua yoo
9532f48515
Fix app not rendering with bad auth storage states (#597)
* render even if session store throws

* handle after timeout

* remove localstorage key

* update tests
2023-02-14 18:35:21 -08:00
sua yoo
d128525e4e
Run unit tests in frontend PR check (#569) 2023-02-06 17:47:15 -08:00
sua yoo
17e1628d2d
Allow superadmins to create org from UI (#563) 2023-02-06 14:58:28 -08:00
sua yoo
4875d7727d
Fix invite accept in UI (#560) 2023-02-06 12:18:24 -08:00
sua yoo
10c96ed2ae
Update tab access by user role (#549)
* update types

* update user org type

* update tabs
2023-02-02 22:26:22 -08:00
sua yoo
8957eda966
Improve org routing & performance (#520) 2023-01-26 15:02:27 -08:00
Tessa Walsh
0fa60ebc45
Rename archives/teams -> orgs in codebase + add db migration (#486)
* Rename archives to orgs and aid to oid on backend

* Rename archive to org and aid to oid in frontend

* Remove translation artifact

* Rename team -> organization

* Add database migrations and run once on startup

* This commit also applies the new by_one_worker decorator to other
asyncio tasks to prevent heavy tasks from being run in each worker.

* Run black, pylint, and husky via pre-commit

* Set db version and use in migrations

* Update and prepare database in single task

* Migrate k8s configmaps
2023-01-18 14:51:04 -08:00
sua yoo
4a23dd12cb
Crawl config detail view & edit workflow UI updates (#415) 2022-12-22 09:37:43 -08:00
sua yoo
28346e0a54
New create crawl config user workflow (#391) 2022-12-12 13:50:33 -08:00
sua yoo
e7f1a00411
Fix authentication getting out of sync between tabs (#380)
Fixes regression to #361 found after increasing the token timeout by preventing app load until the authentication service is initialized (and finishing check if another tab is logged in.)
2022-11-23 23:36:36 -08:00
sua yoo
321f78b861
Upgrade Shoelace 2.0.0-beta.61 -> 2.0.0-beta.83 (#358) 2022-11-21 08:16:51 -08:00
sua yoo
4d4ce40443
Refactor & sync user session across tab/windows (#370) 2022-11-15 19:49:18 -08:00
sua yoo
1ef9f7df6d
Fix auth not persisting on reload (#360) 2022-11-15 13:17:29 -08:00
sua yoo
97eb17784d
Display exclusions & list of URLs in crawl queue (#337)
- including pagination of queue results (30 results per page currently)
- show numbering on paginated results
- allow user navigation to each result page
2022-10-12 20:19:13 -07:00
sua yoo
9606d59c3d
Improve format of crawl template config error from server (#281)
* better display of api errors, such as fields missing or invalid urls, addresses #280
2022-06-29 17:57:03 -07:00
sua yoo
d144591dbf
Display & edit crawl schedule in user local time (#271)
closes #255
2022-06-27 13:01:20 -07:00
sua yoo
f90ef071de
enable opening crawl in new tab 2022-04-11 13:03:10 -07:00
sua yoo
c577e36b74
add debug for access token 2022-02-08 17:52:27 -08:00
sua yoo
02f46f108b
Crawl & crawl config UX improvements (#136) 2022-02-01 14:28:07 -08:00
sua yoo
d7f58c964c
Fix in-app link UX (#132)
closes #130, closes #113
2022-01-31 17:36:50 -08:00
sua yoo
2666b6f6aa
Duplicate crawl config from list (#99) 2022-01-25 17:07:54 -08:00
sua yoo
3a461d86d4
Crawl config detail views (#97) 2022-01-25 11:56:34 -08:00
sua yoo
cb5cf55c69
Add helper for dispatching notify events (#92) 2022-01-19 21:01:47 -08:00
sua yoo
c3edb4bba4
Allow user to configure crawls with JSON (#86) 2022-01-18 19:58:55 -08:00
sua yoo
ff77a92108
Schedule time of day when creating config (#85) 2022-01-18 13:58:28 -08:00
sua yoo
b2088f5634
Add initial crawl template form (#80) 2022-01-16 14:43:33 -08:00
sua yoo
3324bd960f
Refactor to remove sign up and JWT env variables (#65)
closes #63
closes #66
2021-12-06 19:39:04 -08:00
sua yoo
d5c665be8e
Fix archive invite flow (#61)
closes #49
2021-12-05 18:23:24 -08:00
sua yoo
5531c3c33c
hotfix: fix auth persist
Switch lifecycle event to update
2021-12-05 16:52:58 -08:00
sua yoo
9c4bec1411
Add admin page for inviting users (#56)
closes #37
2021-12-05 15:23:28 -08:00
sua yoo
9673794711
Frontend prod build optimizations (#54)
- Leverage webpack chunk splitting to creating more, smaller JS files rather than one large main file (import(file) syntax)
- Enable long-term caching by adding content hash to output file names
- Copy entire /dist folder contents in Dockerfile
- Changed yarn start-dev -> yarn start since there is no prod server
- Reenable locale picker
2021-12-03 18:00:14 -08:00
sua yoo
1d114c682e
hotfix: prevent session from expiring 2021-12-03 17:40:03 -08:00
sua yoo
30e40adddd
Refresh access token in background (#48)
closes #22
2021-12-03 10:00:13 -08:00
sua yoo
c48e870ffe
Refactor frontend auth service (#47) 2021-12-02 16:31:12 -08:00
sua yoo
2396ab9851
Invite user to an archive (#36)
known issue: invalid invitation error shows even when invite is successfully accepted

closes #31
2021-12-02 11:31:03 -08:00
sua yoo
911dc64025
Enable query param routing (#29)
* enable params in url

* rename view state
2021-11-25 12:00:24 -08:00
sua yoo
58eba70c68
Allow users to change password (#25)
wip #21
2021-11-23 17:01:08 -08:00
sua yoo
04fbe6fc4d
Adjust frontend auth behavior (#24) 2021-11-23 16:57:28 -08:00