Commit Graph

154 Commits

Author SHA1 Message Date
sua yoo
52d9ae9661
fix skipping to confirm settings when duplicating config (#454) 2023-01-12 13:57:59 -08:00
sua yoo
a7457ca334
Filter crawls, configs, browser profiles by user ID (#463)
* filter crawls by user id

* filter crawl configs by user id

* text change: Filter My Crawls -> Show Only Mine

Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
2023-01-11 20:32:14 -08:00
Ilya Kreymer
30bda8c75d
VNC-Based Profile Browser (#433)
* profile browser vnc support + fixes:
- switch profile browser rendering to use VNC
- frontend: add @novnc/novnc as dependency, create separate bundle novnc.js to load into vnc browser (to avoid loading from each container)
- frontend: update proxy paths to proxy websocket, index page to crawler
- frontend: allow browser profiles in all browsers, remove browser compatibility check
- frontend: update webpack dev config, apply prettier
- frontend: node version fix
- backend: get vncpassword, build new URL for proxying to crawler iframe
- backend: fix profile / crawl job pull policy from 'Always' -> 'Never', should use existing image for job
- backend: fix kill signal to use bash -c to work with latest backend image
- backend/chart: add 'profile_browser_timeout_seconds' to chart values to control how long profile browser to remain when idle (default to 60)
- backend: remove utils.py, now using secret.token_hex() for random suffix
Co-authored-by: sua yoo <sua@suayoo.com>
2023-01-10 14:42:42 -08:00
sua yoo
303df2869c
Sticky the crawl config progress indicator position (#445)
* fix tabs on scroll

* adjust for smaller creen size

* scroll to top of section on change

* only scroll if needed
2023-01-09 19:58:33 -08:00
sua yoo
2dcf5cb36b
Always sub-navigation bar for selected team (#444)
* move nav

* highlight correct tab
2023-01-09 19:58:19 -08:00
sua yoo
c4fc3146b7
Fix app not loading on older Safari <15.4 (#436) 2023-01-04 15:09:57 -08:00
sua yoo
adc7ea6fe0
Persist currently selected team/archive (#441) 2023-01-04 15:09:23 -08:00
sua yoo
5daf550cb8
Frontend archives -> teams migration (#429) 2023-01-03 15:37:32 -08:00
sua yoo
d33d5f7700
Compute crawl config name from seed URLs (#435) 2023-01-03 15:07:21 -08:00
sua yoo
4a23dd12cb
Crawl config detail view & edit workflow UI updates (#415) 2022-12-22 09:37:43 -08:00
sua yoo
3b1db019c4
disable cache on replay web page (#419) 2022-12-21 10:48:37 -08:00
Ilya Kreymer
dfca09fc9c
Add single crawl info api at /crawls/{crawl_id} (#418)
* backend: crawl info apis:
- add /crawls/{crawl_id} api endpoint which just lists the crawl info, without resolving the individual files
- move /crawls/{crawl_id}.json -> /crawls/{crawl_id}/replay.json for clarity that it's used for replay

* frontend: update api for new replay.json endpoint
2022-12-19 14:54:48 -08:00
sua yoo
28346e0a54
New create crawl config user workflow (#391) 2022-12-12 13:50:33 -08:00
sua yoo
5d18ffa938
Fix language configuration UI (#388) 2022-12-01 10:02:13 -08:00
sua yoo
e7f1a00411
Fix authentication getting out of sync between tabs (#380)
Fixes regression to #361 found after increasing the token timeout by preventing app load until the authentication service is initialized (and finishing check if another tab is logged in.)
2022-11-23 23:36:36 -08:00
sua yoo
da8260a028
Make exclusion table cells editable (#379) 2022-11-23 09:43:52 -08:00
sua yoo
003b3c7a78
Allow users to set crawl config language (#377) 2022-11-22 15:15:35 -08:00
sua yoo
52e6b6199a
fix: prevent login event on storage event 2022-11-21 15:51:47 -08:00
sua yoo
321f78b861
Upgrade Shoelace 2.0.0-beta.61 -> 2.0.0-beta.83 (#358) 2022-11-21 08:16:51 -08:00
sua yoo
13d5ab7d2b
Fix exclusion dropdown & removal in crawl config edit (#372)
* fix dialog closing on regex

* fix unable to delete regex with same text
2022-11-18 14:47:40 -08:00
sua yoo
4d4ce40443
Refactor & sync user session across tab/windows (#370) 2022-11-15 19:49:18 -08:00
sua yoo
40054d1501
Update global footer (#354) 2022-11-15 14:15:04 -08:00
sua yoo
1ef9f7df6d
Fix auth not persisting on reload (#360) 2022-11-15 13:17:29 -08:00
sua yoo
8c4348b9f8
Show exclusion editor when creating & editing crawl templates (#353) 2022-11-14 19:34:15 -08:00
sua yoo
d41b582ef6
Remove exclusion from running crawl (#352) 2022-11-14 10:58:33 -08:00
sua yoo
95ec1599ef
Add exclusion to running crawl (#347) 2022-11-08 18:09:11 -06:00
sua yoo
baacbbdc52
Highlight regular expression syntax in Exclusions Table (#341) 2022-11-01 15:31:01 -07:00
sua yoo
97eb17784d
Display exclusions & list of URLs in crawl queue (#337)
- including pagination of queue results (30 results per page currently)
- show numbering on paginated results
- allow user navigation to each result page
2022-10-12 20:19:13 -07:00
sua yoo
8708c24a74
Improve crawl elapsed time UX (#323)
Smoother elapsed crawl timer:
- Crawls list: show seconds increment up to 2 minutes, then show minutes only
- Crawls detail: show seconds increment up to one day
2022-10-05 21:12:31 -04:00
sua yoo
0bbb7905bd
Add crawl queue editor UI components (#331)
WIP #304
2022-10-04 13:13:40 -07:00
sua yoo
e696104ffa
Update crawl template copy (#325) 2022-09-27 19:49:24 -07:00
sua yoo
709936dfa7
hotfix: decrease size of running crawl action button 2022-09-27 19:09:49 -07:00
sua yoo
63ada3e5b3
Update base fonts and text sizes (#327) 2022-09-27 14:32:57 -07:00
sua yoo
94e3dff27f
update sentry CDN script 2022-09-27 12:29:02 -07:00
sua yoo
20bd8ceecb
Fix browser profile table alignment (#322) 2022-09-26 17:14:08 -07:00
sua yoo
2ebd1eb2f6
Continue to watch crawl while stopping (#316)
* show when running

* redirect after done

* show banner that crawl is stopping
2022-09-21 12:39:00 -07:00
sua yoo
319a8a3c07
make clearer that profile selection is optional and that a default profile is used by default (#290)
- Rename 'Select Profile' -> 'Default Profile'
- Rename 'No Profiles' -> 'No Additional Profiles'
2022-08-10 15:54:39 -07:00
sua yoo
ee6161ad43
Frontend browser profile editor enhancements (#288)
- add button to duplicate profile from main view
- add save / cancel button when editing
- change location of 'full screen' button
2022-08-10 15:51:34 -07:00
sua yoo
9606d59c3d
Improve format of crawl template config error from server (#281)
* better display of api errors, such as fields missing or invalid urls, addresses #280
2022-06-29 17:57:03 -07:00
sua yoo
301b05ff4e
Refactor screencast websocket connection and retry (#276)
* replace ip with index and retry connection, fixes #252
2022-06-29 17:55:32 -07:00
Ilya Kreymer
2717a60763
improvements / bug fixes for stop/cancel handling: (#279)
- only send signal if stopping, no need for canceling as pods/containers will be removed
- refactor stop/cancel handling to be unified in manager, separate in job
- when stopping / graceful shutdown, return false if sending signal fails
- return success=true in json response if and only if stop/cancel actually succeeds, return 'error' message in error, should fix #270
- allow canceling after stopping / if stopping fails
- ensure finished time is set in case of cancelation before crawl starts, should fix #273
2022-06-29 17:47:25 -07:00
sua yoo
1c52902ea0
Update crawl scale label for UI consistency (#275)
closes #254
2022-06-29 16:14:03 -07:00
sua yoo
92292591ad
Re-run crawl from detail view + handle inactive crawl template (#268)
closes #253
2022-06-29 14:17:09 -07:00
sua yoo
d144591dbf
Display & edit crawl schedule in user local time (#271)
closes #255
2022-06-27 13:01:20 -07:00
sua yoo
c2aa4e6319
Fix AM/PM toggle (#272) 2022-06-23 16:35:47 -07:00
sua yoo
c2be1a27ce
Handle stopping state in UI (#269)
closes #262
2022-06-23 16:35:03 -07:00
sua yoo
b40765134c
Re-run crawl from crawls list view (#264)
* run crawl from crawls list, and show link to newly started crawl
* if crawl is already running, show link to previously running crawl
2022-06-15 18:54:57 -07:00
sua yoo
a8757e2e50
Screencast UX enhancements (#251)
* animate starting state
* consistent fixed-size slots for each browser (url + screencast)
* add tooltip for expected number of browsers (workers x scale)
2022-06-15 18:50:14 -07:00
Ilya Kreymer
9fce8cfc1d frontend: fix missed renames 2022-06-11 12:37:24 -07:00
sua yoo
710639365b
adjust no files message (#250)
Change 'no files yet' -> 'no files to replay' when there are no files available for replay.
2022-06-07 22:59:34 -07:00