tea/browsertrix
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
761 Commits 2 Branches 0 Tags 15 MiB
e7f2d93f80
Commit Graph

337 Commits

Author SHA1 Message Date
sua yoo
52e6b6199a
fix: prevent login event on storage event 2022-11-21 15:51:47 -08:00
sua yoo
321f78b861
Upgrade Shoelace 2.0.0-beta.61 -> 2.0.0-beta.83 (#358) 2022-11-21 08:16:51 -08:00
sua yoo
13d5ab7d2b
Fix exclusion dropdown & removal in crawl config edit (#372) 
* fix dialog closing on regex

* fix unable to delete regex with same text
 2022-11-18 14:47:40 -08:00
sua yoo
4d4ce40443
Refactor & sync user session across tab/windows (#370) 2022-11-15 19:49:18 -08:00
sua yoo
40054d1501
Update global footer (#354) 2022-11-15 14:15:04 -08:00
sua yoo
1ef9f7df6d
Fix auth not persisting on reload (#360) 2022-11-15 13:17:29 -08:00
sua yoo
8c4348b9f8
Show exclusion editor when creating & editing crawl templates (#353) 2022-11-14 19:34:15 -08:00
sua yoo
d41b582ef6
Remove exclusion from running crawl (#352) 2022-11-14 10:58:33 -08:00
sua yoo
95ec1599ef
Add exclusion to running crawl (#347) 2022-11-08 18:09:11 -06:00
sua yoo
baacbbdc52
Highlight regular expression syntax in Exclusions Table (#341) 2022-11-01 15:31:01 -07:00
sua yoo
97eb17784d
Display exclusions & list of URLs in crawl queue (#337) 
- including pagination of queue results (30 results per page currently)
- show numbering on paginated results
- allow user navigation to each result page
 2022-10-12 20:19:13 -07:00
sua yoo
8708c24a74
Improve crawl elapsed time UX (#323) 
Smoother elapsed crawl timer:
- Crawls list: show seconds increment up to 2 minutes, then show minutes only
- Crawls detail: show seconds increment up to one day
 2022-10-05 21:12:31 -04:00
sua yoo
0bbb7905bd
Add crawl queue editor UI components (#331) 
WIP #304
 2022-10-04 13:13:40 -07:00
sua yoo
e696104ffa
Update crawl template copy (#325) 2022-09-27 19:49:24 -07:00
sua yoo
709936dfa7
hotfix: decrease size of running crawl action button 2022-09-27 19:09:49 -07:00
sua yoo
63ada3e5b3
Update base fonts and text sizes (#327) 2022-09-27 14:32:57 -07:00
sua yoo
94e3dff27f
update sentry CDN script 2022-09-27 12:29:02 -07:00
sua yoo
20bd8ceecb
Fix browser profile table alignment (#322) 2022-09-26 17:14:08 -07:00
sua yoo
2ebd1eb2f6
Continue to watch crawl while stopping (#316) 
* show when running

* redirect after done

* show banner that crawl is stopping
 2022-09-21 12:39:00 -07:00
sua yoo
319a8a3c07
make clearer that profile selection is optional and that a default profile is used by default (#290) 
- Rename 'Select Profile' -> 'Default Profile'
- Rename 'No Profiles' -> 'No Additional Profiles'
 2022-08-10 15:54:39 -07:00
sua yoo
ee6161ad43
Frontend browser profile editor enhancements (#288) 
- add button to duplicate profile from main view
- add save / cancel button when editing
- change location of 'full screen' button
 2022-08-10 15:51:34 -07:00
sua yoo
9606d59c3d
Improve format of crawl template config error from server (#281) 
* better display of api errors, such as fields missing or invalid urls, addresses #280
 2022-06-29 17:57:03 -07:00
sua yoo
301b05ff4e
Refactor screencast websocket connection and retry (#276) 
* replace ip with index and retry connection, fixes #252
 2022-06-29 17:55:32 -07:00
Ilya Kreymer
2717a60763
improvements / bug fixes for stop/cancel handling: (#279) 
- only send signal if stopping, no need for canceling as pods/containers will be removed
- refactor stop/cancel handling to be unified in manager, separate in job
- when stopping / graceful shutdown, return false if sending signal fails
- return success=true in json response if and only if stop/cancel actually succeeds, return 'error' message in error, should fix #270
- allow canceling after stopping / if stopping fails
- ensure finished time is set in case of cancelation before crawl starts, should fix #273
 2022-06-29 17:47:25 -07:00
sua yoo
1c52902ea0
Update crawl scale label for UI consistency (#275) 
closes #254
 2022-06-29 16:14:03 -07:00
sua yoo
92292591ad
Re-run crawl from detail view + handle inactive crawl template (#268) 
closes #253
 2022-06-29 14:17:09 -07:00
sua yoo
d144591dbf
Display & edit crawl schedule in user local time (#271) 
closes #255
 2022-06-27 13:01:20 -07:00
sua yoo
c2aa4e6319
Fix AM/PM toggle (#272) 2022-06-23 16:35:47 -07:00
sua yoo
c2be1a27ce
Handle stopping state in UI (#269) 
closes #262
 2022-06-23 16:35:03 -07:00
sua yoo
b40765134c
Re-run crawl from crawls list view (#264) 
* run crawl from crawls list, and show link to newly started crawl
* if crawl is already running, show link to previously running crawl
 2022-06-15 18:54:57 -07:00
sua yoo
a8757e2e50
Screencast UX enhancements (#251) 
* animate starting state
* consistent fixed-size slots for each browser (url + screencast)
* add tooltip for expected number of browsers (workers x scale)
 2022-06-15 18:50:14 -07:00
Ilya Kreymer
9fce8cfc1d frontend: fix missed renames 2022-06-11 12:37:24 -07:00
sua yoo
710639365b
adjust no files message (#250) 
Change 'no files yet' -> 'no files to replay' when there are no files available for replay.
 2022-06-07 22:59:34 -07:00
sua yoo
fa4b71288c
Fix watch crawl running state (#249) 2022-06-07 12:04:35 -07:00
sua yoo
502d687620
Enable duplicating and editing browser profile (#237) 
* ensure editing other config options does not lose profile
* support adding/editing/removing profile of existing config
* when duplicating config, ensure profile setting is also copied in the duplicate
 2022-06-04 08:26:19 -07:00
sua yoo
0c1dc2a1d1
Show crawl replay for running crawls (#235) 
* show replay and watch at same time

* add separate section for watch

* only show replay if crawl has files, otherwise show 'no files' message
 2022-06-04 08:19:09 -07:00
sua yoo
6a78bcd4aa
Delete browser profile (#243) 
- delete browser profile, if not in use
- if in use, show error message, listing crawl configs that use the profile
- backend: fix check for confirming profile deletion
 2022-06-01 19:18:41 -07:00
sua yoo
9cf1ed7d4d
copy yaml (#239) 2022-06-01 19:06:52 -07:00
sua yoo
2355de3067
docs: remove extra comment 2022-05-31 14:13:17 -07:00
sua yoo
6e19e854be
Fix "Run now" button (#234) 2022-05-30 16:15:10 -07:00
Ilya Kreymer
955197579e frontend: support multi wacz replay using the crawl json as input 2022-05-20 09:11:23 -07:00
sua yoo
bda817dadd
View and edit browser profile (#218) 2022-04-23 20:12:16 -07:00
sua yoo
f157e2031f
Filter and sort crawl templates (#217) 2022-04-23 20:11:53 -07:00
sua yoo
cb80c6767e
hotfix: update profile ID in crawl template 2022-04-20 19:40:30 -07:00
Ilya Kreymer
38869cdd24
crawl templates: check that lastCrawlState is not null (#220) 2022-04-20 19:17:24 -07:00
sua yoo
db27b6aaaf
View and edit browser profile (#214) 2022-04-19 10:44:21 -07:00
sua yoo
71eec4d915
Create crawl template with browser profile (#215) 2022-04-18 10:36:28 -07:00
Ilya Kreymer
73b8c64ba4 frontend profile browser: cover devtools sidebar with profile sidebar, add try/catch for localStorage override 2022-04-13 21:41:51 -07:00
sua yoo
f5993e8ad8
Create browser profile UI (#211) 2022-04-13 21:11:13 -07:00
sua yoo
d2653ae835
View browser profiles in UI (#209) 2022-04-13 21:10:22 -07:00
sua yoo
238ee8f7ee
delete unused component file 2022-04-11 13:18:23 -07:00
sua yoo
8828681e8e
hotfix: fix crawl sort control alignment 2022-04-11 13:13:53 -07:00
sua yoo
d4b3ae3795
delete unused component file 2022-04-11 13:10:23 -07:00
sua yoo
5307138202
enable opening crawl template in new tab 2022-04-11 13:03:19 -07:00
sua yoo
f90ef071de
enable opening crawl in new tab 2022-04-11 13:03:10 -07:00
sua yoo
29b586b03f
Edit crawl config as YAML (#207) 2022-04-06 17:40:25 -07:00
Ilya Kreymer
9a6483630e
Support for Admin interface for viewing web archives (#198) 
* backend api
- superadmin has admin access to all archives
- new superadmin endpoints: /archives/all/crawls and /archives/all/crawls/<crawl_id>.json for list all running crawls
and loading crawl data by id

- frontend superadmin view (fixes #201)
* show all archives on superadmin home page
* show jump to crawl for super admin (#200)
* navbar links for: all archives, all running crawls and jump to crawl

Co-authored-by: sua yoo <sua@suayoo.com>
 2022-04-06 12:42:04 -07:00
sua yoo
ec3a77b71e
Mobile layout fixes (#206) 
closes #202
 2022-03-30 15:54:25 -07:00
sua yoo
9e2274f612
remove temp file 2022-03-30 13:51:02 -07:00
Ilya Kreymer
9e45dc35d2
minor frontend-tweaks: (#196) 
* frontend-tweaks:
- treat 'starting' state same as 'running'
- default to no schedule instead of weekly for default
- add 'Domain' scopeType

* backend: also allow 'domain' as a scopeType
 2022-03-15 21:19:23 -07:00
sua yoo
8863776c54
Define websocket host in common webpack config (#195) 
* move websocket host var to common config, better fix for #193
 2022-03-15 18:34:49 -07:00
Ilya Kreymer
912004751d quickfix: partial mitigation for #193, use current host for websock address 2022-03-14 15:29:35 -07:00
sua yoo
6fabea3e7a
Frontend build fixes (#191) 
* copy specific files
* replace api host env var
* remove unused dotenv
* Update frontend/webpack.dev.js
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
 2022-03-10 23:26:21 -08:00
sua yoo
4190e40964
Show last crawl state in UI (#192) 
* update crawl list status

* show on detail page
 2022-03-10 23:25:42 -08:00
sua yoo
edf6b9ded7
Update home page routing (#186) 
closes #183
 2022-03-04 16:18:41 -08:00
sua yoo
0fe54653be
Fix unable to save edits to simple view (#185) 2022-03-04 16:17:57 -08:00
sua yoo
f2f67c34af
Copy extra hops value when duplicating crawl config (#184) 
closes #158
 2022-03-04 16:17:37 -08:00
sua yoo
4383c5e8d8
Disable error tracking in prod (#182) 
closes #161
 2022-03-04 16:17:05 -08:00
sua yoo
c18418ff09
Show invite message to super admin & layout fixes (#181) 2022-03-02 18:09:26 -08:00
sua yoo
fe31f551b2
Add "crawler" role to members (#174) 
closes #139
 2022-03-02 18:09:10 -08:00
sua yoo
c888a45d97
Fix seed URLs reset on JSON view toggle (#172) 
closes #160
 2022-03-02 18:08:45 -08:00
sua yoo
373c489b00
Watch crawl from crawl detail page (#156) 
closes #164
closes #134 

Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
 2022-03-02 18:08:08 -08:00
sua yoo
83ded98081
Set and update crawl scale (#162) 
closes #143
 2022-02-28 09:14:27 -08:00
sua yoo
3fe3691e74
Update crawl run duration at intervals (#155) 
fixes #138
 2022-02-23 16:14:01 -08:00
sua yoo
4af30a02be
Archive and crawl navigation improvements (#154) 2022-02-23 09:19:48 -08:00
sua yoo
b5874c3f8c
call super disconnected callback after custom callback 2022-02-22 15:59:55 -08:00
sua yoo
c563216582
Allow user to edit crawl template (#147) 
closes #144
 2022-02-22 13:54:25 -08:00
sua yoo
f30b398fea
Deactivate crawl templates in UI (#145) 
wip #144
 2022-02-21 11:37:15 -08:00
sua yoo
aa645d9b15
Enable frontend exception tracking (#140) 2022-02-18 10:34:07 -08:00
Ilya Kreymer
e9d6c68f6a frontend: replay: use single wacz replay for now (using first wacz file) 2022-02-15 08:34:14 -08:00
sua yoo
c577e36b74
add debug for access token 2022-02-08 17:52:27 -08:00
sua yoo
02f46f108b
Crawl & crawl config UX improvements (#136) 2022-02-01 14:28:07 -08:00
sua yoo
d7f58c964c
Fix in-app link UX (#132) 
closes #130, closes #113
 2022-01-31 17:36:50 -08:00
Ilya Kreymer
adb5c835f2
Presign and replay (#127) 
* support for replay via replayweb.page embed, fixes #124

backend:
- pre-sign all files urls
- cache pre-signed urls in redis, presign again when expired (default duration 3600, settable via PRESIGN_DURATION_SECONDS env var)
- change files output -> resources to confirm to Data Package spec supported by replayweb.page
- add CrawlFileOut which contains 'name' (file id), 'path' (presigned url), 'hash', and 'size'
- add /replay/sw.js endpoint to import sw.js from latest replay-web-page release
- update to fastapi-users 9.2.2
- customize backend auth to allow authentication to check 'auth_bearer' query arg if 'Authorization' header not set
- remove sw.js endpoint, handling in frontend

frontend:
- add <replay-web-page> to frontend, include rwp ui.js from latest release in index.html for now
- update crawl api endpoint to end in json
- replay-web-page loads the api endpoint directly!
- update Crawl type to use new format, 'resources' -> instead of 'files', each file has 'name' and 'path'

- nginx: add endpoint to serve the replay sw.js endpoint
- add defer attr to ui.js
- move 'Download' to 'Download Files'

* frontend: support customizing replayweb.page loading url via RWP_BASE_URL env var in Dockerfile
- default prod value set in frontend Dockerfile (set to upcoming 1.5.8 release needed for multi-wacz-file support) (can be overridden during image build via --build-arg)
- rename index.html -> index.ejs to allow interpolation
- RWP_BASE_URL defaults to latest https://replayweb.page/ for testing
- for local testing, add sw.js loading via devServer, also using RWP_BASE_URL (#131)

Co-authored-by: sua yoo <sua@suayoo.com>
 2022-01-31 17:02:15 -08:00
sua yoo
336cf11521
Fix "View crawl" links (#129) 
* update key

* update in crawl config
 2022-01-31 15:45:48 -08:00
sua yoo
d7c0877403
Refactor archive tabs & navigation improvements (#123) 
closes #112
 2022-01-31 15:45:36 -08:00
sua yoo
9de1a3a003
fix stopping gracefully feedback 2022-01-31 12:02:10 -08:00
sua yoo
be4bf3742f
Initial crawl detail page (#108) 2022-01-30 18:36:43 -08:00
sua yoo
7c067ffe36
Crawl template enhancements (#114) 
closes #100
 2022-01-30 18:30:54 -08:00
sua yoo
b93ca4e833
Add empty state for crawls (#121) 2022-01-29 15:55:44 -08:00
sua yoo
7777a22829
Poll crawls list & add additional details (#116) 2022-01-29 14:37:16 -08:00
sua yoo
2636f33123
Make crawl list interactive (#109) 
- Cancel and stop crawl
- Sorts crawls by start time, status and crawl template ID
- Filters crawls by crawl template ID
- Adds shortcut to copy template ID
 2022-01-29 10:38:58 -08:00
sua yoo
28b59130a6
Initial list of crawls (#105) 2022-01-27 19:28:19 -08:00
sua yoo
5fccd07329
Edit crawl schedule (#103) 2022-01-26 22:11:32 -08:00
Ilya Kreymer
0bea0cfff2
crawl config new template: add support for 'extraHops' config option (available in browsertrix-crawler 0.5.0) (#104) 
frontend:
- add checkbox to basic crawl config component which sets 'extraHops' to 1, otherwise to 0
- text tweaks: rename Scope Type -> Crawl Scope, capitalization

backend: add 'extraHops' to CrawlConfig
fixes #102
 2022-01-26 21:18:22 -08:00
sua yoo
2666b6f6aa
Duplicate crawl config from list (#99) 2022-01-25 17:07:54 -08:00
sua yoo
3a461d86d4
Crawl config detail views (#97) 2022-01-25 11:56:34 -08:00
sua yoo
9ed216ba05
Run and delete crawl templates from list view (#94) 2022-01-22 14:18:02 -08:00
sua yoo
cb5cf55c69
Add helper for dispatching notify events (#92) 2022-01-19 21:01:47 -08:00
sua yoo
22942527e9
Refactor crawl config into multiple components (#91) 2022-01-19 18:43:19 -08:00
sua yoo
3645e3b096
Create crawl config UX enhancements (#90) 
closes #87
 2022-01-19 11:01:17 -08:00
sua yoo
c3edb4bba4
Allow user to configure crawls with JSON (#86) 2022-01-18 19:58:55 -08:00
sua yoo
ff77a92108
Schedule time of day when creating config (#85) 2022-01-18 13:58:28 -08:00
sua yoo
b2088f5634
Add initial crawl template form (#80) 2022-01-16 14:43:33 -08:00
sua yoo
b9622df9e6
Update app layout (#78) 
closes #75
 2022-01-13 17:52:59 -08:00
sua yoo
5492eca117
Fix login page update side effect (#77) 
closes #76
 2022-01-12 16:43:34 -08:00
sua yoo
5258149a34
Fix password autocomplete and suggestions (#73) 
closes #72
 2022-01-08 14:28:54 -08:00
sua yoo
2c3debfec6
Show invite info to user (#68) 2021-12-07 09:36:03 -08:00
sua yoo
3324bd960f
Refactor to remove sign up and JWT env variables (#65) 
closes #63
closes #66
 2021-12-06 19:39:04 -08:00
sua yoo
e787e751d9
update alert import (#64) 2021-12-06 18:49:42 -08:00
sua yoo
ddf1b63313
fix invite token logic 2021-12-05 19:01:21 -08:00
Ilya Kreymer
efaaece842
Optimizing Frontend Dockerfile (#60) 
* optimizing frontend dockerfile:
- run install first to cache node_modules
- don't pass node_modules to image
- add only needed files before build

* remove language file generation from build step

Co-authored-by: sua yoo <sua@suayoo.com>
 2021-12-05 18:39:45 -08:00
sua yoo
d5c665be8e
Fix archive invite flow (#61) 
closes #49
 2021-12-05 18:23:24 -08:00
sua yoo
ba69ac02bc
Update superadmin invite URL (#62) 2021-12-05 18:22:59 -08:00
sua yoo
5531c3c33c
hotfix: fix auth persist 
Switch lifecycle event to update
 2021-12-05 16:52:58 -08:00
sua yoo
9c4bec1411
Add admin page for inviting users (#56) 
closes #37
 2021-12-05 15:23:28 -08:00
sua yoo
9673794711
Frontend prod build optimizations (#54) 
- Leverage webpack chunk splitting to creating more, smaller JS files rather than one large main file (import(file) syntax)
- Enable long-term caching by adding content hash to output file names
- Copy entire /dist folder contents in Dockerfile
- Changed yarn start-dev -> yarn start since there is no prod server
- Reenable locale picker
 2021-12-03 18:00:14 -08:00
sua yoo
1d114c682e
hotfix: prevent session from expiring 2021-12-03 17:40:03 -08:00
sua yoo
87bef6d967
Use settings to determine if sign up route is enabled (#51) 2021-12-03 10:05:55 -08:00
sua yoo
30e40adddd
Refresh access token in background (#48) 
closes #22
 2021-12-03 10:00:13 -08:00
sua yoo
c48e870ffe
Refactor frontend auth service (#47) 2021-12-02 16:31:12 -08:00
sua yoo
2396ab9851
Invite user to an archive (#36) 
known issue: invalid invitation error shows even when invite is successfully accepted

closes #31
 2021-12-02 11:31:03 -08:00
sua yoo
9d7596589b
Show sign up confirmation message (#42) 
closes #41
 2021-12-02 10:28:22 -08:00
sua yoo
8bcdc8877f
Frontend verification UX fixes (#40) 
- Show toast alert when user is verified
- Redirect to correct page on verified
- Update already-logged in user info on verify
- Adds new toast component

closes #39
 2021-12-01 11:56:09 -08:00
sua yoo
6e193b1157
redirect to log in on reset password 2021-12-01 11:04:38 -08:00
sua yoo
3fa85c83f2
Allow users to sign up through UI (#30) 
closes #28
 2021-11-30 08:57:53 -08:00
sua yoo
50e93724fc
Allow users to reset password from UI (#27) 
closes #21
 2021-11-30 08:53:39 -08:00
sua yoo
911dc64025
Enable query param routing (#29) 
* enable params in url

* rename view state
 2021-11-25 12:00:24 -08:00
sua yoo
58eba70c68
Allow users to change password (#25) 
wip #21
 2021-11-23 17:01:08 -08:00
sua yoo
04fbe6fc4d
Adjust frontend auth behavior (#24) 2021-11-23 16:57:28 -08:00
sua yoo
5b8440f295
Fix frontend test mocks (#23) 2021-11-22 16:31:35 -08:00
sua yoo
5722909157
Frontend responsive UI tweaks (#20) 2021-11-22 10:25:34 -08:00
sua yoo
14f2d13a73
Add frontend localization support (#18) 
closes #17
 2021-11-20 07:44:21 -08:00
sua yoo
76e5ceb864
Replace daisy UI with shoelace (#16) 
closes #15
 2021-11-19 19:38:58 -08:00
sua yoo
316a91f612
Switch frontend to use Typescript (#12) 
closes #11
 2021-11-19 14:07:13 -08:00
sua yoo
0f97724ad0
Set up frontend dev tooling (#6) 
* Add eslint for linting js (https://github.com/ikreymer/browsertrix-cloud/pull/7)
* Add prettier for formatting frontend files (https://github.com/ikreymer/browsertrix-cloud/pull/9)
* Add frontend testing framework (https://github.com/ikreymer/browsertrix-cloud/pull/10)

closes #4, closes #5
 2021-11-18 17:26:10 -08:00
Ilya Kreymer
666becdb65 initial pass on frontend: using tailwindcss + daisyui + litelement with webpack build + dev server 2021-10-10 12:22:36 -07:00