Commit Graph

179 Commits

Author SHA1 Message Date
sua yoo
28346e0a54
New create crawl config user workflow (#391) 2022-12-12 13:50:33 -08:00
Ilya Kreymer
2d93cef966
CI: Add K3D CI test (#405)
- add testing with K3D cluster
- bump backend image to python 3.10-slim for newer python, smaller image.
- bump to 1.2.0-beta.0
2022-12-07 23:26:16 -08:00
Ilya Kreymer
0aa09be8c3
README + CHANGES + doc tweaks for 1.1.0 release (#402)
- update README + docs with deprecation of non-k8s deployment
- add CHANGES.md
- bump version to 1.1.0
2022-12-06 12:27:27 -08:00
Ilya Kreymer
ba3e7d048f
build: increase network timeout for yarn for frontend build for arm64 build (#401)
- build: increase network timeout for frontend build for arm64 build (see nodejs/docker-node#1335)
- also bump microk8s action version to fix warning
2022-12-06 10:19:27 -08:00
Ilya Kreymer
829548af0f doc tweaks:
- fix typos in docs
- update prod deployment info
- update minikube info
- add info on how to run with local images
- bump version to 1.1.0-beta.3 for testing multiarch build
2022-12-05 18:14:19 -08:00
Henry Wilkinson
a74d88dcda
mkdocs setup (deploy, dev, user-guide) (#375)
* Initial docs move
* Setup mkdocs
* Adds instructions for building docs
* add new deployment docs, local and prod
* set up three sections: deployment, dev and user guide
* remove old deployment docs
* ci: mkdocs gh-pages publish

Co-authored-by: sua yoo <sua@suayoo.com>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
2022-12-05 16:41:37 -08:00
Ilya Kreymer
82ffc0dfbc
Local Deployment Work: Support running locally + test cluster on CI (#396)
* k8s local deployment work:
- make it easier to deploy w/o ingress by setting 'local_service_port' (suggested port 30870)
- if using local minio, ensure file endpoints set to /data/ and /data/ proxies correctly to local bucket
- if not using minio, ensure file endpoints point to correct access / endpoint url.
- setup should work with docker desktop, minikube, microk8s and k3s!
- nginx chart: bump nginx memory limit to 20Mi
- nginx image: 00-default-override-resolver-config -> 00-browsertrix-nginx-init for clarity
- nginx image: use default nginx.conf, pin to nginx 1.23.2
- mongo: readd readiness probe, bump connect wait timeout (needed for ci)
- config: set superadmin username to 'admin'
- config schema: set 'name' as required 
- add sample chart values overrides:
- chart values: local-config.yaml for running locally with 'local_service_port'
- chart values: add microk8s-hosted.yaml for configuring a hosted microk8s setup
- chart values: add microk8s-ci.yaml for ci tests
- ci: remove docker swarm tests
- ci: add microk8s integration tests: launching cluster, logging in, running a crawl of example.com, downloading/checking WACZ
- bump to 1.1.0-beta.2
2022-12-02 19:58:34 -08:00
sua yoo
5d18ffa938
Fix language configuration UI (#388) 2022-12-01 10:02:13 -08:00
Ilya Kreymer
afe536e568 version: bump to 1.1.0-beta.1 2022-11-23 23:37:33 -08:00
sua yoo
e7f1a00411
Fix authentication getting out of sync between tabs (#380)
Fixes regression to #361 found after increasing the token timeout by preventing app load until the authentication service is initialized (and finishing check if another tab is logged in.)
2022-11-23 23:36:36 -08:00
sua yoo
da8260a028
Make exclusion table cells editable (#379) 2022-11-23 09:43:52 -08:00
sua yoo
3ad47a4c8c
Frontend Node version support (#382) 2022-11-23 09:05:02 -08:00
sua yoo
003b3c7a78
Allow users to set crawl config language (#377) 2022-11-22 15:15:35 -08:00
sua yoo
52e6b6199a
fix: prevent login event on storage event 2022-11-21 15:51:47 -08:00
sua yoo
77fa6ae691
chore: update license to valid expression 2022-11-21 09:19:11 -08:00
sua yoo
321f78b861
Upgrade Shoelace 2.0.0-beta.61 -> 2.0.0-beta.83 (#358) 2022-11-21 08:16:51 -08:00
Ilya Kreymer
d6386b7051
Release Build + Versioning (#373)
- Adds version to version.txt in root
- adds update-version.sh which updates version in frontend/package.json and backend/btrixcloud/version.py
- frontend: loads version from $VERSION env var, ../version.txt or package.json
- ci: on new github release, pushes webrecorder/browsertrix-backend and webrecorder/browsertrix-frontend images to Dockerhub with current version, as well as latest.
- version set to 1.1.0-beta.0
- closes #357
2022-11-18 17:15:25 -08:00
sua yoo
13d5ab7d2b
Fix exclusion dropdown & removal in crawl config edit (#372)
* fix dialog closing on regex

* fix unable to delete regex with same text
2022-11-18 14:47:40 -08:00
sua yoo
4d4ce40443
Refactor & sync user session across tab/windows (#370) 2022-11-15 19:49:18 -08:00
sua yoo
40054d1501
Update global footer (#354) 2022-11-15 14:15:04 -08:00
sua yoo
1ef9f7df6d
Fix auth not persisting on reload (#360) 2022-11-15 13:17:29 -08:00
sua yoo
8c4348b9f8
Show exclusion editor when creating & editing crawl templates (#353) 2022-11-14 19:34:15 -08:00
sua yoo
d41b582ef6
Remove exclusion from running crawl (#352) 2022-11-14 10:58:33 -08:00
sua yoo
95ec1599ef
Add exclusion to running crawl (#347) 2022-11-08 18:09:11 -06:00
sua yoo
baacbbdc52
Highlight regular expression syntax in Exclusions Table (#341) 2022-11-01 15:31:01 -07:00
sua yoo
97eb17784d
Display exclusions & list of URLs in crawl queue (#337)
- including pagination of queue results (30 results per page currently)
- show numbering on paginated results
- allow user navigation to each result page
2022-10-12 20:19:13 -07:00
sua yoo
8708c24a74
Improve crawl elapsed time UX (#323)
Smoother elapsed crawl timer:
- Crawls list: show seconds increment up to 2 minutes, then show minutes only
- Crawls detail: show seconds increment up to one day
2022-10-05 21:12:31 -04:00
sua yoo
2bfbeab55f
build: copy ts declaration file 2022-10-04 13:38:12 -07:00
sua yoo
0bbb7905bd
Add crawl queue editor UI components (#331)
WIP #304
2022-10-04 13:13:40 -07:00
sua yoo
e696104ffa
Update crawl template copy (#325) 2022-09-27 19:49:24 -07:00
sua yoo
709936dfa7
hotfix: decrease size of running crawl action button 2022-09-27 19:09:49 -07:00
sua yoo
63ada3e5b3
Update base fonts and text sizes (#327) 2022-09-27 14:32:57 -07:00
sua yoo
94e3dff27f
update sentry CDN script 2022-09-27 12:29:02 -07:00
sua yoo
20bd8ceecb
Fix browser profile table alignment (#322) 2022-09-26 17:14:08 -07:00
sua yoo
2ebd1eb2f6
Continue to watch crawl while stopping (#316)
* show when running

* redirect after done

* show banner that crawl is stopping
2022-09-21 12:39:00 -07:00
sua yoo
319a8a3c07
make clearer that profile selection is optional and that a default profile is used by default (#290)
- Rename 'Select Profile' -> 'Default Profile'
- Rename 'No Profiles' -> 'No Additional Profiles'
2022-08-10 15:54:39 -07:00
sua yoo
ee6161ad43
Frontend browser profile editor enhancements (#288)
- add button to duplicate profile from main view
- add save / cancel button when editing
- change location of 'full screen' button
2022-08-10 15:51:34 -07:00
sua yoo
9606d59c3d
Improve format of crawl template config error from server (#281)
* better display of api errors, such as fields missing or invalid urls, addresses #280
2022-06-29 17:57:03 -07:00
sua yoo
301b05ff4e
Refactor screencast websocket connection and retry (#276)
* replace ip with index and retry connection, fixes #252
2022-06-29 17:55:32 -07:00
Ilya Kreymer
2717a60763
improvements / bug fixes for stop/cancel handling: (#279)
- only send signal if stopping, no need for canceling as pods/containers will be removed
- refactor stop/cancel handling to be unified in manager, separate in job
- when stopping / graceful shutdown, return false if sending signal fails
- return success=true in json response if and only if stop/cancel actually succeeds, return 'error' message in error, should fix #270
- allow canceling after stopping / if stopping fails
- ensure finished time is set in case of cancelation before crawl starts, should fix #273
2022-06-29 17:47:25 -07:00
sua yoo
1c52902ea0
Update crawl scale label for UI consistency (#275)
closes #254
2022-06-29 16:14:03 -07:00
Ilya Kreymer
3fec2a9f82
dev server: also proxy /data directory for testing replay from a remote instance locally (#266)
API_BASE_URL will need to be set to 'http://remote.example.com/' instead of 'http://remote.example.com/api/'
2022-06-29 15:47:20 -07:00
sua yoo
92292591ad
Re-run crawl from detail view + handle inactive crawl template (#268)
closes #253
2022-06-29 14:17:09 -07:00
sua yoo
d144591dbf
Display & edit crawl schedule in user local time (#271)
closes #255
2022-06-27 13:01:20 -07:00
sua yoo
c2aa4e6319
Fix AM/PM toggle (#272) 2022-06-23 16:35:47 -07:00
sua yoo
c2be1a27ce
Handle stopping state in UI (#269)
closes #262
2022-06-23 16:35:03 -07:00
raffaele messuti
70767f0ac2
small fixes on docker swarm deployment (#265)
* fix COPY with multiple files

* Update Deployment.md
2022-06-16 19:56:40 -07:00
sua yoo
b40765134c
Re-run crawl from crawls list view (#264)
* run crawl from crawls list, and show link to newly started crawl
* if crawl is already running, show link to previously running crawl
2022-06-15 18:54:57 -07:00
sua yoo
a8757e2e50
Screencast UX enhancements (#251)
* animate starting state
* consistent fixed-size slots for each browser (url + screencast)
* add tooltip for expected number of browsers (workers x scale)
2022-06-15 18:50:14 -07:00
Ilya Kreymer
418c07bf0d
Local swarm + podman support (#261)
* backend: refactor swarm support to also support podman (#260)
- implement podman support as subclass of swarm deployment
- podman is used when 'RUNTIME=podman' env var is set
- podman socket is mapped instead of docker socket
- podman-compose is used instead of docker-compose (though docker-compose works with podman, it does not support secrets, but podman-compose does)
- separate cli utils into SwarmRunner and PodmanRunner which extends it
- using config.yaml and config.env, both copied from sample versions
- work on simplifying config: add docker-compose.podman.yml and docker-compose.swarm.yml and signing and debug configs in ./configs
- add {build,run,stop}-{swarm,podman}.sh in scripts dir
- add init-configs, only copy if configs don't exist
- build local image use current version of podman, to support both podman 3.x and 4.x
- additional fixes for after testing podman on centos
- docs: update Deployment.md to cover swarm, podman, k8s deployment
2022-06-14 00:13:49 -07:00