Commit Graph

414 Commits

Author SHA1 Message Date
sua yoo
6044486190
Add button to download error logs (#1080)
* add button to download logs

* render if logs are present

* add icon
2023-08-15 21:14:32 -07:00
sua yoo
270e134359
Show details in crawl error log (#1079)
Shows crawl error log details in a dialog. Since the detail object does not always follow a specific format, this iteration uses the detail key in uppercase as the label.
2023-08-15 21:14:08 -07:00
Ilya Kreymer
768d1181f8
frontend: fixes for queue / exclusions: (#1076)
- fix 'Edit Crawler Instances' not showing up when crawl running
- urlencode regex params to properly encode '+'
- catch server-side regex error, display 'Invalid Regex'
2023-08-15 13:15:43 -07:00
sua yoo
4c74fadf91
Update frontend local dev guide (#1073)
- Clarifies use case for frontend development server
- Fixes incorrect sample API URLs
- Adds additional detail around requirements and quickstart
- Links back to docs from frontend README
---------

Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
Co-authored-by: Ilya Kreymer <ikreymer@users.noreply.github.com>
2023-08-15 12:03:39 -07:00
sua yoo
89983542f9
Update archived item URLs (#1064)
- Changes to URLs in "Crawling", "All Archived Items", and "Collections":
- Rename Artifacts -> Items
- Unifies view crawl view as loaded from All Archived Items and from Workflows
- Includes redirect for /artifacts/uploads -> /items/uploads to support archiveweb.page usage
2023-08-14 18:28:37 -07:00
sua yoo
ffd0e525d9
Webpack config improvements (#1063)
- Upgrades webpack and webpack-dev-server for bugfixes and performance updates
- Removes unnecessary file watching
- Enables persistent build cache in dev
- Switches to faster dev source map
2023-08-11 13:16:24 -07:00
Ilya Kreymer
d93ddaf620 bump version to 1.6.1 2023-08-11 12:50:41 -07:00
Ilya Kreymer
35ab6d6df6 bump to 1.6.0! 2023-08-09 15:40:27 -07:00
Ilya Kreymer
8ea3dd5dae
terminology tweaks in frontend: (part of #922) (#1062)
* terminology tweaks in frontend: (part of #922)
- use 'crawl workflow' instead of 'workflow' where possible
- use 'replay' instead of 'replay crawl'
- localization: rerun string extraction / processing
- "Review Config" → "Review Settings"
- "Workflow" → "Crawl Workflow" in error message

---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
2023-08-09 15:38:58 -07:00
sua yoo
37733483d5
Standardize archived item filtering, sorting and labels (#1054)
Frontend:
- Renames list view to "All Archived Items"
- Refactors fetches to use single all-crawls endpoints
- Removes search by config ID for more search parity with uploads
- Adds sort by size
- Refactors property and method names to replace crawl*
- Replaces remaining references to "crawl" in copy with "item"'
- Rename Upload Archive button to Upload WACZ
- Fix focusout in item menu so menus close

Backend:
- Filter search values by type as well
- Only get list of cids for crawls in search values
- Don't list crawl/workflow ids in search values

---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
2023-08-09 12:13:55 -07:00
Ilya Kreymer
7a8f370bc2 bump version to 1.6.0-beta.4 for testing 2023-08-09 12:09:37 -07:00
Ilya Kreymer
38f67a6cc0
Optimize Frontend Image Build on CI (#1057)
* Always run yarn only on build platform with --platform=$BUILDPLATFORM
* Remove optional dependencies (playwright + chromium) from build with --ignore-optional and move some devDependencies to be optional
* Disable husky pre-commit hook checks on frontend

Co-authored-by: sua yoo <sua@suayoo.com>
2023-08-09 12:06:20 -07:00
sua yoo
b494070e43
Collection share dialog + copy updates (#1056)
- Always shows primary "Share" action button in Collection detail page.
- Enables toggling shareable status and share info from dialog. Difference from mockups: I made the "Done" button neutral do differentiate from our submit action buttons in the dialog, since toggling will apply changes immediately.
- Menu item: "Go to Public View"/"Go to Shareable View" -> "Visit Shareable URL". 
- Toggle label: "Make Collection Shareable" -> "Collection is Shareable".
- Additional dialog copy: adds "This collection can be viewed by anyone with the link." under "Link to Share" and "Share this collection by embedding it into an existing webpage." under "Embed Collection".
- Moves share status icon to its own column in list view.
- Adds new syntax-highlighted code component that supports js and html.

Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
2023-08-09 10:12:46 -07:00
Anish Lakhwara
9236a07800
fix: run yarn format in frontend dir (#1043) 2023-08-03 19:12:48 -07:00
Ilya Kreymer
362afa47bd
Support for Public / Shareable Collections (#1038)
* collections: support toggling collections public/private, viewable via RWP
- backend: add 'public' to collection model, support patching to update
- backend: add .../collections/<id>/public/replay.json for public access
- backend: add CORS handling for public endpoint
- frontend: support 'make shareable / make private' dropdown actions on collection detail + collection list views
- frontend: show shareable / private icons by collection name on detail + list views
- frontend: link to replayweb.page for standalone browsing
- frontend: add embed code popup when a collection is shareable
- refer to public collections as 'shareable' for now

---------
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
2023-08-03 19:11:01 -07:00
sua yoo
62d3399223
Add info bar to Collection detail view (#1036)
- Adds Collection info bar to detail view
- Update "Web Captures" -> "Archived Items"
- Updates Collection list columns to match
- Refactors `btrix-desc-list` and usage in `workflow-details` to reuse horizontal info bar component
2023-08-03 16:58:56 -07:00
Anish Lakhwara
af09d56ef6
Merge pull request #1035 from webrecorder/backend-init
feat: Display waiting message while backend is initializing
2023-08-02 17:39:47 -07:00
Anish Lakhwara
fa58e77167 fix: remove strange character? 2023-08-02 17:34:09 -07:00
Anish Lakhwara
5ed2faaecc fix: need to use window.timeOut to get a timerId back 2023-08-02 17:31:01 -07:00
Anish Lakhwara
6ecfd8ec24 fix: timerId not timeoutId 2023-08-02 17:28:07 -07:00
Anish Lakhwara
3985cf014e fix: clear timeout on disconnect callback 2023-08-02 17:26:26 -07:00
Anish Lakhwara
196b26c60e fix: center text 2023-08-02 17:21:36 -07:00
Anish Lakhwara
f1d91e3bf9 fix: add styling 2023-08-02 17:18:40 -07:00
Anish Lakhwara
a8bedeffb5 fix: take Sua's suggestons, less code needed 2023-08-02 17:10:45 -07:00
Anish Lakhwara
2f26fcefce fix: make pretty & work correctly 2023-08-02 16:36:28 -07:00
Anish Lakhwara
06918c967b feat: use html dialog instead 2023-08-02 11:37:55 -07:00
Anish Lakhwara
84a60b54e4 feat: Display waiting message while backend is initializing 2023-08-01 17:18:05 -07:00
Ilya Kreymer
45eaa0b3a3 version: bump to 1.6.0-beta.3 2023-08-01 09:48:17 -07:00
sua yoo
cc52dfd940
Sort Collections by size (#1026)
- Adds "Size" column to Collections list view
- Adds "Size" option to sort dropdown
2023-08-01 09:47:47 -07:00
sua yoo
54e2b2c703
List web captures in Collection (#1024)
- Adds tab for "Web Captures" in Collection detail view
- Move Collection description under Replay section
- Fixes app reloading when clicking into a Collection
- Standardizes Web Capture list headers from "Finished -> "Created Date"
2023-08-01 09:14:27 -07:00
Ilya Kreymer
06cf9c7cc3
add crawl ending states: 'generate-wacz', 'uploading-wacz', 'pending-wait' that occur after a crawl is finished or is being stopped (#1022)
operator: ensure transitions from each of these states is supported, including to 'waiting_capacity'
add extra check on stopping to avoid transitioning back to a running state after crawl is finished
ui: add states to UI display, localization, add as active states
fixes #263
2023-08-01 00:15:59 -07:00
Anish Lakhwara
d8502da885
fix(build): use /usr/bin/env bash instead of /bin/bash (#1020)
* fix: add to various other shell scripts
2023-07-28 21:50:04 -07:00
sua yoo
7069b33646
Show only running crawls in superadmin view (#1015)
- Show separate crawls list for admin view, fixes #1010
2023-07-26 15:48:20 -07:00
Ilya Kreymer
6506965d98
Streaming Download for Collections (#1012)
* support streaming download of collections (part of #927)
- WACZ zip created on the fly using stream-zip
- add 'Download Collection' option to collection detail and list
- after editing collection, return to collection view
- tests: add test for streaming download, ensure WACZ files + datapackage present, STORE compression used

---------

Co-authored-by: sua yoo <sua@suayoo.com>
2023-07-26 15:42:17 -07:00
Tessa Walsh
c21153255a
Rename notes to description in frontend and backend (#1011)
- Rename crawl notes to description
- Add migration renaming notes -> description
- Stop inheriting workflow description in crawl
- Update frontend to replace crawl/upload notes with description
- Remove setting of config description from crawl list
- Adjust tests for changes
2023-07-26 13:00:04 -07:00
sua yoo
75b011f951
Upload WACZ via UI (#992)
- Users can now upload .WACZ archives from the "Archived Data" page.
- Can specify name, description, tags and collection(s) to add upload to
- Show progress of upload
- Support canceling upload
2023-07-21 16:45:52 +02:00
sua yoo
85913112a2
Upgrade lit + shoelace to reduce build size (#938)
* upgrade lit

* upgrade shoelace

* upgrade testing libraries

* add webpack bundle analyzer

* revert shoelace changes

* remove bundle analyzer

* remove console log
2023-07-20 11:50:05 +02:00
Tessa Walsh
d5c3a8519f
Add crawler Use Sitemap option to Browsertrix Cloud (#978)
* Add user-guide docs for Use Sitemap option
---------

Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
2023-07-19 13:57:52 -04:00
sua yoo
c5b3be0680
Fix frontend formatting pre-commit (#991)
* update lint staged config

* remove prettier defaults
2023-07-18 17:51:13 +02:00
Ilya Kreymer
2372f43c2c
frontend: fix to collection editor with crawls and uploads (#971)
* frontend:
- follow up to #969, fixes crawl workflows by using crawl-specific endpoint and merging results

* get crawls and uploads concurrently

---------

Co-authored-by: sua yoo <sua@suayoo.com>
2023-07-10 19:29:19 +02:00
sua yoo
f3660839bf
Allow users to add uploads to collections (#968)
* show uploads in 'Select Uploads' section
2023-07-09 22:21:50 -07:00
Henry Wilkinson
d9e73fcbc3
Reorder Limits section (#966)
* Reorder Limits section

- Minor text change to section names
  - "Limit Per Page" → "Per-Page Limits"
  - "Limit Per Crawl" → "Per-Crawl Limits"

* Reorder limits section in documentation
2023-07-08 08:54:30 -07:00
Ilya Kreymer
8eeb66e11f
Frontend more upload path fixes (#961)
* additional fixes for #935:
- don't use artifactType for detail pages, ensure correct artifact selected based on path

* naming tweaks:
- from uploads detail, return to 'All Uploads' with filter
- from crawls detail, return to 'All Crawls' with filter
- rename general to 'All Archived Data'
2023-07-07 15:41:03 -07:00
Ilya Kreymer
d3a757e20b
partial fix for: #935: (#960)
- add route for /artifacts/upload/<id> to be used for uploads
- link uploads to /artifacts/upload/<id> instead of /artifacts/crawl/<id>
2023-07-07 14:23:26 -07:00
sua yoo
de4b18aa67
List crawls, uploads, and all objects in UI (#941)
- Adds top-level "Archived Data" view, replacing "Finished Crawls" and moving it as "Crawls" into view
- Adds list for viewing all artifacts/data
- Adds list for viewing all uploaded crawls
- Updates crawl detail view to show upload details
- Edit upload metadata, including 'name'
- Delete uploads
---------

Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
2023-07-07 13:20:28 -07:00
Ilya Kreymer
00eb62214d
Uploads API: BaseCrawl refactor + Initial support for /uploads endpoint (#937)
* basecrawl refactor: make crawls db more generic, supporting different types of 'base crawls': crawls, uploads, manual archives
- move shared functionality to basecrawl.py
- create a base BaseCrawl object, which contains start / finish time, metadata and files array
- create BaseCrawlOps, base class for CrawlOps, which supports base crawl deletion, querying and collection add/remove

* uploads api: (part of #929)
- new UploadCrawl object which extends BaseCrawl, has name and description
- support multipart form data data upload to /uploads/formdata
- support streaming upload of a single file via /uploads/stream, using botocore multipart upload to upload to s3-endpoint in parts
- require 'filename' param to set upload filename for streaming uploads (otherwise use form data names)
- sanitize filename, place uploads in /uploads/<uuid>/<sanitized-filename>-<random>.wacz
- uploads have internal id 'upload-<uuid>'
- create UploadedCrawl object with CrawlFiles pointing to the newly uploaded files, set state to 'complete'
- handle upload failures, abort multipart upload
- ensure uploads added within org bucket path
- return id / added when adding new UploadedCrawl
- support listing, deleting, and patch /uploads
- support upload details via /replay.json to support for replay
- add support for 'replaceId=<id>', which would remove all previous files in upload after new upload succeeds. if replaceId doesn't exist, create new upload. (only for stream endpoint so far).
- support patching upload metadata: notes, tags and name on uploads (UpdateUpload extends UpdateCrawl and adds 'name')

* base crawls api: Add /all-crawls list and delete endpoints for all crawl types (without resources)
- support all-crawls/<id>/replay.json with resources
- Use ListCrawlOut model for /all-crawls list endpoint
- Extend BaseCrawlOut from ListCrawlOut, add type
- use 'type: crawl' for crawls and 'type: upload' for uploads
- migration: ensure all previous crawl objects / missing type are set to 'type: crawl'
- indexes: add db indices on 'type' field and with 'type' field and oid, cid, finished, state

* tests: add test for multipart and streaming upload, listing uploads, deleting upload
- add sample WACZ for upload testing: 'example.wacz' and 'example-2.wacz'

* collections: support adding and remove both crawls and uploads via base crawl
- include collection_ids in /all-crawls list
- collections replay.json can include both crawls and uploads

bump version to 1.6.0-beta.2
---------

Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
2023-07-07 09:13:26 -07:00
Tessa Walsh
29a6f0f6bc
Fix links in watch crawl after workflow crawl completes (#943) 2023-07-06 15:04:26 -07:00
Henry Wilkinson
8a240ad044
Fixes z-index (#939) 2023-07-04 23:05:09 -04:00
Ilya Kreymer
e37f220d6c version: bump to 1.6.0-beta.1 2023-06-16 18:53:32 -07:00
Tessa Walsh
c7051d5fbf
Backend API consistency pass (#921)
* Make API add and update method returns consistent

- Updates return {"updated": True}
- Adds return {"added": True}
- Both can additionally have other fields as needed, e.g. id or name

- remove Profile response model, as returning added / id only
- reformat

---------

Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
2023-06-16 18:52:46 -07:00