browsertrix/backend
Tessa Walsh 6797b41de0
Add pageCount to crawls and uploads and use in frontend for page counts (#2315)
Fixes #2257 

This is a follow-up to the public collections work, which adds pages to
the database for uploads. All crawls and uploads now have a `pageCount`
field which is populated when the item is successfully added. A new
migration is also added to populate the field for existing archived
items that don't have it set yet.

OrgMetrics have also been modified to include `crawlPageCount` and
`uploadPageCount`, and to include the total of both in `pageCount`, and
all three included in the frontend org dashboard.

The frontend has been updated to use `pageCount` rather than
`stats.done` wherever appropriate, meaning that in archived item lists
and details we now have a consistent page count for both crawls and
uploads.

### New functionality

- Deploy this branch
- Create new crawls and uploads and verify that page count appears
correctly throughout the frontend for all new crawls and uploads

### Migration

- Deploy from latest main
- Create some crawls and uploads
- Change to this branch and re-deploy
- Verify migration ran without errors in backend logs
- Verify that page count has been populated successfully by checking
archived items lists, crawl and upload detail pages, and dashboard to
ensure there are no longer any missing page counts.

---------

Co-authored-by: emma <hi@emma.cafe>
2025-01-16 14:41:14 -08:00
..
btrixcloud Add pageCount to crawls and uploads and use in frontend for page counts (#2315) 2025-01-16 14:41:14 -08:00
test Add pageCount to crawls and uploads and use in frontend for page counts (#2315) 2025-01-16 14:41:14 -08:00
test_nightly Add superuser endpoint to get user emails with org info (#2211) 2024-12-09 16:38:01 -08:00
.pylintrc security: tweak get /invite endpoints / InviteOut to: (#2087) 2024-09-20 11:52:56 -07:00
dev-requirements.txt
Dockerfile
mypy.ini
requirements.txt switch to simpler streaming download + multiwacz metadata improvements: (#1982) 2024-10-03 16:13:31 -07:00
test-requirements.txt Fix nightly tests: Add boto3 as test requirement (#2116) 2024-10-23 13:41:22 -07:00