browsertrix/backend/btrixcloud
Tessa Walsh 6797b41de0
Add pageCount to crawls and uploads and use in frontend for page counts (#2315)
Fixes #2257 

This is a follow-up to the public collections work, which adds pages to
the database for uploads. All crawls and uploads now have a `pageCount`
field which is populated when the item is successfully added. A new
migration is also added to populate the field for existing archived
items that don't have it set yet.

OrgMetrics have also been modified to include `crawlPageCount` and
`uploadPageCount`, and to include the total of both in `pageCount`, and
all three included in the frontend org dashboard.

The frontend has been updated to use `pageCount` rather than
`stats.done` wherever appropriate, meaning that in archived item lists
and details we now have a consistent page count for both crawls and
uploads.

### New functionality

- Deploy this branch
- Create new crawls and uploads and verify that page count appears
correctly throughout the frontend for all new crawls and uploads

### Migration

- Deploy from latest main
- Create some crawls and uploads
- Change to this branch and re-deploy
- Verify migration ran without errors in backend logs
- Verify that page count has been populated successfully by checking
archived items lists, crawl and upload detail pages, and dashboard to
ensure there are no longer any missing page counts.

---------

Co-authored-by: emma <hi@emma.cafe>
2025-01-16 14:41:14 -08:00
..
migrations Add pageCount to crawls and uploads and use in frontend for page counts (#2315) 2025-01-16 14:41:14 -08:00
operator Add pageCount to crawls and uploads and use in frontend for page counts (#2315) 2025-01-16 14:41:14 -08:00
__init__.py
auth.py Include user and user org info in login response (#2014) 2024-08-12 18:51:42 -07:00
background_jobs.py Add missing os import 2025-01-13 15:15:48 -08:00
basecrawls.py Add last crawl and subscription status indicators to org list (#2273) 2025-01-14 10:57:06 -05:00
colls.py feat: Add slug to collections and use it in public collection URLs (#2301) 2025-01-15 22:44:32 -08:00
crawlconfigs.py Fix nightly tests: modify kubectl exec syntax for creating new minio bucket (#2097) 2024-10-21 17:41:19 -07:00
crawlmanager.py Backend work for public collections (#2198) 2025-01-13 15:15:48 -08:00
crawls.py Backend work for public collections (#2198) 2025-01-13 15:15:48 -08:00
db.py Add pageCount to crawls and uploads and use in frontend for page counts (#2315) 2025-01-16 14:41:14 -08:00
emailsender.py Send subscription cancelation email (#2234) 2024-12-12 11:52:38 -08:00
invites.py fix: firstOrgAdmin being set to true even if invite was not for an admin (#2110) 2024-10-08 16:42:30 -07:00
k8sapi.py Configure browsertrix proxies (#1847) 2024-10-02 18:35:45 -07:00
main_bg.py Backend work for public collections (#2198) 2025-01-13 15:15:48 -08:00
main_op.py Backend work for public collections (#2198) 2025-01-13 15:15:48 -08:00
main.py Backend work for public collections (#2198) 2025-01-13 15:15:48 -08:00
models.py Add pageCount to crawls and uploads and use in frontend for page counts (#2315) 2025-01-16 14:41:14 -08:00
ops.py Backend work for public collections (#2198) 2025-01-13 15:15:48 -08:00
orgs.py Add pageCount to crawls and uploads and use in frontend for page counts (#2315) 2025-01-16 14:41:14 -08:00
pages.py Add pageCount to crawls and uploads and use in frontend for page counts (#2315) 2025-01-16 14:41:14 -08:00
pagination.py
profiles.py Backend work for public collections (#2198) 2025-01-13 15:15:48 -08:00
storages.py Backend work for public collections (#2198) 2025-01-13 15:15:48 -08:00
subs.py Send subscription cancelation email (#2234) 2024-12-12 11:52:38 -08:00
uploads.py Backend work for public collections (#2198) 2025-01-13 15:15:48 -08:00
users.py Add superuser endpoint to get user emails with org info (#2211) 2024-12-09 16:38:01 -08:00
utils.py feat: Add slug to collections and use it in public collection URLs (#2301) 2025-01-15 22:44:32 -08:00
version.py version: bump to 1.14.0-beta.0 for public collections! 2025-01-13 19:29:54 -08:00
webhooks.py