browsertrix/backend/test
Tessa Walsh 6797b41de0
Add pageCount to crawls and uploads and use in frontend for page counts (#2315)
Fixes #2257 

This is a follow-up to the public collections work, which adds pages to
the database for uploads. All crawls and uploads now have a `pageCount`
field which is populated when the item is successfully added. A new
migration is also added to populate the field for existing archived
items that don't have it set yet.

OrgMetrics have also been modified to include `crawlPageCount` and
`uploadPageCount`, and to include the total of both in `pageCount`, and
all three included in the frontend org dashboard.

The frontend has been updated to use `pageCount` rather than
`stats.done` wherever appropriate, meaning that in archived item lists
and details we now have a consistent page count for both crawls and
uploads.

### New functionality

- Deploy this branch
- Create new crawls and uploads and verify that page count appears
correctly throughout the frontend for all new crawls and uploads

### Migration

- Deploy from latest main
- Create some crawls and uploads
- Change to this branch and re-deploy
- Verify migration ran without errors in backend logs
- Verify that page count has been populated successfully by checking
archived items lists, crawl and upload detail pages, and dashboard to
ensure there are no longer any missing page counts.

---------

Co-authored-by: emma <hi@emma.cafe>
2025-01-16 14:41:14 -08:00
..
data Backend work for public collections (#2198) 2025-01-13 15:15:48 -08:00
__init__.py
conftest.py Move org storage recalculation into background job (#2138) 2024-11-19 17:32:57 -05:00
echo_server.py Adds Subscription API (#1914) 2024-07-10 17:41:16 -07:00
test_api.py quickfix: update test_api.py to match all locales enabled by default (#2241) 2024-12-13 20:30:06 -08:00
test_collections.py feat: Add slug to collections and use it in public collection URLs (#2301) 2025-01-15 22:44:32 -08:00
test_crawl_config_search_values.py Pydantic 2.x update + type fixes + python 3.12 (#1947) 2024-07-22 17:23:03 -07:00
test_crawl_config_tags.py Backend API consistency pass (#921) 2023-06-16 18:52:46 -07:00
test_crawlconfigs.py Serialize datetimes with Z suffix (#2058) 2024-09-12 16:16:13 -07:00
test_filter_sort_results.py Separate seeds into a new endpoints (#1217) 2023-10-02 10:56:12 -07:00
test_login.py User refactor + remove fastapi_users dependency + update fastapi (#1290) 2023-10-18 10:49:23 -07:00
test_org_subs.py security: tweak get /invite endpoints / InviteOut to: (#2087) 2024-09-20 11:52:56 -07:00
test_org.py Add last crawl and subscription status indicators to org list (#2273) 2025-01-14 10:57:06 -05:00
test_permissions.py Paginate API list endpoints (#659) 2023-03-06 14:41:25 -05:00
test_profiles.py Serialize datetimes with Z suffix (#2058) 2024-09-12 16:16:13 -07:00
test_qa.py Move org storage recalculation into background job (#2138) 2024-11-19 17:32:57 -05:00
test_run_crawl.py Add pageCount to crawls and uploads and use in frontend for page counts (#2315) 2025-01-16 14:41:14 -08:00
test_stop_cancel_crawl.py Replace partial_complete -> stopped_by_user or stopped_quota_reached + operator edge cases (#1368) 2023-11-14 11:17:16 -08:00
test_uploads.py Add pageCount to crawls and uploads and use in frontend for page counts (#2315) 2025-01-16 14:41:14 -08:00
test_users.py Add superuser endpoint to get user emails with org info (#2211) 2024-12-09 16:38:01 -08:00
test_utils.py Format backend with Black 24 (#1507) 2024-02-07 11:35:34 -08:00
test_webhooks.py Add webhooks for qaAnalysisStarted, qaAnalysisFinished, and crawlReviewed (#1974) 2024-07-25 16:53:49 -07:00
test_workflow_auto_add_to_collection.py feat: implement 'collections' array with {name, id} for archived item details (#1098) 2023-08-25 00:26:46 -07:00
test_y_org_import_export.py Add API endpoint to delete org (#1448) 2024-07-03 16:00:11 -04:00
test_z_delete_org.py Move org storage recalculation into background job (#2138) 2024-11-19 17:32:57 -05:00
utils.py Uploads API: BaseCrawl refactor + Initial support for /uploads endpoint (#937) 2023-07-07 09:13:26 -07:00