tea/browsertrix
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8a507f0473
browsertrix/backend/btrixcloud
History
Ilya Kreymer 8a507f0473
Consolidate list page endpoints + better QA sorting + optimize pages fix (#2417) 
- consolidate list_pages() and list_replay_query_pages() into
list_pages()
- to keep backwards compatibility, add <crawl>/pagesSearch that does not
include page totals, keep <crawl>/pages with page total (slower)
- qa frontend: add default 'Crawl Order' sort order, to better show
pages in QA view
- bgjob: account for parallelism in bgjobs, add logging if succeeded
mismatches parallelism
- QA sorting: default to 'crawl order' by default to get better results.
- Optimize pages job: also cover crawls that may not have any pages but have pages listed in done stats
- Bgjobs: give custom op jobs more memory
2025-02-21 13:47:20 -08:00
..
migrations Rework crawl page migration + MongoDB Query Optimizations (#2412) 2025-02-20 15:26:11 -08:00
operator Consolidate list page endpoints + better QA sorting + optimize pages fix (#2417) 2025-02-21 13:47:20 -08:00
__init__.py
auth.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
background_jobs.py Rework crawl page migration + MongoDB Query Optimizations (#2412) 2025-02-20 15:26:11 -08:00
basecrawls.py Consolidate list page endpoints + better QA sorting + optimize pages fix (#2417) 2025-02-21 13:47:20 -08:00
colls.py Consolidate list page endpoints + better QA sorting + optimize pages fix (#2417) 2025-02-21 13:47:20 -08:00
crawlconfigs.py Fix max pages quota setting and display (#2370) 2025-02-10 16:15:21 -08:00
crawlmanager.py Consolidate list page endpoints + better QA sorting + optimize pages fix (#2417) 2025-02-21 13:47:20 -08:00
crawls.py Rework crawl page migration + MongoDB Query Optimizations (#2412) 2025-02-20 15:26:11 -08:00
db.py Add WACZ filename, depth, favIconUrl, isSeed to pages (#2352) 2025-02-05 15:50:04 -05:00
emailsender.py Rework crawl page migration + MongoDB Query Optimizations (#2412) 2025-02-20 15:26:11 -08:00
invites.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
k8sapi.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
main_bg.py Rework crawl page migration + MongoDB Query Optimizations (#2412) 2025-02-20 15:26:11 -08:00
main_op.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
main.py Add collection page list/search endpoint (#2354) 2025-02-10 16:44:37 -08:00
models.py Rework crawl page migration + MongoDB Query Optimizations (#2412) 2025-02-20 15:26:11 -08:00
ops.py Modify page upload migration (#2400) 2025-02-17 16:47:58 -08:00
orgs.py Rework crawl page migration + MongoDB Query Optimizations (#2412) 2025-02-20 15:26:11 -08:00
pages.py Consolidate list page endpoints + better QA sorting + optimize pages fix (#2417) 2025-02-21 13:47:20 -08:00
pagination.py
profiles.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
storages.py Rework crawl page migration + MongoDB Query Optimizations (#2412) 2025-02-20 15:26:11 -08:00
subs.py
uploads.py Rework crawl page migration + MongoDB Query Optimizations (#2412) 2025-02-20 15:26:11 -08:00
users.py
utils.py Reformat with Black for 2025 ruleset (#2349) 2025-01-29 16:57:06 -05:00
version.py Consolidate list page endpoints + better QA sorting + optimize pages fix (#2417) 2025-02-21 13:47:20 -08:00
webhooks.py