tea/browsertrix
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

limit: set default page limit to 50k pages (#1211)

Browse Source
This commit is contained in:
Ilya Kreymer 2023-09-26 10:29:03 -07:00 committed by GitHub
parent e9f004ad5e
commit 18b2c1abfc
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
chart/values.yaml
View File

@ -22,8 +22,9 @@ crawler_extract_full_text: false
# max pages per crawl
# set to non-zero value to enforce global max pages per crawl limit
# if 0, there is no page limit (may need to adjust crawler/redis settings for larger crawls)
# if set, each workflow can have a lower limit, but not higher
max_pages_per_crawl: 0
max_pages_per_crawl: 50000
# User Agent Options
# set to add suffix to default browser User Agent