c134b576ae
Fixes #2515. This PR introduces a significantly optimized logic for presigning URLs for crawls and collections. - For collections, the files needed from all crawls are looked up, and then the 'presign_urls' table is merged in one pass, resulting in a unified iterator containing files and presign urls for those files. - For crawls, the presign URLs are also looked up once, and the same iterator is used for a single crawl with passed in list of CrawlFiles - URLs that are already signed are added to the return list. - For any remaining URLs to be signed, a bulk presigning function is added, which shares an HTTP connection and signing 8 files in parallels (customizable via helm chart, though may not be needed). This function is used to call the presigning API in parallel. |
||
|---|---|---|
| .. | ||
| admin/logging | ||
| app-templates | ||
| btrix-crds | ||
| charts | ||
| email-templates | ||
| examples | ||
| proxies | ||
| templates | ||
| test | ||
| .helmignore | ||
| Chart.lock | ||
| Chart.yaml | ||
| README.md | ||
| values.yaml | ||
Update Helm dependencies
- It needs to update Helm charts after changing its dependencies (e.g. logging)
$ helm dependency update .
Update metacontroller
#!/bin/bash
# intall metacontroller
git clone --depth=1 https://github.com/metacontroller/metacontroller.git
cd metacontroller
helm package deploy/helm/metacontroller --destination deploy/helm
cd ..
# update dependency
helm dependency update
- Bump up the metacontroller version in
Chart.yaml