c134b576ae
Fixes #2515. This PR introduces a significantly optimized logic for presigning URLs for crawls and collections. - For collections, the files needed from all crawls are looked up, and then the 'presign_urls' table is merged in one pass, resulting in a unified iterator containing files and presign urls for those files. - For crawls, the presign URLs are also looked up once, and the same iterator is used for a single crawl with passed in list of CrawlFiles - URLs that are already signed are added to the return list. - For any remaining URLs to be signed, a bulk presigning function is added, which shares an HTTP connection and signing 8 files in parallels (customizable via helm chart, though may not be needed). This function is used to call the presigning API in parallel. |
||
|---|---|---|
| .. | ||
| backend.yaml | ||
| configmap.yaml | ||
| frontend.yaml | ||
| ingress.yaml | ||
| minio.yaml | ||
| mongo.yaml | ||
| namespaces.yaml | ||
| networkpolicies.yaml | ||
| operators.yaml | ||
| priorities.yaml | ||
| role.yaml | ||
| secrets.yaml | ||
| service.yaml | ||
| signer.yaml | ||