c134b576ae
Fixes #2515. This PR introduces a significantly optimized logic for presigning URLs for crawls and collections. - For collections, the files needed from all crawls are looked up, and then the 'presign_urls' table is merged in one pass, resulting in a unified iterator containing files and presign urls for those files. - For crawls, the presign URLs are also looked up once, and the same iterator is used for a single crawl with passed in list of CrawlFiles - URLs that are already signed are added to the return list. - For any remaining URLs to be signed, a bulk presigning function is added, which shares an HTTP connection and signing 8 files in parallels (customizable via helm chart, though may not be needed). This function is used to call the presigning API in parallel. |
||
|---|---|---|
| .. | ||
| btrixcloud | ||
| test | ||
| test_nightly | ||
| .pylintrc | ||
| dev-requirements.txt | ||
| Dockerfile | ||
| mypy.ini | ||
| requirements.txt | ||
| test-requirements.txt | ||