* Track collections in Crawl rather than crawls in Collection * Add delete collection API endpoint and tests * Precompute collection crawlCount, pageCount, and tags and add them to GET collection responses * Add modified field to Collection * Update collection replay.json method * Make add and remove crawls accept list of crawl ids * Auto-add new workflow crawls to collections when they successfully complete via CrawlConfig.autoAddCollections field * Move long-running post-crawl operator tasks into asyncio task * Make CrawlConfig.autoAddCollections updatable via /update API endpoint |
||
---|---|---|
.. | ||
__init__.py | ||
migration_0001_archives_to_orgs.py | ||
migration_0002_crawlconfig_crawlstats.py | ||
migration_0003_mutable_crawl_configs.py | ||
migration_0004_config_seeds.py | ||
migration_0005_operator_scheduled_jobs.py | ||
migration_0006_precompute_crawl_stats.py | ||
migration_0007_colls_and_config_update.py |