Explains what Huginn does (Common Crawl-backed site index for
competitor discovery), the two ingestion paths (query API vs bulk CDX
shard download), the Site/SitePage data model, and how to set up and
run the app locally.
Adds Site/SitePage models and the crawlindex:fetch command, which pulls
CDX index records from Common Crawl and optionally enriches them with
page title/description/keywords/body text extracted from the WARC
archive. CDX/WARC access and the site/page upsert logic live in
CommonCrawlIndexService and WarcRecordFetcher.
m105