Huginn

Huginn builds a local, searchable index of websites sourced from Common Crawl — a public archive of billions of web pages. It's the data layer for a competitor-discovery tool: point it at a product and a market (e.g. "who sells honey with a .es domain?") and it should be able to answer from the indexed data, without live-scraping the web yourself.

How it works

Common Crawl publishes a monthly URL index ("CDX") that maps every page it captured to metadata (status, mime type, timestamp) and a location inside one of its WARC archive files. Huginn pulls from that index two ways:

1. index.commoncrawl.org — the query API. Good for targeted lookups: a specific URL, host, domain, or an entire TLD, paginated.
2. data.commoncrawl.org — the raw index files. The same data as static, downloadable shard files (cdx-*.gz), listed per collection in cc-index.paths.gz. Good for bulk ingestion, and doesn't depend on the query API being up.

Either way, matching captures are grouped into Sites (one per domain) and SitePages (one per unique URL within that site), and optionally enriched with the page's title/description/keywords/body text — fetched straight out of the relevant WARC file via an HTTP range request, so only the bytes for that one page are downloaded.

Data model

• Site — a domain (e.g. example.com) within a given Common Crawl collection, with its homepage's title/description/keywords.
• SitePage — one captured URL belonging to a Site: its CDX metadata (status, mime, timestamp, WARC filename/offset/length) plus, if fetched, its title/description/keywords/body text.

Artisan commands

crawlindex:fetch — query the CDX API for a specific URL/host/domain, or browse a whole TLD, and save the results.

# A specific domain
php artisan crawlindex:fetch example.com

# Every page under example.com (prefix match)
php artisan crawlindex:fetch "example.com/*" --match-type=prefix

# Browse an entire TLD, walking every index page, enriching each page
php artisan crawlindex:fetch --tld=es --all-pages --with-metadata

Key options: --collection (defaults to the latest), --match-type (exact|prefix|host|domain), --page/--pages/--all-pages for pagination, --with-metadata to fetch page content (slower — one extra request per page).

crawlindex:bulk-fetch — walk Common Crawl's CDX shard files directly from data.commoncrawl.org, bypassing the query API entirely. Useful for bulk backfill/refresh of the database, or as a fallback when the query API is offline.

# One shard, filtered locally to .es domains
php artisan crawlindex:bulk-fetch --collection=CC-MAIN-2026-25 --tld=es --shard-count=1

# Every shard in the collection (large — see caveat below)
php artisan crawlindex:bulk-fetch --collection=CC-MAIN-2026-25 --all-shards

Key options: --collection, --tld (filters locally, not via the API), --shard-start/--shard-count/--all-shards to bound how many of a collection's ~300 CDX shard files get processed in one run.

Scale caveat: a monthly collection's CDX index is spread across ~300 shard files, each downloaded and decompressed in full. Running --all-shards (or crawlindex:fetch --all-pages without a --tld) over a whole collection means processing on the order of billions of records — pair broad runs with --tld (or a specific URL) to keep them bounded, and be especially careful combining --with-metadata with a broad run, since it adds one extra WARC request per page.

Setup

Requirements: PHP 8.3+, Composer, Node/npm, and a MySQL database.

composer install
npm install

cp .env.example .env
php artisan key:generate

Edit .env and point DB_* at your MySQL instance, then run migrations:

php artisan migrate

Start the app (server, queue worker, and Vite dev server together):

composer run dev

Then populate the database with one of the crawlindex:* commands above.