huginn/README.md

# Huginn

Huginn builds a local, searchable index of websites sourced from
[Common Crawl](https://commoncrawl.org/) — a public archive of billions of
web pages. It's the data layer for a competitor-discovery tool: point it at
a product and a market (e.g. "who sells honey with a `.es` domain?") and it
should be able to answer from the indexed data, without live-scraping the
web yourself.

## How it works

Common Crawl publishes a monthly URL index ("CDX") that maps every page it
captured to metadata (status, mime type, timestamp) and a location inside
one of its WARC archive files. Huginn pulls from that index two ways:

1. **`index.commoncrawl.org` — the query API.** Good for targeted lookups:
   a specific URL, host, domain, or an entire TLD, paginated.
2. **`data.commoncrawl.org` — the raw index files.** The same data as
   static, downloadable shard files (`cdx-*.gz`), listed per collection in
   `cc-index.paths.gz`. Good for bulk ingestion, and doesn't depend on the
   query API being up.

Either way, matching captures are grouped into `Site`s (one per domain) and
`SitePage`s (one per unique URL within that site), and optionally enriched
with the page's title/description/keywords/body text — fetched straight out
of the relevant WARC file via an HTTP range request, so only the bytes for
that one page are downloaded.

### Data model

- **Site** — a domain (e.g. `example.com`) within a given Common Crawl
  collection, with its homepage's title/description/keywords.
- **SitePage** — one captured URL belonging to a Site: its CDX metadata
  (status, mime, timestamp, WARC filename/offset/length) plus, if fetched,
  its title/description/keywords/body text.

### Artisan commands

**`crawlindex:fetch`** — query the CDX API for a specific URL/host/domain,
or browse a whole TLD, and save the results.

```bash
# A specific domain
php artisan crawlindex:fetch example.com

# Every page under example.com (prefix match)
php artisan crawlindex:fetch "example.com/*" --match-type=prefix

# Browse an entire TLD, walking every index page, enriching each page
php artisan crawlindex:fetch --tld=es --all-pages --with-metadata
```

Key options: `--collection` (defaults to the latest), `--match-type`
(`exact|prefix|host|domain`), `--page`/`--pages`/`--all-pages` for
pagination, `--with-metadata` to fetch page content (slower — one extra
request per page).

**`crawlindex:bulk-fetch`** — walk Common Crawl's CDX shard files directly
from `data.commoncrawl.org`, bypassing the query API entirely. Useful for
bulk backfill/refresh of the database, or as a fallback when the query API
is offline.

```bash
# One shard, filtered locally to .es domains
php artisan crawlindex:bulk-fetch --collection=CC-MAIN-2026-25 --tld=es --shard-count=1

# Every shard in the collection (large — see caveat below)
php artisan crawlindex:bulk-fetch --collection=CC-MAIN-2026-25 --all-shards
```

Key options: `--collection`, `--tld` (filters locally, not via the API),
`--shard-start`/`--shard-count`/`--all-shards` to bound how many of a
collection's ~300 CDX shard files get processed in one run.

> **Scale caveat:** a monthly collection's CDX index is spread across
> ~300 shard files, each downloaded and decompressed in full. Running
> `--all-shards` (or `crawlindex:fetch --all-pages` without a `--tld`) over
> a whole collection means processing on the order of billions of records —
> pair broad runs with `--tld` (or a specific URL) to keep them bounded,
> and be especially careful combining `--with-metadata` with a broad run,
> since it adds one extra WARC request per page.

## Setup

Requirements: PHP 8.3+, Composer, Node/npm, and a MySQL database.

```bash
composer install
npm install

cp .env.example .env
php artisan key:generate
```

Edit `.env` and point `DB_*` at your MySQL instance, then run migrations:

```bash
php artisan migrate
```

Start the app (server, queue worker, and Vite dev server together):

```bash
composer run dev
```

Then populate the database with one of the `crawlindex:*` commands above.