m105
4dd5a3c17d
Explains what Huginn does (Common Crawl-backed site index for competitor discovery), the two ingestion paths (query API vs bulk CDX shard download), the Site/SitePage data model, and how to set up and run the app locally.
108 lines
4.0 KiB
Markdown
108 lines
4.0 KiB
Markdown
# Huginn
|
|
|
|
Huginn builds a local, searchable index of websites sourced from
|
|
[Common Crawl](https://commoncrawl.org/) — a public archive of billions of
|
|
web pages. It's the data layer for a competitor-discovery tool: point it at
|
|
a product and a market (e.g. "who sells honey with a `.es` domain?") and it
|
|
should be able to answer from the indexed data, without live-scraping the
|
|
web yourself.
|
|
|
|
## How it works
|
|
|
|
Common Crawl publishes a monthly URL index ("CDX") that maps every page it
|
|
captured to metadata (status, mime type, timestamp) and a location inside
|
|
one of its WARC archive files. Huginn pulls from that index two ways:
|
|
|
|
1. **`index.commoncrawl.org` — the query API.** Good for targeted lookups:
|
|
a specific URL, host, domain, or an entire TLD, paginated.
|
|
2. **`data.commoncrawl.org` — the raw index files.** The same data as
|
|
static, downloadable shard files (`cdx-*.gz`), listed per collection in
|
|
`cc-index.paths.gz`. Good for bulk ingestion, and doesn't depend on the
|
|
query API being up.
|
|
|
|
Either way, matching captures are grouped into `Site`s (one per domain) and
|
|
`SitePage`s (one per unique URL within that site), and optionally enriched
|
|
with the page's title/description/keywords/body text — fetched straight out
|
|
of the relevant WARC file via an HTTP range request, so only the bytes for
|
|
that one page are downloaded.
|
|
|
|
### Data model
|
|
|
|
- **Site** — a domain (e.g. `example.com`) within a given Common Crawl
|
|
collection, with its homepage's title/description/keywords.
|
|
- **SitePage** — one captured URL belonging to a Site: its CDX metadata
|
|
(status, mime, timestamp, WARC filename/offset/length) plus, if fetched,
|
|
its title/description/keywords/body text.
|
|
|
|
### Artisan commands
|
|
|
|
**`crawlindex:fetch`** — query the CDX API for a specific URL/host/domain,
|
|
or browse a whole TLD, and save the results.
|
|
|
|
```bash
|
|
# A specific domain
|
|
php artisan crawlindex:fetch example.com
|
|
|
|
# Every page under example.com (prefix match)
|
|
php artisan crawlindex:fetch "example.com/*" --match-type=prefix
|
|
|
|
# Browse an entire TLD, walking every index page, enriching each page
|
|
php artisan crawlindex:fetch --tld=es --all-pages --with-metadata
|
|
```
|
|
|
|
Key options: `--collection` (defaults to the latest), `--match-type`
|
|
(`exact|prefix|host|domain`), `--page`/`--pages`/`--all-pages` for
|
|
pagination, `--with-metadata` to fetch page content (slower — one extra
|
|
request per page).
|
|
|
|
**`crawlindex:bulk-fetch`** — walk Common Crawl's CDX shard files directly
|
|
from `data.commoncrawl.org`, bypassing the query API entirely. Useful for
|
|
bulk backfill/refresh of the database, or as a fallback when the query API
|
|
is offline.
|
|
|
|
```bash
|
|
# One shard, filtered locally to .es domains
|
|
php artisan crawlindex:bulk-fetch --collection=CC-MAIN-2026-25 --tld=es --shard-count=1
|
|
|
|
# Every shard in the collection (large — see caveat below)
|
|
php artisan crawlindex:bulk-fetch --collection=CC-MAIN-2026-25 --all-shards
|
|
```
|
|
|
|
Key options: `--collection`, `--tld` (filters locally, not via the API),
|
|
`--shard-start`/`--shard-count`/`--all-shards` to bound how many of a
|
|
collection's ~300 CDX shard files get processed in one run.
|
|
|
|
> **Scale caveat:** a monthly collection's CDX index is spread across
|
|
> ~300 shard files, each downloaded and decompressed in full. Running
|
|
> `--all-shards` (or `crawlindex:fetch --all-pages` without a `--tld`) over
|
|
> a whole collection means processing on the order of billions of records —
|
|
> pair broad runs with `--tld` (or a specific URL) to keep them bounded,
|
|
> and be especially careful combining `--with-metadata` with a broad run,
|
|
> since it adds one extra WARC request per page.
|
|
|
|
## Setup
|
|
|
|
Requirements: PHP 8.3+, Composer, Node/npm, and a MySQL database.
|
|
|
|
```bash
|
|
composer install
|
|
npm install
|
|
|
|
cp .env.example .env
|
|
php artisan key:generate
|
|
```
|
|
|
|
Edit `.env` and point `DB_*` at your MySQL instance, then run migrations:
|
|
|
|
```bash
|
|
php artisan migrate
|
|
```
|
|
|
|
Start the app (server, queue worker, and Vite dev server together):
|
|
|
|
```bash
|
|
composer run dev
|
|
```
|
|
|
|
Then populate the database with one of the `crawlindex:*` commands above.
|