From 4dd5a3c17d9463f0d59b41774671ce89d106fc07 Mon Sep 17 00:00:00 2001 From: m105 Date: Mon, 20 Jul 2026 16:07:14 +0300 Subject: [PATCH] Replace default Laravel README with project documentation Explains what Huginn does (Common Crawl-backed site index for competitor discovery), the two ingestion paths (query API vs bulk CDX shard download), the Site/SitePage data model, and how to set up and run the app locally. --- README.md | 121 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 85 insertions(+), 36 deletions(-) diff --git a/README.md b/README.md index 5ad1377..9dcbdc9 100644 --- a/README.md +++ b/README.md @@ -1,58 +1,107 @@ -

Laravel Logo

+# Huginn -

-Build Status -Total Downloads -Latest Stable Version -License -

+Huginn builds a local, searchable index of websites sourced from +[Common Crawl](https://commoncrawl.org/) — a public archive of billions of +web pages. It's the data layer for a competitor-discovery tool: point it at +a product and a market (e.g. "who sells honey with a `.es` domain?") and it +should be able to answer from the indexed data, without live-scraping the +web yourself. -## About Laravel +## How it works -Laravel is a web application framework with expressive, elegant syntax. We believe development must be an enjoyable and creative experience to be truly fulfilling. Laravel takes the pain out of development by easing common tasks used in many web projects, such as: +Common Crawl publishes a monthly URL index ("CDX") that maps every page it +captured to metadata (status, mime type, timestamp) and a location inside +one of its WARC archive files. Huginn pulls from that index two ways: -- [Simple, fast routing engine](https://laravel.com/docs/routing). -- [Powerful dependency injection container](https://laravel.com/docs/container). -- Multiple back-ends for [session](https://laravel.com/docs/session) and [cache](https://laravel.com/docs/cache) storage. -- Expressive, intuitive [database ORM](https://laravel.com/docs/eloquent). -- Database agnostic [schema migrations](https://laravel.com/docs/migrations). -- [Robust background job processing](https://laravel.com/docs/queues). -- [Real-time event broadcasting](https://laravel.com/docs/broadcasting). +1. **`index.commoncrawl.org` — the query API.** Good for targeted lookups: + a specific URL, host, domain, or an entire TLD, paginated. +2. **`data.commoncrawl.org` — the raw index files.** The same data as + static, downloadable shard files (`cdx-*.gz`), listed per collection in + `cc-index.paths.gz`. Good for bulk ingestion, and doesn't depend on the + query API being up. -Laravel is accessible, powerful, and provides tools required for large, robust applications. +Either way, matching captures are grouped into `Site`s (one per domain) and +`SitePage`s (one per unique URL within that site), and optionally enriched +with the page's title/description/keywords/body text — fetched straight out +of the relevant WARC file via an HTTP range request, so only the bytes for +that one page are downloaded. -## Learning Laravel +### Data model -Laravel has the most extensive and thorough [documentation](https://laravel.com/docs) and video tutorial library of all modern web application frameworks, making it a breeze to get started with the framework. +- **Site** — a domain (e.g. `example.com`) within a given Common Crawl + collection, with its homepage's title/description/keywords. +- **SitePage** — one captured URL belonging to a Site: its CDX metadata + (status, mime, timestamp, WARC filename/offset/length) plus, if fetched, + its title/description/keywords/body text. -In addition, [Laracasts](https://laracasts.com) contains thousands of video tutorials on a range of topics including Laravel, modern PHP, unit testing, and JavaScript. Boost your skills by digging into our comprehensive video library. +### Artisan commands -You can also watch bite-sized lessons with real-world projects on [Laravel Learn](https://laravel.com/learn), where you will be guided through building a Laravel application from scratch while learning PHP fundamentals. - -## Agentic Development - -Laravel's predictable structure and conventions make it ideal for AI coding agents like Claude Code, Cursor, and GitHub Copilot. Install [Laravel Boost](https://laravel.com/docs/ai) to supercharge your AI workflow: +**`crawlindex:fetch`** — query the CDX API for a specific URL/host/domain, +or browse a whole TLD, and save the results. ```bash -composer require laravel/boost --dev +# A specific domain +php artisan crawlindex:fetch example.com -php artisan boost:install +# Every page under example.com (prefix match) +php artisan crawlindex:fetch "example.com/*" --match-type=prefix + +# Browse an entire TLD, walking every index page, enriching each page +php artisan crawlindex:fetch --tld=es --all-pages --with-metadata ``` -Boost provides your agent 15+ tools and skills that help agents build Laravel applications while following best practices. +Key options: `--collection` (defaults to the latest), `--match-type` +(`exact|prefix|host|domain`), `--page`/`--pages`/`--all-pages` for +pagination, `--with-metadata` to fetch page content (slower — one extra +request per page). -## Contributing +**`crawlindex:bulk-fetch`** — walk Common Crawl's CDX shard files directly +from `data.commoncrawl.org`, bypassing the query API entirely. Useful for +bulk backfill/refresh of the database, or as a fallback when the query API +is offline. -Thank you for considering contributing to the Laravel framework! The contribution guide can be found in the [Laravel documentation](https://laravel.com/docs/contributions). +```bash +# One shard, filtered locally to .es domains +php artisan crawlindex:bulk-fetch --collection=CC-MAIN-2026-25 --tld=es --shard-count=1 -## Code of Conduct +# Every shard in the collection (large — see caveat below) +php artisan crawlindex:bulk-fetch --collection=CC-MAIN-2026-25 --all-shards +``` -In order to ensure that the Laravel community is welcoming to all, please review and abide by the [Code of Conduct](https://laravel.com/docs/contributions#code-of-conduct). +Key options: `--collection`, `--tld` (filters locally, not via the API), +`--shard-start`/`--shard-count`/`--all-shards` to bound how many of a +collection's ~300 CDX shard files get processed in one run. -## Security Vulnerabilities +> **Scale caveat:** a monthly collection's CDX index is spread across +> ~300 shard files, each downloaded and decompressed in full. Running +> `--all-shards` (or `crawlindex:fetch --all-pages` without a `--tld`) over +> a whole collection means processing on the order of billions of records — +> pair broad runs with `--tld` (or a specific URL) to keep them bounded, +> and be especially careful combining `--with-metadata` with a broad run, +> since it adds one extra WARC request per page. -If you discover a security vulnerability within Laravel, please send an e-mail to Taylor Otwell via [taylor@laravel.com](mailto:taylor@laravel.com). All security vulnerabilities will be promptly addressed. +## Setup -## License +Requirements: PHP 8.3+, Composer, Node/npm, and a MySQL database. -The Laravel framework is open-sourced software licensed under the [MIT license](https://opensource.org/licenses/MIT). +```bash +composer install +npm install + +cp .env.example .env +php artisan key:generate +``` + +Edit `.env` and point `DB_*` at your MySQL instance, then run migrations: + +```bash +php artisan migrate +``` + +Start the app (server, queue worker, and Vite dev server together): + +```bash +composer run dev +``` + +Then populate the database with one of the `crawlindex:*` commands above.