Adds sites.root_domain (indexed string, backfilled for existing rows)
computed via a naive "last two labels" heuristic (App\Support\RootDomain)
— good enough for example.com/somesite.es, but wrong for two-part
public suffixes like .co.uk. Doing that correctly needs the Mozilla
Public Suffix List; flagged as a known simplification rather than
built now.
Deliberately not a hard FK: like Backlink.target_domain, a subdomain
can get indexed before its root domain does (or vice versa), so
Site::subdomains()/rootSite() match on the domain string rather than
requiring the relationship to exist at write time.
Site detail page: a root domain shows a paginated "Subdomains"
section (same guest-preview cap as Pages/Backlinks); a subdomain
shows "Subdomain of {root}" linking back, when the root is itself
indexed.
Verified against real data: 21k.ro correctly lists its two known
subdomains (buzau.21k.ro, timisoara.21k.ro), each of which correctly
links back to 21k.ro. Added SubdomainsTest (root lists subdomains,
subdomain links to root, empty state) — full suite (31 tests) passes.
Hand-rolled login/register/logout on Laravel's built-in Auth facade
rather than Breeze/Jetstream, which scaffold Tailwind v3 config and
would fight our existing Tailwind v4 CSS-first setup. Login/Register
are Livewire components; logout is a plain POST route (auth-gated,
CSRF-protected).
Routing: "/" is now a public landing page describing the project;
the sites list moved from "/" to "/sites" to make room for it.
Data gating: App\Support\GuestPreview::paginate() forces page 1 and a
5-row page size for unauthenticated visitors on Sites, Discover, and
a site's Pages/Backlinks tables, while total() still reflects the
real match count — authenticated users get the existing full
pagination unchanged. A shared guest-preview-notice component shows
"N of M results, sign in to see all" and only renders when something
is actually being hidden.
Testing: pointed the test suite at a dedicated MySQL database
(huginn_testing) instead of the default in-memory SQLite, since the
existing fulltext index migration isn't SQLite-compatible — this app
is MySQL-specific by now regardless. Added AuthFlowTest (register,
login, wrong-password rejection, logout) and GuestPreviewTest (guest
capped to a preview, authenticated sees everything), all passing.
Also fixed a real bug the tests caught: request()->session() isn't
reliably attached during Livewire's in-process test action calls;
switched Login/Register to the session() helper, which resolves the
session store directly and works in both contexts.
WarcRecordFetcher was splitting out and discarding the HTTP response
headers to get at the HTML body — now it keeps and parses them, so
Server/X-Powered-By headers get captured alongside the <meta
name="generator"> tag already sitting in the parsed DOM. No extra
requests: same WARC segment already fetched for --with-metadata.
Site copies these from its homepage capture, same as
title/description/keywords. New badges on the site detail page show
the detected platform (e.g. "Wix.com Website Builder") and server
signature at a glance.
Verified against real .es data: correctly detected Wix
("00padel.es"), nginx/Apache/Cloudflare server headers, and PHP
X-Powered-By headers across the batch.
Adds a backlinks table: one row per <a href> on a captured page that
points at a different domain (same-site nav links are skipped).
Piggybacks on the existing --with-metadata WARC fetch/DOM-parse in
WarcRecordFetcher rather than a separate command/job, since extracting
links needs the same HTML already being downloaded for
title/description/content — no extra requests.
WarcRecordFetcher gains href resolution (absolute, protocol-relative,
root-relative, query-only, and plain relative hrefs) against the
page's own URL. CommonCrawlIndexService::saveBacklinks() replaces a
page's link set wholesale on each (re)fetch, so reruns stay
idempotent. Site gets outboundLinks()/inboundLinks() relations, the
latter keyed by domain string since a link's target site may not be
indexed yet.
Verified against real .es data: 781 backlinks extracted across 205
distinct target domains (social platforms, Trustpilot, genuine
inter-site links with correct anchor text), and a rerun of the same
fetch left the count unchanged.
Adds Site/SitePage models and the crawlindex:fetch command, which pulls
CDX index records from Common Crawl and optionally enriches them with
page title/description/keywords/body text extracted from the WARC
archive. CDX/WARC access and the site/page upsert logic live in
CommonCrawlIndexService and WarcRecordFetcher.
m105