Adds sites.root_domain (indexed string, backfilled for existing rows)
computed via a naive "last two labels" heuristic (App\Support\RootDomain)
— good enough for example.com/somesite.es, but wrong for two-part
public suffixes like .co.uk. Doing that correctly needs the Mozilla
Public Suffix List; flagged as a known simplification rather than
built now.
Deliberately not a hard FK: like Backlink.target_domain, a subdomain
can get indexed before its root domain does (or vice versa), so
Site::subdomains()/rootSite() match on the domain string rather than
requiring the relationship to exist at write time.
Site detail page: a root domain shows a paginated "Subdomains"
section (same guest-preview cap as Pages/Backlinks); a subdomain
shows "Subdomain of {root}" linking back, when the root is itself
indexed.
Verified against real data: 21k.ro correctly lists its two known
subdomains (buzau.21k.ro, timisoara.21k.ro), each of which correctly
links back to 21k.ro. Added SubdomainsTest (root lists subdomains,
subdomain links to root, empty state) — full suite (31 tests) passes.
Adds the auth Breeze scaffolds that were missing before: password
reset and email verification, plus a profile page (update name/email,
change password, delete account). User now implements MustVerifyEmail
so verification is actually enforced, not just scaffolded.
This does mean accepting Tailwind v3 (postcss/tailwind.config.js)
over the v4 CSS-first setup the app started with, and Livewire drops
from ^4.3 to ^3.6.4 — both forced by Breeze/Volt's current
compatibility requirements. Restored the Instrument Sans font Breeze's
installer dropped so styling stays close to what we had.
Reconciliation, since Breeze's installer assumes a blank-slate app:
- Deleted the hand-rolled Login/Register Livewire components/views/
test — superseded by Breeze's Volt-based auth pages
(resources/views/livewire/pages/auth/*), which also cover
forgot-password, reset-password, verify-email, and confirm-password.
- Deleted Breeze's own default welcome/dashboard views and its
separate app layout + nav component — kept our existing
components/layouts/app.blade.php as the one real app layout, used
Breeze's minimal layouts/guest.blade.php only for the auth pages.
- Rewrote routes/web.php merging our routes (home, sites, discover)
with routes/auth.php; replaced every hardcoded route('dashboard')
reference (5 Blade files plus, easy to miss, a plain PHP controller
VerifyEmailController) with sites.index.
- Logout: Breeze's Actions\Logout::__invoke() returns void, so wired
directly as a route it silently no-ops instead of redirecting;
fixed with a thin closure that calls it then redirects.
- Fixed Breeze's own generated tests (Auth/AuthenticationTest,
RegistrationTest, EmailVerificationTest, PasswordConfirmationTest)
where they assumed the deleted dashboard route/nav component.
Verified: full suite (28 tests, including Breeze's Registration/
Authentication/PasswordReset/PasswordConfirmation/PasswordUpdate/
EmailVerification/Profile tests plus our own GuestPreviewTest) passes
against the huginn_testing MySQL database. Also smoke-tested for real:
all key pages return 200, /profile redirects guests to /login, guest
preview capping still works, and a real password-reset email was
generated and logged with correct branding and a working link.
WarcRecordFetcher was splitting out and discarding the HTTP response
headers to get at the HTML body — now it keeps and parses them, so
Server/X-Powered-By headers get captured alongside the <meta
name="generator"> tag already sitting in the parsed DOM. No extra
requests: same WARC segment already fetched for --with-metadata.
Site copies these from its homepage capture, same as
title/description/keywords. New badges on the site detail page show
the detected platform (e.g. "Wix.com Website Builder") and server
signature at a glance.
Verified against real .es data: correctly detected Wix
("00padel.es"), nginx/Apache/Cloudflare server headers, and PHP
X-Powered-By headers across the batch.
Adds a backlinks table: one row per <a href> on a captured page that
points at a different domain (same-site nav links are skipped).
Piggybacks on the existing --with-metadata WARC fetch/DOM-parse in
WarcRecordFetcher rather than a separate command/job, since extracting
links needs the same HTML already being downloaded for
title/description/content — no extra requests.
WarcRecordFetcher gains href resolution (absolute, protocol-relative,
root-relative, query-only, and plain relative hrefs) against the
page's own URL. CommonCrawlIndexService::saveBacklinks() replaces a
page's link set wholesale on each (re)fetch, so reruns stay
idempotent. Site gets outboundLinks()/inboundLinks() relations, the
latter keyed by domain string since a link's target site may not be
indexed yet.
Verified against real .es data: 781 backlinks extracted across 205
distinct target domains (social platforms, Trustpilot, genuine
inter-site links with correct anchor text), and a rerun of the same
fetch left the count unchanged.
crawlindex:bulk-fetch walks Common Crawl's CDX shard files directly on
data.commoncrawl.org (via cc-index.paths.gz), instead of the
index.commoncrawl.org query API — usable for bulk backfill/refresh or
as a fallback when that API is down. Shares the group/save pipeline
with crawlindex:fetch via a new SavesCommonCrawlCaptures trait.
Also fixes a real bug this surfaced under load: site_pages.urlkey was
a varchar(255), which is too narrow for real-world captured URLs
(long query strings on spam/malformed pages overflow it). Widened to
text and switched the uniqueness constraint to a sha1 hash column,
since MySQL can't fully index a TEXT column.
Adds Site/SitePage models and the crawlindex:fetch command, which pulls
CDX index records from Common Crawl and optionally enriches them with
page title/description/keywords/body text extracted from the WARC
archive. CDX/WARC access and the site/page upsert logic live in
CommonCrawlIndexService and WarcRecordFetcher.
m105