Sitemap lists the static pages plus one URL per indexed site (644
URLs currently); not sharded — fine at this scale, but the sitemap
protocol caps a single file at 50,000 URLs, so this will need a
sitemap index if the site count ever gets there. robots.txt moved
from a static public file to a route so it can point at the real
sitemap URL in any environment (a static file would also silently
shadow the dynamic route in production, since webservers serve
existing public/ files before hitting Laravel's router at all).
Site detail page: guests now get one prominent banner up top ("you're
viewing limited data — register or sign in to see everything") in
addition to the existing per-section "N of M results" notices, so the
overall gating is obvious immediately rather than only on scroll.
Also gave site pages a real per-page meta description (using the
site's own description/title) instead of falling back to the generic
site-wide one — worth doing now that these pages are actually in the
sitemap and indexable.
Caught and fixed a real Blade bug along the way: an inline
:description="..." expression on <x-layouts.app> with escaped
double-quoted PHP strings corrupted the compiler's directive parsing
in a way that only surfaced as a 500 on a page rendering the guest
@guest block below it (mismatched @if/@endif at compile time) — not
on the expression itself. Fixed by computing the description in a
@php block first and passing a plain variable instead.
Verified over real HTTP: sitemap.xml validates as well-formed XML
(644 <url> entries), robots.txt correctly emits the live sitemap URL,
the guest banner and site-specific meta description both render
correctly. Full test suite (31 tests) passes.
Adds sites.root_domain (indexed string, backfilled for existing rows)
computed via a naive "last two labels" heuristic (App\Support\RootDomain)
— good enough for example.com/somesite.es, but wrong for two-part
public suffixes like .co.uk. Doing that correctly needs the Mozilla
Public Suffix List; flagged as a known simplification rather than
built now.
Deliberately not a hard FK: like Backlink.target_domain, a subdomain
can get indexed before its root domain does (or vice versa), so
Site::subdomains()/rootSite() match on the domain string rather than
requiring the relationship to exist at write time.
Site detail page: a root domain shows a paginated "Subdomains"
section (same guest-preview cap as Pages/Backlinks); a subdomain
shows "Subdomain of {root}" linking back, when the root is itself
indexed.
Verified against real data: 21k.ro correctly lists its two known
subdomains (buzau.21k.ro, timisoara.21k.ro), each of which correctly
links back to 21k.ro. Added SubdomainsTest (root lists subdomains,
subdomain links to root, empty state) — full suite (31 tests) passes.
Adds the auth Breeze scaffolds that were missing before: password
reset and email verification, plus a profile page (update name/email,
change password, delete account). User now implements MustVerifyEmail
so verification is actually enforced, not just scaffolded.
This does mean accepting Tailwind v3 (postcss/tailwind.config.js)
over the v4 CSS-first setup the app started with, and Livewire drops
from ^4.3 to ^3.6.4 — both forced by Breeze/Volt's current
compatibility requirements. Restored the Instrument Sans font Breeze's
installer dropped so styling stays close to what we had.
Reconciliation, since Breeze's installer assumes a blank-slate app:
- Deleted the hand-rolled Login/Register Livewire components/views/
test — superseded by Breeze's Volt-based auth pages
(resources/views/livewire/pages/auth/*), which also cover
forgot-password, reset-password, verify-email, and confirm-password.
- Deleted Breeze's own default welcome/dashboard views and its
separate app layout + nav component — kept our existing
components/layouts/app.blade.php as the one real app layout, used
Breeze's minimal layouts/guest.blade.php only for the auth pages.
- Rewrote routes/web.php merging our routes (home, sites, discover)
with routes/auth.php; replaced every hardcoded route('dashboard')
reference (5 Blade files plus, easy to miss, a plain PHP controller
VerifyEmailController) with sites.index.
- Logout: Breeze's Actions\Logout::__invoke() returns void, so wired
directly as a route it silently no-ops instead of redirecting;
fixed with a thin closure that calls it then redirects.
- Fixed Breeze's own generated tests (Auth/AuthenticationTest,
RegistrationTest, EmailVerificationTest, PasswordConfirmationTest)
where they assumed the deleted dashboard route/nav component.
Verified: full suite (28 tests, including Breeze's Registration/
Authentication/PasswordReset/PasswordConfirmation/PasswordUpdate/
EmailVerification/Profile tests plus our own GuestPreviewTest) passes
against the huginn_testing MySQL database. Also smoke-tested for real:
all key pages return 200, /profile redirects guests to /login, guest
preview capping still works, and a real password-reset email was
generated and logged with correct branding and a working link.
Hand-rolled login/register/logout on Laravel's built-in Auth facade
rather than Breeze/Jetstream, which scaffold Tailwind v3 config and
would fight our existing Tailwind v4 CSS-first setup. Login/Register
are Livewire components; logout is a plain POST route (auth-gated,
CSRF-protected).
Routing: "/" is now a public landing page describing the project;
the sites list moved from "/" to "/sites" to make room for it.
Data gating: App\Support\GuestPreview::paginate() forces page 1 and a
5-row page size for unauthenticated visitors on Sites, Discover, and
a site's Pages/Backlinks tables, while total() still reflects the
real match count — authenticated users get the existing full
pagination unchanged. A shared guest-preview-notice component shows
"N of M results, sign in to see all" and only renders when something
is actually being hidden.
Testing: pointed the test suite at a dedicated MySQL database
(huginn_testing) instead of the default in-memory SQLite, since the
existing fulltext index migration isn't SQLite-compatible — this app
is MySQL-specific by now regardless. Added AuthFlowTest (register,
login, wrong-password rejection, logout) and GuestPreviewTest (guest
capped to a preview, authenticated sees everything), all passing.
Also fixed a real bug the tests caught: request()->session() isn't
reliably attached during Livewire's in-process test action calls;
switched Login/Register to the session() helper, which resolves the
session store directly and works in both contexts.
The CDX languages field (ISO 639 codes, e.g. "spa,eng") has been
captured on every page since the first ingestion command, but never
surfaced anywhere. Filters via FIND_IN_SET for an exact code match
rather than a LIKE substring match, since the column holds a
comma-separated list.
Verified against real data: lang=spa returns exactly the expected 47
sites (25+22 across two pages), a bogus code renders the empty state,
and lang=spa combined with tld=es correctly narrows to 20.
Filters sites by server/powered_by/generator (e.g. "Shopify",
"WordPress", "nginx"), combinable with the existing keyword/TLD
filters. Results table now also shows each site's detected platform.
Verified: platform=Wix correctly isolates the one known Wix site,
tld=es+platform=nginx narrows to the 12 real nginx-hosted .es sites,
and the no-match case renders the empty state correctly.
WarcRecordFetcher was splitting out and discarding the HTTP response
headers to get at the HTML body — now it keeps and parses them, so
Server/X-Powered-By headers get captured alongside the <meta
name="generator"> tag already sitting in the parsed DOM. No extra
requests: same WARC segment already fetched for --with-metadata.
Site copies these from its homepage capture, same as
title/description/keywords. New badges on the site detail page show
the detected platform (e.g. "Wix.com Website Builder") and server
signature at a glance.
Verified against real .es data: correctly detected Wix
("00padel.es"), nginx/Apache/Cloudflare server headers, and PHP
X-Powered-By headers across the batch.
Shows inbound links (who links to this site, via Site::inboundLinks())
with the referring site, source page, and anchor text — paginated
separately from the Pages table via a distinct page-name query param
so both can paginate independently. Also surfaces a simple outbound
link count and a "N backlinks from M referring domains" summary.
Verified against real data: crawled podcastyradio.es (previously only
known as a backlink target from 00podcast.es) and confirmed its detail
page correctly shows that inbound link with the right anchor text.
Adds a backlinks table: one row per <a href> on a captured page that
points at a different domain (same-site nav links are skipped).
Piggybacks on the existing --with-metadata WARC fetch/DOM-parse in
WarcRecordFetcher rather than a separate command/job, since extracting
links needs the same HTML already being downloaded for
title/description/content — no extra requests.
WarcRecordFetcher gains href resolution (absolute, protocol-relative,
root-relative, query-only, and plain relative hrefs) against the
page's own URL. CommonCrawlIndexService::saveBacklinks() replaces a
page's link set wholesale on each (re)fetch, so reruns stay
idempotent. Site gets outboundLinks()/inboundLinks() relations, the
latter keyed by domain string since a link's target site may not be
indexed yet.
Verified against real .es data: 781 backlinks extracted across 205
distinct target domains (social platforms, Trustpilot, genuine
inter-site links with correct anchor text), and a rerun of the same
fetch left the count unchanged.
The actual "who sells honey with a .es domain" feature: search sites
by keyword — matched against page title/description/keywords/body
text via a new MySQL fulltext index on site_pages — combined with an
optional TLD filter on the domain. Results are sites ranked by number
of matching pages. Shows nothing until a filter is entered, so it
doesn't just duplicate the plain Sites browse page.
Verified against real .es data pulled in for the test (dominio ->
4 matching sites, correctly ranked and TLD-filtered).
Home page is now a Livewire-powered, searchable/paginated Site list
(search by domain) instead of the stock Laravel welcome page. Clicking
a site goes to a plain controller-rendered detail page listing its
pages. Shared layout at resources/views/components/layouts/app.blade.php.
crawlindex:bulk-fetch walks Common Crawl's CDX shard files directly on
data.commoncrawl.org (via cc-index.paths.gz), instead of the
index.commoncrawl.org query API — usable for bulk backfill/refresh or
as a fallback when that API is down. Shares the group/save pipeline
with crawlindex:fetch via a new SavesCommonCrawlCaptures trait.
Also fixes a real bug this surfaced under load: site_pages.urlkey was
a varchar(255), which is too narrow for real-world captured URLs
(long query strings on spam/malformed pages overflow it). Widened to
text and switched the uniqueness constraint to a sha1 hash column,
since MySQL can't fully index a TEXT column.
Adds Site/SitePage models and the crawlindex:fetch command, which pulls
CDX index records from Common Crawl and optionally enriches them with
page title/description/keywords/body text extracted from the WARC
archive. CDX/WARC access and the site/page upsert logic live in
CommonCrawlIndexService and WarcRecordFetcher.
m105