Adds the auth Breeze scaffolds that were missing before: password
reset and email verification, plus a profile page (update name/email,
change password, delete account). User now implements MustVerifyEmail
so verification is actually enforced, not just scaffolded.
This does mean accepting Tailwind v3 (postcss/tailwind.config.js)
over the v4 CSS-first setup the app started with, and Livewire drops
from ^4.3 to ^3.6.4 — both forced by Breeze/Volt's current
compatibility requirements. Restored the Instrument Sans font Breeze's
installer dropped so styling stays close to what we had.
Reconciliation, since Breeze's installer assumes a blank-slate app:
- Deleted the hand-rolled Login/Register Livewire components/views/
test — superseded by Breeze's Volt-based auth pages
(resources/views/livewire/pages/auth/*), which also cover
forgot-password, reset-password, verify-email, and confirm-password.
- Deleted Breeze's own default welcome/dashboard views and its
separate app layout + nav component — kept our existing
components/layouts/app.blade.php as the one real app layout, used
Breeze's minimal layouts/guest.blade.php only for the auth pages.
- Rewrote routes/web.php merging our routes (home, sites, discover)
with routes/auth.php; replaced every hardcoded route('dashboard')
reference (5 Blade files plus, easy to miss, a plain PHP controller
VerifyEmailController) with sites.index.
- Logout: Breeze's Actions\Logout::__invoke() returns void, so wired
directly as a route it silently no-ops instead of redirecting;
fixed with a thin closure that calls it then redirects.
- Fixed Breeze's own generated tests (Auth/AuthenticationTest,
RegistrationTest, EmailVerificationTest, PasswordConfirmationTest)
where they assumed the deleted dashboard route/nav component.
Verified: full suite (28 tests, including Breeze's Registration/
Authentication/PasswordReset/PasswordConfirmation/PasswordUpdate/
EmailVerification/Profile tests plus our own GuestPreviewTest) passes
against the huginn_testing MySQL database. Also smoke-tested for real:
all key pages return 200, /profile redirects guests to /login, guest
preview capping still works, and a real password-reset email was
generated and logged with correct branding and a working link.
Hand-rolled login/register/logout on Laravel's built-in Auth facade
rather than Breeze/Jetstream, which scaffold Tailwind v3 config and
would fight our existing Tailwind v4 CSS-first setup. Login/Register
are Livewire components; logout is a plain POST route (auth-gated,
CSRF-protected).
Routing: "/" is now a public landing page describing the project;
the sites list moved from "/" to "/sites" to make room for it.
Data gating: App\Support\GuestPreview::paginate() forces page 1 and a
5-row page size for unauthenticated visitors on Sites, Discover, and
a site's Pages/Backlinks tables, while total() still reflects the
real match count — authenticated users get the existing full
pagination unchanged. A shared guest-preview-notice component shows
"N of M results, sign in to see all" and only renders when something
is actually being hidden.
Testing: pointed the test suite at a dedicated MySQL database
(huginn_testing) instead of the default in-memory SQLite, since the
existing fulltext index migration isn't SQLite-compatible — this app
is MySQL-specific by now regardless. Added AuthFlowTest (register,
login, wrong-password rejection, logout) and GuestPreviewTest (guest
capped to a preview, authenticated sees everything), all passing.
Also fixed a real bug the tests caught: request()->session() isn't
reliably attached during Livewire's in-process test action calls;
switched Login/Register to the session() helper, which resolves the
session store directly and works in both contexts.
The CDX languages field (ISO 639 codes, e.g. "spa,eng") has been
captured on every page since the first ingestion command, but never
surfaced anywhere. Filters via FIND_IN_SET for an exact code match
rather than a LIKE substring match, since the column holds a
comma-separated list.
Verified against real data: lang=spa returns exactly the expected 47
sites (25+22 across two pages), a bogus code renders the empty state,
and lang=spa combined with tld=es correctly narrows to 20.
The rel attribute has been captured on every backlink since that
table was added, but never displayed — a dofollow vs. nofollow
backlink is a real difference for SEO/competitive analysis, not just
a link count. Links with none of those tokens show as "dofollow".
Verified against real data (existing noopener-only link renders as
dofollow) plus a temporary synthetic nofollow+sponsored row to
confirm that branch too, removed after confirming.
Filters sites by server/powered_by/generator (e.g. "Shopify",
"WordPress", "nginx"), combinable with the existing keyword/TLD
filters. Results table now also shows each site's detected platform.
Verified: platform=Wix correctly isolates the one known Wix site,
tld=es+platform=nginx narrows to the 12 real nginx-hosted .es sites,
and the no-match case renders the empty state correctly.
WarcRecordFetcher was splitting out and discarding the HTTP response
headers to get at the HTML body — now it keeps and parses them, so
Server/X-Powered-By headers get captured alongside the <meta
name="generator"> tag already sitting in the parsed DOM. No extra
requests: same WARC segment already fetched for --with-metadata.
Site copies these from its homepage capture, same as
title/description/keywords. New badges on the site detail page show
the detected platform (e.g. "Wix.com Website Builder") and server
signature at a glance.
Verified against real .es data: correctly detected Wix
("00padel.es"), nginx/Apache/Cloudflare server headers, and PHP
X-Powered-By headers across the batch.
Shows inbound links (who links to this site, via Site::inboundLinks())
with the referring site, source page, and anchor text — paginated
separately from the Pages table via a distinct page-name query param
so both can paginate independently. Also surfaces a simple outbound
link count and a "N backlinks from M referring domains" summary.
Verified against real data: crawled podcastyradio.es (previously only
known as a backlink target from 00podcast.es) and confirmed its detail
page correctly shows that inbound link with the right anchor text.
Adds a backlinks table: one row per <a href> on a captured page that
points at a different domain (same-site nav links are skipped).
Piggybacks on the existing --with-metadata WARC fetch/DOM-parse in
WarcRecordFetcher rather than a separate command/job, since extracting
links needs the same HTML already being downloaded for
title/description/content — no extra requests.
WarcRecordFetcher gains href resolution (absolute, protocol-relative,
root-relative, query-only, and plain relative hrefs) against the
page's own URL. CommonCrawlIndexService::saveBacklinks() replaces a
page's link set wholesale on each (re)fetch, so reruns stay
idempotent. Site gets outboundLinks()/inboundLinks() relations, the
latter keyed by domain string since a link's target site may not be
indexed yet.
Verified against real .es data: 781 backlinks extracted across 205
distinct target domains (social platforms, Trustpilot, genuine
inter-site links with correct anchor text), and a rerun of the same
fetch left the count unchanged.
The actual "who sells honey with a .es domain" feature: search sites
by keyword — matched against page title/description/keywords/body
text via a new MySQL fulltext index on site_pages — combined with an
optional TLD filter on the domain. Results are sites ranked by number
of matching pages. Shows nothing until a filter is entered, so it
doesn't just duplicate the plain Sites browse page.
Verified against real .es data pulled in for the test (dominio ->
4 matching sites, correctly ranked and TLD-filtered).
Home page is now a Livewire-powered, searchable/paginated Site list
(search by domain) instead of the stock Laravel welcome page. Clicking
a site goes to a plain controller-rendered detail page listing its
pages. Shared layout at resources/views/components/layouts/app.blade.php.
Explains what Huginn does (Common Crawl-backed site index for
competitor discovery), the two ingestion paths (query API vs bulk CDX
shard download), the Site/SitePage data model, and how to set up and
run the app locally.
crawlindex:bulk-fetch walks Common Crawl's CDX shard files directly on
data.commoncrawl.org (via cc-index.paths.gz), instead of the
index.commoncrawl.org query API — usable for bulk backfill/refresh or
as a fallback when that API is down. Shares the group/save pipeline
with crawlindex:fetch via a new SavesCommonCrawlCaptures trait.
Also fixes a real bug this surfaced under load: site_pages.urlkey was
a varchar(255), which is too narrow for real-world captured URLs
(long query strings on spam/malformed pages overflow it). Widened to
text and switched the uniqueness constraint to a sha1 hash column,
since MySQL can't fully index a TEXT column.
Adds Site/SitePage models and the crawlindex:fetch command, which pulls
CDX index records from Common Crawl and optionally enriches them with
page title/description/keywords/body text extracted from the WARC
archive. CDX/WARC access and the site/page upsert logic live in
CommonCrawlIndexService and WarcRecordFetcher.
m105