Sitemap lists the static pages plus one URL per indexed site (644
URLs currently); not sharded — fine at this scale, but the sitemap
protocol caps a single file at 50,000 URLs, so this will need a
sitemap index if the site count ever gets there. robots.txt moved
from a static public file to a route so it can point at the real
sitemap URL in any environment (a static file would also silently
shadow the dynamic route in production, since webservers serve
existing public/ files before hitting Laravel's router at all).
Site detail page: guests now get one prominent banner up top ("you're
viewing limited data — register or sign in to see everything") in
addition to the existing per-section "N of M results" notices, so the
overall gating is obvious immediately rather than only on scroll.
Also gave site pages a real per-page meta description (using the
site's own description/title) instead of falling back to the generic
site-wide one — worth doing now that these pages are actually in the
sitemap and indexable.
Caught and fixed a real Blade bug along the way: an inline
:description="..." expression on <x-layouts.app> with escaped
double-quoted PHP strings corrupted the compiler's directive parsing
in a way that only surfaced as a 500 on a page rendering the guest
@guest block below it (mismatched @if/@endif at compile time) — not
on the expression itself. Fixed by computing the description in a
@php block first and passing a plain variable instead.
Verified over real HTTP: sitemap.xml validates as well-formed XML
(644 <url> entries), robots.txt correctly emits the live sitemap URL,
the guest banner and site-specific meta description both render
correctly. Full test suite (31 tests) passes.
Adds Site/SitePage models and the crawlindex:fetch command, which pulls
CDX index records from Common Crawl and optionally enriches them with
page title/description/keywords/body text extracted from the WARC
archive. CDX/WARC access and the site/page upsert logic live in
CommonCrawlIndexService and WarcRecordFetcher.
m105