Add tech-stack fingerprinting (server, powered-by, generator)

WarcRecordFetcher was splitting out and discarding the HTTP response
headers to get at the HTML body — now it keeps and parses them, so
Server/X-Powered-By headers get captured alongside the <meta
name="generator"> tag already sitting in the parsed DOM. No extra
requests: same WARC segment already fetched for --with-metadata.

Site copies these from its homepage capture, same as
title/description/keywords. New badges on the site detail page show
the detected platform (e.g. "Wix.com Website Builder") and server
signature at a glance.

Verified against real .es data: correctly detected Wix
("00padel.es"), nginx/Apache/Cloudflare server headers, and PHP
X-Powered-By headers across the batch.
This commit is contained in:
m105
2026-07-22 22:14:40 +03:00
parent 6e6be7d7c6
commit c7299fe928
6 changed files with 112 additions and 9 deletions
@@ -125,6 +125,9 @@ class CommonCrawlIndexService
'title' => $content['title'],
'description' => $content['description'],
'keywords' => $content['keywords'],
'server' => $content['server'],
'powered_by' => $content['powered_by'],
'generator' => $content['generator'],
]);
}
}