Initial commit: Laravel scaffold + Common Crawl site indexing

Adds Site/SitePage models and the crawlindex:fetch command, which pulls
CDX index records from Common Crawl and optionally enriches them with
page title/description/keywords/body text extracted from the WARC
archive. CDX/WARC access and the site/page upsert logic live in
CommonCrawlIndexService and WarcRecordFetcher.
This commit is contained in:
m105
2026-07-20 00:04:45 +03:00
commit 1c9fc5d423
64 changed files with 11338 additions and 0 deletions
+16
View File
@@ -0,0 +1,16 @@
{
"$schema": "https://www.schemastore.org/package.json",
"private": true,
"type": "module",
"scripts": {
"build": "vite build",
"dev": "vite"
},
"devDependencies": {
"@tailwindcss/vite": "^4.0.0",
"concurrently": "^9.0.1",
"laravel-vite-plugin": "^3.1",
"tailwindcss": "^4.0.0",
"vite": "^8.0.0"
}
}