Initial commit: Laravel scaffold + Common Crawl site indexing
Adds Site/SitePage models and the crawlindex:fetch command, which pulls CDX index records from Common Crawl and optionally enriches them with page title/description/keywords/body text extracted from the WARC archive. CDX/WARC access and the site/page upsert logic live in CommonCrawlIndexService and WarcRecordFetcher.
This commit is contained in:
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"$schema": "https://www.schemastore.org/package.json",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"build": "vite build",
|
||||
"dev": "vite"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@tailwindcss/vite": "^4.0.0",
|
||||
"concurrently": "^9.0.1",
|
||||
"laravel-vite-plugin": "^3.1",
|
||||
"tailwindcss": "^4.0.0",
|
||||
"vite": "^8.0.0"
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user