Initial commit: Laravel scaffold + Common Crawl site indexing
Adds Site/SitePage models and the crawlindex:fetch command, which pulls CDX index records from Common Crawl and optionally enriches them with page title/description/keywords/body text extracted from the WARC archive. CDX/WARC access and the site/page upsert logic live in CommonCrawlIndexService and WarcRecordFetcher.
This commit is contained in:
+27
@@ -0,0 +1,27 @@
|
||||
*.log
|
||||
.DS_Store
|
||||
.env
|
||||
.env.backup
|
||||
.env.production
|
||||
.phpactor.json
|
||||
.phpunit.result.cache
|
||||
/.codex
|
||||
/.cursor/
|
||||
/.idea
|
||||
/.nova
|
||||
/.phpunit.cache
|
||||
/.vscode
|
||||
/.zed
|
||||
/auth.json
|
||||
/node_modules
|
||||
/public/build
|
||||
/public/fonts-manifest.dev.json
|
||||
/public/hot
|
||||
/public/storage
|
||||
/storage/*.key
|
||||
/storage/pail
|
||||
/vendor
|
||||
_ide_helper.php
|
||||
Homestead.json
|
||||
Homestead.yaml
|
||||
Thumbs.db
|
||||
Reference in New Issue
Block a user