Initial commit: Laravel scaffold + Common Crawl site indexing
Adds Site/SitePage models and the crawlindex:fetch command, which pulls CDX index records from Common Crawl and optionally enriches them with page title/description/keywords/body text extracted from the WARC archive. CDX/WARC access and the site/page upsert logic live in CommonCrawlIndexService and WarcRecordFetcher.
This commit is contained in:
@@ -0,0 +1,11 @@
|
||||
* text=auto eol=lf
|
||||
|
||||
*.blade.php diff=html
|
||||
*.css diff=css
|
||||
*.html diff=html
|
||||
*.md diff=markdown
|
||||
*.php diff=php
|
||||
|
||||
/.github export-ignore
|
||||
CHANGELOG.md export-ignore
|
||||
.styleci.yml export-ignore
|
||||
Reference in New Issue
Block a user