m105
1c9fc5d423
Adds Site/SitePage models and the crawlindex:fetch command, which pulls CDX index records from Common Crawl and optionally enriches them with page title/description/keywords/body text extracted from the WARC archive. CDX/WARC access and the site/page upsert logic live in CommonCrawlIndexService and WarcRecordFetcher.
12 lines
186 B
Plaintext
12 lines
186 B
Plaintext
* text=auto eol=lf
|
|
|
|
*.blade.php diff=html
|
|
*.css diff=css
|
|
*.html diff=html
|
|
*.md diff=markdown
|
|
*.php diff=php
|
|
|
|
/.github export-ignore
|
|
CHANGELOG.md export-ignore
|
|
.styleci.yml export-ignore
|