Initial commit: Laravel scaffold + Common Crawl site indexing
Adds Site/SitePage models and the crawlindex:fetch command, which pulls CDX index records from Common Crawl and optionally enriches them with page title/description/keywords/body text extracted from the WARC archive. CDX/WARC access and the site/page upsert logic live in CommonCrawlIndexService and WarcRecordFetcher.
This commit is contained in:
@@ -0,0 +1,8 @@
|
||||
<?php
|
||||
|
||||
use Illuminate\Foundation\Inspiring;
|
||||
use Illuminate\Support\Facades\Artisan;
|
||||
|
||||
Artisan::command('inspire', function () {
|
||||
$this->comment(Inspiring::quote());
|
||||
})->purpose('Display an inspiring quote');
|
||||
Reference in New Issue
Block a user