Initial commit: Laravel scaffold + Common Crawl site indexing
Adds Site/SitePage models and the crawlindex:fetch command, which pulls CDX index records from Common Crawl and optionally enriches them with page title/description/keywords/body text extracted from the WARC archive. CDX/WARC access and the site/page upsert logic live in CommonCrawlIndexService and WarcRecordFetcher.
This commit is contained in:
@@ -0,0 +1,18 @@
|
||||
#!/usr/bin/env php
|
||||
<?php
|
||||
|
||||
use Illuminate\Foundation\Application;
|
||||
use Symfony\Component\Console\Input\ArgvInput;
|
||||
|
||||
define('LARAVEL_START', microtime(true));
|
||||
|
||||
// Register the Composer autoloader...
|
||||
require __DIR__.'/vendor/autoload.php';
|
||||
|
||||
// Bootstrap Laravel and handle the command...
|
||||
/** @var Application $app */
|
||||
$app = require_once __DIR__.'/bootstrap/app.php';
|
||||
|
||||
$status = $app->handleCommand(new ArgvInput);
|
||||
|
||||
exit($status);
|
||||
Reference in New Issue
Block a user