Initial commit: Laravel scaffold + Common Crawl site indexing
Adds Site/SitePage models and the crawlindex:fetch command, which pulls CDX index records from Common Crawl and optionally enriches them with page title/description/keywords/body text extracted from the WARC archive. CDX/WARC access and the site/page upsert logic live in CommonCrawlIndexService and WarcRecordFetcher.
This commit is contained in:
@@ -0,0 +1,25 @@
|
||||
<IfModule mod_rewrite.c>
|
||||
<IfModule mod_negotiation.c>
|
||||
Options -MultiViews -Indexes
|
||||
</IfModule>
|
||||
|
||||
RewriteEngine On
|
||||
|
||||
# Handle Authorization Header
|
||||
RewriteCond %{HTTP:Authorization} .
|
||||
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
|
||||
|
||||
# Handle X-XSRF-Token Header
|
||||
RewriteCond %{HTTP:x-xsrf-token} .
|
||||
RewriteRule .* - [E=HTTP_X_XSRF_TOKEN:%{HTTP:X-XSRF-Token}]
|
||||
|
||||
# Redirect Trailing Slashes If Not A Folder...
|
||||
RewriteCond %{REQUEST_FILENAME} !-d
|
||||
RewriteCond %{REQUEST_URI} (.+)/$
|
||||
RewriteRule ^ %1 [L,R=301]
|
||||
|
||||
# Send Requests To Front Controller...
|
||||
RewriteCond %{REQUEST_FILENAME} !-d
|
||||
RewriteCond %{REQUEST_FILENAME} !-f
|
||||
RewriteRule ^ index.php [L]
|
||||
</IfModule>
|
||||
Reference in New Issue
Block a user