Add bulk CDX shard ingestion command as an API-independent fallback

crawlindex:bulk-fetch walks Common Crawl's CDX shard files directly on
data.commoncrawl.org (via cc-index.paths.gz), instead of the
index.commoncrawl.org query API — usable for bulk backfill/refresh or
as a fallback when that API is down. Shares the group/save pipeline
with crawlindex:fetch via a new SavesCommonCrawlCaptures trait.

Also fixes a real bug this surfaced under load: site_pages.urlkey was
a varchar(255), which is too narrow for real-world captured URLs
(long query strings on spam/malformed pages overflow it). Widened to
text and switched the uniqueness constraint to a sha1 hash column,
since MySQL can't fully index a TEXT column.

This commit is contained in:

m105

2026-07-20 00:26:27 +03:00

parent 1c9fc5d423

commit 4dc9b84eb8

6 changed files with 324 additions and 24 deletions

									
										app/Models/SitePage.php
									
		+1
		
												View File
												
				@@ -10,6 +10,7 @@ class SitePage extends Model

				    protected $fillable = [

				        'site_id',

				        'urlkey',

				        'urlkey_hash',

				        'url',

				        'timestamp',

				        'mime',