Add bulk-run actions and per-source rate limiting

Run-all buttons (per target and per investigation) queue every applicable,
not-already-in-flight source via a new FindingDispatcher, re-attempting
prior failures automatically. RunFinding now applies a per-source
RateLimited queue middleware so a bulk run can't hammer the free public
APIs (ip-api, crt.sh, whois, sherlock, theharvester) it depends on;
$tries raised to 30 to survive rate-limiter releases, with a failed()
handler so exhausted retries still resolve the Finding instead of leaving
it stuck pending.

Also fixes CrtShRunner treating crt.sh's intermittent 404/502 (a known
flaky free service) as fatal on the first try — added a bounded retry.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
cedry2k1992
2026-07-20 11:29:26 +03:00
co-authored by Claude Sonnet 5
parent 853686f3f3
commit e4fa57c3d6
13 changed files with 387 additions and 5 deletions
+37 -1
View File
@@ -6,18 +6,41 @@ use App\Models\Finding;
use App\Osint\RunnerRegistry;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Queue\Middleware\RateLimited;
class RunFinding implements ShouldQueue
{
use Queueable;
public int $tries = 1;
/**
* Business-logic failures (a runner throwing) are caught in handle()
* below and never bubble up to the queue, so this only needs to cover
* genuine retries: rate-limiter releases (see middleware()) and worker
* crashes. A bulk run against a busy investigation can easily queue
* dozens of same-source lookups behind one rate limit, so this is set
* high enough to ride that out rather than give up mid-queue.
*/
public int $tries = 30;
public function __construct(
public readonly int $findingId,
) {
}
/**
* @return array<int, object>
*/
public function middleware(): array
{
$sourceKey = Finding::find($this->findingId)?->source?->key;
if ($sourceKey === null) {
return [];
}
return [new RateLimited('osint-runner:'.$sourceKey)];
}
public function handle(RunnerRegistry $registry): void
{
$finding = Finding::with(['target', 'source'])->findOrFail($this->findingId);
@@ -41,4 +64,17 @@ class RunFinding implements ShouldQueue
]);
}
}
/**
* Only reached if the job itself fails outside handle()'s own
* try/catch — e.g. retries exhausted while rate-limited.
*/
public function failed(\Throwable $exception): void
{
Finding::where('id', $this->findingId)->update([
'status' => 'failed',
'error' => 'Gave up after retries: '.$exception->getMessage(),
'ran_at' => now(),
]);
}
}