Add bulk-run actions and per-source rate limiting

Run-all buttons (per target and per investigation) queue every applicable,
not-already-in-flight source via a new FindingDispatcher, re-attempting
prior failures automatically. RunFinding now applies a per-source
RateLimited queue middleware so a bulk run can't hammer the free public
APIs (ip-api, crt.sh, whois, sherlock, theharvester) it depends on;
$tries raised to 30 to survive rate-limiter releases, with a failed()
handler so exhausted retries still resolve the Finding instead of leaving
it stuck pending.

Also fixes CrtShRunner treating crt.sh's intermittent 404/502 (a known
flaky free service) as fatal on the first try — added a bounded retry.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
cedry2k1992
2026-07-20 11:29:26 +03:00
co-authored by Claude Sonnet 5
parent 853686f3f3
commit e4fa57c3d6
13 changed files with 387 additions and 5 deletions
+50
View File
@@ -160,6 +160,56 @@ commands).
worker process directly and confirmed systemd auto-restarted it
within seconds. See `deploy/systemd/README.md` for install/operate
commands.
9. **Bulk actions + rate limiting** (done) — the original 8 phases
were all complete; this closed the biggest remaining workflow
friction (running each source against each target one at a time).
- `App\Osint\FindingDispatcher::dispatchAllApplicable()` — queues
every enabled, registered source that applies to a target's type
and doesn't already have a `pending`/`running` finding for it
(a `completed`/`failed` one *is* re-queued, so bulk-running
naturally retries prior failures). Shared by a "Run all" button
per target and a "Run all targets" button per investigation.
- Per-source rate limiting via Laravel's `RateLimited` queue
middleware (`RunFinding::middleware()`), limits defined in
`AppServiceProvider::boot()`: ip-api.com 45/min (their published
limit), crt.sh 30/min, WHOIS 20/min, Sherlock 10/min,
theHarvester 5/min (all courtesy limits of ours, protecting the
free infra these runners depend on from a bulk run hammering it).
- `RunFinding::$tries` raised from 1 to 30: a rate-limiter release
counts as a queue attempt, and a bulk run can easily queue dozens
of same-source lookups behind one rate-limit window — needs
enough attempts to ride that out. Added `RunFinding::failed()` so
a job that does exhaust retries still marks its `Finding` as
`failed` with a clear error instead of leaving it stuck at
`pending` forever. (Business-logic failures — a runner throwing —
are unaffected; `handle()` already catches those internally and
never lets them bubble up to Laravel's retry mechanism.)
- Bug found and fixed via this work: `CrtShRunner` treated any
non-2xx as fatal, but crt.sh (a free, frequently overloaded
public service) intermittently returns 404/502/503/timeouts for
queries that succeed moments later — confirmed by hand, same
query, 404 then 200 with no code changes in between. Added
`->retry(3, 2000, throw: false)`. This doesn't fully solve it —
hit a sustained bad patch during testing where even 3 retries
over ~6s didn't survive it — but it's real instability on crt.sh's
end, not something worth over-engineering client-side retries
around; the failure path (clear error, re-runnable, doesn't block
other queued jobs) already handles it reasonably.
- Also reconciled `deploy/systemd/osint-queue-worker.service`'s
`--tries=1` → `--tries=3`: a job's own `$tries` property overrides
this CLI default, so `RunFinding`'s `$tries=30` was never at risk,
but leaving the generic default at 1 was a footgun for any future
job type that doesn't set its own `$tries` (Laravel's real
default if `--tries` were omitted entirely is unlimited retries).
- 8 new tests (75/75 passing): `FindingDispatcher` skip/re-queue
logic, `RunFinding::middleware()`/`failed()`, HTTP-level bulk-run
authorization. Plus a full manual pass against the real dev
server + the live systemd worker (not `queue:work --once`):
bulk-ran 3 sources against one target, bulk-ran across two
targets in one investigation, watched theHarvester/crt.sh/whois
resolve sequentially through the single worker, hit and diagnosed
the crt.sh flakiness live, confirmed a retried lookup completing
cleanly. All test data cleaned up afterward.
## Open questions
- Any legal/ToS constraints on the sources to be queried?