Add bulk-run actions and per-source rate limiting
Run-all buttons (per target and per investigation) queue every applicable, not-already-in-flight source via a new FindingDispatcher, re-attempting prior failures automatically. RunFinding now applies a per-source RateLimited queue middleware so a bulk run can't hammer the free public APIs (ip-api, crt.sh, whois, sherlock, theharvester) it depends on; $tries raised to 30 to survive rate-limiter releases, with a failed() handler so exhausted retries still resolve the Finding instead of leaving it stuck pending. Also fixes CrtShRunner treating crt.sh's intermittent 404/502 (a known flaky free service) as fatal on the first try — added a bounded retry. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
853686f3f3
commit
e4fa57c3d6
@@ -160,6 +160,56 @@ commands).
|
||||
worker process directly and confirmed systemd auto-restarted it
|
||||
within seconds. See `deploy/systemd/README.md` for install/operate
|
||||
commands.
|
||||
9. **Bulk actions + rate limiting** (done) — the original 8 phases
|
||||
were all complete; this closed the biggest remaining workflow
|
||||
friction (running each source against each target one at a time).
|
||||
- `App\Osint\FindingDispatcher::dispatchAllApplicable()` — queues
|
||||
every enabled, registered source that applies to a target's type
|
||||
and doesn't already have a `pending`/`running` finding for it
|
||||
(a `completed`/`failed` one *is* re-queued, so bulk-running
|
||||
naturally retries prior failures). Shared by a "Run all" button
|
||||
per target and a "Run all targets" button per investigation.
|
||||
- Per-source rate limiting via Laravel's `RateLimited` queue
|
||||
middleware (`RunFinding::middleware()`), limits defined in
|
||||
`AppServiceProvider::boot()`: ip-api.com 45/min (their published
|
||||
limit), crt.sh 30/min, WHOIS 20/min, Sherlock 10/min,
|
||||
theHarvester 5/min (all courtesy limits of ours, protecting the
|
||||
free infra these runners depend on from a bulk run hammering it).
|
||||
- `RunFinding::$tries` raised from 1 to 30: a rate-limiter release
|
||||
counts as a queue attempt, and a bulk run can easily queue dozens
|
||||
of same-source lookups behind one rate-limit window — needs
|
||||
enough attempts to ride that out. Added `RunFinding::failed()` so
|
||||
a job that does exhaust retries still marks its `Finding` as
|
||||
`failed` with a clear error instead of leaving it stuck at
|
||||
`pending` forever. (Business-logic failures — a runner throwing —
|
||||
are unaffected; `handle()` already catches those internally and
|
||||
never lets them bubble up to Laravel's retry mechanism.)
|
||||
- Bug found and fixed via this work: `CrtShRunner` treated any
|
||||
non-2xx as fatal, but crt.sh (a free, frequently overloaded
|
||||
public service) intermittently returns 404/502/503/timeouts for
|
||||
queries that succeed moments later — confirmed by hand, same
|
||||
query, 404 then 200 with no code changes in between. Added
|
||||
`->retry(3, 2000, throw: false)`. This doesn't fully solve it —
|
||||
hit a sustained bad patch during testing where even 3 retries
|
||||
over ~6s didn't survive it — but it's real instability on crt.sh's
|
||||
end, not something worth over-engineering client-side retries
|
||||
around; the failure path (clear error, re-runnable, doesn't block
|
||||
other queued jobs) already handles it reasonably.
|
||||
- Also reconciled `deploy/systemd/osint-queue-worker.service`'s
|
||||
`--tries=1` → `--tries=3`: a job's own `$tries` property overrides
|
||||
this CLI default, so `RunFinding`'s `$tries=30` was never at risk,
|
||||
but leaving the generic default at 1 was a footgun for any future
|
||||
job type that doesn't set its own `$tries` (Laravel's real
|
||||
default if `--tries` were omitted entirely is unlimited retries).
|
||||
- 8 new tests (75/75 passing): `FindingDispatcher` skip/re-queue
|
||||
logic, `RunFinding::middleware()`/`failed()`, HTTP-level bulk-run
|
||||
authorization. Plus a full manual pass against the real dev
|
||||
server + the live systemd worker (not `queue:work --once`):
|
||||
bulk-ran 3 sources against one target, bulk-ran across two
|
||||
targets in one investigation, watched theHarvester/crt.sh/whois
|
||||
resolve sequentially through the single worker, hit and diagnosed
|
||||
the crt.sh flakiness live, confirmed a retried lookup completing
|
||||
cleanly. All test data cleaned up afterward.
|
||||
|
||||
## Open questions
|
||||
- Any legal/ToS constraints on the sources to be queried?
|
||||
|
||||
Reference in New Issue
Block a user