osint/PLAN.md

OSINT Project Plan

Overview

Laravel-based OSINT (Open Source Intelligence) platform. Laravel is the core app (UI, auth, data storage, orchestration); heavier or specialized collection/enrichment work is delegated to standalone Node and Python scripts invoked by Laravel (via Process, queued jobs, or scheduled commands).

Stack

• Backend: Laravel 12 (PHP 8.4)
• Database: MySQL, osint schema on 192.168.0.155:3306 (same server used by huginn / muninn)
• Frontend: Laravel default (Blade + Vite), upgrade to Inertia/Vue or Livewire if the UI grows past simple CRUD
• Queue: database driver initially, move to Redis if job volume grows
• Scripts: scripts/node/ and scripts/python/ at the project root for standalone collectors/enrichers, each callable from Laravel as a subprocess or via a small internal HTTP/CLI contract

Directory conventions

• scripts/python/setup.sh — recreates everything below in one step; run it on any new machine (see scripts/python/README.md)
• scripts/python/.venv — shared venv for lightweight Python collectors (Sherlock, python-whois); not committed
• scripts/python/.venv-theharvester — separate venv for theHarvester (heavy deps); not committed
• scripts/python/theharvester-src/ — vendored theHarvester source pinned at tag 4.6.0; not committed
• scripts/python/whois_lookup.py — thin wrapper normalizing python-whois output to JSON for WhoisRunner to parse
• scripts/node/<tool>/ — same pattern for Node-based collectors, own package.json, not yet used
• app/Osint/ — Runner interface, RunnerRegistry (source key → runner class), Runners/ implementations (SherlockRunner, CrtShRunner, WhoisRunner, IpApiRunner, TheHarvesterRunner)
• app/Jobs/RunFinding.php — queued job that resolves a runner via the registry and updates the Finding row
• storage/app/osint/runs/<uuid>/ — scratch dir a runner works in while a lookup is in progress; deleted when the run finishes

Core domain

• Investigations — a case/project; has an owner_id (full control), plus per-investigation access grants to individual users or whole groups, each with a role of viewer or editor. Owner always wins; a direct user grant beats a group grant; among multiple group grants editor beats viewer. See Investigation::roleFor().
• Groups — simple teams (owner + members) that can be granted access to an investigation as a unit
• Targets — the entity being investigated (person, domain, email, username, IP, org), scoped to one investigation
• Sources — catalog of OSINT tools/techniques (seeded: Sherlock, theHarvester, crt.sh, WHOIS, ip-api.com, Holehe, Gravatar, DNS Records, OpenCorporates — 9 total, 8 runnable out of the box; OpenCorporates needs a free API token, see phase 12). sources.runner is a display-only label; the real mapping is App\Osint\RunnerRegistry
• Findings — result of running a source against a target; raw JSON payload preserved plus a parsed summary

Phases

1. Scaffolding (done) — Laravel app created, DB connected, plan committed

2. Auth + base UI (done) — Laravel Breeze (Blade stack) installed; register/login/logout/password reset/email verify/profile all wired and manually verified against the real DB; test suite green against a dedicated osint_testing MySQL DB

3. Domain models (done) — migrations + models for Groups, Investigations (+ user/group access grants), Targets, Sources, Findings; InvestigationPolicy enforcing owner/editor/viewer; sources table seeded with the 5 starter tools; 8 new tests covering the permission matrix (33/33 passing)

4. First script integration (done) — Sherlock wired end-to-end: App\Osint\Runner interface + RunnerRegistry (maps a source's key to its runner class — this is how future collectors plug in), SherlockRunner shells out to a venv'd sherlock binary and parses its --csv output, App\Jobs\RunFinding runs on the database queue and updates the Finding row (pending → running → completed/failed). Minimal UI built alongside it to actually exercise the pipeline: investigations list/create/show, add-target form, per-target "run source" form (only offers sources whose target_types match and that have a registered runner). Manually verified end-to-end against the real dev DB: registered a user, created an investigation, added a username target, ran Sherlock, processed the queue, confirmed 35 matches rendered as links in the UI, then cleaned up the test data.

5. Investigation UI polish (done) — /groups (create, view members, owner adds/removes members by email — GroupPolicy); /investigations/ {id}/access (owner grants/revokes per-user and per-group roles — InvestigationAccessController + InvestigationPolicy::manageAccess, only groups the owner belongs to are selectable); investigation and target delete (owner-only / editor-or-owner respectively, DB-level cascade cleans up targets/findings/grants); target value validation is now type-specific (Target::valueRulesFor() — email format, IP format, domain regex, username charset matching what SherlockRunner accepts). 19 new feature tests (52/52 passing) plus a full manual pass against the real dev DB: registered two users, confirmed the grantee got 403 before any grant, granted direct editor access and confirmed 200 + edit UI, revoked it and confirmed 403 again, then repeated the same check through a group grant (viewer role correctly hides the edit form), and confirmed a path-traversal-style username target is rejected by validation before it ever reaches SherlockRunner.

6. Expand collectors (done) — 4 new runners registered in RunnerRegistry:

   • CrtShRunner (domain) — plain HTTP call to crt.sh's JSON API, no script needed; dedupes subdomains from certificate SAN entries
   • IpApiRunner (ip) — plain HTTP call to ip-api.com; also pure PHP
   • WhoisRunner (domain only — python-whois doesn't do IP RDAP, so sources.target_types for whois was narrowed from [domain, ip] to [domain]) — shells out to scripts/python/whois_lookup.py, a small helper that normalizes python-whois's output (datetimes → ISO strings) to JSON on stdout
   • TheHarvesterRunner (domain) — shells out to a vendored theHarvester (pinned to tag 4.6.0; 4.7+ requires Python 3.12, which isn't available here) in its own venv (.venv-theharvester) since it pulls in much heavier deps (playwright, censys, shodan, fastapi) than Sherlock; queries only free, no-API-key sources (crt.sh, OTX, HackerTarget, CertSpotter, ThreatMiner, RapidDNS, Anubis)

   scripts/python/setup.sh now sets up both venvs + vendors theHarvester in one step (documented in scripts/python/README.md). Show page updated to render any list-shaped raw_payload field (subdomains, hosts, emails, ...) as a collapsible list, not just Sherlock's matches — capped at 50 items with a "+N more" note. 7 new unit tests for the two HTTP-only runners using Http::fake() (59/59 passing); Whois/theHarvester aren't unit-tested (real process + network, same as Sherlock) but were verified end-to-end against the real dev DB: example.com → crt.sh found 11 subdomains, WHOIS returned registrar/dates/nameservers, theHarvester found 502 hosts; 8.8.8.8 → ip-api geolocated to Ashburn, VA. All test data cleaned up afterward.

7. Search/reporting (done) — /search (SearchController) finds investigations by name and targets by value, both scoped through Investigation::accessibleBy() so results never leak data the user can't see. Per-investigation /report (printable HTML, browser print-to-PDF, no PDF library dependency), /export.csv (findings, one row per target × source), /export.json (full structured dump). CSV export escapes values starting with =+-@/tab/CR to prevent Excel/Sheets formula injection from a user-controlled target value or a finding summary sourced from external tool output. 10 new tests (67/67 passing) plus a manual pass against the real dev server: searched by investigation name and by target value, opened the printable report, downloaded and inspected both CSV and JSON exports. All test data cleaned up afterward.

8. Queue worker process management (done) — deploy/systemd/ osint-queue-worker.service, a systemd --user unit (no sudo needed; this box already has lingering enabled so it survives logout) running queue:work --timeout=180 --max-time=3600 --tries=1 with Restart=always. --timeout is set above config('osint.run_timeout') (120s) — the worker's timeout must exceed a runner's own timeout or the worker kills the job first. --max-time restarts the worker hourly to pick up code/.env changes and avoid long-lived-process memory growth. Installed and verified live: dispatched a real job via the web app without manually running queue:work, confirmed the persistent service picked it up and completed it (ip-api lookup, ~200ms); killed the worker process directly and confirmed systemd auto-restarted it within seconds. See deploy/systemd/README.md for install/operate commands.

9. Bulk actions + rate limiting (done) — the original 8 phases were all complete; this closed the biggest remaining workflow friction (running each source against each target one at a time).

   • App\Osint\FindingDispatcher::dispatchAllApplicable() — queues every enabled, registered source that applies to a target's type and doesn't already have a pending/running finding for it (a completed/failed one is re-queued, so bulk-running naturally retries prior failures). Shared by a "Run all" button per target and a "Run all targets" button per investigation.
   • Per-source rate limiting via Laravel's RateLimited queue middleware (RunFinding::middleware()), limits defined in AppServiceProvider::boot(): ip-api.com 45/min (their published limit), crt.sh 30/min, WHOIS 20/min, Sherlock 10/min, theHarvester 5/min (all courtesy limits of ours, protecting the free infra these runners depend on from a bulk run hammering it).
   • RunFinding::$tries raised from 1 to 30: a rate-limiter release counts as a queue attempt, and a bulk run can easily queue dozens of same-source lookups behind one rate-limit window — needs enough attempts to ride that out. Added RunFinding::failed() so a job that does exhaust retries still marks its Finding as failed with a clear error instead of leaving it stuck at pending forever. (Business-logic failures — a runner throwing — are unaffected; handle() already catches those internally and never lets them bubble up to Laravel's retry mechanism.)
   • Bug found and fixed via this work: CrtShRunner treated any non-2xx as fatal, but crt.sh (a free, frequently overloaded public service) intermittently returns 404/502/503/timeouts for queries that succeed moments later — confirmed by hand, same query, 404 then 200 with no code changes in between. Added ->retry(3, 2000, throw: false). This doesn't fully solve it — hit a sustained bad patch during testing where even 3 retries over ~6s didn't survive it — but it's real instability on crt.sh's end, not something worth over-engineering client-side retries around; the failure path (clear error, re-runnable, doesn't block other queued jobs) already handles it reasonably.
   • Also reconciled deploy/systemd/osint-queue-worker.service's --tries=1 → --tries=3: a job's own $tries property overrides this CLI default, so RunFinding's $tries=30 was never at risk, but leaving the generic default at 1 was a footgun for any future job type that doesn't set its own $tries (Laravel's real default if --tries were omitted entirely is unlimited retries).
   • 8 new tests (75/75 passing): FindingDispatcher skip/re-queue logic, RunFinding::middleware()/failed(), HTTP-level bulk-run authorization. Plus a full manual pass against the real dev server + the live systemd worker (not queue:work --once): bulk-ran 3 sources against one target, bulk-ran across two targets in one investigation, watched theHarvester/crt.sh/whois resolve sequentially through the single worker, hit and diagnosed the crt.sh flakiness live, confirmed a retried lookup completing cleanly. All test data cleaned up afterward.

10. Holehe collector (done) — 6th source, closes the email target-type gap (previously zero collectors for it). Checks whether an email is registered on 120+ sites via the password-reset-oracle technique; same venv as Sherlock/whois (scripts/python/setup.sh updated). HoleheRunner follows the Sherlock/CSV pattern but with one wrinkle discovered while verifying it end-to-end: holehe's CLI calls Python's exit("message") on its normal success path right after writing the CSV (core.py: export_csv), which always yields exit code 1 — a real successful run and a genuine crash are indistinguishable by exit code alone. Fixed by gating on whether the CSV file actually exists instead of successful(). Courtesy rate limit: 15/min (osint-runner:holehe). Not unit-tested (real process + network, same as Sherlock/whois/theHarvester); verified by hand instead: registered a user, added an email target, ran Holehe through the live systemd worker, watched it fail with the exit-code-1 quirk before the fix and complete cleanly after, confirmed the summary rendered correctly in the UI. Test data cleaned up afterward.

11. Gravatar + DNS Records collectors (done) — 7th and 8th sources, both pure PHP (no Python/venv, no external process at all for DNS).

    • GravatarRunner (email) — MD5-hashes the email, checks gravatar.com/avatar/{hash}?d=404 for a 200 vs 404. Gravatar's public profile JSON endpoint ({hash}.json) turned out to now require API auth — confirmed by hand against a real profile (Matt Mullenweg's, Gravatar's own creator: avatar check returned 200, .json returned 404 regardless) — so this stays scoped to the binary "does an avatar exist" signal rather than pulling profile details. Unit-tested with Http::fake() (avatar found/not-found/error, invalid email, wrong target type).
    • DnsRunner (domain) — pure PHP dns_get_record(), no HTTP client or subprocess. Looks up A/AAAA/MX/NS/TXT/CNAME. Its raw payload is naturally keyed by record type (A, MX, ...), which the existing generic list-renderer in the investigation show page turns into one collapsible section per record type automatically — no UI changes needed. Unit-tested against a real DNS lookup (example.com, IANA-controlled, stable for decades) plus a nonexistent-domain case, since dns_get_record() isn't mockable through Http::fake().
    • Courtesy rate limits: Gravatar 45/min (cheap single GET), DNS 60/min (local resolver, cheapest operation in the app).
    • 9 new tests (84/84 passing). Manually verified end-to-end through the live systemd worker: Gravatar against a real email with a known public avatar (found it), DNS against google.com (22 records across all 5 present types, each rendering as its own labeled section in the UI with correct counts and values). Test data cleaned up afterward.

12. OpenCorporates collector (done) — 9th source, first attempt at closing the org gap. Correction to what was assumed when this was first suggested: OpenCorporates now requires an API token even on the free tier — confirmed by hand (keyless request → 401 Invalid Api Token). Free tier still exists (50 req/day, 200/month, register at opencorporates.com/api_accounts/new) but isn't fully keyless like the others.

    • services.opencorporates.token (OPENCORPORATES_API_TOKEN in .env) holds the token; left blank by default.
    • RunnerRegistry::has('opencorporates') additionally checks the token is configured — without one, the source is hidden from the "runnable" list (shows "not yet integrated", same as any other unbuilt runner) instead of exposing a "Run" button that always fails. Verified both layers by hand: the UI correctly grays it out, and a direct POST bypassing the UI still gets rejected server-side with a clear error and creates no Finding row.
    • Rate limit is their real quota (50/day) via Limit::perDay(50), not a courtesy limit like the other sources — protects the configured token from getting exhausted or suspended.
    • Hit one real bug while wiring the token-gating: PHP class constants can't hold closures (Constant expression contains invalid operations) — an earlier REQUIRES map of source key → closure crashed every test process on file load. Fixed by inlining the one special case directly in has() instead of building a small closure-map abstraction for what's currently a single entry.
    • 8 new tests (93/93 passing) — OpenCorporatesRunner via Http::fake() (token missing, results found/not-found, wrong target type, HTTP failure), RunnerRegistry gating logic directly. Not runnable end-to-end without a real token (none configured in this environment) — verified what can be verified without one: the gating behavior in both the UI and the controller. Whoever wants this source live needs to register for a free OpenCorporates account and set OPENCORPORATES_API_TOKEN in .env.

Open questions

• Any legal/ToS constraints on the sources to be queried?

Environment notes

• .env points at the shared MySQL server (192.168.0.155), db osint, credentials match the huginn/muninn projects (admin/admin)
• phpunit.xml points at a separate osint_testing DB on the same server so php artisan test never touches real data
• There's no .env.testing file, so artisan migrate --env=testing does not target osint_testing — it silently runs against the real osint DB. Always use php artisan test (which honors phpunit.xml's DB overrides) for anything testing-related, never --env=testing directly against migrate/seed
• No sudo access in the dev sandbox that created this project; SQLite extension isn't installed for PHP 8.4 here, so MySQL was used instead
• QUEUE_CONNECTION=database; the queue worker is supervised by systemd --user (see deploy/systemd/) and running continuously — no more need to manually run php artisan queue:work in dev
• The two Python venvs are ~390MB combined (.venv 166MB, .venv-theharvester 223MB, mostly playwright/censys/shodan/fastapi pulled in by theHarvester) — worth knowing before assuming this is a lightweight checkout