osint/PLAN.md

OSINT Project Plan

Overview

Laravel-based OSINT (Open Source Intelligence) platform. Laravel is the core app (UI, auth, data storage, orchestration); heavier or specialized collection/enrichment work is delegated to standalone Node and Python scripts invoked by Laravel (via Process, queued jobs, or scheduled commands).

Stack

• Backend: Laravel 12 (PHP 8.4)
• Database: MySQL, osint schema on 192.168.0.155:3306 (same server used by huginn / muninn)
• Frontend: Laravel default (Blade + Vite), upgrade to Inertia/Vue or Livewire if the UI grows past simple CRUD
• Queue: database driver initially, move to Redis if job volume grows
• Scripts: scripts/node/ and scripts/python/ at the project root for standalone collectors/enrichers, each callable from Laravel as a subprocess or via a small internal HTTP/CLI contract

Directory conventions

• scripts/python/setup.sh — recreates everything below in one step; run it on any new machine (see scripts/python/README.md)
• scripts/python/.venv — shared venv for lightweight Python collectors (Sherlock, python-whois); not committed
• scripts/python/.venv-theharvester — separate venv for theHarvester (heavy deps); not committed
• scripts/python/theharvester-src/ — vendored theHarvester source pinned at tag 4.6.0; not committed
• scripts/python/whois_lookup.py — thin wrapper normalizing python-whois output to JSON for WhoisRunner to parse
• scripts/node/<tool>/ — same pattern for Node-based collectors, own package.json, not yet used
• app/Osint/ — Runner interface, RunnerRegistry (source key → runner class), Runners/ implementations (SherlockRunner, CrtShRunner, WhoisRunner, IpApiRunner, TheHarvesterRunner)
• app/Jobs/RunFinding.php — queued job that resolves a runner via the registry and updates the Finding row
• storage/app/osint/runs/<uuid>/ — scratch dir a runner works in while a lookup is in progress; deleted when the run finishes

Core domain

• Investigations — a case/project; has an owner_id (full control), plus per-investigation access grants to individual users or whole groups, each with a role of viewer or editor. Owner always wins; a direct user grant beats a group grant; among multiple group grants editor beats viewer. See Investigation::roleFor().
• Groups — simple teams (owner + members) that can be granted access to an investigation as a unit
• Targets — the entity being investigated (person, domain, email, username, IP, org), scoped to one investigation
• Sources — catalog of OSINT tools/techniques (seeded: Sherlock, theHarvester, crt.sh, WHOIS, ip-api.com — all 5 now have a working runner). sources.runner is a display-only label; the real mapping is App\Osint\RunnerRegistry
• Findings — result of running a source against a target; raw JSON payload preserved plus a parsed summary

Phases

1. Scaffolding (done) — Laravel app created, DB connected, plan committed

2. Auth + base UI (done) — Laravel Breeze (Blade stack) installed; register/login/logout/password reset/email verify/profile all wired and manually verified against the real DB; test suite green against a dedicated osint_testing MySQL DB

3. Domain models (done) — migrations + models for Groups, Investigations (+ user/group access grants), Targets, Sources, Findings; InvestigationPolicy enforcing owner/editor/viewer; sources table seeded with the 5 starter tools; 8 new tests covering the permission matrix (33/33 passing)

4. First script integration (done) — Sherlock wired end-to-end: App\Osint\Runner interface + RunnerRegistry (maps a source's key to its runner class — this is how future collectors plug in), SherlockRunner shells out to a venv'd sherlock binary and parses its --csv output, App\Jobs\RunFinding runs on the database queue and updates the Finding row (pending → running → completed/failed). Minimal UI built alongside it to actually exercise the pipeline: investigations list/create/show, add-target form, per-target "run source" form (only offers sources whose target_types match and that have a registered runner). Manually verified end-to-end against the real dev DB: registered a user, created an investigation, added a username target, ran Sherlock, processed the queue, confirmed 35 matches rendered as links in the UI, then cleaned up the test data.

5. Investigation UI polish (done) — /groups (create, view members, owner adds/removes members by email — GroupPolicy); /investigations/ {id}/access (owner grants/revokes per-user and per-group roles — InvestigationAccessController + InvestigationPolicy::manageAccess, only groups the owner belongs to are selectable); investigation and target delete (owner-only / editor-or-owner respectively, DB-level cascade cleans up targets/findings/grants); target value validation is now type-specific (Target::valueRulesFor() — email format, IP format, domain regex, username charset matching what SherlockRunner accepts). 19 new feature tests (52/52 passing) plus a full manual pass against the real dev DB: registered two users, confirmed the grantee got 403 before any grant, granted direct editor access and confirmed 200 + edit UI, revoked it and confirmed 403 again, then repeated the same check through a group grant (viewer role correctly hides the edit form), and confirmed a path-traversal-style username target is rejected by validation before it ever reaches SherlockRunner.

6. Expand collectors (done) — 4 new runners registered in RunnerRegistry:

   • CrtShRunner (domain) — plain HTTP call to crt.sh's JSON API, no script needed; dedupes subdomains from certificate SAN entries
   • IpApiRunner (ip) — plain HTTP call to ip-api.com; also pure PHP
   • WhoisRunner (domain only — python-whois doesn't do IP RDAP, so sources.target_types for whois was narrowed from [domain, ip] to [domain]) — shells out to scripts/python/whois_lookup.py, a small helper that normalizes python-whois's output (datetimes → ISO strings) to JSON on stdout
   • TheHarvesterRunner (domain) — shells out to a vendored theHarvester (pinned to tag 4.6.0; 4.7+ requires Python 3.12, which isn't available here) in its own venv (.venv-theharvester) since it pulls in much heavier deps (playwright, censys, shodan, fastapi) than Sherlock; queries only free, no-API-key sources (crt.sh, OTX, HackerTarget, CertSpotter, ThreatMiner, RapidDNS, Anubis)

   scripts/python/setup.sh now sets up both venvs + vendors theHarvester in one step (documented in scripts/python/README.md). Show page updated to render any list-shaped raw_payload field (subdomains, hosts, emails, ...) as a collapsible list, not just Sherlock's matches — capped at 50 items with a "+N more" note. 7 new unit tests for the two HTTP-only runners using Http::fake() (59/59 passing); Whois/theHarvester aren't unit-tested (real process + network, same as Sherlock) but were verified end-to-end against the real dev DB: example.com → crt.sh found 11 subdomains, WHOIS returned registrar/dates/nameservers, theHarvester found 502 hosts; 8.8.8.8 → ip-api geolocated to Ashburn, VA. All test data cleaned up afterward.

7. Search/reporting (done) — /search (SearchController) finds investigations by name and targets by value, both scoped through Investigation::accessibleBy() so results never leak data the user can't see. Per-investigation /report (printable HTML, browser print-to-PDF, no PDF library dependency), /export.csv (findings, one row per target × source), /export.json (full structured dump). CSV export escapes values starting with =+-@/tab/CR to prevent Excel/Sheets formula injection from a user-controlled target value or a finding summary sourced from external tool output. 10 new tests (67/67 passing) plus a manual pass against the real dev server: searched by investigation name and by target value, opened the printable report, downloaded and inspected both CSV and JSON exports. All test data cleaned up afterward.

8. Queue worker process management — QUEUE_CONNECTION=database needs a supervised queue:work process (supervisor/systemd) before this is usable outside manual dev testing (see Environment notes)

Open questions

• Any legal/ToS constraints on the sources to be queried?

Environment notes

• .env points at the shared MySQL server (192.168.0.155), db osint, credentials match the huginn/muninn projects (admin/admin)
• phpunit.xml points at a separate osint_testing DB on the same server so php artisan test never touches real data
• There's no .env.testing file, so artisan migrate --env=testing does not target osint_testing — it silently runs against the real osint DB. Always use php artisan test (which honors phpunit.xml's DB overrides) for anything testing-related, never --env=testing directly against migrate/seed
• No sudo access in the dev sandbox that created this project; SQLite extension isn't installed for PHP 8.4 here, so MySQL was used instead
• QUEUE_CONNECTION=database, so lookups won't actually run unless a queue worker is running (php artisan queue:work). No supervisor/ systemd unit set up for this yet — needed before this is usable outside of manual dev testing
• The two Python venvs are ~390MB combined (.venv 166MB, .venv-theharvester 223MB, mostly playwright/censys/shodan/fastapi pulled in by theHarvester) — worth knowing before assuming this is a lightweight checkout