osint/PLAN.md

# OSINT Project Plan

## Overview
Laravel-based OSINT (Open Source Intelligence) platform. Laravel is the
core app (UI, auth, data storage, orchestration); heavier or specialized
collection/enrichment work is delegated to standalone Node and Python
scripts invoked by Laravel (via `Process`, queued jobs, or scheduled
commands).

## Stack
- **Backend**: Laravel 12 (PHP 8.4)
- **Database**: MySQL, `osint` schema on `192.168.0.155:3306` (same
  server used by `huginn` / `muninn`)
- **Frontend**: Laravel default (Blade + Vite), upgrade to
  Inertia/Vue or Livewire if the UI grows past simple CRUD
- **Queue**: `database` driver initially, move to Redis if job volume
  grows
- **Scripts**: `scripts/node/` and `scripts/python/` at the project
  root for standalone collectors/enrichers, each callable from Laravel
  as a subprocess or via a small internal HTTP/CLI contract

## Directory conventions
- `scripts/python/setup.sh` — recreates everything below in one step;
  run it on any new machine (see `scripts/python/README.md`)
- `scripts/python/.venv` — shared venv for lightweight Python
  collectors (Sherlock, `python-whois`); not committed
- `scripts/python/.venv-theharvester` — separate venv for theHarvester
  (heavy deps); not committed
- `scripts/python/theharvester-src/` — vendored theHarvester source
  pinned at tag `4.6.0`; not committed
- `scripts/python/whois_lookup.py` — thin wrapper normalizing
  `python-whois` output to JSON for `WhoisRunner` to parse
- `scripts/node/<tool>/` — same pattern for Node-based collectors, own
  `package.json`, not yet used
- `app/Osint/` — `Runner` interface, `RunnerRegistry` (source key →
  runner class), `Runners/` implementations (`SherlockRunner`,
  `CrtShRunner`, `WhoisRunner`, `IpApiRunner`, `TheHarvesterRunner`)
- `app/Jobs/RunFinding.php` — queued job that resolves a runner via
  the registry and updates the `Finding` row
- `storage/app/osint/runs/<uuid>/` — scratch dir a runner works in
  while a lookup is in progress; deleted when the run finishes

## Core domain
- **Investigations** — a case/project; has an `owner_id` (full control),
  plus per-investigation access grants to individual users or whole
  groups, each with a `role` of `viewer` or `editor`. Owner always wins;
  a direct user grant beats a group grant; among multiple group grants
  `editor` beats `viewer`. See `Investigation::roleFor()`.
- **Groups** — simple teams (owner + members) that can be granted access
  to an investigation as a unit
- **Targets** — the entity being investigated (person, domain, email,
  username, IP, org), scoped to one investigation
- **Sources** — catalog of OSINT tools/techniques (seeded: Sherlock,
  theHarvester, crt.sh, WHOIS, ip-api.com — all 5 now have a working
  runner). `sources.runner` is a display-only label; the real mapping
  is `App\Osint\RunnerRegistry`
- **Findings** — result of running a source against a target; raw JSON
  payload preserved plus a parsed summary

## Phases
1. **Scaffolding** (done) — Laravel app created, DB connected, plan
   committed
2. **Auth + base UI** (done) — Laravel Breeze (Blade stack) installed;
   register/login/logout/password reset/email verify/profile all wired
   and manually verified against the real DB; test suite green against
   a dedicated `osint_testing` MySQL DB
3. **Domain models** (done) — migrations + models for Groups,
   Investigations (+ user/group access grants), Targets, Sources,
   Findings; `InvestigationPolicy` enforcing owner/editor/viewer;
   `sources` table seeded with the 5 starter tools; 8 new tests
   covering the permission matrix (33/33 passing)
4. **First script integration** (done) — Sherlock wired end-to-end:
   `App\Osint\Runner` interface + `RunnerRegistry` (maps a source's
   `key` to its runner class — this is how future collectors plug in),
   `SherlockRunner` shells out to a venv'd `sherlock` binary and parses
   its `--csv` output, `App\Jobs\RunFinding` runs on the `database`
   queue and updates the `Finding` row (pending → running →
   completed/failed). Minimal UI built alongside it to actually
   exercise the pipeline: investigations list/create/show, add-target
   form, per-target "run source" form (only offers sources whose
   `target_types` match and that have a registered runner). Manually
   verified end-to-end against the real dev DB: registered a user,
   created an investigation, added a `username` target, ran Sherlock,
   processed the queue, confirmed 35 matches rendered as links in the
   UI, then cleaned up the test data.
5. **Investigation UI polish** (done) — `/groups` (create, view members,
   owner adds/removes members by email — `GroupPolicy`); `/investigations/
   {id}/access` (owner grants/revokes per-user and per-group roles —
   `InvestigationAccessController` + `InvestigationPolicy::manageAccess`,
   only groups the owner belongs to are selectable); investigation and
   target delete (owner-only / editor-or-owner respectively, DB-level
   cascade cleans up targets/findings/grants); target `value` validation
   is now type-specific (`Target::valueRulesFor()` — email format, IP
   format, domain regex, username charset matching what `SherlockRunner`
   accepts). 19 new feature tests (52/52 passing) plus a full manual
   pass against the real dev DB: registered two users, confirmed the
   grantee got 403 before any grant, granted direct `editor` access
   and confirmed 200 + edit UI, revoked it and confirmed 403 again,
   then repeated the same check through a group grant (`viewer` role
   correctly hides the edit form), and confirmed a path-traversal-style
   username target is rejected by validation before it ever reaches
   `SherlockRunner`.
6. **Expand collectors** (done) — 4 new runners registered in
   `RunnerRegistry`:
   - `CrtShRunner` (domain) — plain HTTP call to crt.sh's JSON API, no
     script needed; dedupes subdomains from certificate SAN entries
   - `IpApiRunner` (ip) — plain HTTP call to ip-api.com; also pure PHP
   - `WhoisRunner` (domain only — `python-whois` doesn't do IP RDAP,
     so `sources.target_types` for `whois` was narrowed from
     `[domain, ip]` to `[domain]`) — shells out to
     `scripts/python/whois_lookup.py`, a small helper that normalizes
     `python-whois`'s output (datetimes → ISO strings) to JSON on
     stdout
   - `TheHarvesterRunner` (domain) — shells out to a vendored
     `theHarvester` (pinned to tag `4.6.0`; 4.7+ requires Python 3.12,
     which isn't available here) in its own venv
     (`.venv-theharvester`) since it pulls in much heavier deps
     (playwright, censys, shodan, fastapi) than Sherlock; queries only
     free, no-API-key sources (crt.sh, OTX, HackerTarget, CertSpotter,
     ThreatMiner, RapidDNS, Anubis)

   `scripts/python/setup.sh` now sets up both venvs + vendors
   theHarvester in one step (documented in `scripts/python/README.md`).
   Show page updated to render any list-shaped `raw_payload` field
   (subdomains, hosts, emails, ...) as a collapsible list, not just
   Sherlock's `matches` — capped at 50 items with a "+N more" note.
   7 new unit tests for the two HTTP-only runners using `Http::fake()`
   (59/59 passing); Whois/theHarvester aren't unit-tested (real
   process + network, same as Sherlock) but were verified end-to-end
   against the real dev DB: `example.com` → crt.sh found 11
   subdomains, WHOIS returned registrar/dates/nameservers, theHarvester
   found 502 hosts; `8.8.8.8` → ip-api geolocated to Ashburn, VA. All
   test data cleaned up afterward.
7. **Search/reporting** (done) — `/search` (`SearchController`) finds
   investigations by name and targets by value, both scoped through
   `Investigation::accessibleBy()` so results never leak data the
   user can't see. Per-investigation `/report` (printable HTML,
   browser print-to-PDF, no PDF library dependency), `/export.csv`
   (findings, one row per target × source), `/export.json` (full
   structured dump). CSV export escapes values starting with
   `=+-@`/tab/CR to prevent Excel/Sheets formula injection from a
   user-controlled target value or a finding summary sourced from
   external tool output. 10 new tests (67/67 passing) plus a manual
   pass against the real dev server: searched by investigation name
   and by target value, opened the printable report, downloaded and
   inspected both CSV and JSON exports. All test data cleaned up
   afterward.
8. **Queue worker process management** (done) — `deploy/systemd/
   osint-queue-worker.service`, a `systemd --user` unit (no sudo
   needed; this box already has lingering enabled so it survives
   logout) running `queue:work --timeout=180 --max-time=3600
   --tries=1` with `Restart=always`. `--timeout` is set above
   `config('osint.run_timeout')` (120s) — the worker's timeout must
   exceed a runner's own timeout or the worker kills the job first.
   `--max-time` restarts the worker hourly to pick up code/`.env`
   changes and avoid long-lived-process memory growth. Installed and
   verified live: dispatched a real job via the web app *without*
   manually running `queue:work`, confirmed the persistent service
   picked it up and completed it (ip-api lookup, ~200ms); killed the
   worker process directly and confirmed systemd auto-restarted it
   within seconds. See `deploy/systemd/README.md` for install/operate
   commands.
9. **Bulk actions + rate limiting** (done) — the original 8 phases
   were all complete; this closed the biggest remaining workflow
   friction (running each source against each target one at a time).
   - `App\Osint\FindingDispatcher::dispatchAllApplicable()` — queues
     every enabled, registered source that applies to a target's type
     and doesn't already have a `pending`/`running` finding for it
     (a `completed`/`failed` one *is* re-queued, so bulk-running
     naturally retries prior failures). Shared by a "Run all" button
     per target and a "Run all targets" button per investigation.
   - Per-source rate limiting via Laravel's `RateLimited` queue
     middleware (`RunFinding::middleware()`), limits defined in
     `AppServiceProvider::boot()`: ip-api.com 45/min (their published
     limit), crt.sh 30/min, WHOIS 20/min, Sherlock 10/min,
     theHarvester 5/min (all courtesy limits of ours, protecting the
     free infra these runners depend on from a bulk run hammering it).
   - `RunFinding::$tries` raised from 1 to 30: a rate-limiter release
     counts as a queue attempt, and a bulk run can easily queue dozens
     of same-source lookups behind one rate-limit window — needs
     enough attempts to ride that out. Added `RunFinding::failed()` so
     a job that does exhaust retries still marks its `Finding` as
     `failed` with a clear error instead of leaving it stuck at
     `pending` forever. (Business-logic failures — a runner throwing —
     are unaffected; `handle()` already catches those internally and
     never lets them bubble up to Laravel's retry mechanism.)
   - Bug found and fixed via this work: `CrtShRunner` treated any
     non-2xx as fatal, but crt.sh (a free, frequently overloaded
     public service) intermittently returns 404/502/503/timeouts for
     queries that succeed moments later — confirmed by hand, same
     query, 404 then 200 with no code changes in between. Added
     `->retry(3, 2000, throw: false)`. This doesn't fully solve it —
     hit a sustained bad patch during testing where even 3 retries
     over ~6s didn't survive it — but it's real instability on crt.sh's
     end, not something worth over-engineering client-side retries
     around; the failure path (clear error, re-runnable, doesn't block
     other queued jobs) already handles it reasonably.
   - Also reconciled `deploy/systemd/osint-queue-worker.service`'s
     `--tries=1` → `--tries=3`: a job's own `$tries` property overrides
     this CLI default, so `RunFinding`'s `$tries=30` was never at risk,
     but leaving the generic default at 1 was a footgun for any future
     job type that doesn't set its own `$tries` (Laravel's real
     default if `--tries` were omitted entirely is unlimited retries).
   - 8 new tests (75/75 passing): `FindingDispatcher` skip/re-queue
     logic, `RunFinding::middleware()`/`failed()`, HTTP-level bulk-run
     authorization. Plus a full manual pass against the real dev
     server + the live systemd worker (not `queue:work --once`):
     bulk-ran 3 sources against one target, bulk-ran across two
     targets in one investigation, watched theHarvester/crt.sh/whois
     resolve sequentially through the single worker, hit and diagnosed
     the crt.sh flakiness live, confirmed a retried lookup completing
     cleanly. All test data cleaned up afterward.

## Open questions
- Any legal/ToS constraints on the sources to be queried?

## Environment notes
- `.env` points at the shared MySQL server (`192.168.0.155`), db
  `osint`, credentials match the `huginn`/`muninn` projects (`admin`/`admin`)
- `phpunit.xml` points at a separate `osint_testing` DB on the same
  server so `php artisan test` never touches real data
- There's no `.env.testing` file, so `artisan migrate --env=testing`
  does **not** target `osint_testing` — it silently runs against the
  real `osint` DB. Always use `php artisan test` (which honors
  `phpunit.xml`'s DB overrides) for anything testing-related, never
  `--env=testing` directly against migrate/seed
- No sudo access in the dev sandbox that created this project; SQLite
  extension isn't installed for PHP 8.4 here, so MySQL was used instead
- `QUEUE_CONNECTION=database`; the queue worker is supervised by
  `systemd --user` (see `deploy/systemd/`) and running continuously —
  no more need to manually run `php artisan queue:work` in dev
- The two Python venvs are ~390MB combined (`.venv` 166MB,
  `.venv-theharvester` 223MB, mostly playwright/censys/shodan/fastapi
  pulled in by theHarvester) — worth knowing before assuming this is
  a lightweight checkout