Initial commit: Laravel OSINT platform

Multi-user OSINT investigation tool with owner-controlled per-user and per-group access grants. Includes Sherlock, theHarvester, crt.sh, WHOIS, and ip-api.com collectors wired through a pluggable Runner/queue pipeline, investigation search, and printable/CSV/JSON reporting. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:22:05 +03:00
commit 873699e23e
165 changed files with 18507 additions and 0 deletions
@@ -0,0 +1,174 @@
+# OSINT Project Plan
+
+## Overview
+Laravel-based OSINT (Open Source Intelligence) platform. Laravel is the
+core app (UI, auth, data storage, orchestration); heavier or specialized
+collection/enrichment work is delegated to standalone Node and Python
+scripts invoked by Laravel (via `Process`, queued jobs, or scheduled
+commands).
+
+## Stack
+- **Backend**: Laravel 12 (PHP 8.4)
+- **Database**: MySQL, `osint` schema on `192.168.0.155:3306` (same
+  server used by `huginn` / `muninn`)
+- **Frontend**: Laravel default (Blade + Vite), upgrade to
+  Inertia/Vue or Livewire if the UI grows past simple CRUD
+- **Queue**: `database` driver initially, move to Redis if job volume
+  grows
+- **Scripts**: `scripts/node/` and `scripts/python/` at the project
+  root for standalone collectors/enrichers, each callable from Laravel
+  as a subprocess or via a small internal HTTP/CLI contract
+
+## Directory conventions
+- `scripts/python/setup.sh` — recreates everything below in one step;
+  run it on any new machine (see `scripts/python/README.md`)
+- `scripts/python/.venv` — shared venv for lightweight Python
+  collectors (Sherlock, `python-whois`); not committed
+- `scripts/python/.venv-theharvester` — separate venv for theHarvester
+  (heavy deps); not committed
+- `scripts/python/theharvester-src/` — vendored theHarvester source
+  pinned at tag `4.6.0`; not committed
+- `scripts/python/whois_lookup.py` — thin wrapper normalizing
+  `python-whois` output to JSON for `WhoisRunner` to parse
+- `scripts/node/<tool>/` — same pattern for Node-based collectors, own
+  `package.json`, not yet used
+- `app/Osint/` — `Runner` interface, `RunnerRegistry` (source key →
+  runner class), `Runners/` implementations (`SherlockRunner`,
+  `CrtShRunner`, `WhoisRunner`, `IpApiRunner`, `TheHarvesterRunner`)
+- `app/Jobs/RunFinding.php` — queued job that resolves a runner via
+  the registry and updates the `Finding` row
+- `storage/app/osint/runs/<uuid>/` — scratch dir a runner works in
+  while a lookup is in progress; deleted when the run finishes
+
+## Core domain
+- **Investigations** — a case/project; has an `owner_id` (full control),
+  plus per-investigation access grants to individual users or whole
+  groups, each with a `role` of `viewer` or `editor`. Owner always wins;
+  a direct user grant beats a group grant; among multiple group grants
+  `editor` beats `viewer`. See `Investigation::roleFor()`.
+- **Groups** — simple teams (owner + members) that can be granted access
+  to an investigation as a unit
+- **Targets** — the entity being investigated (person, domain, email,
+  username, IP, org), scoped to one investigation
+- **Sources** — catalog of OSINT tools/techniques (seeded: Sherlock,
+  theHarvester, crt.sh, WHOIS, ip-api.com — all 5 now have a working
+  runner). `sources.runner` is a display-only label; the real mapping
+  is `App\Osint\RunnerRegistry`
+- **Findings** — result of running a source against a target; raw JSON
+  payload preserved plus a parsed summary
+
+## Phases
+1. **Scaffolding** (done) — Laravel app created, DB connected, plan
+   committed
+2. **Auth + base UI** (done) — Laravel Breeze (Blade stack) installed;
+   register/login/logout/password reset/email verify/profile all wired
+   and manually verified against the real DB; test suite green against
+   a dedicated `osint_testing` MySQL DB
+3. **Domain models** (done) — migrations + models for Groups,
+   Investigations (+ user/group access grants), Targets, Sources,
+   Findings; `InvestigationPolicy` enforcing owner/editor/viewer;
+   `sources` table seeded with the 5 starter tools; 8 new tests
+   covering the permission matrix (33/33 passing)
+4. **First script integration** (done) — Sherlock wired end-to-end:
+   `App\Osint\Runner` interface + `RunnerRegistry` (maps a source's
+   `key` to its runner class — this is how future collectors plug in),
+   `SherlockRunner` shells out to a venv'd `sherlock` binary and parses
+   its `--csv` output, `App\Jobs\RunFinding` runs on the `database`
+   queue and updates the `Finding` row (pending → running →
+   completed/failed). Minimal UI built alongside it to actually
+   exercise the pipeline: investigations list/create/show, add-target
+   form, per-target "run source" form (only offers sources whose
+   `target_types` match and that have a registered runner). Manually
+   verified end-to-end against the real dev DB: registered a user,
+   created an investigation, added a `username` target, ran Sherlock,
+   processed the queue, confirmed 35 matches rendered as links in the
+   UI, then cleaned up the test data.
+5. **Investigation UI polish** (done) — `/groups` (create, view members,
+   owner adds/removes members by email — `GroupPolicy`); `/investigations/
+   {id}/access` (owner grants/revokes per-user and per-group roles —
+   `InvestigationAccessController` + `InvestigationPolicy::manageAccess`,
+   only groups the owner belongs to are selectable); investigation and
+   target delete (owner-only / editor-or-owner respectively, DB-level
+   cascade cleans up targets/findings/grants); target `value` validation
+   is now type-specific (`Target::valueRulesFor()` — email format, IP
+   format, domain regex, username charset matching what `SherlockRunner`
+   accepts). 19 new feature tests (52/52 passing) plus a full manual
+   pass against the real dev DB: registered two users, confirmed the
+   grantee got 403 before any grant, granted direct `editor` access
+   and confirmed 200 + edit UI, revoked it and confirmed 403 again,
+   then repeated the same check through a group grant (`viewer` role
+   correctly hides the edit form), and confirmed a path-traversal-style
+   username target is rejected by validation before it ever reaches
+   `SherlockRunner`.
+6. **Expand collectors** (done) — 4 new runners registered in
+   `RunnerRegistry`:
+   - `CrtShRunner` (domain) — plain HTTP call to crt.sh's JSON API, no
+     script needed; dedupes subdomains from certificate SAN entries
+   - `IpApiRunner` (ip) — plain HTTP call to ip-api.com; also pure PHP
+   - `WhoisRunner` (domain only — `python-whois` doesn't do IP RDAP,
+     so `sources.target_types` for `whois` was narrowed from
+     `[domain, ip]` to `[domain]`) — shells out to
+     `scripts/python/whois_lookup.py`, a small helper that normalizes
+     `python-whois`'s output (datetimes → ISO strings) to JSON on
+     stdout
+   - `TheHarvesterRunner` (domain) — shells out to a vendored
+     `theHarvester` (pinned to tag `4.6.0`; 4.7+ requires Python 3.12,
+     which isn't available here) in its own venv
+     (`.venv-theharvester`) since it pulls in much heavier deps
+     (playwright, censys, shodan, fastapi) than Sherlock; queries only
+     free, no-API-key sources (crt.sh, OTX, HackerTarget, CertSpotter,
+     ThreatMiner, RapidDNS, Anubis)
+
+   `scripts/python/setup.sh` now sets up both venvs + vendors
+   theHarvester in one step (documented in `scripts/python/README.md`).
+   Show page updated to render any list-shaped `raw_payload` field
+   (subdomains, hosts, emails, ...) as a collapsible list, not just
+   Sherlock's `matches` — capped at 50 items with a "+N more" note.
+   7 new unit tests for the two HTTP-only runners using `Http::fake()`
+   (59/59 passing); Whois/theHarvester aren't unit-tested (real
+   process + network, same as Sherlock) but were verified end-to-end
+   against the real dev DB: `example.com` → crt.sh found 11
+   subdomains, WHOIS returned registrar/dates/nameservers, theHarvester
+   found 502 hosts; `8.8.8.8` → ip-api geolocated to Ashburn, VA. All
+   test data cleaned up afterward.
+7. **Search/reporting** (done) — `/search` (`SearchController`) finds
+   investigations by name and targets by value, both scoped through
+   `Investigation::accessibleBy()` so results never leak data the
+   user can't see. Per-investigation `/report` (printable HTML,
+   browser print-to-PDF, no PDF library dependency), `/export.csv`
+   (findings, one row per target × source), `/export.json` (full
+   structured dump). CSV export escapes values starting with
+   `=+-@`/tab/CR to prevent Excel/Sheets formula injection from a
+   user-controlled target value or a finding summary sourced from
+   external tool output. 10 new tests (67/67 passing) plus a manual
+   pass against the real dev server: searched by investigation name
+   and by target value, opened the printable report, downloaded and
+   inspected both CSV and JSON exports. All test data cleaned up
+   afterward.
+8. **Queue worker process management** — `QUEUE_CONNECTION=database`
+   needs a supervised `queue:work` process (supervisor/systemd) before
+   this is usable outside manual dev testing (see Environment notes)
+
+## Open questions
+- Any legal/ToS constraints on the sources to be queried?
+
+## Environment notes
+- `.env` points at the shared MySQL server (`192.168.0.155`), db
+  `osint`, credentials match the `huginn`/`muninn` projects (`admin`/`admin`)
+- `phpunit.xml` points at a separate `osint_testing` DB on the same
+  server so `php artisan test` never touches real data
+- There's no `.env.testing` file, so `artisan migrate --env=testing`
+  does **not** target `osint_testing` — it silently runs against the
+  real `osint` DB. Always use `php artisan test` (which honors
+  `phpunit.xml`'s DB overrides) for anything testing-related, never
+  `--env=testing` directly against migrate/seed
+- No sudo access in the dev sandbox that created this project; SQLite
+  extension isn't installed for PHP 8.4 here, so MySQL was used instead
+- `QUEUE_CONNECTION=database`, so lookups won't actually run unless a
+  queue worker is running (`php artisan queue:work`). No supervisor/
+  systemd unit set up for this yet — needed before this is usable
+  outside of manual dev testing
+- The two Python venvs are ~390MB combined (`.venv` 166MB,
+  `.venv-theharvester` 223MB, mostly playwright/censys/shodan/fastapi
+  pulled in by theHarvester) — worth knowing before assuming this is
+  a lightweight checkout