# OSINT Project Plan ## Overview Laravel-based OSINT (Open Source Intelligence) platform. Laravel is the core app (UI, auth, data storage, orchestration); heavier or specialized collection/enrichment work is delegated to standalone Node and Python scripts invoked by Laravel (via `Process`, queued jobs, or scheduled commands). ## Stack - **Backend**: Laravel 12 (PHP 8.4) - **Database**: MySQL, `osint` schema on `192.168.0.155:3306` (same server used by `huginn` / `muninn`) - **Frontend**: Laravel default (Blade + Vite), upgrade to Inertia/Vue or Livewire if the UI grows past simple CRUD - **Queue**: `database` driver initially, move to Redis if job volume grows - **Scripts**: `scripts/node/` and `scripts/python/` at the project root for standalone collectors/enrichers, each callable from Laravel as a subprocess or via a small internal HTTP/CLI contract ## Directory conventions - `scripts/python/setup.sh` — recreates everything below in one step; run it on any new machine (see `scripts/python/README.md`) - `scripts/python/.venv` — shared venv for lightweight Python collectors (Sherlock, `python-whois`); not committed - `scripts/python/.venv-theharvester` — separate venv for theHarvester (heavy deps); not committed - `scripts/python/theharvester-src/` — vendored theHarvester source pinned at tag `4.6.0`; not committed - `scripts/python/whois_lookup.py` — thin wrapper normalizing `python-whois` output to JSON for `WhoisRunner` to parse - `scripts/node//` — same pattern for Node-based collectors, own `package.json`, not yet used - `app/Osint/` — `Runner` interface, `RunnerRegistry` (source key → runner class), `Runners/` implementations (`SherlockRunner`, `CrtShRunner`, `WhoisRunner`, `IpApiRunner`, `TheHarvesterRunner`) - `app/Jobs/RunFinding.php` — queued job that resolves a runner via the registry and updates the `Finding` row - `storage/app/osint/runs//` — scratch dir a runner works in while a lookup is in progress; deleted when the run finishes ## Core domain - **Investigations** — a case/project; has an `owner_id` (full control), plus per-investigation access grants to individual users or whole groups, each with a `role` of `viewer` or `editor`. Owner always wins; a direct user grant beats a group grant; among multiple group grants `editor` beats `viewer`. See `Investigation::roleFor()`. - **Groups** — simple teams (owner + members) that can be granted access to an investigation as a unit - **Targets** — the entity being investigated (person, domain, email, username, IP, org), scoped to one investigation - **Sources** — catalog of OSINT tools/techniques (seeded: Sherlock, theHarvester, crt.sh, WHOIS, ip-api.com — all 5 now have a working runner). `sources.runner` is a display-only label; the real mapping is `App\Osint\RunnerRegistry` - **Findings** — result of running a source against a target; raw JSON payload preserved plus a parsed summary ## Phases 1. **Scaffolding** (done) — Laravel app created, DB connected, plan committed 2. **Auth + base UI** (done) — Laravel Breeze (Blade stack) installed; register/login/logout/password reset/email verify/profile all wired and manually verified against the real DB; test suite green against a dedicated `osint_testing` MySQL DB 3. **Domain models** (done) — migrations + models for Groups, Investigations (+ user/group access grants), Targets, Sources, Findings; `InvestigationPolicy` enforcing owner/editor/viewer; `sources` table seeded with the 5 starter tools; 8 new tests covering the permission matrix (33/33 passing) 4. **First script integration** (done) — Sherlock wired end-to-end: `App\Osint\Runner` interface + `RunnerRegistry` (maps a source's `key` to its runner class — this is how future collectors plug in), `SherlockRunner` shells out to a venv'd `sherlock` binary and parses its `--csv` output, `App\Jobs\RunFinding` runs on the `database` queue and updates the `Finding` row (pending → running → completed/failed). Minimal UI built alongside it to actually exercise the pipeline: investigations list/create/show, add-target form, per-target "run source" form (only offers sources whose `target_types` match and that have a registered runner). Manually verified end-to-end against the real dev DB: registered a user, created an investigation, added a `username` target, ran Sherlock, processed the queue, confirmed 35 matches rendered as links in the UI, then cleaned up the test data. 5. **Investigation UI polish** (done) — `/groups` (create, view members, owner adds/removes members by email — `GroupPolicy`); `/investigations/ {id}/access` (owner grants/revokes per-user and per-group roles — `InvestigationAccessController` + `InvestigationPolicy::manageAccess`, only groups the owner belongs to are selectable); investigation and target delete (owner-only / editor-or-owner respectively, DB-level cascade cleans up targets/findings/grants); target `value` validation is now type-specific (`Target::valueRulesFor()` — email format, IP format, domain regex, username charset matching what `SherlockRunner` accepts). 19 new feature tests (52/52 passing) plus a full manual pass against the real dev DB: registered two users, confirmed the grantee got 403 before any grant, granted direct `editor` access and confirmed 200 + edit UI, revoked it and confirmed 403 again, then repeated the same check through a group grant (`viewer` role correctly hides the edit form), and confirmed a path-traversal-style username target is rejected by validation before it ever reaches `SherlockRunner`. 6. **Expand collectors** (done) — 4 new runners registered in `RunnerRegistry`: - `CrtShRunner` (domain) — plain HTTP call to crt.sh's JSON API, no script needed; dedupes subdomains from certificate SAN entries - `IpApiRunner` (ip) — plain HTTP call to ip-api.com; also pure PHP - `WhoisRunner` (domain only — `python-whois` doesn't do IP RDAP, so `sources.target_types` for `whois` was narrowed from `[domain, ip]` to `[domain]`) — shells out to `scripts/python/whois_lookup.py`, a small helper that normalizes `python-whois`'s output (datetimes → ISO strings) to JSON on stdout - `TheHarvesterRunner` (domain) — shells out to a vendored `theHarvester` (pinned to tag `4.6.0`; 4.7+ requires Python 3.12, which isn't available here) in its own venv (`.venv-theharvester`) since it pulls in much heavier deps (playwright, censys, shodan, fastapi) than Sherlock; queries only free, no-API-key sources (crt.sh, OTX, HackerTarget, CertSpotter, ThreatMiner, RapidDNS, Anubis) `scripts/python/setup.sh` now sets up both venvs + vendors theHarvester in one step (documented in `scripts/python/README.md`). Show page updated to render any list-shaped `raw_payload` field (subdomains, hosts, emails, ...) as a collapsible list, not just Sherlock's `matches` — capped at 50 items with a "+N more" note. 7 new unit tests for the two HTTP-only runners using `Http::fake()` (59/59 passing); Whois/theHarvester aren't unit-tested (real process + network, same as Sherlock) but were verified end-to-end against the real dev DB: `example.com` → crt.sh found 11 subdomains, WHOIS returned registrar/dates/nameservers, theHarvester found 502 hosts; `8.8.8.8` → ip-api geolocated to Ashburn, VA. All test data cleaned up afterward. 7. **Search/reporting** (done) — `/search` (`SearchController`) finds investigations by name and targets by value, both scoped through `Investigation::accessibleBy()` so results never leak data the user can't see. Per-investigation `/report` (printable HTML, browser print-to-PDF, no PDF library dependency), `/export.csv` (findings, one row per target × source), `/export.json` (full structured dump). CSV export escapes values starting with `=+-@`/tab/CR to prevent Excel/Sheets formula injection from a user-controlled target value or a finding summary sourced from external tool output. 10 new tests (67/67 passing) plus a manual pass against the real dev server: searched by investigation name and by target value, opened the printable report, downloaded and inspected both CSV and JSON exports. All test data cleaned up afterward. 8. **Queue worker process management** (done) — `deploy/systemd/ osint-queue-worker.service`, a `systemd --user` unit (no sudo needed; this box already has lingering enabled so it survives logout) running `queue:work --timeout=180 --max-time=3600 --tries=1` with `Restart=always`. `--timeout` is set above `config('osint.run_timeout')` (120s) — the worker's timeout must exceed a runner's own timeout or the worker kills the job first. `--max-time` restarts the worker hourly to pick up code/`.env` changes and avoid long-lived-process memory growth. Installed and verified live: dispatched a real job via the web app *without* manually running `queue:work`, confirmed the persistent service picked it up and completed it (ip-api lookup, ~200ms); killed the worker process directly and confirmed systemd auto-restarted it within seconds. See `deploy/systemd/README.md` for install/operate commands. 9. **Bulk actions + rate limiting** (done) — the original 8 phases were all complete; this closed the biggest remaining workflow friction (running each source against each target one at a time). - `App\Osint\FindingDispatcher::dispatchAllApplicable()` — queues every enabled, registered source that applies to a target's type and doesn't already have a `pending`/`running` finding for it (a `completed`/`failed` one *is* re-queued, so bulk-running naturally retries prior failures). Shared by a "Run all" button per target and a "Run all targets" button per investigation. - Per-source rate limiting via Laravel's `RateLimited` queue middleware (`RunFinding::middleware()`), limits defined in `AppServiceProvider::boot()`: ip-api.com 45/min (their published limit), crt.sh 30/min, WHOIS 20/min, Sherlock 10/min, theHarvester 5/min (all courtesy limits of ours, protecting the free infra these runners depend on from a bulk run hammering it). - `RunFinding::$tries` raised from 1 to 30: a rate-limiter release counts as a queue attempt, and a bulk run can easily queue dozens of same-source lookups behind one rate-limit window — needs enough attempts to ride that out. Added `RunFinding::failed()` so a job that does exhaust retries still marks its `Finding` as `failed` with a clear error instead of leaving it stuck at `pending` forever. (Business-logic failures — a runner throwing — are unaffected; `handle()` already catches those internally and never lets them bubble up to Laravel's retry mechanism.) - Bug found and fixed via this work: `CrtShRunner` treated any non-2xx as fatal, but crt.sh (a free, frequently overloaded public service) intermittently returns 404/502/503/timeouts for queries that succeed moments later — confirmed by hand, same query, 404 then 200 with no code changes in between. Added `->retry(3, 2000, throw: false)`. This doesn't fully solve it — hit a sustained bad patch during testing where even 3 retries over ~6s didn't survive it — but it's real instability on crt.sh's end, not something worth over-engineering client-side retries around; the failure path (clear error, re-runnable, doesn't block other queued jobs) already handles it reasonably. - Also reconciled `deploy/systemd/osint-queue-worker.service`'s `--tries=1` → `--tries=3`: a job's own `$tries` property overrides this CLI default, so `RunFinding`'s `$tries=30` was never at risk, but leaving the generic default at 1 was a footgun for any future job type that doesn't set its own `$tries` (Laravel's real default if `--tries` were omitted entirely is unlimited retries). - 8 new tests (75/75 passing): `FindingDispatcher` skip/re-queue logic, `RunFinding::middleware()`/`failed()`, HTTP-level bulk-run authorization. Plus a full manual pass against the real dev server + the live systemd worker (not `queue:work --once`): bulk-ran 3 sources against one target, bulk-ran across two targets in one investigation, watched theHarvester/crt.sh/whois resolve sequentially through the single worker, hit and diagnosed the crt.sh flakiness live, confirmed a retried lookup completing cleanly. All test data cleaned up afterward. ## Open questions - Any legal/ToS constraints on the sources to be queried? ## Environment notes - `.env` points at the shared MySQL server (`192.168.0.155`), db `osint`, credentials match the `huginn`/`muninn` projects (`admin`/`admin`) - `phpunit.xml` points at a separate `osint_testing` DB on the same server so `php artisan test` never touches real data - There's no `.env.testing` file, so `artisan migrate --env=testing` does **not** target `osint_testing` — it silently runs against the real `osint` DB. Always use `php artisan test` (which honors `phpunit.xml`'s DB overrides) for anything testing-related, never `--env=testing` directly against migrate/seed - No sudo access in the dev sandbox that created this project; SQLite extension isn't installed for PHP 8.4 here, so MySQL was used instead - `QUEUE_CONNECTION=database`; the queue worker is supervised by `systemd --user` (see `deploy/systemd/`) and running continuously — no more need to manually run `php artisan queue:work` in dev - The two Python venvs are ~390MB combined (`.venv` 166MB, `.venv-theharvester` 223MB, mostly playwright/censys/shodan/fastapi pulled in by theHarvester) — worth knowing before assuming this is a lightweight checkout