osint/README.md

OSINT

A multi-user OSINT (Open Source Intelligence) investigation platform built on Laravel. Investigators create investigations, add targets (a username, domain, email, IP, or org name), and run free/open-source OSINT tools against them. Results are collected as findings, searchable across everything you have access to, and exportable as a printable report or CSV/JSON.

Multi-user by design: each investigation has an owner who controls access, granting viewer or editor roles to individual users or whole groups.

Features

• Investigations — cases that group targets and findings, with owner-controlled access grants (per-user or per-group, viewer/editor)

• Groups — simple teams that can be granted access to an investigation as a unit

• Targets — username, domain, email, ip, org, or person, each with type-specific input validation

• 9 OSINT collectors, all free/open-source, run as background jobs through a shared queue worker:

  Source	Target type	What it does
  Sherlock	username	Checks presence across 400+ social/community sites
  theHarvester	domain	Emails/subdomains/hosts from crt.sh, OTX, HackerTarget, and more
  crt.sh	domain	Subdomain enumeration via certificate transparency logs
  WHOIS	domain	Registrar, dates, name servers
  DNS Records	domain	A/AAAA/MX/NS/TXT/CNAME lookup
  Holehe	email	Checks if an email is registered on 120+ sites
  Gravatar	email	Checks for a registered Gravatar image
  ip-api.com	ip	IP geolocation
  OpenCorporates	org	Company registration search — needs a free API token, see below

• Bulk actions — run every applicable source against a target, or every target in an investigation, in one click; skips sources already in flight and automatically retries prior failures

• Search — find investigations by name or targets by value, scoped to what you have access to

• Reporting — printable HTML report (browser print-to-PDF), CSV export, JSON export

Stack

• Laravel 13 (PHP 8.4), Breeze (Blade + Vite) for auth
• MySQL for storage, database queue driver
• Python 3.10 venvs for the Python-based collectors (Sherlock, Holehe, WHOIS, theHarvester); everything else is pure PHP (HTTP calls or dns_get_record()), no external process needed

See PLAN.md for the full build history, design decisions, and known gotchas — it's the more detailed companion to this file.

Setup

1. PHP dependencies

composer install
cp .env.example .env
php artisan key:generate

2. Database

Point .env at a MySQL database (DB_* variables) and run:

php artisan migrate
php artisan db:seed --class=SourceSeeder

3. Frontend assets

npm install
npm run build   # or `npm run dev` while developing

4. Python collectors

scripts/python/setup.sh

Sets up two venvs (scripts/python/.venv for Sherlock/whois/Holehe, scripts/python/.venv-theharvester for theHarvester, which has much heavier dependencies) and vendors theHarvester's source. See scripts/python/README.md for details.

5. Queue worker

Lookups run as background jobs — nothing executes until a worker is running. For local development:

php artisan queue:work

For anything longer-lived, install the supervised systemd --user service in deploy/systemd/ instead.

6. Optional: OpenCorporates

The org collector needs a free API token. Register at opencorporates.com/api_accounts/new and set OPENCORPORATES_API_TOKEN in .env. Without one, that source is hidden rather than shown broken. Restart the queue worker after setting it.

7. Run it

php artisan serve

Testing

php artisan test

Tests run against a separate osint_testing database, configured in phpunit.xml — never against your real data.

Project layout

• app/Osint/ — the collector framework: Runner interface, RunnerRegistry (maps a source's key to its implementation), Runners/ (one class per collector), FindingDispatcher (bulk-run logic)
• app/Jobs/RunFinding.php — queued job that executes a runner and updates the Finding row, with per-source rate limiting
• app/Models/ — Investigation, Target, Source, Finding, Group, User
• scripts/python/ — Python collectors and their setup script
• deploy/systemd/ — the queue worker's supervised-process definition