osint/README.md

# OSINT

A multi-user OSINT (Open Source Intelligence) investigation platform built on
Laravel. Investigators create **investigations**, add **targets** (a
username, domain, email, IP, or org name), and run free/open-source OSINT
tools against them. Results are collected as **findings**, searchable across
everything you have access to, and exportable as a printable report or
CSV/JSON.

Multi-user by design: each investigation has an owner who controls access,
granting `viewer` or `editor` roles to individual users or whole groups.

## Features

- **Investigations** — cases that group targets and findings, with
  owner-controlled access grants (per-user or per-group, `viewer`/`editor`)
- **Groups** — simple teams that can be granted access to an investigation
  as a unit
- **Targets** — `username`, `domain`, `email`, `ip`, `org`, or `person`,
  each with type-specific input validation
- **9 OSINT collectors**, all free/open-source, run as background jobs
  through a shared queue worker:

  | Source | Target type | What it does |
  |---|---|---|
  | [Sherlock](https://github.com/sherlock-project/sherlock) | `username` | Checks presence across 400+ social/community sites |
  | [theHarvester](https://github.com/laramies/theHarvester) | `domain` | Emails/subdomains/hosts from crt.sh, OTX, HackerTarget, and more |
  | [crt.sh](https://crt.sh) | `domain` | Subdomain enumeration via certificate transparency logs |
  | WHOIS | `domain` | Registrar, dates, name servers |
  | DNS Records | `domain` | A/AAAA/MX/NS/TXT/CNAME lookup |
  | [Holehe](https://github.com/megadose/holehe) | `email` | Checks if an email is registered on 120+ sites |
  | Gravatar | `email` | Checks for a registered Gravatar image |
  | [ip-api.com](https://ip-api.com) | `ip` | IP geolocation |
  | [OpenCorporates](https://opencorporates.com) | `org` | Company registration search — needs a free API token, see below |

- **Bulk actions** — run every applicable source against a target, or every
  target in an investigation, in one click; skips sources already in
  flight and automatically retries prior failures
- **Search** — find investigations by name or targets by value, scoped to
  what you have access to
- **Reporting** — printable HTML report (browser print-to-PDF), CSV export,
  JSON export

## Stack

- Laravel 13 (PHP 8.4), Breeze (Blade + Vite) for auth
- MySQL for storage, `database` queue driver
- Python 3.10 venvs for the Python-based collectors (Sherlock, Holehe,
  WHOIS, theHarvester); everything else is pure PHP (HTTP calls or
  `dns_get_record()`), no external process needed

See [`PLAN.md`](PLAN.md) for the full build history, design decisions, and
known gotchas — it's the more detailed companion to this file.

## Setup

### 1. PHP dependencies

```bash
composer install
cp .env.example .env
php artisan key:generate
```

### 2. Database

Point `.env` at a MySQL database (`DB_*` variables) and run:

```bash
php artisan migrate
php artisan db:seed --class=SourceSeeder
```

### 3. Frontend assets

```bash
npm install
npm run build   # or `npm run dev` while developing
```

### 4. Python collectors

```bash
scripts/python/setup.sh
```

Sets up two venvs (`scripts/python/.venv` for Sherlock/whois/Holehe,
`scripts/python/.venv-theharvester` for theHarvester, which has much
heavier dependencies) and vendors theHarvester's source. See
[`scripts/python/README.md`](scripts/python/README.md) for details.

### 5. Queue worker

Lookups run as background jobs — nothing executes until a worker is
running. For local development:

```bash
php artisan queue:work
```

For anything longer-lived, install the supervised `systemd --user` service
in [`deploy/systemd/`](deploy/systemd/README.md) instead.

### 6. Optional: OpenCorporates

The `org` collector needs a free API token. Register at
[opencorporates.com/api_accounts/new](https://opencorporates.com/api_accounts/new)
and set `OPENCORPORATES_API_TOKEN` in `.env`. Without one, that source is
hidden rather than shown broken. Restart the queue worker after setting it.

### 7. Run it

```bash
php artisan serve
```

## Testing

```bash
php artisan test
```

Tests run against a separate `osint_testing` database, configured in
`phpunit.xml` — never against your real data.

## Project layout

- `app/Osint/` — the collector framework: `Runner` interface,
  `RunnerRegistry` (maps a source's `key` to its implementation),
  `Runners/` (one class per collector), `FindingDispatcher` (bulk-run logic)
- `app/Jobs/RunFinding.php` — queued job that executes a runner and updates
  the `Finding` row, with per-source rate limiting
- `app/Models/` — `Investigation`, `Target`, `Source`, `Finding`, `Group`,
  `User`
- `scripts/python/` — Python collectors and their setup script
- `deploy/systemd/` — the queue worker's supervised-process definition