cedry2k1992andClaude Sonnet 5
5e8b1c8a15
Describes what this actually is (multi-user OSINT investigation platform), the 9 collectors, and step-by-step setup (DB, Python venvs, queue worker, optional OpenCorporates token). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
137 lines
4.7 KiB
Markdown
137 lines
4.7 KiB
Markdown
# OSINT
|
|
|
|
A multi-user OSINT (Open Source Intelligence) investigation platform built on
|
|
Laravel. Investigators create **investigations**, add **targets** (a
|
|
username, domain, email, IP, or org name), and run free/open-source OSINT
|
|
tools against them. Results are collected as **findings**, searchable across
|
|
everything you have access to, and exportable as a printable report or
|
|
CSV/JSON.
|
|
|
|
Multi-user by design: each investigation has an owner who controls access,
|
|
granting `viewer` or `editor` roles to individual users or whole groups.
|
|
|
|
## Features
|
|
|
|
- **Investigations** — cases that group targets and findings, with
|
|
owner-controlled access grants (per-user or per-group, `viewer`/`editor`)
|
|
- **Groups** — simple teams that can be granted access to an investigation
|
|
as a unit
|
|
- **Targets** — `username`, `domain`, `email`, `ip`, `org`, or `person`,
|
|
each with type-specific input validation
|
|
- **9 OSINT collectors**, all free/open-source, run as background jobs
|
|
through a shared queue worker:
|
|
|
|
| Source | Target type | What it does |
|
|
|---|---|---|
|
|
| [Sherlock](https://github.com/sherlock-project/sherlock) | `username` | Checks presence across 400+ social/community sites |
|
|
| [theHarvester](https://github.com/laramies/theHarvester) | `domain` | Emails/subdomains/hosts from crt.sh, OTX, HackerTarget, and more |
|
|
| [crt.sh](https://crt.sh) | `domain` | Subdomain enumeration via certificate transparency logs |
|
|
| WHOIS | `domain` | Registrar, dates, name servers |
|
|
| DNS Records | `domain` | A/AAAA/MX/NS/TXT/CNAME lookup |
|
|
| [Holehe](https://github.com/megadose/holehe) | `email` | Checks if an email is registered on 120+ sites |
|
|
| Gravatar | `email` | Checks for a registered Gravatar image |
|
|
| [ip-api.com](https://ip-api.com) | `ip` | IP geolocation |
|
|
| [OpenCorporates](https://opencorporates.com) | `org` | Company registration search — needs a free API token, see below |
|
|
|
|
- **Bulk actions** — run every applicable source against a target, or every
|
|
target in an investigation, in one click; skips sources already in
|
|
flight and automatically retries prior failures
|
|
- **Search** — find investigations by name or targets by value, scoped to
|
|
what you have access to
|
|
- **Reporting** — printable HTML report (browser print-to-PDF), CSV export,
|
|
JSON export
|
|
|
|
## Stack
|
|
|
|
- Laravel 13 (PHP 8.4), Breeze (Blade + Vite) for auth
|
|
- MySQL for storage, `database` queue driver
|
|
- Python 3.10 venvs for the Python-based collectors (Sherlock, Holehe,
|
|
WHOIS, theHarvester); everything else is pure PHP (HTTP calls or
|
|
`dns_get_record()`), no external process needed
|
|
|
|
See [`PLAN.md`](PLAN.md) for the full build history, design decisions, and
|
|
known gotchas — it's the more detailed companion to this file.
|
|
|
|
## Setup
|
|
|
|
### 1. PHP dependencies
|
|
|
|
```bash
|
|
composer install
|
|
cp .env.example .env
|
|
php artisan key:generate
|
|
```
|
|
|
|
### 2. Database
|
|
|
|
Point `.env` at a MySQL database (`DB_*` variables) and run:
|
|
|
|
```bash
|
|
php artisan migrate
|
|
php artisan db:seed --class=SourceSeeder
|
|
```
|
|
|
|
### 3. Frontend assets
|
|
|
|
```bash
|
|
npm install
|
|
npm run build # or `npm run dev` while developing
|
|
```
|
|
|
|
### 4. Python collectors
|
|
|
|
```bash
|
|
scripts/python/setup.sh
|
|
```
|
|
|
|
Sets up two venvs (`scripts/python/.venv` for Sherlock/whois/Holehe,
|
|
`scripts/python/.venv-theharvester` for theHarvester, which has much
|
|
heavier dependencies) and vendors theHarvester's source. See
|
|
[`scripts/python/README.md`](scripts/python/README.md) for details.
|
|
|
|
### 5. Queue worker
|
|
|
|
Lookups run as background jobs — nothing executes until a worker is
|
|
running. For local development:
|
|
|
|
```bash
|
|
php artisan queue:work
|
|
```
|
|
|
|
For anything longer-lived, install the supervised `systemd --user` service
|
|
in [`deploy/systemd/`](deploy/systemd/README.md) instead.
|
|
|
|
### 6. Optional: OpenCorporates
|
|
|
|
The `org` collector needs a free API token. Register at
|
|
[opencorporates.com/api_accounts/new](https://opencorporates.com/api_accounts/new)
|
|
and set `OPENCORPORATES_API_TOKEN` in `.env`. Without one, that source is
|
|
hidden rather than shown broken. Restart the queue worker after setting it.
|
|
|
|
### 7. Run it
|
|
|
|
```bash
|
|
php artisan serve
|
|
```
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
php artisan test
|
|
```
|
|
|
|
Tests run against a separate `osint_testing` database, configured in
|
|
`phpunit.xml` — never against your real data.
|
|
|
|
## Project layout
|
|
|
|
- `app/Osint/` — the collector framework: `Runner` interface,
|
|
`RunnerRegistry` (maps a source's `key` to its implementation),
|
|
`Runners/` (one class per collector), `FindingDispatcher` (bulk-run logic)
|
|
- `app/Jobs/RunFinding.php` — queued job that executes a runner and updates
|
|
the `Finding` row, with per-source rate limiting
|
|
- `app/Models/` — `Investigation`, `Target`, `Source`, `Finding`, `Group`,
|
|
`User`
|
|
- `scripts/python/` — Python collectors and their setup script
|
|
- `deploy/systemd/` — the queue worker's supervised-process definition
|