Add Gravatar and DNS Records collectors

Two pure-PHP collectors, no Python/venv/subprocess needed:
- GravatarRunner (email): checks gravatar.com/avatar/{md5}?d=404 for a
  200 vs 404. Scoped to that binary signal since Gravatar's profile
  JSON endpoint now requires API auth even for real, public profiles
  (confirmed against Gravatar's own creator's email).
- DnsRunner (domain): dns_get_record() for A/AAAA/MX/NS/TXT/CNAME. Raw
  payload is naturally keyed by record type, which the existing
  generic list-renderer on the investigation page turns into one
  collapsible section per type with no UI changes needed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
cedry2k1992
2026-07-20 11:59:20 +03:00
co-authored by Claude Sonnet 5
parent bb15f13bac
commit 86f73c46b2
8 changed files with 296 additions and 3 deletions
+32 -3
View File
@@ -51,9 +51,9 @@ commands).
- **Targets** — the entity being investigated (person, domain, email, - **Targets** — the entity being investigated (person, domain, email,
username, IP, org), scoped to one investigation username, IP, org), scoped to one investigation
- **Sources** — catalog of OSINT tools/techniques (seeded: Sherlock, - **Sources** — catalog of OSINT tools/techniques (seeded: Sherlock,
theHarvester, crt.sh, WHOIS, ip-api.com, Holehe — all 6 now have a theHarvester, crt.sh, WHOIS, ip-api.com, Holehe, Gravatar, DNS
working runner). `sources.runner` is a display-only label; the real Records — all 8 now have a working runner). `sources.runner` is a
mapping is `App\Osint\RunnerRegistry` display-only label; the real mapping is `App\Osint\RunnerRegistry`
- **Findings** — result of running a source against a target; raw JSON - **Findings** — result of running a source against a target; raw JSON
payload preserved plus a parsed summary payload preserved plus a parsed summary
@@ -228,6 +228,35 @@ commands).
live systemd worker, watched it fail with the exit-code-1 quirk live systemd worker, watched it fail with the exit-code-1 quirk
before the fix and complete cleanly after, confirmed the summary before the fix and complete cleanly after, confirmed the summary
rendered correctly in the UI. Test data cleaned up afterward. rendered correctly in the UI. Test data cleaned up afterward.
11. **Gravatar + DNS Records collectors** (done) — 7th and 8th
sources, both pure PHP (no Python/venv, no external process at
all for DNS).
- `GravatarRunner` (email) — MD5-hashes the email, checks
`gravatar.com/avatar/{hash}?d=404` for a 200 vs 404. Gravatar's
public profile JSON endpoint (`{hash}.json`) turned out to now
require API auth — confirmed by hand against a real profile
(Matt Mullenweg's, Gravatar's own creator: avatar check returned
200, `.json` returned 404 regardless) — so this stays scoped to
the binary "does an avatar exist" signal rather than pulling
profile details. Unit-tested with `Http::fake()` (avatar
found/not-found/error, invalid email, wrong target type).
- `DnsRunner` (domain) — pure PHP `dns_get_record()`, no HTTP
client or subprocess. Looks up A/AAAA/MX/NS/TXT/CNAME. Its raw
payload is naturally keyed by record type (`A`, `MX`, ...),
which the existing generic list-renderer in the investigation
show page turns into one collapsible section per record type
automatically — no UI changes needed. Unit-tested against a real
DNS lookup (`example.com`, IANA-controlled, stable for decades)
plus a nonexistent-domain case, since `dns_get_record()` isn't
mockable through `Http::fake()`.
- Courtesy rate limits: Gravatar 45/min (cheap single GET), DNS
60/min (local resolver, cheapest operation in the app).
- 9 new tests (84/84 passing). Manually verified end-to-end
through the live systemd worker: Gravatar against a real email
with a known public avatar (found it), DNS against `google.com`
(22 records across all 5 present types, each rendering as its
own labeled section in the UI with correct counts and values).
Test data cleaned up afterward.
## Open questions ## Open questions
- Any legal/ToS constraints on the sources to be queried? - Any legal/ToS constraints on the sources to be queried?
+4
View File
@@ -3,6 +3,8 @@
namespace App\Osint; namespace App\Osint;
use App\Osint\Runners\CrtShRunner; use App\Osint\Runners\CrtShRunner;
use App\Osint\Runners\DnsRunner;
use App\Osint\Runners\GravatarRunner;
use App\Osint\Runners\HoleheRunner; use App\Osint\Runners\HoleheRunner;
use App\Osint\Runners\IpApiRunner; use App\Osint\Runners\IpApiRunner;
use App\Osint\Runners\SherlockRunner; use App\Osint\Runners\SherlockRunner;
@@ -25,6 +27,8 @@ class RunnerRegistry
'ip-api' => IpApiRunner::class, 'ip-api' => IpApiRunner::class,
'theharvester' => TheHarvesterRunner::class, 'theharvester' => TheHarvesterRunner::class,
'holehe' => HoleheRunner::class, 'holehe' => HoleheRunner::class,
'gravatar' => GravatarRunner::class,
'dns' => DnsRunner::class,
]; ];
public function has(string $sourceKey): bool public function has(string $sourceKey): bool
+74
View File
@@ -0,0 +1,74 @@
<?php
namespace App\Osint\Runners;
use App\Models\Target;
use App\Osint\Runner;
use App\Osint\RunnerException;
use App\Osint\RunnerResult;
class DnsRunner implements Runner
{
/**
* @var array<string, int>
*/
private const RECORD_TYPES = [
'A' => DNS_A,
'AAAA' => DNS_AAAA,
'MX' => DNS_MX,
'NS' => DNS_NS,
'TXT' => DNS_TXT,
'CNAME' => DNS_CNAME,
];
public function run(Target $target): RunnerResult
{
if ($target->type !== 'domain') {
throw new RunnerException('DNS lookup only supports "domain" targets.');
}
if (! preg_match('/^([a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}$/', $target->value)) {
throw new RunnerException('Domain contains characters that are not safe to look up.');
}
$records = [];
foreach (self::RECORD_TYPES as $label => $type) {
// dns_get_record() emits a warning (not an exception) when a
// record type doesn't resolve for the domain — expected for
// most domains on most types, not a real error.
$found = @dns_get_record($target->value, $type);
if ($found === false || empty($found)) {
continue;
}
$records[$label] = array_values(array_unique(array_map(
fn (array $record) => $this->formatRecord($label, $record),
$found
)));
}
$total = array_sum(array_map('count', $records));
$summary = $total > 0
? sprintf('Found %d DNS record%s across %d type%s (%s).',
$total, $total === 1 ? '' : 's',
count($records), count($records) === 1 ? '' : 's',
implode(', ', array_keys($records)))
: 'No DNS records found.';
return new RunnerResult(raw: $records, summary: $summary);
}
private function formatRecord(string $label, array $record): string
{
return match ($label) {
'A', 'AAAA' => $record['ip'] ?? '',
'MX' => sprintf('%d %s', $record['pri'] ?? 0, $record['target'] ?? ''),
'NS', 'CNAME' => $record['target'] ?? '',
'TXT' => $record['txt'] ?? '',
default => '',
};
}
}
+49
View File
@@ -0,0 +1,49 @@
<?php
namespace App\Osint\Runners;
use App\Models\Target;
use App\Osint\Runner;
use App\Osint\RunnerException;
use App\Osint\RunnerResult;
use Illuminate\Support\Facades\Http;
class GravatarRunner implements Runner
{
public function run(Target $target): RunnerResult
{
if ($target->type !== 'email') {
throw new RunnerException('Gravatar only supports "email" targets.');
}
if (! filter_var($target->value, FILTER_VALIDATE_EMAIL)) {
throw new RunnerException('Value is not a valid email address.');
}
// Gravatar's public profile JSON endpoint now requires API auth
// (returns 404 even for real profiles without a key), but the
// plain avatar-existence check is still open and reliable.
$hash = md5(strtolower(trim($target->value)));
$avatarUrl = "https://www.gravatar.com/avatar/{$hash}";
$response = Http::timeout((int) config('osint.run_timeout'))
->get($avatarUrl, ['d' => '404']);
if ($response->status() !== 200 && $response->status() !== 404) {
throw new RunnerException('Gravatar request failed with status '.$response->status().'.');
}
$hasAvatar = $response->status() === 200;
return new RunnerResult(
raw: [
'has_avatar' => $hasAvatar,
'avatar_url' => $hasAvatar ? $avatarUrl : null,
'profile_url' => "https://gravatar.com/{$hash}",
],
summary: $hasAvatar
? 'Gravatar image found — this email is registered with Gravatar.'
: 'No Gravatar found for this email.',
);
}
}
+2
View File
@@ -30,5 +30,7 @@ class AppServiceProvider extends ServiceProvider
RateLimiter::for('osint-runner:sherlock', fn () => Limit::perMinute(10)); RateLimiter::for('osint-runner:sherlock', fn () => Limit::perMinute(10));
RateLimiter::for('osint-runner:theharvester', fn () => Limit::perMinute(5)); RateLimiter::for('osint-runner:theharvester', fn () => Limit::perMinute(5));
RateLimiter::for('osint-runner:holehe', fn () => Limit::perMinute(15)); RateLimiter::for('osint-runner:holehe', fn () => Limit::perMinute(15));
RateLimiter::for('osint-runner:gravatar', fn () => Limit::perMinute(45));
RateLimiter::for('osint-runner:dns', fn () => Limit::perMinute(60));
} }
} }
+14
View File
@@ -56,6 +56,20 @@ class SourceSeeder extends Seeder
'target_types' => ['email'], 'target_types' => ['email'],
'runner' => 'python:holehe', 'runner' => 'python:holehe',
], ],
[
'key' => 'gravatar',
'name' => 'Gravatar',
'description' => 'Checks whether an email address has a registered Gravatar image.',
'target_types' => ['email'],
'runner' => 'http:gravatar',
],
[
'key' => 'dns',
'name' => 'DNS Records',
'description' => 'Looks up A, AAAA, MX, NS, TXT, and CNAME records for a domain.',
'target_types' => ['domain'],
'runner' => 'dns:records',
],
]; ];
foreach ($sources as $source) { foreach ($sources as $source) {
+54
View File
@@ -0,0 +1,54 @@
<?php
namespace Tests\Unit\Osint;
use App\Models\Target;
use App\Osint\Runners\DnsRunner;
use App\Osint\RunnerException;
use Tests\TestCase;
class DnsRunnerTest extends TestCase
{
public function test_rejects_non_domain_targets(): void
{
$target = new Target(['type' => 'ip', 'value' => '8.8.8.8']);
$this->expectException(RunnerException::class);
(new DnsRunner)->run($target);
}
public function test_rejects_unsafe_domain_format(): void
{
$target = new Target(['type' => 'domain', 'value' => '../../etc/passwd']);
$this->expectException(RunnerException::class);
(new DnsRunner)->run($target);
}
/**
* example.com is IANA-controlled and has had stable NS/A records for
* decades — safe to rely on for a real (non-mocked) DNS lookup.
*/
public function test_resolves_a_real_stable_domain(): void
{
$target = new Target(['type' => 'domain', 'value' => 'example.com']);
$result = (new DnsRunner)->run($target);
$this->assertArrayHasKey('NS', $result->raw);
$this->assertNotEmpty($result->raw['NS']);
$this->assertStringContainsString('DNS record', $result->summary);
}
public function test_reports_no_records_for_a_nonexistent_domain(): void
{
$target = new Target(['type' => 'domain', 'value' => 'this-domain-should-not-exist-osint-test.invalid']);
$result = (new DnsRunner)->run($target);
$this->assertSame([], $result->raw);
$this->assertSame('No DNS records found.', $result->summary);
}
}
+67
View File
@@ -0,0 +1,67 @@
<?php
namespace Tests\Unit\Osint;
use App\Models\Target;
use App\Osint\Runners\GravatarRunner;
use App\Osint\RunnerException;
use Illuminate\Support\Facades\Http;
use Tests\TestCase;
class GravatarRunnerTest extends TestCase
{
public function test_reports_avatar_found(): void
{
Http::fake(['gravatar.com/avatar/*' => Http::response('', 200)]);
$target = new Target(['type' => 'email', 'value' => 'test@example.com']);
$result = (new GravatarRunner)->run($target);
$this->assertTrue($result->raw['has_avatar']);
$this->assertNotNull($result->raw['avatar_url']);
$this->assertStringContainsString('Gravatar image found', $result->summary);
}
public function test_reports_no_avatar(): void
{
Http::fake(['gravatar.com/avatar/*' => Http::response('', 404)]);
$target = new Target(['type' => 'email', 'value' => 'test@example.com']);
$result = (new GravatarRunner)->run($target);
$this->assertFalse($result->raw['has_avatar']);
$this->assertNull($result->raw['avatar_url']);
$this->assertSame('No Gravatar found for this email.', $result->summary);
}
public function test_rejects_non_email_targets(): void
{
$target = new Target(['type' => 'domain', 'value' => 'example.com']);
$this->expectException(RunnerException::class);
(new GravatarRunner)->run($target);
}
public function test_rejects_invalid_email_format(): void
{
$target = new Target(['type' => 'email', 'value' => 'not-an-email']);
$this->expectException(RunnerException::class);
(new GravatarRunner)->run($target);
}
public function test_throws_on_unexpected_status(): void
{
Http::fake(['gravatar.com/avatar/*' => Http::response('', 500)]);
$target = new Target(['type' => 'email', 'value' => 'test@example.com']);
$this->expectException(RunnerException::class);
(new GravatarRunner)->run($target);
}
}