trackbugs/PLAN.md

# Error Tracker — Build Plan

A self-hosted error tracking application (inspired by [Bugsink](https://www.bugsink.com/) / Sentry) built on this Laravel 13 app.

## Overview

This is an **error tracking application similar to Sentry**, designed to receive and aggregate errors from applications running in production. Users integrate SDKs into their applications (PHP, JavaScript, Python, etc.), and those SDKs send error events **directly** to our tracking server in real-time when errors occur.

### How It Works

1. **SDK Integration**: Developers install a Sentry-compatible SDK in their application (e.g., `sentry/sentry-laravel` for Laravel, `@sentry/browser` for JavaScript, `sentry-python` for Python).
2. **Configuration**: They configure the SDK with a DSN (Data Source Name) pointing to our tracking server.
3. **Automatic Error Capture**: When an error occurs in their application, the SDK automatically captures the error context (stacktrace, breadcrumbs, user info, request details) and sends it to our server.
4. **Real-Time Processing**: Our server receives the error payload, groups similar errors into issues, and displays them in a web dashboard.
5. **Notifications**: Teams are notified when new issues occur or when resolved issues regress.

This architecture means we don't need to build custom SDKs — we leverage the entire ecosystem of existing Sentry SDKs by implementing a Sentry-compatible ingestion API.

## Stack Summary

| Layer | Technology |
|-------|-----------|
| Backend | Laravel 13, PHP 8.3+ |
| Database | SQLite (dev), easily swappable to Postgres/MySQL |
| Frontend | Blade + Tailwind CSS 4 + Vite 8 |
| Queue | Database driver (upgrade to Redis for production) |
| Auth | Laravel Breeze (installed) |

## SDK Support & Multi-Language Integration

### Supported Languages & Frameworks

Our tracking server accepts error events from **any Sentry-compatible SDK**, enabling broad language and framework support without custom development:

| Language/Platform | Official Sentry SDK | Notes |
|-------------------|---------------------|-------|
| **PHP** | `sentry/sentry-laravel`, `sentry/sentry-symfony`, `sentry/sentry` | Full Laravel, Symfony, and native PHP support |
| **JavaScript/Node.js** | `@sentry/browser`, `@sentry/node`, `@sentry/react`, `@sentry/vue`, `@sentry/angular` | Frontend and backend JS, plus framework-specific integrations |
| **Python** | `sentry-sdk` | Django, Flask, FastAPI, etc. |
| **Ruby** | `sentry-ruby`, `sentry-rails` | Rails and native Ruby support |
| **Java** | `io.sentry:sentry` | Spring, Android, and JVM languages |
| **Go** | `github.com/getsentry/sentry-go` | Native Go support |
| **.NET/C#** | `Sentry.AspNetCore`, `Sentry` | ASP.NET Core, .NET Framework |
| **Mobile** | `@sentry/react-native`, `sentry-android`, `sentry-cocoa` | React Native, Android (Java/Kotlin), iOS (Swift/Obj-C) |
| **Rust** | `sentry` | Native Rust support |
| **Elixir** | `sentry` | Phoenix and Elixir support |

### SDK Configuration Example

Users simply point the official Sentry SDK to our server:

**PHP (Laravel):**
```php
// config/sentry.php
'dsn' => 'https://<public_key>@trackbugs.example.com/api/<project_id>/store/',
```

**JavaScript:**
```javascript
Sentry.init({
  dsn: 'https://<public_key>@trackbugs.example.com/api/<project_id>/store/',
});
```

**Python:**
```python
import sentry_sdk
sentry_sdk.init(
    dsn="https://<public_key>@trackbugs.example.com/api/<project_id>/store/"
)
```

No custom SDK development is required — our server implements the Sentry protocol, so all existing SDKs work out of the box.

---

## Phase 1 — Foundation & Auth

### 1.1 Install Auth Scaffolding
- [x] Install Laravel Breeze (Blade + Tailwind stack) for login, registration, password reset, email verification.
- [x] Authenticated layout, middleware, and user management.

### 1.2 Core Data Models & Migrations

- [x] `projects` migration & `Project` model
- [x] `project_user` pivot migration
- [x] `issues` migration & `Issue` model
- [x] `events` migration & `Event` model
- [x] `User` model updated with `projects()` and `ownedProjects()` relationships

#### `projects` table
| Column | Type | Notes |
|--------|------|-------|
| id | ulid (pk) | |
| name | string | |
| platform | string (nullable) | e.g. "php", "javascript", "python" |
| dsn_public_key | string (unique) | Auto-generated UUID used in the DSN |
| dsn_secret_key | string (unique) | Auto-generated secret for server-side SDKs |
| created_at / updated_at | timestamps | |

#### `project_user` pivot table
| Column | Type | Notes |
|--------|------|-------|
| project_id | foreign | |
| user_id | foreign | |
| role | string | "owner", "member" |

#### `issues` table (groups of similar errors)
| Column | Type | Notes |
|--------|------|-------|
| id | ulid (pk) | |
| project_id | foreign | |
| fingerprint | string (index) | Hash used to group events |
| title | string | Short description, e.g. exception class + message |
| culprit | string (nullable) | File/function where it occurred |
| level | string | "fatal", "error", "warning", "info", "debug" |
| status | string | "unresolved", "resolved", "ignored", "muted" |
| first_seen_at | datetime | |
| last_seen_at | datetime | |
| times_seen | integer | Counter cache |
| is_regression | boolean | Re-opened after being resolved |
| metadata | json (nullable) | Platform, runtime, SDK info |
| created_at / updated_at | timestamps | |

#### `events` table (individual error occurrences)
| Column | Type | Notes |
|--------|------|-------|
| id | ulid (pk) | |
| issue_id | foreign | |
| project_id | foreign | |
| event_id | string (unique) | UUID sent by the SDK |
| level | string | |
| message | text (nullable) | |
| platform | string (nullable) | |
| timestamp | datetime | Event timestamp from SDK |
| server_name | string (nullable) | |
| release | string (nullable) | App version/release tag |
| environment | string (nullable) | "production", "staging", etc. |
| exception | json | Structured exception data (type, value, stacktrace) |
| breadcrumbs | json (nullable) | Trail of events leading up to the error |
| contexts | json (nullable) | OS, browser, device, runtime, etc. |
| tags | json (nullable) | Key-value pairs for filtering |
| extra | json (nullable) | Arbitrary extra data |
| user_context | json (nullable) | Info about the affected user |
| request_context | json (nullable) | HTTP method, URL, headers, body |
| sdk | json (nullable) | SDK name + version |
| raw_payload | json | Full original envelope for debugging |
| created_at | timestamp | |

---

## Phase 2 — Event Ingestion API

### 2.1 Sentry-Compatible Ingest Endpoint

Implement a `/api/{project_id}/store/` endpoint that accepts the Sentry event payload format. This allows using existing Sentry SDKs (for PHP, JS, Python, etc.) as clients without any custom SDK work.

- [x] `POST /api/{project_id}/store/` route
- [x] Validate DSN public key via `sentry_key` query param or `X-Sentry-Auth` header
- [x] `AuthenticateDsn` middleware
- [x] `EventIngestController`

### 2.2 Event Processing Pipeline

- [x] `ProcessEventJob` (queued)
- [x] `PayloadNormalizer` service
- [x] `EventProcessor` service (find or create Issue, store Event, update counters)
- [x] Regression detection (re-open resolved issues)

```
Ingest Request
  → Authenticate DSN key
  → Validate & normalize payload
  → Dispatch job to queue
      → Compute fingerprint (grouping)
      → Find or create Issue
      → Store Event
      → Update Issue counters (last_seen_at, times_seen)
      → Check if regression (status was "resolved")
      → Trigger notifications if needed
```

### 2.3 Fingerprinting / Grouping Strategy

- [x] `Fingerprinter` service

Group events into issues using a fingerprint hash built from:
1. Exception type (class name)
2. Exception message (with variable parts stripped, e.g. IDs, numbers)
3. Top N frames of the stacktrace (file + function)

Users can override with a custom `fingerprint` array in the event payload (Sentry SDK standard).

### 2.4 DSN Format

```
https://<public_key>@<host>/api/<project_id>/store/
```

Clients configure this DSN in their Sentry SDK — no changes needed on the SDK side.

---

## Phase 3 — Web Dashboard

### 3.1 Layouts & Navigation

- [x] Authenticated layout with sidebar navigation
- [x] Global project switcher in the sidebar/topbar
- [x] Sections: **Issues**, **Project Settings**, **Team Members**

### 3.2 Issues List Page (`/projects/{project}/issues`)

- [x] Table/list of issues sorted by last seen (most recent first)
- [x] Columns: status badge, title, culprit, event count sparkline, times seen, last seen
- [x] Filters: status (unresolved/resolved/ignored), level, environment, date range
- [x] Search: full-text search on title and culprit
- [x] Bulk actions: resolve, ignore, mute, delete
- [x] Pagination (cursor-based for performance)

### 3.3 Issue Detail Page (`/projects/{project}/issues/{issue}`)

- [x] Header: title, status controls (resolve / ignore / unresolve), level badge, first/last seen, times seen
- [x] **Stacktrace viewer:** collapsible frames, source context (if available), highlighted app frames vs vendor frames
- [x] **Breadcrumbs timeline:** chronological list of breadcrumb entries
- [x] **Tags panel:** key-value tag list with links to filter by tag
- [x] **Context panels:** device, OS, browser, runtime, user info, request details
- [x] **Events sub-list:** paginated list of individual occurrences with ability to view each one
- [ ] **Activity:** timeline of status changes (resolved, regressed, etc.)

### 3.4 Event Detail Page (`/projects/{project}/issues/{issue}/events/{event}`)

- [x] Full event data: raw JSON viewer + structured panels
- [x] Navigation: previous/next event within the issue

### 3.5 Project Settings Page

- [x] Rename project, change platform
- [x] DSN display with copy button
- [x] Regenerate keys
- [x] Delete project (with confirmation)

### 3.6 Dashboard / Overview Page

- [x] Summary cards: total unresolved issues, events in last 24h, most active projects
- [x] Line chart: events over time (last 7/14/30 days)

---

## Phase 4 — Notifications & Alerting

### 4.1 Notification Channels

- [x] **Email:** on first occurrence of a new issue (not on every event)
- [x] **Database (in-app):** notification bell with unread count
- [ ] Extensible to Slack/webhooks later

### 4.2 Alert Rules (per project)

- [x] Notify on: new issue, regression, spike (N events in M minutes)
- [x] Per-user mute controls

---

## Phase 5 — Polish & Advanced Features

### 5.1 Release Tracking

- [x] Parse `release` field from events
- [x] Show which release introduced an issue
- [x] Mark issues as resolved-in-release; auto-regress if they appear in a newer release

### 5.2 Environment Filtering

- [x] Global environment selector (production / staging / etc.)
- [x] Filter issues and events by environment

### 5.3 Retention & Cleanup

- [x] Artisan command / scheduled job to delete events older than N days (configurable)
- [x] Keep issue summaries even after events are pruned

### 5.4 Source Maps / Source Context (optional, stretch)

- [ ] Upload source maps for JS projects
- [ ] Store and display source context lines around each stacktrace frame

### 5.5 API Tokens & External API

- [x] Personal API tokens for programmatic access
- [x] REST API for listing/resolving issues (useful for CI/CD integrations)

---

## Implementation Order & Milestones

### Milestone 1 — MVP (walking skeleton)
- [x] 1. Install Breeze, set up auth
- [x] 2. Create migrations for `projects`, `issues`, `events`, `project_user`
- [x] 3. Build models with relationships
- [x] 4. Build the ingest endpoint (`POST /api/{project_id}/store/`)
- [x] 5. Implement fingerprinting and the `ProcessEventJob`
- [x] 6. Build a minimal issues list page and issue detail page
- [ ] 7. Test with a real Sentry SDK (e.g. `sentry/sentry-laravel` on another app pointing at this tracker)

### Milestone 2 — Usable product
- [x] 8. Project CRUD and settings pages
- [x] 9. Team/member management (invite users to projects)
- [x] 10. Stacktrace viewer with collapsible frames
- [x] 11. Breadcrumbs, tags, context panels on issue detail
- [x] 12. Filters, search, bulk actions on issues list
- [x] 13. Status management (resolve, ignore, mute, auto-regression detection)

### Milestone 3 — Notifications & dashboard
- [x] 14. Email notifications on new issue / regression
- [x] 15. In-app notification system
- [x] 16. Overview dashboard with charts (events over time)

### Milestone 4 — Production-readiness
- [ ] 17. Rate limiting on ingest
- [x] 18. Event retention / cleanup scheduler
- [x] 19. Release tracking
- [x] 20. Environment filtering
- [ ] 21. Performance optimization (indexes, query tuning, caching)

### Milestone 5 — Advanced (stretch)
- [ ] 22. Source map upload & processing
- [x] 23. API tokens & external REST API
- [ ] 24. Webhook / Slack notifications
- [ ] 25. CQRS-style read models for heavy dashboards (if needed)

---

## File / Directory Structure (planned)

```
app/
├── Http/
│   ├── Controllers/
│   │   ├── DashboardController.php
│   │   ├── ProjectController.php
│   │   ├── IssueController.php
│   │   ├── EventController.php
│   │   └── Api/
│   │       └── EventIngestController.php
│   ├── Middleware/
│   │   └── AuthenticateDsn.php
│   └── Requests/
│       ├── StoreProjectRequest.php
│       └── UpdateProjectRequest.php
├── Jobs/
│   └── ProcessEventJob.php
├── Models/
│   ├── Project.php
│   ├── Issue.php
│   └── Event.php
├── Notifications/
│   ├── NewIssueNotification.php
│   └── IssueRegressionNotification.php
├── Services/
│   ├── EventProcessor.php
│   ├── Fingerprinter.php
│   └── PayloadNormalizer.php
├── Policies/
│   ├── ProjectPolicy.php
│   └── IssuePolicy.php
database/
├── migrations/
│   ├── xxxx_create_projects_table.php
│   ├── xxxx_create_project_user_table.php
│   ├── xxxx_create_issues_table.php
│   └── xxxx_create_events_table.php
resources/
├── views/
│   ├── dashboard.blade.php
│   ├── projects/
│   │   ├── index.blade.php
│   │   ├── show.blade.php
│   │   ├── create.blade.php
│   │   └── edit.blade.php
│   ├── issues/
│   │   ├── index.blade.php
│   │   └── show.blade.php
│   └── events/
│       └── show.blade.php
routes/
├── web.php        (dashboard, projects, issues, events)
└── api.php        (event ingest endpoint)
```

---

## Key Design Decisions

1. **Sentry-compatible protocol** — Reuse the massive ecosystem of existing Sentry SDKs instead of building custom clients.
2. **Queue-based processing** — The ingest endpoint returns `202 Accepted` immediately and processes asynchronously, so SDK clients never block.
3. **ULID primary keys** — Sortable, time-ordered, no sequential ID leakage.
4. **Fingerprint-based grouping** — Same approach as Sentry/Bugsink; keeps the issues list manageable.
5. **Blade + Tailwind (no SPA)** — Keep complexity low; server-rendered pages with sprinkles of Alpine.js where needed (e.g. collapsible stacktrace frames, modals).
6. **SQLite for dev, Postgres for prod** — SQLite is fine for development and small-scale use. The JSON columns and indexing strategy work on both.