analytics/STATUS.md

# Analytics Platform — Project Status

> Last updated: 2026-05-25

---

## What Is This

A self-hosted web analytics platform built with Laravel 11 — a privacy-focused alternative to Google Analytics or Plausible. Users add a lightweight JavaScript snippet to their website and get real-time dashboards, AI-ready data pipelines, goal/funnel tracking, and a full REST API.

---

## Tech Stack

| Layer | Choice |
|---|---|
| Backend | Laravel 11, PHP 8.2+ |
| Auth | Laravel Fortify (2FA ready) |
| Frontend | Blade + Alpine.js + Tailwind CSS |
| Charts | Chart.js |
| Database | SQLite (dev) / MySQL or PostgreSQL (prod) |
| Queue | Database driver (dev) / Redis + Horizon (prod) |
| API | Laravel Sanctum (token auth) |
| Geolocation | stevebauman/location |
| Bot detection | jaybizzle/crawler-detect |
| Build | Vite |

---

## Feature Status

### ✅ Done — Core Platform

**Authentication & Users**
- Registration, login, password reset, email verification via Fortify
- Two-factor authentication support
- Profile management
- API token management (Sanctum)

**Website Management**
- Full CRUD for tracked websites
- Embed snippet page with copy-ready `<script>` tag
- WebsitePolicy — all queries scoped to authenticated owner

**Data Collection Pipeline**
- `POST /api/e` and `POST /api/collect` endpoints
- Async ingestion via `ProcessTrackingPayload` queued job
- Bot filtering (CrawlerDetect)
- Domain validation (rejects spoofed tracking IDs)
- IP geolocation → country, region, city on Visitor
- CORS middleware for cross-origin beacons
- Tracking pixel (1×1 GIF) as no-JS fallback
- Rate limiting (120 req/min per IP)
- Ad-blocker resistant alternate endpoints (`/cdn/feedback`, `/cdn/resources/`)

**Dashboard**
- Date-range picker (today / 7d / 30d / custom)
- Metric cards: page views, unique visitors, sessions, bounce rate, avg session duration, pages/session
- Visitors & page views chart (daily/hourly granularity)
- Real-time active visitors widget (last 5 minutes)
- Comparison mode vs. previous period with delta %
- Goal conversion sparklines

**Reports — Audience**
- Geographic: visitors by country, region, city (table + world map)
- Technology: browser, OS, device type, screen resolution
- Language: browser language breakdown
- New vs. returning visitors

**Reports — Acquisition**
- Traffic sources (direct, organic, referral, social)
- UTM campaigns (source, medium, campaign)
- Top referrers

**Reports — Behavior**
- Top pages (views, unique views, avg time, bounce rate)
- Landing pages (entry pages with session count and bounce rate)
- Exit pages

**Reports — Events**
- Events overview (grouped by name/category/action)
- Event detail drill-down (label/value breakdown)
- Event trending chart

**Reports — Conversions**
- Goal completions, conversion rate, goal value totals

**Goals & Conversions**
- 4 goal types: URL destination, event match, duration, pages per session
- GoalEvaluationService — evaluates goals on every incoming payload
- Conversion rate reporting

**Advanced Features**
- Funnels — multi-step conversion tracking with drop-off visualization
- Segments — filter all reports by visitor attributes
- Custom dashboards — pin widgets to a personal overview
- Alerts — notify on traffic spikes/drops/goal thresholds (email + in-app)
- Annotations — add notes to specific dates on charts
- Data export — CSV export for any report
- UTM builder — in-app campaign URL generator

**API (v1)**
- `GET /api/v1/websites` — list websites
- `GET /api/v1/websites/{id}/stats` — summary metrics with date-range params
- `GET /api/v1/websites/{id}/pages` — top pages
- `GET /api/v1/websites/{id}/referrers` — traffic sources
- `GET /api/v1/websites/{id}/events` — events
- `GET /api/v1/websites/{id}/visitors` — audience breakdown
- JSON:API resource classes for consistent formatting
- Rate limiting per token

**Data Aggregation**
- `daily_aggregates` table — pre-computed per-website daily metrics
- `page_daily_aggregates` table — per-page daily metrics
- `analytics:aggregate` Artisan command (runs daily at 02:00)
- Hybrid query strategy: aggregates for past days, raw tables for today
- `analytics:purge` command — purge raw data older than retention window
- Configurable `data_retention_days` per website (default: 365)

**Performance & Indexes**
- Composite indexes on all date-range + website_id query paths
- SQLite-compatible index migrations (also works with MySQL/PostgreSQL)

**Testing**
- 80 tests, 0 failures
- Feature tests: `CollectController`, `WebsiteController`, Auth flows
- Unit tests: `AnalyticsQueryService`, `GoalEvaluationService`
- Model factories for all analytics models
- `withoutVite()` in base TestCase — tests run without a built frontend

---

### 🔲 Not Yet Done

| Item | Notes |
|---|---|
| **User Flow / Sankey diagram** | Intentionally deferred — complex to visualize, low priority for v1 |
| **Production database** | SQLite is dev-only; set up MySQL or PostgreSQL before launch |
| **Redis + Horizon** | Replace database queue driver for real traffic; Horizon gives a monitoring UI |
| **Cron / scheduler** | Configure `schedule:run` on the server — needed for aggregation, purge, and alerts |
| **HTTPS + CDN for tracker JS** | Tracker script should load from a CDN subdomain to avoid ad-blockers |
| **Error monitoring** | Wire up Sentry or Flare for production error tracking |
| **Database backups** | Automate daily dumps with offsite storage |
| **Dusk browser tests** | UI interaction tests deferred — requires Dusk + ChromeDriver setup |

---

## Running Locally

```bash
cd ~/Projects/analytics
composer install
cp .env.example .env
php artisan key:generate
php artisan migrate
npm install && npm run build
composer run dev       # starts web server + queue worker + log viewer + Vite
```

App runs at `http://localhost:8000`.

### Background processes required

```bash
# Queue worker — processes tracking events (critical)
php artisan queue:work database --tries=3

# Scheduler — aggregation, purge, alerts (critical)
php artisan schedule:work

# OR add to crontab for production:
# * * * * * cd /path/to/analytics && php artisan schedule:run >> /dev/null 2>&1
```

---

## Deployment Checklist

- [ ] Provision server (Ubuntu 22+ recommended)
- [ ] Install PHP 8.2+, Nginx, MySQL/PostgreSQL, Redis, Node
- [ ] Clone repo, `composer install --no-dev`, `npm ci && npm run build`
- [ ] Set `APP_ENV=production`, `APP_DEBUG=false`, configure `.env`
- [ ] Run `php artisan migrate --force`
- [ ] Run `php artisan config:cache route:cache view:cache`
- [ ] Configure Supervisor for queue worker (see `HOWTO.md`)
- [ ] Add crontab entry for `schedule:run`
- [ ] Point Nginx document root to `public/`
- [ ] Set up SSL (Let's Encrypt)
- [ ] Set up CNAME for tracker script (e.g. `track.yourdomain.com`)
- [ ] Configure Sentry DSN in `.env`
- [ ] Set up automated database backups

---

## Key Artisan Commands

| Command | Description | Schedule |
|---|---|---|
| `analytics:aggregate` | Roll up yesterday's raw data into aggregates | Daily 02:00 |
| `analytics:purge --force` | Delete raw data older than retention window | Daily 03:00 |
| `analytics:check-alerts` | Evaluate alert rules and send notifications | Every hour |
| `queue:work database --tries=3` | Process tracking payloads | Always-on process |

---

## Architecture Notes

### Data Flow
```
Client website (JS snippet)
  → POST /api/e  (CollectController)
  → ProcessTrackingPayload job (queued)
  → Creates/updates: Visitor, AnalyticsSession, PageView or Event
  → GoalEvaluationService checks active goals
  → Daily: analytics:aggregate rolls raw → daily_aggregates
  → Dashboard queries: aggregates for history + raw tables for today
```

### Collector Extraction (future)
The `collector_app_plan.md` documents a plan to split the data collection endpoints into a separate Laravel app for independent scaling. This is not needed yet but the code is structured to make it straightforward — `CollectController`, `TrackingScriptController`, `ProcessTrackingPayload`, and `GoalEvaluationService` are all self-contained.

---

## Database Schema (Core Tables)

| Table | Purpose |
|---|---|
| `websites` | Tracked websites, settings, tracking ID |
| `visitors` | Unique visitors with device, browser, location |
| `analytics_sessions` | Sessions with UTM params, entry/exit page, duration |
| `page_views` | Individual page visits |
| `events` | Custom events with category, label, value |
| `goals` | Conversion goal definitions (4 types) |
| `goal_conversions` | Goal completion records |
| `funnels` | Multi-step funnel definitions |
| `segments` | Custom visitor segment definitions |
| `alerts` | Alert rule definitions |
| `alert_notifications` | Triggered alert history |
| `annotations` | Date annotations visible on charts |
| `dashboard_widgets` | Custom dashboard layout per user |
| `daily_aggregates` | Pre-computed daily metrics per website |
| `page_daily_aggregates` | Pre-computed daily metrics per page |