analytics/PLAN.md

# Analytics Platform — Implementation Plan

## Current State

The foundation is in place: 7 models with migrations (`Website`, `Visitor`, `AnalyticsSession`, `PageView`, `Event`, `Goal`, `GoalConversion`), a JavaScript tracker served dynamically per tracking ID, and a `POST /api/collect` endpoint that ingests page views and events. No authentication, no dashboard, no frontend yet.

---

## Phase 1: Authentication & Website Management

The platform needs user accounts and CRUD for tracked websites before anything else is usable.

- [x] Install Laravel Breeze (or Fortify) for registration, login, password reset, email verification
- [x] Build a `WebsiteController` with full CRUD (index, create, store, show, edit, update, destroy)
- [x] Create views/pages: list of websites, add/edit website form, website settings page
- [x] Build the "embed snippet" page — show the user the `<script>` tag to copy for each website
- [x] Add form request validation (`StoreWebsiteRequest`, `UpdateWebsiteRequest`)
- [x] Scope all website queries to `auth()->user()` (policy or middleware)
- [x] Add a `WebsitePolicy` to authorize show/edit/delete actions

---

## Phase 2: Dashboard — Real-Time Overview

The main dashboard a user sees after selecting a website, showing key metrics at a glance.

- [x] Create `DashboardController` scoped to a single website
- [x] Build a date-range picker component (today, last 7 days, last 30 days, custom range)
- [x] **Metric cards** — display for the selected period:
  - Total page views
  - Unique visitors
  - Total sessions
  - Bounce rate (% of sessions with 1 page view)
  - Average session duration
  - Average pages per session
- [x] **Visitors chart** — line/area chart of visitors and page views over time (daily or hourly granularity)
- [x] **Real-time widget** — show active visitors in the last 5 minutes (poll or WebSocket)
- [x] Create an `AnalyticsQueryService` (or similar) to encapsulate all aggregation queries, keeping controllers thin

---

## Phase 3: Detailed Reports

Individual report pages accessible from the dashboard sidebar.

### 3a — Audience Reports

- [x] **Geographic report** — visitors by country, region, city (table + world map visualization)
- [x] **Technology report** — breakdown by browser, OS, device type, screen resolution
- [x] **Language report** — visitors by browser language
- [x] **New vs. returning visitors** — segment visitors who have more than one session

### 3b — Acquisition Reports

- [x] **Traffic sources** — breakdown by referrer domain (direct, organic, referral, social)
- [x] **Campaigns (UTM)** — table of sessions grouped by `utm_source`, `utm_medium`, `utm_campaign`
- [x] **Referrers** — top referring URLs and domains

### 3c — Behavior Reports

- [x] **Pages report** — top pages by views, unique views, avg. time on page, bounce rate
- [x] **Landing pages** — top entry pages with session count and bounce rate
- [x] **Exit pages** — top exit pages
- [ ] **User flow** — simplified Sankey or funnel showing common page sequences (can be deferred)

### 3d — Events Report

- [x] **Events overview** — table of events grouped by name, category, action
- [x] **Event detail** — drill-down into a single event name with label/value breakdown
- [x] **Event trending** — chart of event counts over time

---

## Phase 4: Goals & Conversions

Allow users to define goals and track conversion rates.

- [x] Build `GoalController` CRUD (create, edit, delete goals per website)
- [x] Support goal types:
  - **URL destination** — session visited a specific path
  - **Event match** — a specific event name/category was fired
  - **Duration** — session lasted longer than N seconds
  - **Pages per session** — session hit N+ page views
- [x] Create a `GoalEvaluationService` that checks incoming data against active goals and records `GoalConversion` entries
- [x] Hook goal evaluation into `CollectController` after recording page view / event
- [x] Build a **Conversions report** — goal completion count, conversion rate (completions / sessions), goal value totals
- [x] Add goal completion sparklines to the main dashboard

---

## Phase 5: Collection Pipeline Hardening

The `CollectController` currently does synchronous DB writes. This won't scale under real traffic.

- [x] Move ingestion to a queued job (`ProcessTrackingPayload`) dispatched from the controller so the endpoint returns instantly
- [x] Add rate limiting middleware to `POST /api/collect` (e.g., 120 req/min per IP)
- [x] Add CORS headers to the collect endpoint so browsers allow cross-origin beacons
- [x] Validate the incoming `referrer` / `url` against the website's `domain` to reject spoofed traffic
- [x] Add IP-based geolocation (use MaxMind GeoLite2 via `stevebauman/location`) to populate `country`, `region`, `city` on the `Visitor`
- [x] Handle bot filtering — skip known bot user agents (using `jaybizzle/crawler-detect`)
- [x] Add a `tracking_pixel` endpoint (1x1 GIF) as a no-JS fallback

---

## Phase 6: Data Aggregation & Performance

Raw event tables grow fast. Pre-aggregate data for fast dashboard loads.

- [x] Create `daily_aggregates` table (migration + model):
  - `website_id`, `date`, `visitors`, `sessions`, `page_views`, `bounce_count`, `total_duration`
- [x] Create `page_daily_aggregates` table:
  - `website_id`, `date`, `path`, `views`, `unique_views`, `total_time_on_page`, `bounces`
- [x] Create a scheduled Artisan command (`analytics:aggregate`) that rolls up yesterday's raw data into aggregate tables — run daily via `schedule:run`
- [x] Refactor dashboard and report queries to read from aggregates for past days, and query raw tables only for "today"
- [x] Add database indexes review — ensure all date-range + website_id queries are covered
- [x] Add `data_retention_days` setting per website (stored in `settings` JSON) and a cleanup command (`analytics:purge`) to purge old raw rows

---

## Phase 7: Frontend & UI

Choose a frontend approach and build a polished interface.

- [x] Pick a stack:
  - **Option A:** Blade + Alpine.js (server-rendered, simpler) — selected
- [x] Install Tailwind CSS (already in the Laravel scaffold via Vite) + Inter font
- [x] Build a **layout** with sidebar navigation (websites list, reports sections) and top bar (date picker, user menu)
- [x] Integrate a charting library (Chart.js + chartjs-chart-geo) for:
  - Line charts (visitors over time)
  - Bar charts (top pages, browsers, sources)
  - Pie/donut charts (device type, traffic source split, new vs returning)
  - World map (geographic report — choropleth)
- [x] Build **comparison mode** — compare current period vs. previous period with delta percentages on metric cards
- [x] Add loading states, empty states, and error handling throughout
- [x] Make the dashboard responsive (mobile-friendly) — collapsible sidebar with overlay on mobile

---

## Phase 8: API for Programmatic Access ✅

Let users pull their analytics data via an API.

- [x] Create API token management (Laravel Sanctum) — UI page to generate/revoke API tokens
- [x] Build versioned API routes (`/api/v1/...`)
- [x] Endpoints:
  - `GET /api/v1/websites` — list websites
  - `GET /api/v1/websites/{id}/stats` — summary metrics with date-range params
  - `GET /api/v1/websites/{id}/pages` — top pages
  - `GET /api/v1/websites/{id}/referrers` — traffic sources
  - `GET /api/v1/websites/{id}/events` — events report
  - `GET /api/v1/websites/{id}/visitors` — audience breakdown
- [x] Add API resource classes for consistent JSON formatting
- [x] Rate limit API endpoints per token

---

## Phase 9: Advanced Features ✅

Features that differentiate from a basic analytics tool.

- [x] **Funnels** — let users define a sequence of pages/events and visualize drop-off at each step
- [x] **Segments** — filter all reports by visitor attributes (e.g., mobile users from Germany)
- [x] **Custom dashboards** — let users pin specific widgets/reports to a custom overview
- [x] **Alerts / Notifications** — notify users when traffic spikes, drops, or a goal hits a threshold (email and/or in-app)
- [x] **Data export** — CSV/PDF export for any report
- [x] **UTM builder** — in-app tool to generate campaign URLs with UTM parameters
- [x] **Annotations** — let users add notes on specific dates (e.g., "launched ad campaign") visible on charts

---

## Phase 10: Testing & Quality ✅

- [x] Write Feature tests for `CollectController` (valid payloads, invalid tracking IDs, rate limiting)
- [x] Write Feature tests for `WebsiteController` CRUD and authorization
- [x] Write Unit tests for `AnalyticsQueryService` aggregation logic
- [x] Write Unit tests for `GoalEvaluationService`
- [ ] Write browser tests (Dusk) for dashboard interactions and date picker (deferred — requires Dusk package + browser driver)
- [x] Add CI pipeline (GitHub Actions): run tests, Pint linting, Larastan static analysis
- [x] Create model factories for all analytics models
- [x] Fix base Controller `AuthorizesRequests` trait
- [x] Fix DailyAggregate/PageDailyAggregate date cast for SQLite compatibility


---

## Phase 11: Deployment & Operations

- [ ] Set up production database (PostgreSQL recommended for analytical queries, or MySQL)
- [ ] Configure queue worker (Redis + Horizon) for async ingestion jobs
- [ ] Set up the scheduler (`php artisan schedule:run`) for daily aggregation
- [ ] Configure caching (Redis) for dashboard queries and script delivery
- [ ] Set up HTTPS and CDN for the tracker JS file
- [ ] Add application monitoring (Sentry or similar for error tracking)
- [ ] Configure backups for the database
- [ ] Write a deployment script or use Laravel Forge / Vapor

---

## Suggested Order of Execution

| Priority | Phase | Reason |
|----------|-------|--------|
| 1 | Phase 1 | Can't use anything without auth and website management |
| 2 | Phase 7 (partial) | Pick frontend stack and build layout early — everything else renders into it |
| 3 | Phase 2 | The dashboard is the core product |
| 4 | Phase 5 | Harden collection before driving real traffic |
| 5 | Phase 3 | Detailed reports give the platform actual value |
| 6 | Phase 4 | Goals & conversions are a key differentiator |
| 7 | Phase 6 | Aggregation becomes necessary as data grows |
| 8 | Phase 10 | Tests before shipping to production |
| 9 | Phase 8 | API access for power users |
| 10 | Phase 9 | Advanced features for product maturity |
| 11 | Phase 11 | Production deployment |