analytics/PLAN.md

# Analytics Platform — Implementation Plan

## Current State

The foundation is in place: 7 models with migrations (`Website`, `Visitor`, `AnalyticsSession`, `PageView`, `Event`, `Goal`, `GoalConversion`), a JavaScript tracker served dynamically per tracking ID, and a `POST /api/collect` endpoint that ingests page views and events. No authentication, no dashboard, no frontend yet.

---

## Phase 1: Authentication & Website Management

The platform needs user accounts and CRUD for tracked websites before anything else is usable.

- [x] Install Laravel Breeze (or Fortify) for registration, login, password reset, email verification
- [x] Build a `WebsiteController` with full CRUD (index, create, store, show, edit, update, destroy)
- [x] Create views/pages: list of websites, add/edit website form, website settings page
- [x] Build the "embed snippet" page — show the user the `<script>` tag to copy for each website
- [x] Add form request validation (`StoreWebsiteRequest`, `UpdateWebsiteRequest`)
- [x] Scope all website queries to `auth()->user()` (policy or middleware)
- [x] Add a `WebsitePolicy` to authorize show/edit/delete actions

---

## Phase 2: Dashboard — Real-Time Overview

The main dashboard a user sees after selecting a website, showing key metrics at a glance.

- [x] Create `DashboardController` scoped to a single website
- [x] Build a date-range picker component (today, last 7 days, last 30 days, custom range)
- [x] **Metric cards** — display for the selected period:
  - Total page views
  - Unique visitors
  - Total sessions
  - Bounce rate (% of sessions with 1 page view)
  - Average session duration
  - Average pages per session
- [x] **Visitors chart** — line/area chart of visitors and page views over time (daily or hourly granularity)
- [x] **Real-time widget** — show active visitors in the last 5 minutes (poll or WebSocket)
- [x] Create an `AnalyticsQueryService` (or similar) to encapsulate all aggregation queries, keeping controllers thin

---

## Phase 3: Detailed Reports

Individual report pages accessible from the dashboard sidebar.

### 3a — Audience Reports

- [x] **Geographic report** — visitors by country, region, city (table + world map visualization)
- [x] **Technology report** — breakdown by browser, OS, device type, screen resolution
- [x] **Language report** — visitors by browser language
- [x] **New vs. returning visitors** — segment visitors who have more than one session

### 3b — Acquisition Reports

- [x] **Traffic sources** — breakdown by referrer domain (direct, organic, referral, social)
- [x] **Campaigns (UTM)** — table of sessions grouped by `utm_source`, `utm_medium`, `utm_campaign`
- [x] **Referrers** — top referring URLs and domains

### 3c — Behavior Reports

- [x] **Pages report** — top pages by views, unique views, avg. time on page, bounce rate
- [x] **Landing pages** — top entry pages with session count and bounce rate
- [x] **Exit pages** — top exit pages
- [ ] **User flow** — simplified Sankey or funnel showing common page sequences (can be deferred)

### 3d — Events Report

- [x] **Events overview** — table of events grouped by name, category, action
- [x] **Event detail** — drill-down into a single event name with label/value breakdown
- [x] **Event trending** — chart of event counts over time

---

## Phase 4: Goals & Conversions

Allow users to define goals and track conversion rates.

- [ ] Build `GoalController` CRUD (create, edit, delete goals per website)
- [ ] Support goal types:
  - **URL destination** — session visited a specific path
  - **Event match** — a specific event name/category was fired
  - **Duration** — session lasted longer than N seconds
  - **Pages per session** — session hit N+ page views
- [ ] Create a `GoalEvaluationService` that checks incoming data against active goals and records `GoalConversion` entries
- [ ] Hook goal evaluation into `CollectController` after recording page view / event
- [ ] Build a **Conversions report** — goal completion count, conversion rate (completions / sessions), goal value totals
- [ ] Add goal completion sparklines to the main dashboard

---

## Phase 5: Collection Pipeline Hardening

The `CollectController` currently does synchronous DB writes. This won't scale under real traffic.

- [ ] Move ingestion to a queued job (`ProcessTrackingPayload`) dispatched from the controller so the endpoint returns instantly
- [ ] Add rate limiting middleware to `POST /api/collect` (e.g., 60 req/min per IP)
- [ ] Add CORS headers to the collect endpoint so browsers allow cross-origin beacons
- [ ] Validate the incoming `referrer` / `url` against the website's `domain` to reject spoofed traffic
- [ ] Add IP-based geolocation (use MaxMind GeoLite2 via `stevebauman/location` or similar) to populate `country`, `region`, `city` on the `Visitor`
- [ ] Handle bot filtering — skip known bot user agents (use `jaybizzle/crawler-detect`)
- [ ] Add a `tracking_pixel` endpoint (1x1 GIF) as a no-JS fallback

---

## Phase 6: Data Aggregation & Performance

Raw event tables grow fast. Pre-aggregate data for fast dashboard loads.

- [ ] Create `daily_aggregates` table (migration + model):
  - `website_id`, `date`, `visitors`, `sessions`, `page_views`, `bounce_count`, `total_duration`
- [ ] Create `page_daily_aggregates` table:
  - `website_id`, `date`, `path`, `views`, `unique_views`, `total_time_on_page`, `bounces`
- [ ] Create a scheduled Artisan command (`analytics:aggregate`) that rolls up yesterday's raw data into aggregate tables — run daily via `schedule:run`
- [ ] Refactor dashboard and report queries to read from aggregates for past days, and query raw tables only for "today"
- [ ] Add database indexes review — ensure all date-range + website_id queries are covered
- [ ] Consider adding a `data_retention_days` setting per website and a cleanup command to purge old raw rows

---

## Phase 7: Frontend & UI

Choose a frontend approach and build a polished interface.

- [ ] Pick a stack:
  - **Option A:** Blade + Livewire + Alpine.js (server-rendered, simpler)
  - **Option B:** Inertia.js + Vue/React (SPA feel, more flexibility)
- [ ] Install Tailwind CSS (already in the Laravel scaffold via Vite)
- [ ] Build a **layout** with sidebar navigation (websites list, reports sections) and top bar (date picker, user menu)
- [ ] Integrate a charting library (Chart.js, ApexCharts, or ECharts) for:
  - Line charts (visitors over time)
  - Bar charts (top pages, browsers)
  - Pie/donut charts (device type, traffic source split)
  - World map (geographic report)
- [ ] Build **comparison mode** — compare current period vs. previous period with delta percentages
- [ ] Add loading states, empty states, and error handling throughout
- [ ] Make the dashboard responsive (mobile-friendly)

---

## Phase 8: API for Programmatic Access

Let users pull their analytics data via an API.

- [ ] Create API token management (Laravel Sanctum) — UI page to generate/revoke API tokens
- [ ] Build versioned API routes (`/api/v1/...`)
- [ ] Endpoints:
  - `GET /api/v1/websites` — list websites
  - `GET /api/v1/websites/{id}/stats` — summary metrics with date-range params
  - `GET /api/v1/websites/{id}/pages` — top pages
  - `GET /api/v1/websites/{id}/referrers` — traffic sources
  - `GET /api/v1/websites/{id}/events` — events report
  - `GET /api/v1/websites/{id}/visitors` — audience breakdown
- [ ] Add API resource classes for consistent JSON formatting
- [ ] Rate limit API endpoints per token

---

## Phase 9: Advanced Features

Features that differentiate from a basic analytics tool.

- [ ] **Funnels** — let users define a sequence of pages/events and visualize drop-off at each step
- [ ] **Segments** — filter all reports by visitor attributes (e.g., mobile users from Germany)
- [ ] **Custom dashboards** — let users pin specific widgets/reports to a custom overview
- [ ] **Alerts / Notifications** — notify users when traffic spikes, drops, or a goal hits a threshold (email and/or in-app)
- [ ] **Data export** — CSV/PDF export for any report
- [ ] **UTM builder** — in-app tool to generate campaign URLs with UTM parameters
- [ ] **Annotations** — let users add notes on specific dates (e.g., "launched ad campaign") visible on charts

---

## Phase 10: Testing & Quality

- [ ] Write Feature tests for `CollectController` (valid payloads, invalid tracking IDs, rate limiting)
- [ ] Write Feature tests for `WebsiteController` CRUD and authorization
- [ ] Write Unit tests for `AnalyticsQueryService` aggregation logic
- [ ] Write Unit tests for `GoalEvaluationService`
- [ ] Write browser tests (Dusk) for dashboard interactions and date picker
- [ ] Add CI pipeline (GitHub Actions): run tests, Pint linting, Larastan static analysis

---

## Phase 11: Deployment & Operations

- [ ] Set up production database (PostgreSQL recommended for analytical queries, or MySQL)
- [ ] Configure queue worker (Redis + Horizon) for async ingestion jobs
- [ ] Set up the scheduler (`php artisan schedule:run`) for daily aggregation
- [ ] Configure caching (Redis) for dashboard queries and script delivery
- [ ] Set up HTTPS and CDN for the tracker JS file
- [ ] Add application monitoring (Sentry or similar for error tracking)
- [ ] Configure backups for the database
- [ ] Write a deployment script or use Laravel Forge / Vapor

---

## Suggested Order of Execution

| Priority | Phase | Reason |
|----------|-------|--------|
| 1 | Phase 1 | Can't use anything without auth and website management |
| 2 | Phase 7 (partial) | Pick frontend stack and build layout early — everything else renders into it |
| 3 | Phase 2 | The dashboard is the core product |
| 4 | Phase 5 | Harden collection before driving real traffic |
| 5 | Phase 3 | Detailed reports give the platform actual value |
| 6 | Phase 4 | Goals & conversions are a key differentiator |
| 7 | Phase 6 | Aggregation becomes necessary as data grows |
| 8 | Phase 10 | Tests before shipping to production |
| 9 | Phase 8 | API access for power users |
| 10 | Phase 9 | Advanced features for product maturity |
| 11 | Phase 11 | Production deployment |