214 lines
10 KiB
Markdown
214 lines
10 KiB
Markdown
|
# Analytics Platform — Implementation Plan
|
||
|
|
|
||
|
|
## Current State
|
||
|
|
|
||
|
|
The foundation is in place: 7 models with migrations (`Website`, `Visitor`, `AnalyticsSession`, `PageView`, `Event`, `Goal`, `GoalConversion`), a JavaScript tracker served dynamically per tracking ID, and a `POST /api/collect` endpoint that ingests page views and events. No authentication, no dashboard, no frontend yet.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 1: Authentication & Website Management
|
||
|
|
|
||
|
|
The platform needs user accounts and CRUD for tracked websites before anything else is usable.
|
||
|
|
|
||
|
|
- [x] Install Laravel Breeze (or Fortify) for registration, login, password reset, email verification
|
||
|
|
- [x] Build a `WebsiteController` with full CRUD (index, create, store, show, edit, update, destroy)
|
||
|
|
- [x] Create views/pages: list of websites, add/edit website form, website settings page
|
||
|
|
- [x] Build the "embed snippet" page — show the user the `<script>` tag to copy for each website
|
||
|
|
- [x] Add form request validation (`StoreWebsiteRequest`, `UpdateWebsiteRequest`)
|
||
|
|
- [x] Scope all website queries to `auth()->user()` (policy or middleware)
|
||
|
|
- [x] Add a `WebsitePolicy` to authorize show/edit/delete actions
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 2: Dashboard — Real-Time Overview
|
||
|
|
|
||
|
|
The main dashboard a user sees after selecting a website, showing key metrics at a glance.
|
||
|
|
|
||
|
|
- [x] Create `DashboardController` scoped to a single website
|
||
|
|
- [x] Build a date-range picker component (today, last 7 days, last 30 days, custom range)
|
||
|
|
- [x] **Metric cards** — display for the selected period:
|
||
|
|
- Total page views
|
||
|
|
- Unique visitors
|
||
|
|
- Total sessions
|
||
|
|
- Bounce rate (% of sessions with 1 page view)
|
||
|
|
- Average session duration
|
||
|
|
- Average pages per session
|
||
|
|
- [x] **Visitors chart** — line/area chart of visitors and page views over time (daily or hourly granularity)
|
||
|
|
- [x] **Real-time widget** — show active visitors in the last 5 minutes (poll or WebSocket)
|
||
|
|
- [x] Create an `AnalyticsQueryService` (or similar) to encapsulate all aggregation queries, keeping controllers thin
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 3: Detailed Reports
|
||
|
|
|
||
|
|
Individual report pages accessible from the dashboard sidebar.
|
||
|
|
|
||
|
|
### 3a — Audience Reports
|
||
|
|
|
||
|
|
- [x] **Geographic report** — visitors by country, region, city (table + world map visualization)
|
||
|
|
- [x] **Technology report** — breakdown by browser, OS, device type, screen resolution
|
||
|
|
- [x] **Language report** — visitors by browser language
|
||
|
|
- [x] **New vs. returning visitors** — segment visitors who have more than one session
|
||
|
|
|
||
|
|
### 3b — Acquisition Reports
|
||
|
|
|
||
|
|
- [x] **Traffic sources** — breakdown by referrer domain (direct, organic, referral, social)
|
||
|
|
- [x] **Campaigns (UTM)** — table of sessions grouped by `utm_source`, `utm_medium`, `utm_campaign`
|
||
|
|
- [x] **Referrers** — top referring URLs and domains
|
||
|
|
|
||
|
|
### 3c — Behavior Reports
|
||
|
|
|
||
|
|
- [x] **Pages report** — top pages by views, unique views, avg. time on page, bounce rate
|
||
|
|
- [x] **Landing pages** — top entry pages with session count and bounce rate
|
||
|
|
- [x] **Exit pages** — top exit pages
|
||
|
|
- [ ] **User flow** — simplified Sankey or funnel showing common page sequences (can be deferred)
|
||
|
|
|
||
|
|
### 3d — Events Report
|
||
|
|
|
||
|
|
- [x] **Events overview** — table of events grouped by name, category, action
|
||
|
|
- [x] **Event detail** — drill-down into a single event name with label/value breakdown
|
||
|
|
- [x] **Event trending** — chart of event counts over time
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 4: Goals & Conversions
|
||
|
|
|
||
|
|
Allow users to define goals and track conversion rates.
|
||
|
|
|
||
|
|
- [x] Build `GoalController` CRUD (create, edit, delete goals per website)
|
||
|
|
- [x] Support goal types:
|
||
|
|
- **URL destination** — session visited a specific path
|
||
|
|
- **Event match** — a specific event name/category was fired
|
||
|
|
- **Duration** — session lasted longer than N seconds
|
||
|
|
- **Pages per session** — session hit N+ page views
|
||
|
|
- [x] Create a `GoalEvaluationService` that checks incoming data against active goals and records `GoalConversion` entries
|
||
|
|
- [x] Hook goal evaluation into `CollectController` after recording page view / event
|
||
|
|
- [x] Build a **Conversions report** — goal completion count, conversion rate (completions / sessions), goal value totals
|
||
|
|
- [x] Add goal completion sparklines to the main dashboard
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 5: Collection Pipeline Hardening
|
||
|
|
|
||
|
|
The `CollectController` currently does synchronous DB writes. This won't scale under real traffic.
|
||
|
|
|
||
|
|
- [x] Move ingestion to a queued job (`ProcessTrackingPayload`) dispatched from the controller so the endpoint returns instantly
|
||
|
|
- [x] Add rate limiting middleware to `POST /api/collect` (e.g., 120 req/min per IP)
|
||
|
|
- [x] Add CORS headers to the collect endpoint so browsers allow cross-origin beacons
|
||
|
|
- [x] Validate the incoming `referrer` / `url` against the website's `domain` to reject spoofed traffic
|
||
|
|
- [x] Add IP-based geolocation (use MaxMind GeoLite2 via `stevebauman/location`) to populate `country`, `region`, `city` on the `Visitor`
|
||
|
|
- [x] Handle bot filtering — skip known bot user agents (using `jaybizzle/crawler-detect`)
|
||
|
|
- [x] Add a `tracking_pixel` endpoint (1x1 GIF) as a no-JS fallback
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 6: Data Aggregation & Performance
|
||
|
|
|
||
|
|
Raw event tables grow fast. Pre-aggregate data for fast dashboard loads.
|
||
|
|
|
||
|
|
- [x] Create `daily_aggregates` table (migration + model):
|
||
|
|
- `website_id`, `date`, `visitors`, `sessions`, `page_views`, `bounce_count`, `total_duration`
|
||
|
|
- [x] Create `page_daily_aggregates` table:
|
||
|
|
- `website_id`, `date`, `path`, `views`, `unique_views`, `total_time_on_page`, `bounces`
|
||
|
|
- [x] Create a scheduled Artisan command (`analytics:aggregate`) that rolls up yesterday's raw data into aggregate tables — run daily via `schedule:run`
|
||
|
|
- [x] Refactor dashboard and report queries to read from aggregates for past days, and query raw tables only for "today"
|
||
|
|
- [x] Add database indexes review — ensure all date-range + website_id queries are covered
|
||
|
|
- [x] Add `data_retention_days` setting per website (stored in `settings` JSON) and a cleanup command (`analytics:purge`) to purge old raw rows
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 7: Frontend & UI
|
||
|
|
|
||
|
|
Choose a frontend approach and build a polished interface.
|
||
|
|
|
||
|
|
- [x] Pick a stack:
|
||
|
|
- **Option A:** Blade + Alpine.js (server-rendered, simpler) — selected
|
||
|
|
- [x] Install Tailwind CSS (already in the Laravel scaffold via Vite) + Inter font
|
||
|
|
- [x] Build a **layout** with sidebar navigation (websites list, reports sections) and top bar (date picker, user menu)
|
||
|
|
- [x] Integrate a charting library (Chart.js + chartjs-chart-geo) for:
|
||
|
|
- Line charts (visitors over time)
|
||
|
|
- Bar charts (top pages, browsers, sources)
|
||
|
|
- Pie/donut charts (device type, traffic source split, new vs returning)
|
||
|
|
- World map (geographic report — choropleth)
|
||
|
|
- [x] Build **comparison mode** — compare current period vs. previous period with delta percentages on metric cards
|
||
|
|
- [x] Add loading states, empty states, and error handling throughout
|
||
|
|
- [x] Make the dashboard responsive (mobile-friendly) — collapsible sidebar with overlay on mobile
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 8: API for Programmatic Access ✅
|
||
|
|
|
||
|
|
Let users pull their analytics data via an API.
|
||
|
|
|
||
|
|
- [x] Create API token management (Laravel Sanctum) — UI page to generate/revoke API tokens
|
||
|
|
- [x] Build versioned API routes (`/api/v1/...`)
|
||
|
|
- [x] Endpoints:
|
||
|
|
- `GET /api/v1/websites` — list websites
|
||
|
|
- `GET /api/v1/websites/{id}/stats` — summary metrics with date-range params
|
||
|
|
- `GET /api/v1/websites/{id}/pages` — top pages
|
||
|
|
- `GET /api/v1/websites/{id}/referrers` — traffic sources
|
||
|
|
- `GET /api/v1/websites/{id}/events` — events report
|
||
|
|
- `GET /api/v1/websites/{id}/visitors` — audience breakdown
|
||
|
|
- [x] Add API resource classes for consistent JSON formatting
|
||
|
|
- [x] Rate limit API endpoints per token
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 9: Advanced Features ✅
|
||
|
|
|
||
|
|
Features that differentiate from a basic analytics tool.
|
||
|
|
|
||
|
|
- [x] **Funnels** — let users define a sequence of pages/events and visualize drop-off at each step
|
||
|
|
- [x] **Segments** — filter all reports by visitor attributes (e.g., mobile users from Germany)
|
||
|
|
- [x] **Custom dashboards** — let users pin specific widgets/reports to a custom overview
|
||
|
|
- [x] **Alerts / Notifications** — notify users when traffic spikes, drops, or a goal hits a threshold (email and/or in-app)
|
||
|
|
- [x] **Data export** — CSV/PDF export for any report
|
||
|
|
- [x] **UTM builder** — in-app tool to generate campaign URLs with UTM parameters
|
||
|
|
- [x] **Annotations** — let users add notes on specific dates (e.g., "launched ad campaign") visible on charts
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 10: Testing & Quality ✅
|
||
|
|
|
||
|
|
- [x] Write Feature tests for `CollectController` (valid payloads, invalid tracking IDs, rate limiting)
|
||
|
|
- [x] Write Feature tests for `WebsiteController` CRUD and authorization
|
||
|
|
- [x] Write Unit tests for `AnalyticsQueryService` aggregation logic
|
||
|
|
- [x] Write Unit tests for `GoalEvaluationService`
|
||
|
|
- [ ] Write browser tests (Dusk) for dashboard interactions and date picker (deferred — requires Dusk package + browser driver)
|
||
|
|
- [x] Add CI pipeline (GitHub Actions): run tests, Pint linting, Larastan static analysis
|
||
|
|
- [x] Create model factories for all analytics models
|
||
|
|
- [x] Fix base Controller `AuthorizesRequests` trait
|
||
|
|
- [x] Fix DailyAggregate/PageDailyAggregate date cast for SQLite compatibility
|
||
|
|
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 11: Deployment & Operations
|
||
|
|
|
||
|
|
- [ ] Set up production database (PostgreSQL recommended for analytical queries, or MySQL)
|
||
|
|
- [ ] Configure queue worker (Redis + Horizon) for async ingestion jobs
|
||
|
|
- [ ] Set up the scheduler (`php artisan schedule:run`) for daily aggregation
|
||
|
|
- [ ] Configure caching (Redis) for dashboard queries and script delivery
|
||
|
|
- [ ] Set up HTTPS and CDN for the tracker JS file
|
||
|
|
- [ ] Add application monitoring (Sentry or similar for error tracking)
|
||
|
|
- [ ] Configure backups for the database
|
||
|
|
- [ ] Write a deployment script or use Laravel Forge / Vapor
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Suggested Order of Execution
|
||
|
|
|
||
|
|
| Priority | Phase | Reason |
|
||
|
|
|----------|-------|--------|
|
||
|
|
| 1 | Phase 1 | Can't use anything without auth and website management |
|
||
|
|
| 2 | Phase 7 (partial) | Pick frontend stack and build layout early — everything else renders into it |
|
||
|
|
| 3 | Phase 2 | The dashboard is the core product |
|
||
|
|
| 4 | Phase 5 | Harden collection before driving real traffic |
|
||
|
|
| 5 | Phase 3 | Detailed reports give the platform actual value |
|
||
|
|
| 6 | Phase 4 | Goals & conversions are a key differentiator |
|
||
|
|
| 7 | Phase 6 | Aggregation becomes necessary as data grows |
|
||
|
|
| 8 | Phase 10 | Tests before shipping to production |
|
||
|
|
| 9 | Phase 8 | API access for power users |
|
||
|
|
| 10 | Phase 9 | Advanced features for product maturity |
|
||
|
|
| 11 | Phase 11 | Production deployment |
|