analytics/STATUS.md

Analytics Platform — Project Status

Last updated: 2026-05-25

---

What Is This

A self-hosted web analytics platform built with Laravel 11 — a privacy-focused alternative to Google Analytics or Plausible. Users add a lightweight JavaScript snippet to their website and get real-time dashboards, AI-ready data pipelines, goal/funnel tracking, and a full REST API.

---

Tech Stack

Layer	Choice
Backend	Laravel 11, PHP 8.2+
Auth	Laravel Fortify (2FA ready)
Frontend	Blade + Alpine.js + Tailwind CSS
Charts	Chart.js
Database	SQLite (dev) / MySQL or PostgreSQL (prod)
Queue	Database driver (dev) / Redis + Horizon (prod)
API	Laravel Sanctum (token auth)
Geolocation	stevebauman/location
Bot detection	jaybizzle/crawler-detect
Build	Vite

---

Feature Status

✅ Done — Core Platform

Authentication & Users

• Registration, login, password reset, email verification via Fortify
• Two-factor authentication support
• Profile management
• API token management (Sanctum)

Website Management

• Full CRUD for tracked websites
• Embed snippet page with copy-ready <script> tag
• WebsitePolicy — all queries scoped to authenticated owner

Data Collection Pipeline

• POST /api/e and POST /api/collect endpoints
• Async ingestion via ProcessTrackingPayload queued job
• Bot filtering (CrawlerDetect)
• Domain validation (rejects spoofed tracking IDs)
• IP geolocation → country, region, city on Visitor
• CORS middleware for cross-origin beacons
• Tracking pixel (1×1 GIF) as no-JS fallback
• Rate limiting (120 req/min per IP)
• Ad-blocker resistant alternate endpoints (/cdn/feedback, /cdn/resources/)

Dashboard

• Date-range picker (today / 7d / 30d / custom)
• Metric cards: page views, unique visitors, sessions, bounce rate, avg session duration, pages/session
• Visitors & page views chart (daily/hourly granularity)
• Real-time active visitors widget (last 5 minutes)
• Comparison mode vs. previous period with delta %
• Goal conversion sparklines

Reports — Audience

• Geographic: visitors by country, region, city (table + world map)
• Technology: browser, OS, device type, screen resolution
• Language: browser language breakdown
• New vs. returning visitors

Reports — Acquisition

• Traffic sources (direct, organic, referral, social)
• UTM campaigns (source, medium, campaign)
• Top referrers

Reports — Behavior

• Top pages (views, unique views, avg time, bounce rate)
• Landing pages (entry pages with session count and bounce rate)
• Exit pages

Reports — Events

• Events overview (grouped by name/category/action)
• Event detail drill-down (label/value breakdown)
• Event trending chart

Reports — Conversions

• Goal completions, conversion rate, goal value totals

Goals & Conversions

• 4 goal types: URL destination, event match, duration, pages per session
• GoalEvaluationService — evaluates goals on every incoming payload
• Conversion rate reporting

Advanced Features

• Funnels — multi-step conversion tracking with drop-off visualization
• Segments — filter all reports by visitor attributes
• Custom dashboards — pin widgets to a personal overview
• Alerts — notify on traffic spikes/drops/goal thresholds (email + in-app)
• Annotations — add notes to specific dates on charts
• Data export — CSV export for any report
• UTM builder — in-app campaign URL generator

API (v1)

• GET /api/v1/websites — list websites
• GET /api/v1/websites/{id}/stats — summary metrics with date-range params
• GET /api/v1/websites/{id}/pages — top pages
• GET /api/v1/websites/{id}/referrers — traffic sources
• GET /api/v1/websites/{id}/events — events
• GET /api/v1/websites/{id}/visitors — audience breakdown
• JSON:API resource classes for consistent formatting
• Rate limiting per token

Data Aggregation

• daily_aggregates table — pre-computed per-website daily metrics
• page_daily_aggregates table — per-page daily metrics
• analytics:aggregate Artisan command (runs daily at 02:00)
• Hybrid query strategy: aggregates for past days, raw tables for today
• analytics:purge command — purge raw data older than retention window
• Configurable data_retention_days per website (default: 365)

Performance & Indexes

• Composite indexes on all date-range + website_id query paths
• SQLite-compatible index migrations (also works with MySQL/PostgreSQL)

Testing

• 80 tests, 0 failures
• Feature tests: CollectController, WebsiteController, Auth flows
• Unit tests: AnalyticsQueryService, GoalEvaluationService
• Model factories for all analytics models
• withoutVite() in base TestCase — tests run without a built frontend

---

🔲 Not Yet Done

Item	Notes
User Flow / Sankey diagram	Intentionally deferred — complex to visualize, low priority for v1
Production database	SQLite is dev-only; set up MySQL or PostgreSQL before launch
Redis + Horizon	Replace database queue driver for real traffic; Horizon gives a monitoring UI
Cron / scheduler	Configure schedule:run on the server — needed for aggregation, purge, and alerts
HTTPS + CDN for tracker JS	Tracker script should load from a CDN subdomain to avoid ad-blockers
Error monitoring	Wire up Sentry or Flare for production error tracking
Database backups	Automate daily dumps with offsite storage
Dusk browser tests	UI interaction tests deferred — requires Dusk + ChromeDriver setup

---

Running Locally

cd ~/Projects/analytics
composer install
cp .env.example .env
php artisan key:generate
php artisan migrate
npm install && npm run build
composer run dev       # starts web server + queue worker + log viewer + Vite

App runs at http://localhost:8000.

Background processes required

# Queue worker — processes tracking events (critical)
php artisan queue:work database --tries=3

# Scheduler — aggregation, purge, alerts (critical)
php artisan schedule:work

# OR add to crontab for production:
# * * * * * cd /path/to/analytics && php artisan schedule:run >> /dev/null 2>&1

---

Deployment Checklist

• Provision server (Ubuntu 22+ recommended)
• Install PHP 8.2+, Nginx, MySQL/PostgreSQL, Redis, Node
• Clone repo, composer install --no-dev, npm ci && npm run build
• Set APP_ENV=production, APP_DEBUG=false, configure .env
• Run php artisan migrate --force
• Run php artisan config:cache route:cache view:cache
• Configure Supervisor for queue worker (see HOWTO.md)
• Add crontab entry for schedule:run
• Point Nginx document root to public/
• Set up SSL (Let's Encrypt)
• Set up CNAME for tracker script (e.g. track.yourdomain.com)
• Configure Sentry DSN in .env
• Set up automated database backups

---

Key Artisan Commands

Command	Description	Schedule
analytics:aggregate	Roll up yesterday's raw data into aggregates	Daily 02:00
analytics:purge --force	Delete raw data older than retention window	Daily 03:00
analytics:check-alerts	Evaluate alert rules and send notifications	Every hour
queue:work database --tries=3	Process tracking payloads	Always-on process

---

Architecture Notes

Data Flow

Client website (JS snippet)
  → POST /api/e  (CollectController)
  → ProcessTrackingPayload job (queued)
  → Creates/updates: Visitor, AnalyticsSession, PageView or Event
  → GoalEvaluationService checks active goals
  → Daily: analytics:aggregate rolls raw → daily_aggregates
  → Dashboard queries: aggregates for history + raw tables for today

Collector Extraction (future)

The collector_app_plan.md documents a plan to split the data collection endpoints into a separate Laravel app for independent scaling. This is not needed yet but the code is structured to make it straightforward — CollectController, TrackingScriptController, ProcessTrackingPayload, and GoalEvaluationService are all self-contained.

---

Database Schema (Core Tables)

Table	Purpose
websites	Tracked websites, settings, tracking ID
visitors	Unique visitors with device, browser, location
analytics_sessions	Sessions with UTM params, entry/exit page, duration
page_views	Individual page visits
events	Custom events with category, label, value
goals	Conversion goal definitions (4 types)
goal_conversions	Goal completion records
funnels	Multi-step funnel definitions
segments	Custom visitor segment definitions
alerts	Alert rule definitions
alert_notifications	Triggered alert history
annotations	Date annotations visible on charts
dashboard_widgets	Custom dashboard layout per user
daily_aggregates	Pre-computed daily metrics per website
page_daily_aggregates	Pre-computed daily metrics per page