analytics/HOWTO.md

# Analytics Platform - HOWTO Guide

## Table of Contents
1. [What is This Project?](#what-is-this-project)
2. [Features Overview](#features-overview)
3. [Architecture](#architecture)
4. [Installation](#installation)
5. [Configuration](#configuration)
6. [Required Background Tasks](#required-background-tasks)
7. [Tracking Setup](#tracking-setup)
8. [How It Works](#how-it-works)
9. [Database Structure](#database-structure)
10. [Artisan Commands](#artisan-commands)
11. [API Documentation](#api-documentation)
12. [Maintenance](#maintenance)

---

## What is This Project?

This is a **self-hosted web analytics platform** built with Laravel 11. It provides privacy-focused website analytics similar to Google Analytics or Plausible, allowing you to track visitor behavior, conversions, and engagement on your websites without relying on third-party services.

### Key Capabilities
- Real-time visitor tracking
- Goal and conversion tracking
- Custom event tracking
- Funnel analysis
- Automated alerts and notifications
- Custom dashboards
- Export analytics data
- API access for integrations

---

## Features Overview

### 1. **Visitor Tracking**
- Automatic page view tracking
- Session management (30-minute timeout)
- Device, browser, and OS detection
- Geolocation (country, region, city)
- Bot filtering
- UTM parameter tracking (campaign, source, medium, etc.)
- Time on page measurement

### 2. **Analytics Reports**
- **Audience**: Geographic distribution, technology breakdown, languages
- **Acquisition**: Traffic sources, referrers, campaigns
- **Behavior**: Popular pages, landing pages, exit pages
- **Events**: Custom event tracking with categories and labels
- **Conversions**: Goal completions with revenue tracking

### 3. **Goals (4 Types)**
- **URL Destination**: Track when users visit specific pages
- **Event Match**: Track custom events
- **Duration**: Sessions longer than X seconds
- **Pages Per Session**: Sessions with X or more pages

### 4. **Advanced Features**
- **Funnels**: Multi-step conversion tracking
- **Segments**: Custom visitor segments for filtered reporting
- **Alerts**: Automated notifications when metrics exceed thresholds
- **Annotations**: Add notes to specific dates for context
- **Custom Dashboards**: Drag-and-drop widget management

### 5. **Data Management**
- Automatic daily aggregation for performance
- Configurable data retention (default: 365 days)
- CSV export
- API access with token authentication

---

## Architecture

### Technology Stack
- **Backend**: Laravel 11, PHP 8.2+
- **Frontend**: Blade templates, Alpine.js, Tailwind CSS
- **Database**: SQLite (default), MySQL/PostgreSQL compatible
- **Queue System**: Database-backed (Redis/SQS optional)
- **Build Tool**: Vite

### Data Flow

```
Client Website (tracking script)
    ↓ POST /api/collect
CollectController (validates & queues)
    ↓ Queue Job
ProcessTrackingPayload (processes data)
    ↓ Stores in DB
Raw Tables (visitors, sessions, page_views, events)
    ↓ Daily aggregation (02:00 AM)
Aggregated Tables (daily_aggregates, page_daily_aggregates)
    ↓ Used by
Dashboard & Reports (hybrid: aggregates + real-time)
```

---

## Installation

### Development Setup

```bash
# Clone the repository
cd /path/to/analytics

# Install dependencies and setup
composer setup

# This runs:
# - composer install
# - Creates .env from .env.example
# - php artisan key:generate
# - php artisan migrate --force
# - npm install
# - npm run build

# Start development environment (runs 4 processes)
composer run dev
```

This starts:
- Web server: `http://localhost:8000`
- Queue worker (processes tracking data)
- Log viewer (Pail)
- Vite dev server (hot module reload)

### Production Setup

```bash
# Install dependencies
composer install --no-dev --optimize-autoloader

# Generate application key
php artisan key:generate

# Run database migrations
php artisan migrate --force

# Build frontend assets
npm ci
npm run build

# Set proper permissions
chmod -R 755 storage bootstrap/cache
chown -R www-data:www-data storage bootstrap/cache
```

---

## Configuration

### Environment Variables

Copy `.env.example` to `.env` and configure:

```env
# Application
APP_NAME="Analytics Platform"
APP_ENV=production
APP_DEBUG=false
APP_URL=https://yourdomain.com

# Database
DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=analytics
DB_USERNAME=your_user
DB_PASSWORD=your_password

# Queue (use 'redis' for better performance)
QUEUE_CONNECTION=database

# Mail (for alerts and notifications)
MAIL_MAILER=smtp
MAIL_HOST=smtp.mailtrap.io
MAIL_PORT=2525
MAIL_USERNAME=null
MAIL_PASSWORD=null
MAIL_FROM_ADDRESS=noreply@yourdomain.com
MAIL_FROM_NAME="${APP_NAME}"

# Cache (use 'redis' for better performance)
CACHE_STORE=database

# Session
SESSION_DRIVER=database
SESSION_LIFETIME=120
```

### Web Server Configuration

#### For Standard Setup (public/ folder)
Point your web server document root to the `public/` directory.

#### For Shared Hosting (engine/ + public_html/)

If your hosting has separate folders:
- `engine/` - Laravel application files
- `public_html/` - Public web root

**Modify `public_html/index.php`:**

```php
<?php

use Illuminate\Foundation\Application;
use Illuminate\Http\Request;

define('LARAVEL_START', microtime(true));

// Point to engine directory
if (file_exists($maintenance = __DIR__.'/../engine/storage/framework/maintenance.php')) {
    require $maintenance;
}

require __DIR__.'/../engine/vendor/autoload.php';

$app = require_once __DIR__.'/../engine/bootstrap/app.php';

// Tell Laravel the public path is public_html
$app->usePublicPath(__DIR__);

$app->handleRequest(Request::capture());
```

---

## Required Background Tasks

### 1. Queue Worker (CRITICAL)

**Purpose**: Processes tracking data from websites

Without this, tracking events are queued but never persisted to the database.

```bash
# Start queue worker
php artisan queue:work database --tries=3 --timeout=60
```

**Production Setup**: Use a process supervisor

**Using Supervisor** (recommended):

```ini
# /etc/supervisor/conf.d/analytics-queue.conf
[program:analytics-queue]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/analytics/artisan queue:work database --tries=3 --timeout=60
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=1
redirect_stderr=true
stdout_logfile=/path/to/analytics/storage/logs/queue-worker.log
stopwaitsecs=3600
```

Then run:
```bash
sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start analytics-queue:*
```

**Using systemd**:

```ini
# /etc/systemd/system/analytics-queue.service
[Unit]
Description=Analytics Queue Worker
After=network.target

[Service]
Type=simple
User=www-data
WorkingDirectory=/path/to/analytics
ExecStart=/usr/bin/php /path/to/analytics/artisan queue:work database --tries=3 --timeout=60
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
```

Enable and start:
```bash
sudo systemctl enable analytics-queue
sudo systemctl start analytics-queue
```

### 2. Task Scheduler (CRITICAL)

**Purpose**: Runs scheduled tasks (aggregation, purging, alerts)

Add to crontab:

```bash
# Edit crontab
crontab -e

# Add this line
* * * * * cd /path/to/analytics && php artisan schedule:run >> /dev/null 2>&1
```

This executes:
- **Daily at 02:00 AM**: `analytics:aggregate` - Rolls up data into daily aggregates
- **Daily at 03:00 AM**: `analytics:purge --force` - Deletes old raw data
- **Every hour**: `analytics:check-alerts` - Evaluates alerts and sends notifications

Without the scheduler:
- Reports will be slower (no aggregated data)
- Old data will accumulate and bloat the database
- Alerts will never trigger

---

## Tracking Setup

### 1. Create a Website

1. Register/login to the analytics platform
2. Navigate to **Websites** → **Add Website**
3. Enter website name and domain
4. Copy the tracking ID (e.g., `abc123def456`)

### 2. Install Tracking Script

Add this script to your website's HTML (before `</head>`):

**Option 1: JavaScript Snippet**

```html
<script async src="https://analytics.yourdomain.com/api/t/YOUR_TRACKING_ID.js"></script>
```

**Option 2: Self-hosted Script**

```html
<script>
  !function(t,e,a){/* tracking script */}(window,document,"https://analytics.yourdomain.com","YOUR_TRACKING_ID");
</script>
```

**Option 3: No-JavaScript Fallback**

```html
<noscript>
  <img src="https://analytics.yourdomain.com/api/collect/pixel/YOUR_TRACKING_ID"
       alt=""
       style="position:absolute;left:-9999px;" />
</noscript>
```

### 3. Track Custom Events

```javascript
// Track a custom event
ta('event', 'button_click', {
  category: 'engagement',
  label: 'signup_button',
  value: 1
});

// Track a purchase
ta('event', 'purchase', {
  category: 'ecommerce',
  value: 99.99
});
```

### 4. SPA Support

For single-page applications (React, Vue, etc.), the tracker automatically detects route changes via `pushState` and `popstate`. You can also manually track page views:

```javascript
// Manual page view tracking
ta('pageview');
```

---

## How It Works

### Tracking Process

1. **Client-side**: Tracking script loads on visitor's browser
2. **Collection**: Script sends tracking data to `/api/collect`
3. **Validation**: `CollectController` validates tracking ID and domain
4. **Queueing**: Tracking payload is queued as `ProcessTrackingPayload` job
5. **Processing**: Queue worker picks up job and:
   - Creates/updates **Visitor** record (with geolocation)
   - Creates/retrieves **AnalyticsSession** (30-min timeout)
   - Records **PageView** or **Event**
   - Evaluates active **Goals** and records conversions
   - Updates session metrics (duration, bounce status)
6. **Aggregation**: Daily at 02:00 AM, raw data is rolled up into aggregates
7. **Reporting**: Dashboard queries combine aggregated + real-time data

### Session Logic

- **Session Start**: First page view from a visitor
- **Session Timeout**: 30 minutes of inactivity
- **Session Duration**: Time between first and last page view
- **Bounce**: Session with only 1 page view
- **Exit Page**: Last page viewed in session

### Goal Evaluation

Goals are evaluated in real-time during tracking:

1. **URL Destination**: Matches current page URL
   - Exact: URL must match exactly
   - Starts with: URL must start with pattern
   - Contains: URL must contain pattern
   - Regex: URL must match regex pattern

2. **Event Match**: Matches event name (and optionally category)

3. **Duration**: Session duration exceeds threshold

4. **Pages Per Session**: Page count exceeds threshold

### Data Aggregation

Raw tracking data is aggregated daily to improve query performance:

**Daily Aggregates** (per website, per day):
- Total visitors (unique)
- Total sessions
- Total page views
- Bounce count
- Total session duration

**Page Daily Aggregates** (per website, per page, per day):
- Page views
- Unique page views
- Total time on page
- Bounce count

**Query Strategy**:
- Historical dates: Query aggregated tables
- Today: Query raw tables
- Results are merged for complete picture

---

## Database Structure

### Core Tables

| Table | Description |
|-------|-------------|
| `users` | User accounts |
| `websites` | Tracked websites with settings |
| `visitors` | Unique visitors with device/location info |
| `analytics_sessions` | User sessions with UTM data |
| `page_views` | Individual page visits |
| `events` | Custom events with metadata |
| `goals` | Conversion goal definitions |
| `goal_conversions` | Goal completion records |
| `funnels` | Funnel definitions |
| `segments` | Custom visitor segments |
| `alerts` | Alert rule definitions |
| `alert_notifications` | Triggered alert records |
| `annotations` | Date annotations |
| `dashboard_widgets` | Custom dashboard layouts |
| `daily_aggregates` | Pre-computed daily metrics |
| `page_daily_aggregates` | Per-page daily metrics |

### Data Retention

- **Raw Data**: Retained for 365 days (configurable per website)
- **Aggregated Data**: Retained indefinitely
- **Purge Schedule**: Daily at 03:00 AM via `analytics:purge`

---

## Artisan Commands

### Analytics Commands

#### 1. Aggregate Data
```bash
php artisan analytics:aggregate [options]

Options:
  --date=YYYY-MM-DD    Aggregate specific date (default: yesterday)
  --website=ID         Aggregate specific website (default: all)

# Examples
php artisan analytics:aggregate
php artisan analytics:aggregate --date=2026-03-20
php artisan analytics:aggregate --website=1
```

**Important**: Cannot aggregate today's data (prevents incomplete aggregations)

#### 2. Purge Old Data
```bash
php artisan analytics:purge [options]

Options:
  --website=ID    Purge specific website (default: all)
  --force         Skip confirmation prompt

# Examples
php artisan analytics:purge --force
php artisan analytics:purge --website=1
```

Deletes raw data older than retention period (default: 365 days).

#### 3. Check Alerts
```bash
php artisan analytics:check-alerts
```

Evaluates all active alerts and triggers notifications.

### Queue Commands

```bash
# Process queue jobs
php artisan queue:work database --tries=3

# List failed jobs
php artisan queue:failed

# Retry failed job
php artisan queue:retry {job-id}

# Retry all failed jobs
php artisan queue:retry all

# Flush failed jobs
php artisan queue:flush
```

### Cache Commands

```bash
# Clear application cache
php artisan cache:clear

# Clear config cache
php artisan config:clear

# Clear route cache
php artisan route:clear

# Clear view cache
php artisan view:clear

# Cache config for production
php artisan config:cache
php artisan route:cache
php artisan view:cache
```

---

## API Documentation

### Authentication

Generate API token in dashboard: **Profile** → **API Tokens** → **Create Token**

Include in requests:
```bash
Authorization: Bearer YOUR_API_TOKEN
```

### Endpoints

#### GET /api/v1/websites
List all websites

```bash
curl -H "Authorization: Bearer YOUR_TOKEN" \
  https://analytics.yourdomain.com/api/v1/websites
```

#### GET /api/v1/websites/{id}/stats
Get website statistics

```bash
curl -H "Authorization: Bearer YOUR_TOKEN" \
  "https://analytics.yourdomain.com/api/v1/websites/1/stats?from=2026-03-01&to=2026-03-22"
```

Query Parameters:
- `from` (required): Start date (YYYY-MM-DD)
- `to` (required): End date (YYYY-MM-DD)

#### GET /api/v1/websites/{id}/pages
Get page analytics

```bash
curl -H "Authorization: Bearer YOUR_TOKEN" \
  "https://analytics.yourdomain.com/api/v1/websites/1/pages?from=2026-03-01&to=2026-03-22&limit=20"
```

#### GET /api/v1/websites/{id}/referrers
Get referrer data

#### GET /api/v1/websites/{id}/events
Get event data

#### GET /api/v1/websites/{id}/visitors
Get visitor data

---

## Maintenance

### Daily Tasks (Automated)

These run automatically via cron (see [Task Scheduler](#2-task-scheduler-critical)):

- **02:00 AM**: Data aggregation
- **03:00 AM**: Old data purging
- **Every hour**: Alert evaluation

### Weekly Tasks (Manual)

#### Check Queue Status
```bash
# Check if queue worker is running
ps aux | grep "queue:work"

# Check failed jobs
php artisan queue:failed

# Retry failed jobs
php artisan queue:retry all
```

#### Monitor Disk Space
```bash
# Check database size
du -h database/database.sqlite

# Check storage usage
du -sh storage/
```

#### Review Logs
```bash
# Check application logs
tail -f storage/logs/laravel.log

# Check queue worker logs (if using supervisor)
tail -f storage/logs/queue-worker.log
```

### Monthly Tasks

#### Database Optimization

**SQLite**:
```bash
sqlite3 database/database.sqlite "VACUUM;"
```

**MySQL**:
```sql
OPTIMIZE TABLE visitors, analytics_sessions, page_views, events, daily_aggregates, page_daily_aggregates;
```

#### Review Data Retention

Check if retention settings are appropriate:
1. Navigate to **Website Settings**
2. Review **Data Retention** setting (default: 365 days)
3. Adjust based on compliance requirements

### Backup Strategy

#### Database Backup

**SQLite**:
```bash
# Copy database file
cp database/database.sqlite backups/database-$(date +%Y%m%d).sqlite

# Compress
gzip backups/database-$(date +%Y%m%d).sqlite
```

**MySQL**:
```bash
mysqldump -u user -p analytics > backups/analytics-$(date +%Y%m%d).sql
gzip backups/analytics-$(date +%Y%m%d).sql
```

#### Application Backup
```bash
# Backup entire application
tar -czf analytics-backup-$(date +%Y%m%d).tar.gz \
  --exclude=node_modules \
  --exclude=vendor \
  --exclude=storage/logs \
  /path/to/analytics
```

### Troubleshooting

#### Tracking Not Working

1. **Check queue worker is running**:
   ```bash
   ps aux | grep "queue:work"
   ```

2. **Check failed jobs**:
   ```bash
   php artisan queue:failed
   ```

3. **Check tracking domain matches**:
   - Ensure website domain in dashboard matches actual domain
   - Domain validation is case-insensitive but must match

4. **Check browser console**:
   - Look for CORS errors
   - Verify tracking script loads

5. **Check logs**:
   ```bash
   tail -f storage/logs/laravel.log
   ```

#### Dashboard Shows No Data

1. **Check if aggregation ran**:
   ```bash
   php artisan analytics:aggregate --date=yesterday
   ```

2. **Check date range** - today's data may not appear if only querying aggregates

3. **Verify tracking is working** - check raw tables:
   ```sql
   SELECT COUNT(*) FROM page_views WHERE created_at > datetime('now', '-1 hour');
   ```

#### Alerts Not Triggering

1. **Check cron is running**:
   ```bash
   grep CRON /var/log/syslog
   ```

2. **Manually run alert check**:
   ```bash
   php artisan analytics:check-alerts
   ```

3. **Check mail configuration**:
   ```bash
   php artisan tinker
   >>> Mail::raw('Test', function($msg) { $msg->to('test@example.com')->subject('Test'); });
   ```

#### High Disk Usage

1. **Run aggregation** (if not running):
   ```bash
   php artisan analytics:aggregate
   ```

2. **Purge old data**:
   ```bash
   php artisan analytics:purge --force
   ```

3. **Adjust retention settings** in website configuration

---

## Performance Optimization

### 1. Use Redis for Queue & Cache

```env
QUEUE_CONNECTION=redis
CACHE_STORE=redis
SESSION_DRIVER=redis

REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379
```

### 2. Database Indexing

Ensure migrations have run (indexes are included):
```bash
php artisan migrate
```

### 3. Cache Configuration

In production:
```bash
php artisan config:cache
php artisan route:cache
php artisan view:cache
```

### 4. Enable OPcache

Add to `php.ini`:
```ini
opcache.enable=1
opcache.memory_consumption=256
opcache.interned_strings_buffer=16
opcache.max_accelerated_files=10000
opcache.revalidate_freq=2
```

### 5. Queue Workers

Run multiple queue workers for high traffic:
```bash
# Run 3 workers
php artisan queue:work database --tries=3 &
php artisan queue:work database --tries=3 &
php artisan queue:work database --tries=3 &
```

Or use supervisor with `numprocs=3`.

---

## Security Considerations

### 1. Environment File
- Never commit `.env` to version control
- Use strong `APP_KEY` (generated automatically)
- Set `APP_DEBUG=false` in production

### 2. Database Credentials
- Use strong passwords
- Restrict database user permissions
- Use separate database user for application

### 3. HTTPS
- Always use HTTPS in production
- Update `APP_URL` to use `https://`
- Tracking script requires HTTPS for secure cookies

### 4. CORS
- CORS is enabled for `/api/collect` and `/api/t/{id}.js`
- Only these endpoints are publicly accessible
- Dashboard requires authentication

### 5. Rate Limiting
- Tracking endpoints have rate limiting
- API endpoints have Sanctum rate limiting
- Adjust in `app/Http/Middleware` if needed

---

## Support & Resources

### Documentation
- Laravel 11: https://laravel.com/docs/11.x
- Alpine.js: https://alpinejs.dev
- Tailwind CSS: https://tailwindcss.com

### Logs
- Application: `storage/logs/laravel.log`
- Queue Worker: Check supervisor/systemd logs

### Common Paths
- **Application**: `/path/to/analytics`
- **Public**: `/path/to/analytics/public`
- **Storage**: `/path/to/analytics/storage`
- **Database**: `/path/to/analytics/database/database.sqlite`
- **Logs**: `/path/to/analytics/storage/logs/`

---

## Quick Reference

### Essential Commands

```bash
# Development
composer run dev                    # Start dev environment

# Queue
php artisan queue:work database     # Start queue worker
php artisan queue:failed            # List failed jobs
php artisan queue:retry all         # Retry failed jobs

# Analytics
php artisan analytics:aggregate     # Aggregate yesterday's data
php artisan analytics:purge --force # Purge old data
php artisan analytics:check-alerts  # Check alerts

# Cache
php artisan cache:clear             # Clear cache
php artisan config:cache            # Cache config (production)

# Maintenance
php artisan down                    # Enable maintenance mode
php artisan up                      # Disable maintenance mode
```

### Cron Entry

```cron
* * * * * cd /path/to/analytics && php artisan schedule:run >> /dev/null 2>&1
```

### Supervisor Config

```ini
[program:analytics-queue]
command=php /path/to/analytics/artisan queue:work database --tries=3
autostart=true
autorestart=true
user=www-data
```

---

**Last Updated**: 2026-03-22
**Laravel Version**: 11.50.0
**PHP Version**: 8.2+ (Compatible with 8.3)