root/analytics

Fork 0

Files

T

Doar Mihai 7b4c6e59ba downgraded to php 8.3 so it works on host

2026-03-22 17:46:44 +02:00

21 KiB

Raw Blame History

Analytics Platform - HOWTO Guide

What is This Project?
Features Overview
Architecture
Installation
Configuration
Required Background Tasks
Tracking Setup
How It Works
Database Structure
Artisan Commands
API Documentation
Maintenance

What is This Project?

This is a self-hosted web analytics platform built with Laravel 11. It provides privacy-focused website analytics similar to Google Analytics or Plausible, allowing you to track visitor behavior, conversions, and engagement on your websites without relying on third-party services.

Key Capabilities

Real-time visitor tracking
Goal and conversion tracking
Custom event tracking
Funnel analysis
Automated alerts and notifications
Custom dashboards
Export analytics data
API access for integrations

Features Overview

1. Visitor Tracking

Automatic page view tracking
Session management (30-minute timeout)
Device, browser, and OS detection
Geolocation (country, region, city)
Bot filtering
UTM parameter tracking (campaign, source, medium, etc.)
Time on page measurement

2. Analytics Reports

Audience: Geographic distribution, technology breakdown, languages
Acquisition: Traffic sources, referrers, campaigns
Behavior: Popular pages, landing pages, exit pages
Events: Custom event tracking with categories and labels
Conversions: Goal completions with revenue tracking

3. Goals (4 Types)

URL Destination: Track when users visit specific pages
Event Match: Track custom events
Duration: Sessions longer than X seconds
Pages Per Session: Sessions with X or more pages

4. Advanced Features

Funnels: Multi-step conversion tracking
Segments: Custom visitor segments for filtered reporting
Alerts: Automated notifications when metrics exceed thresholds
Annotations: Add notes to specific dates for context
Custom Dashboards: Drag-and-drop widget management

5. Data Management

Automatic daily aggregation for performance
Configurable data retention (default: 365 days)
CSV export
API access with token authentication

Architecture

Technology Stack

Backend: Laravel 11, PHP 8.2+
Frontend: Blade templates, Alpine.js, Tailwind CSS
Database: SQLite (default), MySQL/PostgreSQL compatible
Queue System: Database-backed (Redis/SQS optional)
Build Tool: Vite

Data Flow

Client Website (tracking script)
    ↓ POST /api/collect
CollectController (validates & queues)
    ↓ Queue Job
ProcessTrackingPayload (processes data)
    ↓ Stores in DB
Raw Tables (visitors, sessions, page_views, events)
    ↓ Daily aggregation (02:00 AM)
Aggregated Tables (daily_aggregates, page_daily_aggregates)
    ↓ Used by
Dashboard & Reports (hybrid: aggregates + real-time)

Installation

Development Setup

# Clone the repository
cd /path/to/analytics

# Install dependencies and setup
composer setup

# This runs:
# - composer install
# - Creates .env from .env.example
# - php artisan key:generate
# - php artisan migrate --force
# - npm install
# - npm run build

# Start development environment (runs 4 processes)
composer run dev

This starts:

Web server: http://localhost:8000
Queue worker (processes tracking data)
Log viewer (Pail)
Vite dev server (hot module reload)

Production Setup

# Install dependencies
composer install --no-dev --optimize-autoloader

# Generate application key
php artisan key:generate

# Run database migrations
php artisan migrate --force

# Build frontend assets
npm ci
npm run build

# Set proper permissions
chmod -R 755 storage bootstrap/cache
chown -R www-data:www-data storage bootstrap/cache

Configuration

Environment Variables

Copy .env.example to .env and configure:

# Application
APP_NAME="Analytics Platform"
APP_ENV=production
APP_DEBUG=false
APP_URL=https://yourdomain.com

# Database
DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=analytics
DB_USERNAME=your_user
DB_PASSWORD=your_password

# Queue (use 'redis' for better performance)
QUEUE_CONNECTION=database

# Mail (for alerts and notifications)
MAIL_MAILER=smtp
MAIL_HOST=smtp.mailtrap.io
MAIL_PORT=2525
MAIL_USERNAME=null
MAIL_PASSWORD=null
MAIL_FROM_ADDRESS=noreply@yourdomain.com
MAIL_FROM_NAME="${APP_NAME}"

# Cache (use 'redis' for better performance)
CACHE_STORE=database

# Session
SESSION_DRIVER=database
SESSION_LIFETIME=120

Web Server Configuration

For Standard Setup (public/ folder)

Point your web server document root to the public/ directory.

For Shared Hosting (engine/ + public_html/)

If your hosting has separate folders:

engine/ - Laravel application files
public_html/ - Public web root

Modify public_html/index.php:

<?php

use Illuminate\Foundation\Application;
use Illuminate\Http\Request;

define('LARAVEL_START', microtime(true));

// Point to engine directory
if (file_exists($maintenance = __DIR__.'/../engine/storage/framework/maintenance.php')) {
    require $maintenance;
}

require __DIR__.'/../engine/vendor/autoload.php';

$app = require_once __DIR__.'/../engine/bootstrap/app.php';

// Tell Laravel the public path is public_html
$app->usePublicPath(__DIR__);

$app->handleRequest(Request::capture());

Required Background Tasks

1. Queue Worker (CRITICAL)

Purpose: Processes tracking data from websites

Without this, tracking events are queued but never persisted to the database.

# Start queue worker
php artisan queue:work database --tries=3 --timeout=60

Production Setup: Use a process supervisor

Using Supervisor (recommended):

# /etc/supervisor/conf.d/analytics-queue.conf
[program:analytics-queue]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/analytics/artisan queue:work database --tries=3 --timeout=60
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=1
redirect_stderr=true
stdout_logfile=/path/to/analytics/storage/logs/queue-worker.log
stopwaitsecs=3600

Then run:

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start analytics-queue:*

Using systemd:

# /etc/systemd/system/analytics-queue.service
[Unit]
Description=Analytics Queue Worker
After=network.target

[Service]
Type=simple
User=www-data
WorkingDirectory=/path/to/analytics
ExecStart=/usr/bin/php /path/to/analytics/artisan queue:work database --tries=3 --timeout=60
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl enable analytics-queue
sudo systemctl start analytics-queue

2. Task Scheduler (CRITICAL)

Purpose: Runs scheduled tasks (aggregation, purging, alerts)

Add to crontab:

# Edit crontab
crontab -e

# Add this line
* * * * * cd /path/to/analytics && php artisan schedule:run >> /dev/null 2>&1

This executes:

Daily at 02:00 AM: analytics:aggregate - Rolls up data into daily aggregates
Daily at 03:00 AM: analytics:purge --force - Deletes old raw data
Every hour: analytics:check-alerts - Evaluates alerts and sends notifications

Without the scheduler:

Reports will be slower (no aggregated data)
Old data will accumulate and bloat the database
Alerts will never trigger

Tracking Setup

1. Create a Website

Register/login to the analytics platform
Navigate to Websites → Add Website
Enter website name and domain
Copy the tracking ID (e.g., abc123def456)

2. Install Tracking Script

Add this script to your website's HTML (before </head>):

Option 1: JavaScript Snippet

<script async src="https://analytics.yourdomain.com/api/t/YOUR_TRACKING_ID.js"></script>

Option 2: Self-hosted Script

<script>
  !function(t,e,a){/* tracking script */}(window,document,"https://analytics.yourdomain.com","YOUR_TRACKING_ID");
</script>

Option 3: No-JavaScript Fallback

<noscript>
  <img src="https://analytics.yourdomain.com/api/collect/pixel/YOUR_TRACKING_ID"
       alt=""
       style="position:absolute;left:-9999px;" />
</noscript>

3. Track Custom Events

// Track a custom event
ta('event', 'button_click', {
  category: 'engagement',
  label: 'signup_button',
  value: 1
});

// Track a purchase
ta('event', 'purchase', {
  category: 'ecommerce',
  value: 99.99
});

4. SPA Support

For single-page applications (React, Vue, etc.), the tracker automatically detects route changes via pushState and popstate. You can also manually track page views:

// Manual page view tracking
ta('pageview');

How It Works

Tracking Process

Client-side: Tracking script loads on visitor's browser
Collection: Script sends tracking data to /api/collect
Validation: CollectController validates tracking ID and domain
Queueing: Tracking payload is queued as ProcessTrackingPayload job
Processing: Queue worker picks up job and:
- Creates/updates Visitor record (with geolocation)
- Creates/retrieves AnalyticsSession (30-min timeout)
- Records PageView or Event
- Evaluates active Goals and records conversions
- Updates session metrics (duration, bounce status)
Aggregation: Daily at 02:00 AM, raw data is rolled up into aggregates
Reporting: Dashboard queries combine aggregated + real-time data

Session Logic

Session Start: First page view from a visitor
Session Timeout: 30 minutes of inactivity
Session Duration: Time between first and last page view
Bounce: Session with only 1 page view
Exit Page: Last page viewed in session

Goal Evaluation

Goals are evaluated in real-time during tracking:

URL Destination: Matches current page URL
- Exact: URL must match exactly
- Starts with: URL must start with pattern
- Contains: URL must contain pattern
- Regex: URL must match regex pattern
Event Match: Matches event name (and optionally category)
Duration: Session duration exceeds threshold
Pages Per Session: Page count exceeds threshold

Data Aggregation

Raw tracking data is aggregated daily to improve query performance:

Daily Aggregates (per website, per day):

Total visitors (unique)
Total sessions
Total page views
Bounce count
Total session duration

Page Daily Aggregates (per website, per page, per day):

Page views
Unique page views
Total time on page
Bounce count

Query Strategy:

Historical dates: Query aggregated tables
Today: Query raw tables
Results are merged for complete picture

Database Structure

Core Tables

Table	Description
`users`	User accounts
`websites`	Tracked websites with settings
`visitors`	Unique visitors with device/location info
`analytics_sessions`	User sessions with UTM data
`page_views`	Individual page visits
`events`	Custom events with metadata
`goals`	Conversion goal definitions
`goal_conversions`	Goal completion records
`funnels`	Funnel definitions
`segments`	Custom visitor segments
`alerts`	Alert rule definitions
`alert_notifications`	Triggered alert records
`annotations`	Date annotations
`dashboard_widgets`	Custom dashboard layouts
`daily_aggregates`	Pre-computed daily metrics
`page_daily_aggregates`	Per-page daily metrics

Data Retention

Raw Data: Retained for 365 days (configurable per website)
Aggregated Data: Retained indefinitely
Purge Schedule: Daily at 03:00 AM via analytics:purge

Artisan Commands

Analytics Commands

1. Aggregate Data

php artisan analytics:aggregate [options]

Options:
  --date=YYYY-MM-DD    Aggregate specific date (default: yesterday)
  --website=ID         Aggregate specific website (default: all)

# Examples
php artisan analytics:aggregate
php artisan analytics:aggregate --date=2026-03-20
php artisan analytics:aggregate --website=1

Important: Cannot aggregate today's data (prevents incomplete aggregations)

2. Purge Old Data

php artisan analytics:purge [options]

Options:
  --website=ID    Purge specific website (default: all)
  --force         Skip confirmation prompt

# Examples
php artisan analytics:purge --force
php artisan analytics:purge --website=1

Deletes raw data older than retention period (default: 365 days).

3. Check Alerts

php artisan analytics:check-alerts

Evaluates all active alerts and triggers notifications.

Queue Commands

# Process queue jobs
php artisan queue:work database --tries=3

# List failed jobs
php artisan queue:failed

# Retry failed job
php artisan queue:retry {job-id}

# Retry all failed jobs
php artisan queue:retry all

# Flush failed jobs
php artisan queue:flush

Cache Commands

# Clear application cache
php artisan cache:clear

# Clear config cache
php artisan config:clear

# Clear route cache
php artisan route:clear

# Clear view cache
php artisan view:clear

# Cache config for production
php artisan config:cache
php artisan route:cache
php artisan view:cache

API Documentation

Authentication

Generate API token in dashboard: Profile → API Tokens → Create Token

Include in requests:

Authorization: Bearer YOUR_API_TOKEN

Endpoints

GET /api/v1/websites

List all websites

curl -H "Authorization: Bearer YOUR_TOKEN" \
  https://analytics.yourdomain.com/api/v1/websites

GET /api/v1/websites/{id}/stats

Get website statistics

curl -H "Authorization: Bearer YOUR_TOKEN" \
  "https://analytics.yourdomain.com/api/v1/websites/1/stats?from=2026-03-01&to=2026-03-22"

Query Parameters:

from (required): Start date (YYYY-MM-DD)
to (required): End date (YYYY-MM-DD)

GET /api/v1/websites/{id}/pages

Get page analytics

curl -H "Authorization: Bearer YOUR_TOKEN" \
  "https://analytics.yourdomain.com/api/v1/websites/1/pages?from=2026-03-01&to=2026-03-22&limit=20"

GET /api/v1/websites/{id}/referrers

Get referrer data

GET /api/v1/websites/{id}/events

Get event data

GET /api/v1/websites/{id}/visitors

Get visitor data

Maintenance

Daily Tasks (Automated)

These run automatically via cron (see Task Scheduler):

02:00 AM: Data aggregation
03:00 AM: Old data purging
Every hour: Alert evaluation

Weekly Tasks (Manual)

Check Queue Status

# Check if queue worker is running
ps aux | grep "queue:work"

# Check failed jobs
php artisan queue:failed

# Retry failed jobs
php artisan queue:retry all

Monitor Disk Space

# Check database size
du -h database/database.sqlite

# Check storage usage
du -sh storage/

Review Logs

# Check application logs
tail -f storage/logs/laravel.log

# Check queue worker logs (if using supervisor)
tail -f storage/logs/queue-worker.log

Monthly Tasks

Database Optimization

SQLite:

sqlite3 database/database.sqlite "VACUUM;"

MySQL:

OPTIMIZE TABLE visitors, analytics_sessions, page_views, events, daily_aggregates, page_daily_aggregates;

Review Data Retention

Check if retention settings are appropriate:

Navigate to Website Settings
Review Data Retention setting (default: 365 days)
Adjust based on compliance requirements

Backup Strategy

Database Backup

SQLite:

# Copy database file
cp database/database.sqlite backups/database-$(date +%Y%m%d).sqlite

# Compress
gzip backups/database-$(date +%Y%m%d).sqlite

MySQL:

mysqldump -u user -p analytics > backups/analytics-$(date +%Y%m%d).sql
gzip backups/analytics-$(date +%Y%m%d).sql

Application Backup

# Backup entire application
tar -czf analytics-backup-$(date +%Y%m%d).tar.gz \
  --exclude=node_modules \
  --exclude=vendor \
  --exclude=storage/logs \
  /path/to/analytics

Troubleshooting

Tracking Not Working

Check queue worker is running:
```
ps aux | grep "queue:work"
```
Check failed jobs:
```
php artisan queue:failed
```
Check tracking domain matches:
- Ensure website domain in dashboard matches actual domain
- Domain validation is case-insensitive but must match
Check browser console:
- Look for CORS errors
- Verify tracking script loads
Check logs:
```
tail -f storage/logs/laravel.log
```

Dashboard Shows No Data

Check if aggregation ran:

php artisan analytics:aggregate --date=yesterday

Check date range - today's data may not appear if only querying aggregates

Verify tracking is working - check raw tables:

SELECT COUNT(*) FROM page_views WHERE created_at > datetime('now', '-1 hour');

Alerts Not Triggering

Check cron is running:
```
grep CRON /var/log/syslog
```
Manually run alert check:
```
php artisan analytics:check-alerts
```

Check mail configuration:

php artisan tinker
>>> Mail::raw('Test', function($msg) { $msg->to('test@example.com')->subject('Test'); });

High Disk Usage

Run aggregation (if not running):
```
php artisan analytics:aggregate
```
Purge old data:
```
php artisan analytics:purge --force
```
Adjust retention settings in website configuration

Performance Optimization

1. Use Redis for Queue & Cache

QUEUE_CONNECTION=redis
CACHE_STORE=redis
SESSION_DRIVER=redis

REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379

2. Database Indexing

Ensure migrations have run (indexes are included):

php artisan migrate

3. Cache Configuration

In production:

php artisan config:cache
php artisan route:cache
php artisan view:cache

4. Enable OPcache

Add to php.ini:

opcache.enable=1
opcache.memory_consumption=256
opcache.interned_strings_buffer=16
opcache.max_accelerated_files=10000
opcache.revalidate_freq=2

5. Queue Workers

Run multiple queue workers for high traffic:

# Run 3 workers
php artisan queue:work database --tries=3 &
php artisan queue:work database --tries=3 &
php artisan queue:work database --tries=3 &

Or use supervisor with numprocs=3.

Security Considerations

1. Environment File

Never commit .env to version control
Use strong APP_KEY (generated automatically)
Set APP_DEBUG=false in production

2. Database Credentials

Use strong passwords
Restrict database user permissions
Use separate database user for application

3. HTTPS

Always use HTTPS in production
Update APP_URL to use https://
Tracking script requires HTTPS for secure cookies

4. CORS

CORS is enabled for /api/collect and /api/t/{id}.js
Only these endpoints are publicly accessible
Dashboard requires authentication

5. Rate Limiting

Tracking endpoints have rate limiting
API endpoints have Sanctum rate limiting
Adjust in app/Http/Middleware if needed

Support & Resources

Documentation

Laravel 11: https://laravel.com/docs/11.x
Alpine.js: https://alpinejs.dev
Tailwind CSS: https://tailwindcss.com

Logs

Application: storage/logs/laravel.log
Queue Worker: Check supervisor/systemd logs

Common Paths

Application: /path/to/analytics
Public: /path/to/analytics/public
Storage: /path/to/analytics/storage
Database: /path/to/analytics/database/database.sqlite
Logs: /path/to/analytics/storage/logs/

Quick Reference

Essential Commands

# Development
composer run dev                    # Start dev environment

# Queue
php artisan queue:work database     # Start queue worker
php artisan queue:failed            # List failed jobs
php artisan queue:retry all         # Retry failed jobs

# Analytics
php artisan analytics:aggregate     # Aggregate yesterday's data
php artisan analytics:purge --force # Purge old data
php artisan analytics:check-alerts  # Check alerts

# Cache
php artisan cache:clear             # Clear cache
php artisan config:cache            # Cache config (production)

# Maintenance
php artisan down                    # Enable maintenance mode
php artisan up                      # Disable maintenance mode

Cron Entry

* * * * * cd /path/to/analytics && php artisan schedule:run >> /dev/null 2>&1

Supervisor Config

[program:analytics-queue]
command=php /path/to/analytics/artisan queue:work database --tries=3
autostart=true
autorestart=true
user=www-data

Last Updated: 2026-03-22 Laravel Version: 11.50.0 PHP Version: 8.2+ (Compatible with 8.3)

21 KiB Raw Blame History