analytics/collector_app_plan.md

Analytics Collector App Extraction Plan

Overview

This document outlines the plan for extracting the data collection endpoints from the main analytics application into a separate Laravel application. The collector app will be responsible for receiving tracking data from external websites, processing it, and storing it in the shared database (same database as the main app).

Goals

1. Separation of Concerns: Isolate high-traffic data collection endpoints from the analytics dashboard
2. Scalability: Enable independent scaling of the collector service
3. Reliability: Prevent tracking service disruptions from affecting dashboard availability
4. Performance: Optimize collector for high-throughput, low-latency operations
5. Security: Isolate public-facing endpoints from internal analytics data

---

Architecture Overview

Current State

┌─────────────────────────────────────────┐
│       Single Laravel Application        │
│  ┌─────────────┐    ┌────────────────┐ │
│  │  Dashboard  │    │   Collection   │ │
│  │  (Auth UI)  │    │   Endpoints    │ │
│  └─────────────┘    └────────────────┘ │
│            │               │            │
│            └───────┬───────┘            │
│                    │                    │
│              ┌─────▼─────┐              │
│              │  Database │              │
│              └───────────┘              │
└─────────────────────────────────────────┘

Target State (Shared Database Architecture)

┌──────────────────────┐       ┌──────────────────────┐
│   Analytics App      │       │   Collector App      │
│   (Dashboard/Auth)   │       │   (Data Ingestion)   │
│                      │       │                      │
│  ┌────────────────┐  │       │  ┌────────────────┐  │
│  │  Reports/UI    │  │       │  │   POST /api/e  │  │
│  │  User Mgmt     │  │       │  │  GET /api/p    │  │
│  │  Settings      │  │       │  │  Tracking JS   │  │
│  └────────────────┘  │       │  └────────────────┘  │
│          │           │       │          │           │
│          └───────────┴───────┴──────────┘           │
│                      │                              │
│              ┌───────▼────────┐                     │
│              │  Shared Database│                     │
│              │  (All tables)   │                     │
│              └─────────────────┘                     │
└──────────────────────────────────────────────────────┘

Both apps read/write to the same database:
- No data sync needed
- Real-time data availability
- Simpler architecture

---

Phase 1: Setup New Collector App

1.1 Create New Laravel Application

composer create-project laravel/laravel analytics-collector
cd analytics-collector

1.2 Configure Environment

.env configuration for collector app:

APP_NAME="Analytics Collector"

# IMPORTANT: Same database as main app
DB_CONNECTION=mysql
DB_HOST=127.0.0.1  # Same as main app
DB_PORT=3306
DB_DATABASE=analytics  # Same database name as main app
DB_USERNAME=same_user_as_main_app
DB_PASSWORD=same_password_as_main_app

QUEUE_CONNECTION=redis
REDIS_CLIENT=phpredis

# Rate limiting - can be higher than main app
COLLECTOR_RATE_LIMIT=120  # per minute per IP

# Geolocation
LOCATION_TESTING=false

Key points:

• Database credentials must match the main app exactly
• Both apps will share all tables (websites, visitors, sessions, events, etc.)
• No separate database needed
• No data sync mechanism required

1.3 Install Dependencies

composer require jaybizzle/crawler-detect
composer require stevebauman/location
composer require predis/predis

---

Phase 2: Database Setup (Shared Database)

2.1 No Migrations Needed!

Since both apps use the same database, there's no need to copy or create migrations in the collector app.

What this means:

• The database already has all necessary tables from the main app
• The collector app will simply connect to the existing database
• Both apps read/write to the same tables directly
• No schema changes required

2.2 Skip Running Migrations

Do NOT run migrations on the collector app - the tables already exist from the main app.

Instead:

1. Configure .env to point to the same database
2. Test database connection: php artisan db:show
3. Verify tables exist: php artisan tinker → DB::table('websites')->count()

---

Phase 3: Code Extraction

3.1 Controllers to Copy

File: app/Http/Controllers/CollectController.php

Copy entire controller with modifications:

• Keep all validation logic
• Keep bot detection (isBot(), isDomainValid())
• Dispatch ProcessTrackingPayload job

File: app/Http/Controllers/TrackingScriptController.php

Copy entire controller:

• Serves JavaScript tracking code
• Update collection endpoint URLs to point to collector app

3.2 Middleware to Copy

File: app/Http/Middleware/CollectCors.php

Copy entire middleware for CORS handling.

Register in bootstrap/app.php:

->withMiddleware(function (Middleware $middleware) {
    $middleware->alias([
        'collect.cors' => \App\Http\Middleware\CollectCors::class,
    ]);
})

3.3 Jobs to Copy

File: app/Jobs/ProcessTrackingPayload.php

Copy entire job with modifications:

• Keep all visitor/session/pageview/event processing
• Keep goal evaluation logic
• Add error handling for missing websites (sync issue)

3.4 Models to Copy

Copy the following models exactly as they are from the main app:

• app/Models/Website.php
• app/Models/Visitor.php
• app/Models/AnalyticsSession.php
• app/Models/PageView.php
• app/Models/Event.php
• app/Models/Goal.php
• app/Models/GoalConversion.php

Important notes:

• Copy the models unchanged - they need to work with the same database schema
• Keep all relationships intact
• Keep all casts and accessors
• The only exception: You can remove the User relationship from Website.php if the collector doesn't need it

3.5 Services to Copy

File: app/Services/GoalEvaluationService.php

Copy entire service for goal evaluation logic.

3.6 Views to Copy

File: resources/views/tracking/tracker.blade.php

Copy JavaScript tracking code with modifications:

• Update collection endpoint URLs to collector app
• Keep all tracking logic (visitor ID, session management, event tracking)

---

Phase 4: Routes Configuration

4.1 API Routes

File: routes/api.php

use App\Http\Controllers\CollectController;
use App\Http\Controllers\TrackingScriptController;

// Main collection endpoints (backward compatible)
Route::post('/e', [CollectController::class, 'store'])
    ->middleware(['collect.cors', 'throttle:collect'])
    ->name('collect.store');

Route::get('/p', [CollectController::class, 'pixel'])
    ->middleware(['collect.cors', 'throttle:collect'])
    ->name('collect.pixel');

// Ad-blocker resistant endpoints
Route::post('/cdn/feedback', [CollectController::class, 'store'])
    ->middleware(['collect.cors', 'throttle:collect'])
    ->name('collect.store.alt');

Route::get('/cdn/status.gif', [CollectController::class, 'pixel'])
    ->middleware(['collect.cors', 'throttle:collect'])
    ->name('collect.pixel.alt');

// Tracking script delivery
Route::get('/t/{trackingId}.js', [TrackingScriptController::class, 'script'])
    ->middleware('collect.cors')
    ->name('tracking.script');

Route::get('/cdn/resources/{trackingId}.js', [TrackingScriptController::class, 'script'])
    ->middleware('collect.cors')
    ->name('tracking.script.alt');

4.2 Rate Limiting Configuration

File: app/Providers/AppServiceProvider.php

use Illuminate\Cache\RateLimiting\Limit;
use Illuminate\Support\Facades\RateLimiter;

public function boot(): void
{
    RateLimiter::for('collect', function (Request $request) {
        return Limit::perMinute(config('collector.rate_limit', 120))
            ->by($request->ip());
    });
}

Create config file: config/collector.php

return [
    'rate_limit' => env('COLLECTOR_RATE_LIMIT', 120),
    'main_app_url' => env('MAIN_APP_URL'),
    'api_key' => env('COLLECTOR_API_KEY'),
];

---

Phase 5: No Sync Mechanism Needed! 🎉

5.1 Shared Database = No Sync Required

Since both apps use the same database, there's NO need for a sync mechanism!

What this means:

• When a website is created/updated/deleted in the main app, it's immediately available in the collector app
• When a goal is created/updated/deleted in the main app, the collector app sees it instantly
• When the collector creates tracking data, the main app can query it immediately
• Zero latency between apps
• No complex sync logic needed
• No API endpoints for syncing

Benefits:

• Simpler codebase (no sync controllers, middleware, or API calls)
• Real-time data consistency
• No sync delays or failures to handle
• Reduced network overhead
• Fewer moving parts = fewer potential issues

Only consideration:

• Both apps must have read access to websites and goals tables
• Both apps need write access to visitors, sessions, page_views, events, goal_conversions tables
• Database connection pooling should account for connections from both apps

---

Phase 6: Data Flow - Already Solved! ✅

Shared Database Architecture

Data flow is automatic with shared database:

Collector App                    Main App
     │                              │
     │  Writes tracking data        │  Reads tracking data
     │  (visitors, sessions,        │  (for reports & analytics)
     │   page_views, events)        │
     │                              │
     └──────────┬───────────────────┘
                │
         ┌──────▼───────┐
         │   Database   │
         │  (Shared)    │
         └──────────────┘

How it works:

1. Collector writes → tracking data (visitors, sessions, page_views, events, conversions)
2. Main app reads → same tables for reports, dashboards, analytics
3. Main app writes → configuration data (websites, goals, users, settings)
4. Collector reads → website/goal configs for validation and goal evaluation

No additional implementation needed - this is the beauty of the shared database approach!

---

Phase 7: Deployment Strategy

7.1 Infrastructure Setup

Collector App Requirements:

• High-availability load balancer
• Multiple app instances for horizontal scaling
• Redis for queue and cache
• Database (MySQL/PostgreSQL)
• CDN for tracking script delivery (optional)

Suggested Stack:

┌─────────────────────────┐
│   CDN (CloudFlare)      │
│   /cdn/resources/*.js   │
└────────┬────────────────┘
         │
┌────────▼────────────────┐
│   Load Balancer         │
│   (nginx/HAProxy)       │
└────────┬────────────────┘
         │
    ┌────┴────┐
    │         │
┌───▼───┐ ┌──▼────┐
│ App 1 │ │ App 2 │ ... (N instances)
└───┬───┘ └──┬────┘
    │        │
┌───▼────────▼───┐
│  Redis Queue   │
└───┬────────────┘
    │
┌───▼────────────┐
│  Queue Workers │
└───┬────────────┘
    │
┌───▼────────────┐
│  Database      │
└────────────────┘

7.2 Migration Steps

Step 1: Prepare Collector App

1. Deploy collector app to staging
2. Run migrations
3. Sync initial website/goal data
4. Test collection endpoints

Step 2: Gradual Rollout

1. Update tracking script to include feature flag
2. Route 10% of traffic to collector app
3. Monitor error rates, latency, data accuracy
4. Gradually increase to 100%

Step 3: DNS/Routing Changes

1. Update tracking script URLs to collector domain
2. Setup CNAME: track.yourdomain.com → collector app
3. Update CORS configuration

Step 4: Main App Cleanup

1. Remove collection routes from main app
2. Remove CollectController, TrackingScriptController
3. Keep models and migrations (if shared database)
4. Archive old code for rollback safety

---

Phase 8: Monitoring & Observability

8.1 Metrics to Track

Collector App:

• Request rate (req/sec)
• Response time (p50, p95, p99)
• Error rate by type (bot, validation, domain_mismatch)
• Queue depth and processing rate
• Database connection pool usage
• Bot traffic percentage

Dashboard:

// app/Http/Controllers/HealthController.php
public function stats(): JsonResponse
{
    return response()->json([
        'requests_last_minute' => Cache::get('requests_last_minute', 0),
        'queue_depth' => Queue::size(),
        'db_connections' => DB::connection()->getDoctrineConnection()->getWrappedConnection()->getAttribute(PDO::ATTR_SERVER_INFO),
    ]);
}

8.2 Alerting

Set up alerts for:

• Response time > 200ms (p95)
• Error rate > 1%
• Queue depth > 10,000
• Database connections > 80% of pool
• Disk space < 20%

---

Phase 9: Performance Optimizations

9.1 Database Optimizations

Indexing Strategy:

-- Visitors table
CREATE INDEX idx_visitors_lookup ON visitors(website_id, uid);
CREATE INDEX idx_visitors_last_seen ON visitors(website_id, last_seen_at);

-- Sessions table
CREATE INDEX idx_sessions_lookup ON analytics_sessions(website_id, uid);
CREATE INDEX idx_sessions_started ON analytics_sessions(website_id, started_at);

-- Page views table
CREATE INDEX idx_pageviews_viewed ON page_views(website_id, viewed_at);
CREATE INDEX idx_pageviews_path ON page_views(website_id, path(255));

-- Events table
CREATE INDEX idx_events_occurred ON events(website_id, occurred_at);
CREATE INDEX idx_events_name ON events(website_id, name);

Partitioning (for very high volume):

-- Partition page_views by month
ALTER TABLE page_views PARTITION BY RANGE (YEAR(viewed_at) * 100 + MONTH(viewed_at)) (
    PARTITION p202603 VALUES LESS THAN (202604),
    PARTITION p202604 VALUES LESS THAN (202605),
    -- ... add future partitions
);

9.2 Queue Optimization

Use Redis for queue:

QUEUE_CONNECTION=redis
REDIS_QUEUE=tracking

Tune worker count based on load:

php artisan queue:work redis --queue=tracking --sleep=1 --tries=3 --max-jobs=1000

Use Horizon for queue management:

composer require laravel/horizon
php artisan horizon:install
php artisan horizon

9.3 Caching Strategy

Cache website configs:

// In ProcessTrackingPayload job
protected function getWebsite(string $trackingId): ?Website
{
    return Cache::remember(
        "website:{$trackingId}",
        now()->addHours(1),
        fn() => Website::where('tracking_id', $trackingId)->first()
    );
}

Cache goals:

protected function getActiveGoals(int $websiteId): Collection
{
    return Cache::remember(
        "goals:{$websiteId}",
        now()->addMinutes(5),
        fn() => Goal::where('website_id', $websiteId)
            ->where('is_active', true)
            ->get()
    );
}

---

Phase 10: Security Considerations

10.1 DDoS Protection

Rate Limiting by IP:

• Current: 120 req/min per IP
• Consider: Dynamic rate limiting based on traffic patterns
• Use CloudFlare or similar CDN for DDoS protection

Bot Detection:

• Keep CrawlerDetect package
• Log bot patterns for analysis
• Consider adding honeypot endpoints

10.2 Data Validation

Strict input validation:

• Validate all tracking payload fields
• Sanitize URLs, titles, event names
• Limit string lengths (prevent database bloat)
• Validate JSON structure for event_properties

10.3 API Security

Sync endpoint protection:

• Use strong API keys (32+ characters)
• Rotate keys periodically
• IP whitelist for sync endpoints
• Use HTTPS only

---

Phase 11: Testing Strategy

11.1 Unit Tests

Test coverage for:

• CollectController validation logic
• ProcessTrackingPayload job
• GoalEvaluationService
• Bot detection
• Domain validation

11.2 Integration Tests

Test scenarios:

• End-to-end pageview tracking
• Event tracking and goal conversion
• Session management
• Website sync from main app
• Goal sync from main app

11.3 Load Testing

Use tools:

• Apache Bench (ab)
• wrk
• k6

Test scenarios:

# Simulate 1000 req/sec for 60 seconds
k6 run --vus 100 --duration 60s load-test.js

Load test script example:

// load-test.js
import http from 'k6/http';

export default function () {
  const payload = JSON.stringify({
    tid: 'TA-TEST123456',
    vid: `visitor-${__VU}-${__ITER}`,
    sid: `session-${__VU}`,
    type: 'pageview',
    url: 'https://example.com/test',
    path: '/test',
  });

  http.post('http://collector.local/api/e', payload, {
    headers: { 'Content-Type': 'application/json' },
  });
}

---

Phase 12: Rollback Plan

12.1 Rollback Triggers

Rollback if:

• Error rate > 5%
• Data loss detected (missing tracking data)
• Performance degradation (p95 > 500ms)
• Queue backup (depth > 50,000)

12.2 Rollback Steps

1. Immediate:

   • Update tracking script to point back to main app
   • Disable collector app routing

2. Within 1 hour:

   • Restore collection endpoints in main app
   • Sync any data collected in collector to main app

3. Post-incident:

   • Analyze root cause
   • Fix issues
   • Re-plan rollout

---

Phase 13: Cost Analysis

13.1 Infrastructure Costs

Collector App (estimated monthly costs with shared database):

• Load balancer: $20/month
• App instances (2x): $50/month each = $100/month
• Database: $0 (using main app's database)
• Redis: $30/month
• Queue workers (2x): $30/month each = $60/month
• Total: ~$210/month

Savings from reduced main app load:

• Potential downgrade of main app instance: -$30/month

Net increase: ~$180/month

Database costs:

• No additional database server needed
• May need to upgrade main database instance slightly (+$10-20/month) to handle additional connections from collector app
• Actual net increase: ~$190-200/month

13.2 ROI Analysis

Benefits:

• Reliability: Fewer dashboard outages during traffic spikes
• Scalability: Handle 10x traffic without main app changes
• Development velocity: Deploy collector changes independently
• Security: Better isolation of public endpoints

---

Appendix A: Configuration Files

Main App .env (no changes needed)

Since both apps share the same database, no configuration changes are needed in the main app's .env file.

The main app continues to work exactly as before.

Collector App .env

APP_NAME="Analytics Collector"
APP_ENV=production
APP_DEBUG=false

# SHARED DATABASE - Same as main app
DB_CONNECTION=mysql
DB_HOST=main-db.local  # Same database server as main app
DB_PORT=3306
DB_DATABASE=analytics  # Same database name as main app
DB_USERNAME=analytics_user
DB_PASSWORD=same_password_as_main_app

QUEUE_CONNECTION=redis
REDIS_HOST=collector-redis.local

COLLECTOR_RATE_LIMIT=120

---

Appendix B: nginx Configuration

server {
    listen 80;
    server_name track.yourdomain.com;

    location / {
        return 301 https://$server_name$request_uri;
    }
}

server {
    listen 443 ssl http2;
    server_name track.yourdomain.com;

    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    root /var/www/analytics-collector/public;
    index index.php;

    # Rate limiting
    limit_req_zone $binary_remote_addr zone=collector:10m rate=120r/m;
    limit_req zone=collector burst=20 nodelay;

    # Static assets (tracking script)
    location ~* \.(js|gif)$ {
        expires 1h;
        add_header Cache-Control "public, immutable";
    }

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        fastcgi_pass unix:/var/run/php/php8.2-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        include fastcgi_params;
    }
}

---

Appendix C: Supervisor Configuration (Queue Workers)

[program:analytics-collector-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/analytics-collector/artisan queue:work redis --queue=tracking --sleep=1 --tries=3 --max-jobs=1000 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/analytics-collector/storage/logs/worker.log
stopwaitsecs=3600

---

Timeline Estimate

Phase	Duration	Dependencies	Notes
1. Setup new app	1 day	-	Laravel install + dependencies
2. Database setup	0.5 days	Phase 1	Just configure .env - no migrations!
3. Code extraction	2-3 days	Phase 2	Copy controllers, models, jobs, services
4. Routes configuration	1 day	Phase 3	Set up API routes
5. Sync mechanism	Skipped	-	Not needed with shared DB!
6. Data flow	Skipped	-	Automatic with shared DB!
7. Deployment prep	2 days	Phase 4	Infrastructure setup
8. Monitoring setup	1 day	Phase 7	Metrics & alerting
9. Optimizations	2 days	Phase 7	Caching, queue tuning
10. Security review	1 day	All above	CORS, rate limiting, validation
11. Testing	2-3 days	Phase 10	Unit + integration + load tests
12. Staged rollout	1 week	Phase 11	Gradual traffic migration

Total estimated time: 2-3 weeks (reduced from 3-4 weeks due to no sync mechanism needed!)

---

Success Metrics

Post-Launch (30 days)

• Uptime: > 99.9% for collector app
• Response time: p95 < 100ms for collection endpoints
• Error rate: < 0.5%
• Data accuracy: 100% match vs. pre-migration baseline
• Cost: Within 10% of projections

---

Conclusion

Extracting the data collection endpoints into a separate collector app provides significant benefits in terms of scalability, reliability, and maintainability. With the shared database approach, implementation is significantly simplified:

Key Benefits of Shared Database Architecture

1. Simplicity: No sync mechanism needed - data is immediately available to both apps
2. Real-time: Zero latency between data collection and analytics availability
3. Reliability: No sync failures or delays to handle
4. Cost-effective: Save ~$50/month vs. separate database approach
5. Faster implementation: 2-3 weeks instead of 3-4 weeks

Implementation Summary

1. Database strategy: ✅ Shared database (both apps connect to same DB)
2. Sync mechanism: ✅ Not needed - data is instantly shared
3. Deployment: Gradual rollout with feature flags
4. Monitoring: Comprehensive metrics and alerting from day 1

Trade-offs to Consider

Pros:

• Simplest architecture
• No data sync complexity
• Real-time data availability
• Lower infrastructure costs

Cons:

• Database is shared point of failure (mitigate with replication)
• Both apps depend on same database uptime
• Database connection pool must accommodate both apps
• Less flexibility for future independent scaling of databases

Recommendation

The shared database approach is recommended for your use case because:

• Your traffic volume doesn't require separate databases yet
• The simplicity significantly reduces complexity and development time
• You can always migrate to separate databases later if needed
• The cost savings (~$50/month) and time savings (1-2 weeks) are significant

This plan provides a clear roadmap for the extraction while minimizing risk and maintaining data integrity throughout the migration process.