analytics/collector_app_plan.md

# Analytics Collector App Extraction Plan

## Overview

This document outlines the plan for extracting the data collection endpoints from the main analytics application into a separate Laravel application. The collector app will be responsible for receiving tracking data from external websites, processing it, and storing it in the **shared database** (same database as the main app).

## Goals

1. **Separation of Concerns**: Isolate high-traffic data collection endpoints from the analytics dashboard
2. **Scalability**: Enable independent scaling of the collector service
3. **Reliability**: Prevent tracking service disruptions from affecting dashboard availability
4. **Performance**: Optimize collector for high-throughput, low-latency operations
5. **Security**: Isolate public-facing endpoints from internal analytics data

---

## Architecture Overview

### Current State
```
┌─────────────────────────────────────────┐
│       Single Laravel Application        │
│  ┌─────────────┐    ┌────────────────┐ │
│  │  Dashboard  │    │   Collection   │ │
│  │  (Auth UI)  │    │   Endpoints    │ │
│  └─────────────┘    └────────────────┘ │
│            │               │            │
│            └───────┬───────┘            │
│                    │                    │
│              ┌─────▼─────┐              │
│              │  Database │              │
│              └───────────┘              │
└─────────────────────────────────────────┘
```

### Target State (Shared Database Architecture)
```
┌──────────────────────┐       ┌──────────────────────┐
│   Analytics App      │       │   Collector App      │
│   (Dashboard/Auth)   │       │   (Data Ingestion)   │
│                      │       │                      │
│  ┌────────────────┐  │       │  ┌────────────────┐  │
│  │  Reports/UI    │  │       │  │   POST /api/e  │  │
│  │  User Mgmt     │  │       │  │  GET /api/p    │  │
│  │  Settings      │  │       │  │  Tracking JS   │  │
│  └────────────────┘  │       │  └────────────────┘  │
│          │           │       │          │           │
│          └───────────┴───────┴──────────┘           │
│                      │                              │
│              ┌───────▼────────┐                     │
│              │  Shared Database│                     │
│              │  (All tables)   │                     │
│              └─────────────────┘                     │
└──────────────────────────────────────────────────────┘

Both apps read/write to the same database:
- No data sync needed
- Real-time data availability
- Simpler architecture
```

---

## Phase 1: Setup New Collector App

### 1.1 Create New Laravel Application

```bash
composer create-project laravel/laravel analytics-collector
cd analytics-collector
```

### 1.2 Configure Environment

**`.env` configuration for collector app:**
```env
APP_NAME="Analytics Collector"

# IMPORTANT: Same database as main app
DB_CONNECTION=mysql
DB_HOST=127.0.0.1  # Same as main app
DB_PORT=3306
DB_DATABASE=analytics  # Same database name as main app
DB_USERNAME=same_user_as_main_app
DB_PASSWORD=same_password_as_main_app

QUEUE_CONNECTION=redis
REDIS_CLIENT=phpredis

# Rate limiting - can be higher than main app
COLLECTOR_RATE_LIMIT=120  # per minute per IP

# Geolocation
LOCATION_TESTING=false
```

**Key points:**
- Database credentials must match the main app exactly
- Both apps will share all tables (websites, visitors, sessions, events, etc.)
- No separate database needed
- No data sync mechanism required

### 1.3 Install Dependencies

```bash
composer require jaybizzle/crawler-detect
composer require stevebauman/location
composer require predis/predis
```

---

## Phase 2: Database Setup (Shared Database)

### 2.1 No Migrations Needed!

Since both apps use the **same database**, there's no need to copy or create migrations in the collector app.

**What this means:**
- The database already has all necessary tables from the main app
- The collector app will simply connect to the existing database
- Both apps read/write to the same tables directly
- No schema changes required

### 2.2 Skip Running Migrations

**Do NOT run migrations on the collector app** - the tables already exist from the main app.

Instead:
1. Configure `.env` to point to the same database
2. Test database connection: `php artisan db:show`
3. Verify tables exist: `php artisan tinker` → `DB::table('websites')->count()`

---

## Phase 3: Code Extraction

### 3.1 Controllers to Copy

**File: `app/Http/Controllers/CollectController.php`**

Copy entire controller with modifications:
- Keep all validation logic
- Keep bot detection (`isBot()`, `isDomainValid()`)
- Dispatch `ProcessTrackingPayload` job

**File: `app/Http/Controllers/TrackingScriptController.php`**

Copy entire controller:
- Serves JavaScript tracking code
- Update collection endpoint URLs to point to collector app

### 3.2 Middleware to Copy

**File: `app/Http/Middleware/CollectCors.php`**

Copy entire middleware for CORS handling.

**Register in `bootstrap/app.php`:**
```php
->withMiddleware(function (Middleware $middleware) {
    $middleware->alias([
        'collect.cors' => \App\Http\Middleware\CollectCors::class,
    ]);
})
```

### 3.3 Jobs to Copy

**File: `app/Jobs/ProcessTrackingPayload.php`**

Copy entire job with modifications:
- Keep all visitor/session/pageview/event processing
- Keep goal evaluation logic
- Add error handling for missing websites (sync issue)

### 3.4 Models to Copy

Copy the following models **exactly as they are** from the main app:
- `app/Models/Website.php`
- `app/Models/Visitor.php`
- `app/Models/AnalyticsSession.php`
- `app/Models/PageView.php`
- `app/Models/Event.php`
- `app/Models/Goal.php`
- `app/Models/GoalConversion.php`

**Important notes:**
- Copy the models unchanged - they need to work with the same database schema
- Keep all relationships intact
- Keep all casts and accessors
- The only exception: You can remove the `User` relationship from `Website.php` if the collector doesn't need it

### 3.5 Services to Copy

**File: `app/Services/GoalEvaluationService.php`**

Copy entire service for goal evaluation logic.

### 3.6 Views to Copy

**File: `resources/views/tracking/tracker.blade.php`**

Copy JavaScript tracking code with modifications:
- Update collection endpoint URLs to collector app
- Keep all tracking logic (visitor ID, session management, event tracking)

---

## Phase 4: Routes Configuration

### 4.1 API Routes

**File: `routes/api.php`**

```php
use App\Http\Controllers\CollectController;
use App\Http\Controllers\TrackingScriptController;

// Main collection endpoints (backward compatible)
Route::post('/e', [CollectController::class, 'store'])
    ->middleware(['collect.cors', 'throttle:collect'])
    ->name('collect.store');

Route::get('/p', [CollectController::class, 'pixel'])
    ->middleware(['collect.cors', 'throttle:collect'])
    ->name('collect.pixel');

// Ad-blocker resistant endpoints
Route::post('/cdn/feedback', [CollectController::class, 'store'])
    ->middleware(['collect.cors', 'throttle:collect'])
    ->name('collect.store.alt');

Route::get('/cdn/status.gif', [CollectController::class, 'pixel'])
    ->middleware(['collect.cors', 'throttle:collect'])
    ->name('collect.pixel.alt');

// Tracking script delivery
Route::get('/t/{trackingId}.js', [TrackingScriptController::class, 'script'])
    ->middleware('collect.cors')
    ->name('tracking.script');

Route::get('/cdn/resources/{trackingId}.js', [TrackingScriptController::class, 'script'])
    ->middleware('collect.cors')
    ->name('tracking.script.alt');
```

### 4.2 Rate Limiting Configuration

**File: `app/Providers/AppServiceProvider.php`**

```php
use Illuminate\Cache\RateLimiting\Limit;
use Illuminate\Support\Facades\RateLimiter;

public function boot(): void
{
    RateLimiter::for('collect', function (Request $request) {
        return Limit::perMinute(config('collector.rate_limit', 120))
            ->by($request->ip());
    });
}
```

**Create config file: `config/collector.php`**

```php
return [
    'rate_limit' => env('COLLECTOR_RATE_LIMIT', 120),
    'main_app_url' => env('MAIN_APP_URL'),
    'api_key' => env('COLLECTOR_API_KEY'),
];
```

---

## Phase 5: No Sync Mechanism Needed! 🎉

### 5.1 Shared Database = No Sync Required

Since both apps use the **same database**, there's **NO need for a sync mechanism**!

**What this means:**
- When a website is created/updated/deleted in the main app, it's immediately available in the collector app
- When a goal is created/updated/deleted in the main app, the collector app sees it instantly
- When the collector creates tracking data, the main app can query it immediately
- Zero latency between apps
- No complex sync logic needed
- No API endpoints for syncing

**Benefits:**
- Simpler codebase (no sync controllers, middleware, or API calls)
- Real-time data consistency
- No sync delays or failures to handle
- Reduced network overhead
- Fewer moving parts = fewer potential issues

**Only consideration:**
- Both apps must have read access to `websites` and `goals` tables
- Both apps need write access to `visitors`, `sessions`, `page_views`, `events`, `goal_conversions` tables
- Database connection pooling should account for connections from both apps

---

## Phase 6: Data Flow - Already Solved! ✅

### Shared Database Architecture

**Data flow is automatic with shared database:**

```
Collector App                    Main App
     │                              │
     │  Writes tracking data        │  Reads tracking data
     │  (visitors, sessions,        │  (for reports & analytics)
     │   page_views, events)        │
     │                              │
     └──────────┬───────────────────┘
                │
         ┌──────▼───────┐
         │   Database   │
         │  (Shared)    │
         └──────────────┘
```

**How it works:**
1. **Collector writes** → tracking data (visitors, sessions, page_views, events, conversions)
2. **Main app reads** → same tables for reports, dashboards, analytics
3. **Main app writes** → configuration data (websites, goals, users, settings)
4. **Collector reads** → website/goal configs for validation and goal evaluation

**No additional implementation needed** - this is the beauty of the shared database approach!

---

## Phase 7: Deployment Strategy

### 7.1 Infrastructure Setup

**Collector App Requirements:**
- High-availability load balancer
- Multiple app instances for horizontal scaling
- Redis for queue and cache
- Database (MySQL/PostgreSQL)
- CDN for tracking script delivery (optional)

**Suggested Stack:**
```
┌─────────────────────────┐
│   CDN (CloudFlare)      │
│   /cdn/resources/*.js   │
└────────┬────────────────┘
         │
┌────────▼────────────────┐
│   Load Balancer         │
│   (nginx/HAProxy)       │
└────────┬────────────────┘
         │
    ┌────┴────┐
    │         │
┌───▼───┐ ┌──▼────┐
│ App 1 │ │ App 2 │ ... (N instances)
└───┬───┘ └──┬────┘
    │        │
┌───▼────────▼───┐
│  Redis Queue   │
└───┬────────────┘
    │
┌───▼────────────┐
│  Queue Workers │
└───┬────────────┘
    │
┌───▼────────────┐
│  Database      │
└────────────────┘
```

### 7.2 Migration Steps

**Step 1: Prepare Collector App**
1. Deploy collector app to staging
2. Run migrations
3. Sync initial website/goal data
4. Test collection endpoints

**Step 2: Gradual Rollout**
1. Update tracking script to include feature flag
2. Route 10% of traffic to collector app
3. Monitor error rates, latency, data accuracy
4. Gradually increase to 100%

**Step 3: DNS/Routing Changes**
1. Update tracking script URLs to collector domain
2. Setup CNAME: `track.yourdomain.com` → collector app
3. Update CORS configuration

**Step 4: Main App Cleanup**
1. Remove collection routes from main app
2. Remove `CollectController`, `TrackingScriptController`
3. Keep models and migrations (if shared database)
4. Archive old code for rollback safety

---

## Phase 8: Monitoring & Observability

### 8.1 Metrics to Track

**Collector App:**
- Request rate (req/sec)
- Response time (p50, p95, p99)
- Error rate by type (bot, validation, domain_mismatch)
- Queue depth and processing rate
- Database connection pool usage
- Bot traffic percentage

**Dashboard:**
```php
// app/Http/Controllers/HealthController.php
public function stats(): JsonResponse
{
    return response()->json([
        'requests_last_minute' => Cache::get('requests_last_minute', 0),
        'queue_depth' => Queue::size(),
        'db_connections' => DB::connection()->getDoctrineConnection()->getWrappedConnection()->getAttribute(PDO::ATTR_SERVER_INFO),
    ]);
}
```

### 8.2 Alerting

Set up alerts for:
- Response time > 200ms (p95)
- Error rate > 1%
- Queue depth > 10,000
- Database connections > 80% of pool
- Disk space < 20%

---

## Phase 9: Performance Optimizations

### 9.1 Database Optimizations

**Indexing Strategy:**
```sql
-- Visitors table
CREATE INDEX idx_visitors_lookup ON visitors(website_id, uid);
CREATE INDEX idx_visitors_last_seen ON visitors(website_id, last_seen_at);

-- Sessions table
CREATE INDEX idx_sessions_lookup ON analytics_sessions(website_id, uid);
CREATE INDEX idx_sessions_started ON analytics_sessions(website_id, started_at);

-- Page views table
CREATE INDEX idx_pageviews_viewed ON page_views(website_id, viewed_at);
CREATE INDEX idx_pageviews_path ON page_views(website_id, path(255));

-- Events table
CREATE INDEX idx_events_occurred ON events(website_id, occurred_at);
CREATE INDEX idx_events_name ON events(website_id, name);
```

**Partitioning (for very high volume):**
```sql
-- Partition page_views by month
ALTER TABLE page_views PARTITION BY RANGE (YEAR(viewed_at) * 100 + MONTH(viewed_at)) (
    PARTITION p202603 VALUES LESS THAN (202604),
    PARTITION p202604 VALUES LESS THAN (202605),
    -- ... add future partitions
);
```

### 9.2 Queue Optimization

**Use Redis for queue:**
```env
QUEUE_CONNECTION=redis
REDIS_QUEUE=tracking
```

**Tune worker count based on load:**
```bash
php artisan queue:work redis --queue=tracking --sleep=1 --tries=3 --max-jobs=1000
```

**Use Horizon for queue management:**
```bash
composer require laravel/horizon
php artisan horizon:install
php artisan horizon
```

### 9.3 Caching Strategy

**Cache website configs:**
```php
// In ProcessTrackingPayload job
protected function getWebsite(string $trackingId): ?Website
{
    return Cache::remember(
        "website:{$trackingId}",
        now()->addHours(1),
        fn() => Website::where('tracking_id', $trackingId)->first()
    );
}
```

**Cache goals:**
```php
protected function getActiveGoals(int $websiteId): Collection
{
    return Cache::remember(
        "goals:{$websiteId}",
        now()->addMinutes(5),
        fn() => Goal::where('website_id', $websiteId)
            ->where('is_active', true)
            ->get()
    );
}
```

---

## Phase 10: Security Considerations

### 10.1 DDoS Protection

**Rate Limiting by IP:**
- Current: 120 req/min per IP
- Consider: Dynamic rate limiting based on traffic patterns
- Use CloudFlare or similar CDN for DDoS protection

**Bot Detection:**
- Keep `CrawlerDetect` package
- Log bot patterns for analysis
- Consider adding honeypot endpoints

### 10.2 Data Validation

**Strict input validation:**
- Validate all tracking payload fields
- Sanitize URLs, titles, event names
- Limit string lengths (prevent database bloat)
- Validate JSON structure for event_properties

### 10.3 API Security

**Sync endpoint protection:**
- Use strong API keys (32+ characters)
- Rotate keys periodically
- IP whitelist for sync endpoints
- Use HTTPS only

---

## Phase 11: Testing Strategy

### 11.1 Unit Tests

**Test coverage for:**
- `CollectController` validation logic
- `ProcessTrackingPayload` job
- `GoalEvaluationService`
- Bot detection
- Domain validation

### 11.2 Integration Tests

**Test scenarios:**
- End-to-end pageview tracking
- Event tracking and goal conversion
- Session management
- Website sync from main app
- Goal sync from main app

### 11.3 Load Testing

**Use tools:**
- Apache Bench (ab)
- wrk
- k6

**Test scenarios:**
```bash
# Simulate 1000 req/sec for 60 seconds
k6 run --vus 100 --duration 60s load-test.js
```

**Load test script example:**
```javascript
// load-test.js
import http from 'k6/http';

export default function () {
  const payload = JSON.stringify({
    tid: 'TA-TEST123456',
    vid: `visitor-${__VU}-${__ITER}`,
    sid: `session-${__VU}`,
    type: 'pageview',
    url: 'https://example.com/test',
    path: '/test',
  });

  http.post('http://collector.local/api/e', payload, {
    headers: { 'Content-Type': 'application/json' },
  });
}
```

---

## Phase 12: Rollback Plan

### 12.1 Rollback Triggers

Rollback if:
- Error rate > 5%
- Data loss detected (missing tracking data)
- Performance degradation (p95 > 500ms)
- Queue backup (depth > 50,000)

### 12.2 Rollback Steps

1. **Immediate:**
   - Update tracking script to point back to main app
   - Disable collector app routing

2. **Within 1 hour:**
   - Restore collection endpoints in main app
   - Sync any data collected in collector to main app

3. **Post-incident:**
   - Analyze root cause
   - Fix issues
   - Re-plan rollout

---

## Phase 13: Cost Analysis

### 13.1 Infrastructure Costs

**Collector App (estimated monthly costs with shared database):**
- Load balancer: $20/month
- App instances (2x): $50/month each = $100/month
- ~~Database: $0 (using main app's database)~~
- Redis: $30/month
- Queue workers (2x): $30/month each = $60/month
- **Total: ~$210/month**

**Savings from reduced main app load:**
- Potential downgrade of main app instance: -$30/month

**Net increase: ~$180/month**

**Database costs:**
- No additional database server needed
- May need to upgrade main database instance slightly (+$10-20/month) to handle additional connections from collector app
- **Actual net increase: ~$190-200/month**

### 13.2 ROI Analysis

**Benefits:**
- **Reliability:** Fewer dashboard outages during traffic spikes
- **Scalability:** Handle 10x traffic without main app changes
- **Development velocity:** Deploy collector changes independently
- **Security:** Better isolation of public endpoints

---

## Appendix A: Configuration Files

### Main App `.env` (no changes needed)

Since both apps share the same database, **no configuration changes** are needed in the main app's `.env` file.

The main app continues to work exactly as before.

### Collector App `.env`

```env
APP_NAME="Analytics Collector"
APP_ENV=production
APP_DEBUG=false

# SHARED DATABASE - Same as main app
DB_CONNECTION=mysql
DB_HOST=main-db.local  # Same database server as main app
DB_PORT=3306
DB_DATABASE=analytics  # Same database name as main app
DB_USERNAME=analytics_user
DB_PASSWORD=same_password_as_main_app

QUEUE_CONNECTION=redis
REDIS_HOST=collector-redis.local

COLLECTOR_RATE_LIMIT=120
```

---

## Appendix B: nginx Configuration

```nginx
server {
    listen 80;
    server_name track.yourdomain.com;

    location / {
        return 301 https://$server_name$request_uri;
    }
}

server {
    listen 443 ssl http2;
    server_name track.yourdomain.com;

    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    root /var/www/analytics-collector/public;
    index index.php;

    # Rate limiting
    limit_req_zone $binary_remote_addr zone=collector:10m rate=120r/m;
    limit_req zone=collector burst=20 nodelay;

    # Static assets (tracking script)
    location ~* \.(js|gif)$ {
        expires 1h;
        add_header Cache-Control "public, immutable";
    }

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        fastcgi_pass unix:/var/run/php/php8.2-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        include fastcgi_params;
    }
}
```

---

## Appendix C: Supervisor Configuration (Queue Workers)

```ini
[program:analytics-collector-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/analytics-collector/artisan queue:work redis --queue=tracking --sleep=1 --tries=3 --max-jobs=1000 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/analytics-collector/storage/logs/worker.log
stopwaitsecs=3600
```

---

## Timeline Estimate

| Phase | Duration | Dependencies | Notes |
|-------|----------|--------------|-------|
| 1. Setup new app | 1 day | - | Laravel install + dependencies |
| 2. Database setup | 0.5 days | Phase 1 | Just configure `.env` - no migrations! |
| 3. Code extraction | 2-3 days | Phase 2 | Copy controllers, models, jobs, services |
| 4. Routes configuration | 1 day | Phase 3 | Set up API routes |
| 5. ~~Sync mechanism~~ | ~~Skipped~~ | - | **Not needed with shared DB!** |
| 6. ~~Data flow~~ | ~~Skipped~~ | - | **Automatic with shared DB!** |
| 7. Deployment prep | 2 days | Phase 4 | Infrastructure setup |
| 8. Monitoring setup | 1 day | Phase 7 | Metrics & alerting |
| 9. Optimizations | 2 days | Phase 7 | Caching, queue tuning |
| 10. Security review | 1 day | All above | CORS, rate limiting, validation |
| 11. Testing | 2-3 days | Phase 10 | Unit + integration + load tests |
| 12. Staged rollout | 1 week | Phase 11 | Gradual traffic migration |

**Total estimated time: 2-3 weeks** (reduced from 3-4 weeks due to no sync mechanism needed!)

---

## Success Metrics

### Post-Launch (30 days)

- **Uptime:** > 99.9% for collector app
- **Response time:** p95 < 100ms for collection endpoints
- **Error rate:** < 0.5%
- **Data accuracy:** 100% match vs. pre-migration baseline
- **Cost:** Within 10% of projections

---

## Conclusion

Extracting the data collection endpoints into a separate collector app provides significant benefits in terms of scalability, reliability, and maintainability. With the **shared database approach**, implementation is significantly simplified:

### Key Benefits of Shared Database Architecture

1. **Simplicity:** No sync mechanism needed - data is immediately available to both apps
2. **Real-time:** Zero latency between data collection and analytics availability
3. **Reliability:** No sync failures or delays to handle
4. **Cost-effective:** Save ~$50/month vs. separate database approach
5. **Faster implementation:** 2-3 weeks instead of 3-4 weeks

### Implementation Summary

1. **Database strategy:** ✅ Shared database (both apps connect to same DB)
2. **Sync mechanism:** ✅ Not needed - data is instantly shared
3. **Deployment:** Gradual rollout with feature flags
4. **Monitoring:** Comprehensive metrics and alerting from day 1

### Trade-offs to Consider

**Pros:**
- Simplest architecture
- No data sync complexity
- Real-time data availability
- Lower infrastructure costs

**Cons:**
- Database is shared point of failure (mitigate with replication)
- Both apps depend on same database uptime
- Database connection pool must accommodate both apps
- Less flexibility for future independent scaling of databases

### Recommendation

The **shared database approach is recommended** for your use case because:
- Your traffic volume doesn't require separate databases yet
- The simplicity significantly reduces complexity and development time
- You can always migrate to separate databases later if needed
- The cost savings (~$50/month) and time savings (1-2 weeks) are significant

This plan provides a clear roadmap for the extraction while minimizing risk and maintaining data integrity throughout the migration process.