diff --git a/collector_app_plan.md b/collector_app_plan.md new file mode 100644 index 0000000..9794f79 --- /dev/null +++ b/collector_app_plan.md @@ -0,0 +1,843 @@ +# Analytics Collector App Extraction Plan + +## Overview + +This document outlines the plan for extracting the data collection endpoints from the main analytics application into a separate Laravel application. The collector app will be responsible for receiving tracking data from external websites, processing it, and storing it in the **shared database** (same database as the main app). + +## Goals + +1. **Separation of Concerns**: Isolate high-traffic data collection endpoints from the analytics dashboard +2. **Scalability**: Enable independent scaling of the collector service +3. **Reliability**: Prevent tracking service disruptions from affecting dashboard availability +4. **Performance**: Optimize collector for high-throughput, low-latency operations +5. **Security**: Isolate public-facing endpoints from internal analytics data + +--- + +## Architecture Overview + +### Current State +``` +┌─────────────────────────────────────────┐ +│ Single Laravel Application │ +│ ┌─────────────┐ ┌────────────────┐ │ +│ │ Dashboard │ │ Collection │ │ +│ │ (Auth UI) │ │ Endpoints │ │ +│ └─────────────┘ └────────────────┘ │ +│ │ │ │ +│ └───────┬───────┘ │ +│ │ │ +│ ┌─────▼─────┐ │ +│ │ Database │ │ +│ └───────────┘ │ +└─────────────────────────────────────────┘ +``` + +### Target State (Shared Database Architecture) +``` +┌──────────────────────┐ ┌──────────────────────┐ +│ Analytics App │ │ Collector App │ +│ (Dashboard/Auth) │ │ (Data Ingestion) │ +│ │ │ │ +│ ┌────────────────┐ │ │ ┌────────────────┐ │ +│ │ Reports/UI │ │ │ │ POST /api/e │ │ +│ │ User Mgmt │ │ │ │ GET /api/p │ │ +│ │ Settings │ │ │ │ Tracking JS │ │ +│ └────────────────┘ │ │ └────────────────┘ │ +│ │ │ │ │ │ +│ └───────────┴───────┴──────────┘ │ +│ │ │ +│ ┌───────▼────────┐ │ +│ │ Shared Database│ │ +│ │ (All tables) │ │ +│ └─────────────────┘ │ +└──────────────────────────────────────────────────────┘ + +Both apps read/write to the same database: +- No data sync needed +- Real-time data availability +- Simpler architecture +``` + +--- + +## Phase 1: Setup New Collector App + +### 1.1 Create New Laravel Application + +```bash +composer create-project laravel/laravel analytics-collector +cd analytics-collector +``` + +### 1.2 Configure Environment + +**`.env` configuration for collector app:** +```env +APP_NAME="Analytics Collector" + +# IMPORTANT: Same database as main app +DB_CONNECTION=mysql +DB_HOST=127.0.0.1 # Same as main app +DB_PORT=3306 +DB_DATABASE=analytics # Same database name as main app +DB_USERNAME=same_user_as_main_app +DB_PASSWORD=same_password_as_main_app + +QUEUE_CONNECTION=redis +REDIS_CLIENT=phpredis + +# Rate limiting - can be higher than main app +COLLECTOR_RATE_LIMIT=120 # per minute per IP + +# Geolocation +LOCATION_TESTING=false +``` + +**Key points:** +- Database credentials must match the main app exactly +- Both apps will share all tables (websites, visitors, sessions, events, etc.) +- No separate database needed +- No data sync mechanism required + +### 1.3 Install Dependencies + +```bash +composer require jaybizzle/crawler-detect +composer require stevebauman/location +composer require predis/predis +``` + +--- + +## Phase 2: Database Setup (Shared Database) + +### 2.1 No Migrations Needed! + +Since both apps use the **same database**, there's no need to copy or create migrations in the collector app. + +**What this means:** +- The database already has all necessary tables from the main app +- The collector app will simply connect to the existing database +- Both apps read/write to the same tables directly +- No schema changes required + +### 2.2 Skip Running Migrations + +**Do NOT run migrations on the collector app** - the tables already exist from the main app. + +Instead: +1. Configure `.env` to point to the same database +2. Test database connection: `php artisan db:show` +3. Verify tables exist: `php artisan tinker` → `DB::table('websites')->count()` + +--- + +## Phase 3: Code Extraction + +### 3.1 Controllers to Copy + +**File: `app/Http/Controllers/CollectController.php`** + +Copy entire controller with modifications: +- Keep all validation logic +- Keep bot detection (`isBot()`, `isDomainValid()`) +- Dispatch `ProcessTrackingPayload` job + +**File: `app/Http/Controllers/TrackingScriptController.php`** + +Copy entire controller: +- Serves JavaScript tracking code +- Update collection endpoint URLs to point to collector app + +### 3.2 Middleware to Copy + +**File: `app/Http/Middleware/CollectCors.php`** + +Copy entire middleware for CORS handling. + +**Register in `bootstrap/app.php`:** +```php +->withMiddleware(function (Middleware $middleware) { + $middleware->alias([ + 'collect.cors' => \App\Http\Middleware\CollectCors::class, + ]); +}) +``` + +### 3.3 Jobs to Copy + +**File: `app/Jobs/ProcessTrackingPayload.php`** + +Copy entire job with modifications: +- Keep all visitor/session/pageview/event processing +- Keep goal evaluation logic +- Add error handling for missing websites (sync issue) + +### 3.4 Models to Copy + +Copy the following models **exactly as they are** from the main app: +- `app/Models/Website.php` +- `app/Models/Visitor.php` +- `app/Models/AnalyticsSession.php` +- `app/Models/PageView.php` +- `app/Models/Event.php` +- `app/Models/Goal.php` +- `app/Models/GoalConversion.php` + +**Important notes:** +- Copy the models unchanged - they need to work with the same database schema +- Keep all relationships intact +- Keep all casts and accessors +- The only exception: You can remove the `User` relationship from `Website.php` if the collector doesn't need it + +### 3.5 Services to Copy + +**File: `app/Services/GoalEvaluationService.php`** + +Copy entire service for goal evaluation logic. + +### 3.6 Views to Copy + +**File: `resources/views/tracking/tracker.blade.php`** + +Copy JavaScript tracking code with modifications: +- Update collection endpoint URLs to collector app +- Keep all tracking logic (visitor ID, session management, event tracking) + +--- + +## Phase 4: Routes Configuration + +### 4.1 API Routes + +**File: `routes/api.php`** + +```php +use App\Http\Controllers\CollectController; +use App\Http\Controllers\TrackingScriptController; + +// Main collection endpoints (backward compatible) +Route::post('/e', [CollectController::class, 'store']) + ->middleware(['collect.cors', 'throttle:collect']) + ->name('collect.store'); + +Route::get('/p', [CollectController::class, 'pixel']) + ->middleware(['collect.cors', 'throttle:collect']) + ->name('collect.pixel'); + +// Ad-blocker resistant endpoints +Route::post('/cdn/feedback', [CollectController::class, 'store']) + ->middleware(['collect.cors', 'throttle:collect']) + ->name('collect.store.alt'); + +Route::get('/cdn/status.gif', [CollectController::class, 'pixel']) + ->middleware(['collect.cors', 'throttle:collect']) + ->name('collect.pixel.alt'); + +// Tracking script delivery +Route::get('/t/{trackingId}.js', [TrackingScriptController::class, 'script']) + ->middleware('collect.cors') + ->name('tracking.script'); + +Route::get('/cdn/resources/{trackingId}.js', [TrackingScriptController::class, 'script']) + ->middleware('collect.cors') + ->name('tracking.script.alt'); +``` + +### 4.2 Rate Limiting Configuration + +**File: `app/Providers/AppServiceProvider.php`** + +```php +use Illuminate\Cache\RateLimiting\Limit; +use Illuminate\Support\Facades\RateLimiter; + +public function boot(): void +{ + RateLimiter::for('collect', function (Request $request) { + return Limit::perMinute(config('collector.rate_limit', 120)) + ->by($request->ip()); + }); +} +``` + +**Create config file: `config/collector.php`** + +```php +return [ + 'rate_limit' => env('COLLECTOR_RATE_LIMIT', 120), + 'main_app_url' => env('MAIN_APP_URL'), + 'api_key' => env('COLLECTOR_API_KEY'), +]; +``` + +--- + +## Phase 5: No Sync Mechanism Needed! 🎉 + +### 5.1 Shared Database = No Sync Required + +Since both apps use the **same database**, there's **NO need for a sync mechanism**! + +**What this means:** +- When a website is created/updated/deleted in the main app, it's immediately available in the collector app +- When a goal is created/updated/deleted in the main app, the collector app sees it instantly +- When the collector creates tracking data, the main app can query it immediately +- Zero latency between apps +- No complex sync logic needed +- No API endpoints for syncing + +**Benefits:** +- Simpler codebase (no sync controllers, middleware, or API calls) +- Real-time data consistency +- No sync delays or failures to handle +- Reduced network overhead +- Fewer moving parts = fewer potential issues + +**Only consideration:** +- Both apps must have read access to `websites` and `goals` tables +- Both apps need write access to `visitors`, `sessions`, `page_views`, `events`, `goal_conversions` tables +- Database connection pooling should account for connections from both apps + +--- + +## Phase 6: Data Flow - Already Solved! ✅ + +### Shared Database Architecture + +**Data flow is automatic with shared database:** + +``` +Collector App Main App + │ │ + │ Writes tracking data │ Reads tracking data + │ (visitors, sessions, │ (for reports & analytics) + │ page_views, events) │ + │ │ + └──────────┬───────────────────┘ + │ + ┌──────▼───────┐ + │ Database │ + │ (Shared) │ + └──────────────┘ +``` + +**How it works:** +1. **Collector writes** → tracking data (visitors, sessions, page_views, events, conversions) +2. **Main app reads** → same tables for reports, dashboards, analytics +3. **Main app writes** → configuration data (websites, goals, users, settings) +4. **Collector reads** → website/goal configs for validation and goal evaluation + +**No additional implementation needed** - this is the beauty of the shared database approach! + +--- + +## Phase 7: Deployment Strategy + +### 7.1 Infrastructure Setup + +**Collector App Requirements:** +- High-availability load balancer +- Multiple app instances for horizontal scaling +- Redis for queue and cache +- Database (MySQL/PostgreSQL) +- CDN for tracking script delivery (optional) + +**Suggested Stack:** +``` +┌─────────────────────────┐ +│ CDN (CloudFlare) │ +│ /cdn/resources/*.js │ +└────────┬────────────────┘ + │ +┌────────▼────────────────┐ +│ Load Balancer │ +│ (nginx/HAProxy) │ +└────────┬────────────────┘ + │ + ┌────┴────┐ + │ │ +┌───▼───┐ ┌──▼────┐ +│ App 1 │ │ App 2 │ ... (N instances) +└───┬───┘ └──┬────┘ + │ │ +┌───▼────────▼───┐ +│ Redis Queue │ +└───┬────────────┘ + │ +┌───▼────────────┐ +│ Queue Workers │ +└───┬────────────┘ + │ +┌───▼────────────┐ +│ Database │ +└────────────────┘ +``` + +### 7.2 Migration Steps + +**Step 1: Prepare Collector App** +1. Deploy collector app to staging +2. Run migrations +3. Sync initial website/goal data +4. Test collection endpoints + +**Step 2: Gradual Rollout** +1. Update tracking script to include feature flag +2. Route 10% of traffic to collector app +3. Monitor error rates, latency, data accuracy +4. Gradually increase to 100% + +**Step 3: DNS/Routing Changes** +1. Update tracking script URLs to collector domain +2. Setup CNAME: `track.yourdomain.com` → collector app +3. Update CORS configuration + +**Step 4: Main App Cleanup** +1. Remove collection routes from main app +2. Remove `CollectController`, `TrackingScriptController` +3. Keep models and migrations (if shared database) +4. Archive old code for rollback safety + +--- + +## Phase 8: Monitoring & Observability + +### 8.1 Metrics to Track + +**Collector App:** +- Request rate (req/sec) +- Response time (p50, p95, p99) +- Error rate by type (bot, validation, domain_mismatch) +- Queue depth and processing rate +- Database connection pool usage +- Bot traffic percentage + +**Dashboard:** +```php +// app/Http/Controllers/HealthController.php +public function stats(): JsonResponse +{ + return response()->json([ + 'requests_last_minute' => Cache::get('requests_last_minute', 0), + 'queue_depth' => Queue::size(), + 'db_connections' => DB::connection()->getDoctrineConnection()->getWrappedConnection()->getAttribute(PDO::ATTR_SERVER_INFO), + ]); +} +``` + +### 8.2 Alerting + +Set up alerts for: +- Response time > 200ms (p95) +- Error rate > 1% +- Queue depth > 10,000 +- Database connections > 80% of pool +- Disk space < 20% + +--- + +## Phase 9: Performance Optimizations + +### 9.1 Database Optimizations + +**Indexing Strategy:** +```sql +-- Visitors table +CREATE INDEX idx_visitors_lookup ON visitors(website_id, uid); +CREATE INDEX idx_visitors_last_seen ON visitors(website_id, last_seen_at); + +-- Sessions table +CREATE INDEX idx_sessions_lookup ON analytics_sessions(website_id, uid); +CREATE INDEX idx_sessions_started ON analytics_sessions(website_id, started_at); + +-- Page views table +CREATE INDEX idx_pageviews_viewed ON page_views(website_id, viewed_at); +CREATE INDEX idx_pageviews_path ON page_views(website_id, path(255)); + +-- Events table +CREATE INDEX idx_events_occurred ON events(website_id, occurred_at); +CREATE INDEX idx_events_name ON events(website_id, name); +``` + +**Partitioning (for very high volume):** +```sql +-- Partition page_views by month +ALTER TABLE page_views PARTITION BY RANGE (YEAR(viewed_at) * 100 + MONTH(viewed_at)) ( + PARTITION p202603 VALUES LESS THAN (202604), + PARTITION p202604 VALUES LESS THAN (202605), + -- ... add future partitions +); +``` + +### 9.2 Queue Optimization + +**Use Redis for queue:** +```env +QUEUE_CONNECTION=redis +REDIS_QUEUE=tracking +``` + +**Tune worker count based on load:** +```bash +php artisan queue:work redis --queue=tracking --sleep=1 --tries=3 --max-jobs=1000 +``` + +**Use Horizon for queue management:** +```bash +composer require laravel/horizon +php artisan horizon:install +php artisan horizon +``` + +### 9.3 Caching Strategy + +**Cache website configs:** +```php +// In ProcessTrackingPayload job +protected function getWebsite(string $trackingId): ?Website +{ + return Cache::remember( + "website:{$trackingId}", + now()->addHours(1), + fn() => Website::where('tracking_id', $trackingId)->first() + ); +} +``` + +**Cache goals:** +```php +protected function getActiveGoals(int $websiteId): Collection +{ + return Cache::remember( + "goals:{$websiteId}", + now()->addMinutes(5), + fn() => Goal::where('website_id', $websiteId) + ->where('is_active', true) + ->get() + ); +} +``` + +--- + +## Phase 10: Security Considerations + +### 10.1 DDoS Protection + +**Rate Limiting by IP:** +- Current: 120 req/min per IP +- Consider: Dynamic rate limiting based on traffic patterns +- Use CloudFlare or similar CDN for DDoS protection + +**Bot Detection:** +- Keep `CrawlerDetect` package +- Log bot patterns for analysis +- Consider adding honeypot endpoints + +### 10.2 Data Validation + +**Strict input validation:** +- Validate all tracking payload fields +- Sanitize URLs, titles, event names +- Limit string lengths (prevent database bloat) +- Validate JSON structure for event_properties + +### 10.3 API Security + +**Sync endpoint protection:** +- Use strong API keys (32+ characters) +- Rotate keys periodically +- IP whitelist for sync endpoints +- Use HTTPS only + +--- + +## Phase 11: Testing Strategy + +### 11.1 Unit Tests + +**Test coverage for:** +- `CollectController` validation logic +- `ProcessTrackingPayload` job +- `GoalEvaluationService` +- Bot detection +- Domain validation + +### 11.2 Integration Tests + +**Test scenarios:** +- End-to-end pageview tracking +- Event tracking and goal conversion +- Session management +- Website sync from main app +- Goal sync from main app + +### 11.3 Load Testing + +**Use tools:** +- Apache Bench (ab) +- wrk +- k6 + +**Test scenarios:** +```bash +# Simulate 1000 req/sec for 60 seconds +k6 run --vus 100 --duration 60s load-test.js +``` + +**Load test script example:** +```javascript +// load-test.js +import http from 'k6/http'; + +export default function () { + const payload = JSON.stringify({ + tid: 'TA-TEST123456', + vid: `visitor-${__VU}-${__ITER}`, + sid: `session-${__VU}`, + type: 'pageview', + url: 'https://example.com/test', + path: '/test', + }); + + http.post('http://collector.local/api/e', payload, { + headers: { 'Content-Type': 'application/json' }, + }); +} +``` + +--- + +## Phase 12: Rollback Plan + +### 12.1 Rollback Triggers + +Rollback if: +- Error rate > 5% +- Data loss detected (missing tracking data) +- Performance degradation (p95 > 500ms) +- Queue backup (depth > 50,000) + +### 12.2 Rollback Steps + +1. **Immediate:** + - Update tracking script to point back to main app + - Disable collector app routing + +2. **Within 1 hour:** + - Restore collection endpoints in main app + - Sync any data collected in collector to main app + +3. **Post-incident:** + - Analyze root cause + - Fix issues + - Re-plan rollout + +--- + +## Phase 13: Cost Analysis + +### 13.1 Infrastructure Costs + +**Collector App (estimated monthly costs with shared database):** +- Load balancer: $20/month +- App instances (2x): $50/month each = $100/month +- ~~Database: $0 (using main app's database)~~ +- Redis: $30/month +- Queue workers (2x): $30/month each = $60/month +- **Total: ~$210/month** + +**Savings from reduced main app load:** +- Potential downgrade of main app instance: -$30/month + +**Net increase: ~$180/month** + +**Database costs:** +- No additional database server needed +- May need to upgrade main database instance slightly (+$10-20/month) to handle additional connections from collector app +- **Actual net increase: ~$190-200/month** + +### 13.2 ROI Analysis + +**Benefits:** +- **Reliability:** Fewer dashboard outages during traffic spikes +- **Scalability:** Handle 10x traffic without main app changes +- **Development velocity:** Deploy collector changes independently +- **Security:** Better isolation of public endpoints + +--- + +## Appendix A: Configuration Files + +### Main App `.env` (no changes needed) + +Since both apps share the same database, **no configuration changes** are needed in the main app's `.env` file. + +The main app continues to work exactly as before. + +### Collector App `.env` + +```env +APP_NAME="Analytics Collector" +APP_ENV=production +APP_DEBUG=false + +# SHARED DATABASE - Same as main app +DB_CONNECTION=mysql +DB_HOST=main-db.local # Same database server as main app +DB_PORT=3306 +DB_DATABASE=analytics # Same database name as main app +DB_USERNAME=analytics_user +DB_PASSWORD=same_password_as_main_app + +QUEUE_CONNECTION=redis +REDIS_HOST=collector-redis.local + +COLLECTOR_RATE_LIMIT=120 +``` + +--- + +## Appendix B: nginx Configuration + +```nginx +server { + listen 80; + server_name track.yourdomain.com; + + location / { + return 301 https://$server_name$request_uri; + } +} + +server { + listen 443 ssl http2; + server_name track.yourdomain.com; + + ssl_certificate /path/to/cert.pem; + ssl_certificate_key /path/to/key.pem; + + root /var/www/analytics-collector/public; + index index.php; + + # Rate limiting + limit_req_zone $binary_remote_addr zone=collector:10m rate=120r/m; + limit_req zone=collector burst=20 nodelay; + + # Static assets (tracking script) + location ~* \.(js|gif)$ { + expires 1h; + add_header Cache-Control "public, immutable"; + } + + location / { + try_files $uri $uri/ /index.php?$query_string; + } + + location ~ \.php$ { + fastcgi_pass unix:/var/run/php/php8.2-fpm.sock; + fastcgi_index index.php; + fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; + include fastcgi_params; + } +} +``` + +--- + +## Appendix C: Supervisor Configuration (Queue Workers) + +```ini +[program:analytics-collector-worker] +process_name=%(program_name)s_%(process_num)02d +command=php /var/www/analytics-collector/artisan queue:work redis --queue=tracking --sleep=1 --tries=3 --max-jobs=1000 --max-time=3600 +autostart=true +autorestart=true +stopasgroup=true +killasgroup=true +user=www-data +numprocs=4 +redirect_stderr=true +stdout_logfile=/var/www/analytics-collector/storage/logs/worker.log +stopwaitsecs=3600 +``` + +--- + +## Timeline Estimate + +| Phase | Duration | Dependencies | Notes | +|-------|----------|--------------|-------| +| 1. Setup new app | 1 day | - | Laravel install + dependencies | +| 2. Database setup | 0.5 days | Phase 1 | Just configure `.env` - no migrations! | +| 3. Code extraction | 2-3 days | Phase 2 | Copy controllers, models, jobs, services | +| 4. Routes configuration | 1 day | Phase 3 | Set up API routes | +| 5. ~~Sync mechanism~~ | ~~Skipped~~ | - | **Not needed with shared DB!** | +| 6. ~~Data flow~~ | ~~Skipped~~ | - | **Automatic with shared DB!** | +| 7. Deployment prep | 2 days | Phase 4 | Infrastructure setup | +| 8. Monitoring setup | 1 day | Phase 7 | Metrics & alerting | +| 9. Optimizations | 2 days | Phase 7 | Caching, queue tuning | +| 10. Security review | 1 day | All above | CORS, rate limiting, validation | +| 11. Testing | 2-3 days | Phase 10 | Unit + integration + load tests | +| 12. Staged rollout | 1 week | Phase 11 | Gradual traffic migration | + +**Total estimated time: 2-3 weeks** (reduced from 3-4 weeks due to no sync mechanism needed!) + +--- + +## Success Metrics + +### Post-Launch (30 days) + +- **Uptime:** > 99.9% for collector app +- **Response time:** p95 < 100ms for collection endpoints +- **Error rate:** < 0.5% +- **Data accuracy:** 100% match vs. pre-migration baseline +- **Cost:** Within 10% of projections + +--- + +## Conclusion + +Extracting the data collection endpoints into a separate collector app provides significant benefits in terms of scalability, reliability, and maintainability. With the **shared database approach**, implementation is significantly simplified: + +### Key Benefits of Shared Database Architecture + +1. **Simplicity:** No sync mechanism needed - data is immediately available to both apps +2. **Real-time:** Zero latency between data collection and analytics availability +3. **Reliability:** No sync failures or delays to handle +4. **Cost-effective:** Save ~$50/month vs. separate database approach +5. **Faster implementation:** 2-3 weeks instead of 3-4 weeks + +### Implementation Summary + +1. **Database strategy:** ✅ Shared database (both apps connect to same DB) +2. **Sync mechanism:** ✅ Not needed - data is instantly shared +3. **Deployment:** Gradual rollout with feature flags +4. **Monitoring:** Comprehensive metrics and alerting from day 1 + +### Trade-offs to Consider + +**Pros:** +- Simplest architecture +- No data sync complexity +- Real-time data availability +- Lower infrastructure costs + +**Cons:** +- Database is shared point of failure (mitigate with replication) +- Both apps depend on same database uptime +- Database connection pool must accommodate both apps +- Less flexibility for future independent scaling of databases + +### Recommendation + +The **shared database approach is recommended** for your use case because: +- Your traffic volume doesn't require separate databases yet +- The simplicity significantly reduces complexity and development time +- You can always migrate to separate databases later if needed +- The cost savings (~$50/month) and time savings (1-2 weeks) are significant + +This plan provides a clear roadmap for the extraction while minimizing risk and maintaining data integrity throughout the migration process.