created plan for a collector microservice

2026-03-25 21:05:46 +02:00
parent b5f948433b
commit 8a5aef0b13
1 changed files with 843 additions and 0 deletions
@@ -0,0 +1,843 @@
 # Analytics Collector App Extraction Plan
 ## Overview
 This document outlines the plan for extracting the data collection endpoints from the main analytics application into a separate Laravel application. The collector app will be responsible for receiving tracking data from external websites, processing it, and storing it in the **shared database** (same database as the main app).
 ## Goals
 1. **Separation of Concerns**: Isolate high-traffic data collection endpoints from the analytics dashboard
 2. **Scalability**: Enable independent scaling of the collector service
 3. **Reliability**: Prevent tracking service disruptions from affecting dashboard availability
 4. **Performance**: Optimize collector for high-throughput, low-latency operations
 5. **Security**: Isolate public-facing endpoints from internal analytics data
 ---
 ## Architecture Overview
 ### Current State
 ```
 ┌─────────────────────────────────────────┐
 │       Single Laravel Application        │
 │  ┌─────────────┐    ┌────────────────┐ │
 │  │  Dashboard  │    │   Collection   │ │
 │  │  (Auth UI)  │    │   Endpoints    │ │
 │  └─────────────┘    └────────────────┘ │
 │            │               │            │
 │            └───────┬───────┘            │
 │                    │                    │
 │              ┌─────▼─────┐              │
 │              │  Database │              │
 │              └───────────┘              │
 └─────────────────────────────────────────┘
 ```
 ### Target State (Shared Database Architecture)
 ```
 ┌──────────────────────┐       ┌──────────────────────┐
 │   Analytics App      │       │   Collector App      │
 │   (Dashboard/Auth)   │       │   (Data Ingestion)   │
 │                      │       │                      │
 │  ┌────────────────┐  │       │  ┌────────────────┐  │
 │  │  Reports/UI    │  │       │  │   POST /api/e  │  │
 │  │  User Mgmt     │  │       │  │  GET /api/p    │  │
 │  │  Settings      │  │       │  │  Tracking JS   │  │
 │  └────────────────┘  │       │  └────────────────┘  │
 │          │           │       │          │           │
 │          └───────────┴───────┴──────────┘           │
 │                      │                              │
 │              ┌───────▼────────┐                     │
 │              │  Shared Database│                     │
 │              │  (All tables)   │                     │
 │              └─────────────────┘                     │
 └──────────────────────────────────────────────────────┘
 Both apps read/write to the same database:
 - No data sync needed
 - Real-time data availability
 - Simpler architecture
 ```
 ---
 ## Phase 1: Setup New Collector App
 ### 1.1 Create New Laravel Application
 ```bash
 composer create-project laravel/laravel analytics-collector
 cd analytics-collector
 ```
 ### 1.2 Configure Environment
 **`.env` configuration for collector app:**
 ```env
 APP_NAME="Analytics Collector"
 # IMPORTANT: Same database as main app
 DB_CONNECTION=mysql
 DB_HOST=127.0.0.1  # Same as main app
 DB_PORT=3306
 DB_DATABASE=analytics  # Same database name as main app
 DB_USERNAME=same_user_as_main_app
 DB_PASSWORD=same_password_as_main_app
 QUEUE_CONNECTION=redis
 REDIS_CLIENT=phpredis
 # Rate limiting - can be higher than main app
 COLLECTOR_RATE_LIMIT=120  # per minute per IP
 # Geolocation
 LOCATION_TESTING=false
 ```
 **Key points:**
 - Database credentials must match the main app exactly
 - Both apps will share all tables (websites, visitors, sessions, events, etc.)
 - No separate database needed
 - No data sync mechanism required
 ### 1.3 Install Dependencies
 ```bash
 composer require jaybizzle/crawler-detect
 composer require stevebauman/location
 composer require predis/predis
 ```
 ---
 ## Phase 2: Database Setup (Shared Database)
 ### 2.1 No Migrations Needed!
 Since both apps use the **same database**, there's no need to copy or create migrations in the collector app.
 **What this means:**
 - The database already has all necessary tables from the main app
 - The collector app will simply connect to the existing database
 - Both apps read/write to the same tables directly
 - No schema changes required
 ### 2.2 Skip Running Migrations
 **Do NOT run migrations on the collector app** - the tables already exist from the main app.
 Instead:
 1. Configure `.env` to point to the same database
 2. Test database connection: `php artisan db:show`
 3. Verify tables exist: `php artisan tinker` → `DB::table('websites')->count()`
 ---
 ## Phase 3: Code Extraction
 ### 3.1 Controllers to Copy
 **File: `app/Http/Controllers/CollectController.php`**
 Copy entire controller with modifications:
 - Keep all validation logic
 - Keep bot detection (`isBot()`, `isDomainValid()`)
 - Dispatch `ProcessTrackingPayload` job
 **File: `app/Http/Controllers/TrackingScriptController.php`**
 Copy entire controller:
 - Serves JavaScript tracking code
 - Update collection endpoint URLs to point to collector app
 ### 3.2 Middleware to Copy
 **File: `app/Http/Middleware/CollectCors.php`**
 Copy entire middleware for CORS handling.
 **Register in `bootstrap/app.php`:**
 ```php
 ->withMiddleware(function (Middleware $middleware) {
    $middleware->alias([
        'collect.cors' => \App\Http\Middleware\CollectCors::class,
    ]);
 })
 ```
 ### 3.3 Jobs to Copy
 **File: `app/Jobs/ProcessTrackingPayload.php`**
 Copy entire job with modifications:
 - Keep all visitor/session/pageview/event processing
 - Keep goal evaluation logic
 - Add error handling for missing websites (sync issue)
 ### 3.4 Models to Copy
 Copy the following models **exactly as they are** from the main app:
 - `app/Models/Website.php`
 - `app/Models/Visitor.php`
 - `app/Models/AnalyticsSession.php`
 - `app/Models/PageView.php`
 - `app/Models/Event.php`
 - `app/Models/Goal.php`
 - `app/Models/GoalConversion.php`
 **Important notes:**
 - Copy the models unchanged - they need to work with the same database schema
 - Keep all relationships intact
 - Keep all casts and accessors
 - The only exception: You can remove the `User` relationship from `Website.php` if the collector doesn't need it
 ### 3.5 Services to Copy
 **File: `app/Services/GoalEvaluationService.php`**
 Copy entire service for goal evaluation logic.
 ### 3.6 Views to Copy
 **File: `resources/views/tracking/tracker.blade.php`**
 Copy JavaScript tracking code with modifications:
 - Update collection endpoint URLs to collector app
 - Keep all tracking logic (visitor ID, session management, event tracking)
 ---
 ## Phase 4: Routes Configuration
 ### 4.1 API Routes
 **File: `routes/api.php`**
 ```php
 use App\Http\Controllers\CollectController;
 use App\Http\Controllers\TrackingScriptController;
 // Main collection endpoints (backward compatible)
 Route::post('/e', [CollectController::class, 'store'])
    ->middleware(['collect.cors', 'throttle:collect'])
    ->name('collect.store');
 Route::get('/p', [CollectController::class, 'pixel'])
    ->middleware(['collect.cors', 'throttle:collect'])
    ->name('collect.pixel');
 // Ad-blocker resistant endpoints
 Route::post('/cdn/feedback', [CollectController::class, 'store'])
    ->middleware(['collect.cors', 'throttle:collect'])
    ->name('collect.store.alt');
 Route::get('/cdn/status.gif', [CollectController::class, 'pixel'])
    ->middleware(['collect.cors', 'throttle:collect'])
    ->name('collect.pixel.alt');
 // Tracking script delivery
 Route::get('/t/{trackingId}.js', [TrackingScriptController::class, 'script'])
    ->middleware('collect.cors')
    ->name('tracking.script');
 Route::get('/cdn/resources/{trackingId}.js', [TrackingScriptController::class, 'script'])
    ->middleware('collect.cors')
    ->name('tracking.script.alt');
 ```
 ### 4.2 Rate Limiting Configuration
 **File: `app/Providers/AppServiceProvider.php`**
 ```php
 use Illuminate\Cache\RateLimiting\Limit;
 use Illuminate\Support\Facades\RateLimiter;
 public function boot(): void
 {
    RateLimiter::for('collect', function (Request $request) {
        return Limit::perMinute(config('collector.rate_limit', 120))
            ->by($request->ip());
    });
 }
 ```
 **Create config file: `config/collector.php`**
 ```php
 return [
    'rate_limit' => env('COLLECTOR_RATE_LIMIT', 120),
    'main_app_url' => env('MAIN_APP_URL'),
    'api_key' => env('COLLECTOR_API_KEY'),
 ];
 ```
 ---
 ## Phase 5: No Sync Mechanism Needed! 🎉
 ### 5.1 Shared Database = No Sync Required
 Since both apps use the **same database**, there's **NO need for a sync mechanism**!
 **What this means:**
 - When a website is created/updated/deleted in the main app, it's immediately available in the collector app
 - When a goal is created/updated/deleted in the main app, the collector app sees it instantly
 - When the collector creates tracking data, the main app can query it immediately
 - Zero latency between apps
 - No complex sync logic needed
 - No API endpoints for syncing
 **Benefits:**
 - Simpler codebase (no sync controllers, middleware, or API calls)
 - Real-time data consistency
 - No sync delays or failures to handle
 - Reduced network overhead
 - Fewer moving parts = fewer potential issues
 **Only consideration:**
 - Both apps must have read access to `websites` and `goals` tables
 - Both apps need write access to `visitors`, `sessions`, `page_views`, `events`, `goal_conversions` tables
 - Database connection pooling should account for connections from both apps
 ---
 ## Phase 6: Data Flow - Already Solved! ✅
 ### Shared Database Architecture
 **Data flow is automatic with shared database:**
 ```
 Collector App                    Main App
     │                              │
     │  Writes tracking data        │  Reads tracking data
     │  (visitors, sessions,        │  (for reports & analytics)
     │   page_views, events)        │
     │                              │
     └──────────┬───────────────────┘
                │
         ┌──────▼───────┐
         │   Database   │
         │  (Shared)    │
         └──────────────┘
 ```
 **How it works:**
 1. **Collector writes** → tracking data (visitors, sessions, page_views, events, conversions)
 2. **Main app reads** → same tables for reports, dashboards, analytics
 3. **Main app writes** → configuration data (websites, goals, users, settings)
 4. **Collector reads** → website/goal configs for validation and goal evaluation
 **No additional implementation needed** - this is the beauty of the shared database approach!
 ---
 ## Phase 7: Deployment Strategy
 ### 7.1 Infrastructure Setup
 **Collector App Requirements:**
 - High-availability load balancer
 - Multiple app instances for horizontal scaling
 - Redis for queue and cache
 - Database (MySQL/PostgreSQL)
 - CDN for tracking script delivery (optional)
 **Suggested Stack:**
 ```
 ┌─────────────────────────┐
 │   CDN (CloudFlare)      │
 │   /cdn/resources/*.js   │
 └────────┬────────────────┘
         │
 ┌────────▼────────────────┐
 │   Load Balancer         │
 │   (nginx/HAProxy)       │
 └────────┬────────────────┘
         │
    ┌────┴────┐
    │         │
 ┌───▼───┐ ┌──▼────┐
 │ App 1 │ │ App 2 │ ... (N instances)
 └───┬───┘ └──┬────┘
    │        │
 ┌───▼────────▼───┐
 │  Redis Queue   │
 └───┬────────────┘
    │
 ┌───▼────────────┐
 │  Queue Workers │
 └───┬────────────┘
    │
 ┌───▼────────────┐
 │  Database      │
 └────────────────┘
 ```
 ### 7.2 Migration Steps
 **Step 1: Prepare Collector App**
 1. Deploy collector app to staging
 2. Run migrations
 3. Sync initial website/goal data
 4. Test collection endpoints
 **Step 2: Gradual Rollout**
 1. Update tracking script to include feature flag
 2. Route 10% of traffic to collector app
 3. Monitor error rates, latency, data accuracy
 4. Gradually increase to 100%
 **Step 3: DNS/Routing Changes**
 1. Update tracking script URLs to collector domain
 2. Setup CNAME: `track.yourdomain.com` → collector app
 3. Update CORS configuration
 **Step 4: Main App Cleanup**
 1. Remove collection routes from main app
 2. Remove `CollectController`, `TrackingScriptController`
 3. Keep models and migrations (if shared database)
 4. Archive old code for rollback safety
 ---
 ## Phase 8: Monitoring & Observability
 ### 8.1 Metrics to Track
 **Collector App:**
 - Request rate (req/sec)
 - Response time (p50, p95, p99)
 - Error rate by type (bot, validation, domain_mismatch)
 - Queue depth and processing rate
 - Database connection pool usage
 - Bot traffic percentage
 **Dashboard:**
 ```php
 // app/Http/Controllers/HealthController.php
 public function stats(): JsonResponse
 {
    return response()->json([
        'requests_last_minute' => Cache::get('requests_last_minute', 0),
        'queue_depth' => Queue::size(),
        'db_connections' => DB::connection()->getDoctrineConnection()->getWrappedConnection()->getAttribute(PDO::ATTR_SERVER_INFO),
    ]);
 }
 ```
 ### 8.2 Alerting
 Set up alerts for:
 - Response time > 200ms (p95)
 - Error rate > 1%
 - Queue depth > 10,000
 - Database connections > 80% of pool
 - Disk space < 20%
 ---
 ## Phase 9: Performance Optimizations
 ### 9.1 Database Optimizations
 **Indexing Strategy:**
 ```sql
 -- Visitors table
 CREATE INDEX idx_visitors_lookup ON visitors(website_id, uid);
 CREATE INDEX idx_visitors_last_seen ON visitors(website_id, last_seen_at);
 -- Sessions table
 CREATE INDEX idx_sessions_lookup ON analytics_sessions(website_id, uid);
 CREATE INDEX idx_sessions_started ON analytics_sessions(website_id, started_at);
 -- Page views table
 CREATE INDEX idx_pageviews_viewed ON page_views(website_id, viewed_at);
 CREATE INDEX idx_pageviews_path ON page_views(website_id, path(255));
 -- Events table
 CREATE INDEX idx_events_occurred ON events(website_id, occurred_at);
 CREATE INDEX idx_events_name ON events(website_id, name);
 ```
 **Partitioning (for very high volume):**
 ```sql
 -- Partition page_views by month
 ALTER TABLE page_views PARTITION BY RANGE (YEAR(viewed_at) * 100 + MONTH(viewed_at)) (
    PARTITION p202603 VALUES LESS THAN (202604),
    PARTITION p202604 VALUES LESS THAN (202605),
    -- ... add future partitions
 );
 ```
 ### 9.2 Queue Optimization
 **Use Redis for queue:**
 ```env
 QUEUE_CONNECTION=redis
 REDIS_QUEUE=tracking
 ```
 **Tune worker count based on load:**
 ```bash
 php artisan queue:work redis --queue=tracking --sleep=1 --tries=3 --max-jobs=1000
 ```
 **Use Horizon for queue management:**
 ```bash
 composer require laravel/horizon
 php artisan horizon:install
 php artisan horizon
 ```
 ### 9.3 Caching Strategy
 **Cache website configs:**
 ```php
 // In ProcessTrackingPayload job
 protected function getWebsite(string $trackingId): ?Website
 {
    return Cache::remember(
        "website:{$trackingId}",
        now()->addHours(1),
        fn() => Website::where('tracking_id', $trackingId)->first()
    );
 }
 ```
 **Cache goals:**
 ```php
 protected function getActiveGoals(int $websiteId): Collection
 {
    return Cache::remember(
        "goals:{$websiteId}",
        now()->addMinutes(5),
        fn() => Goal::where('website_id', $websiteId)
            ->where('is_active', true)
            ->get()
    );
 }
 ```
 ---
 ## Phase 10: Security Considerations
 ### 10.1 DDoS Protection
 **Rate Limiting by IP:**
 - Current: 120 req/min per IP
 - Consider: Dynamic rate limiting based on traffic patterns
 - Use CloudFlare or similar CDN for DDoS protection
 **Bot Detection:**
 - Keep `CrawlerDetect` package
 - Log bot patterns for analysis
 - Consider adding honeypot endpoints
 ### 10.2 Data Validation
 **Strict input validation:**
 - Validate all tracking payload fields
 - Sanitize URLs, titles, event names
 - Limit string lengths (prevent database bloat)
 - Validate JSON structure for event_properties
 ### 10.3 API Security
 **Sync endpoint protection:**
 - Use strong API keys (32+ characters)
 - Rotate keys periodically
 - IP whitelist for sync endpoints
 - Use HTTPS only
 ---
 ## Phase 11: Testing Strategy
 ### 11.1 Unit Tests
 **Test coverage for:**
 - `CollectController` validation logic
 - `ProcessTrackingPayload` job
 - `GoalEvaluationService`
 - Bot detection
 - Domain validation
 ### 11.2 Integration Tests
 **Test scenarios:**
 - End-to-end pageview tracking
 - Event tracking and goal conversion
 - Session management
 - Website sync from main app
 - Goal sync from main app
 ### 11.3 Load Testing
 **Use tools:**
 - Apache Bench (ab)
 - wrk
 - k6
 **Test scenarios:**
 ```bash
 # Simulate 1000 req/sec for 60 seconds
 k6 run --vus 100 --duration 60s load-test.js
 ```
 **Load test script example:**
 ```javascript
 // load-test.js
 import http from 'k6/http';
 export default function () {
  const payload = JSON.stringify({
    tid: 'TA-TEST123456',
    vid: `visitor-${__VU}-${__ITER}`,
    sid: `session-${__VU}`,
    type: 'pageview',
    url: 'https://example.com/test',
    path: '/test',
  });
  http.post('http://collector.local/api/e', payload, {
    headers: { 'Content-Type': 'application/json' },
  });
 }
 ```
 ---
 ## Phase 12: Rollback Plan
 ### 12.1 Rollback Triggers
 Rollback if:
 - Error rate > 5%
 - Data loss detected (missing tracking data)
 - Performance degradation (p95 > 500ms)
 - Queue backup (depth > 50,000)
 ### 12.2 Rollback Steps
 1. **Immediate:**
   - Update tracking script to point back to main app
   - Disable collector app routing
 2. **Within 1 hour:**
   - Restore collection endpoints in main app
   - Sync any data collected in collector to main app
 3. **Post-incident:**
   - Analyze root cause
   - Fix issues
   - Re-plan rollout
 ---
 ## Phase 13: Cost Analysis
 ### 13.1 Infrastructure Costs
 **Collector App (estimated monthly costs with shared database):**
 - Load balancer: $20/month
 - App instances (2x): $50/month each = $100/month
 - ~~Database: $0 (using main app's database)~~
 - Redis: $30/month
 - Queue workers (2x): $30/month each = $60/month
 - **Total: ~$210/month**
 **Savings from reduced main app load:**
 - Potential downgrade of main app instance: -$30/month
 **Net increase: ~$180/month**
 **Database costs:**
 - No additional database server needed
 - May need to upgrade main database instance slightly (+$10-20/month) to handle additional connections from collector app
 - **Actual net increase: ~$190-200/month**
 ### 13.2 ROI Analysis
 **Benefits:**
 - **Reliability:** Fewer dashboard outages during traffic spikes
 - **Scalability:** Handle 10x traffic without main app changes
 - **Development velocity:** Deploy collector changes independently
 - **Security:** Better isolation of public endpoints
 ---
 ## Appendix A: Configuration Files
 ### Main App `.env` (no changes needed)
 Since both apps share the same database, **no configuration changes** are needed in the main app's `.env` file.
 The main app continues to work exactly as before.
 ### Collector App `.env`
 ```env
 APP_NAME="Analytics Collector"
 APP_ENV=production
 APP_DEBUG=false
 # SHARED DATABASE - Same as main app
 DB_CONNECTION=mysql
 DB_HOST=main-db.local  # Same database server as main app
 DB_PORT=3306
 DB_DATABASE=analytics  # Same database name as main app
 DB_USERNAME=analytics_user
 DB_PASSWORD=same_password_as_main_app
 QUEUE_CONNECTION=redis
 REDIS_HOST=collector-redis.local
 COLLECTOR_RATE_LIMIT=120
 ```
 ---
 ## Appendix B: nginx Configuration
 ```nginx
 server {
    listen 80;
    server_name track.yourdomain.com;
    location / {
        return 301 https://$server_name$request_uri;
    }
 }
 server {
    listen 443 ssl http2;
    server_name track.yourdomain.com;
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    root /var/www/analytics-collector/public;
    index index.php;
    # Rate limiting
    limit_req_zone $binary_remote_addr zone=collector:10m rate=120r/m;
    limit_req zone=collector burst=20 nodelay;
    # Static assets (tracking script)
    location ~* \.(js|gif)$ {
        expires 1h;
        add_header Cache-Control "public, immutable";
    }
    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }
    location ~ \.php$ {
        fastcgi_pass unix:/var/run/php/php8.2-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        include fastcgi_params;
    }
 }
 ```
 ---
 ## Appendix C: Supervisor Configuration (Queue Workers)
 ```ini
 [program:analytics-collector-worker]
 process_name=%(program_name)s_%(process_num)02d
 command=php /var/www/analytics-collector/artisan queue:work redis --queue=tracking --sleep=1 --tries=3 --max-jobs=1000 --max-time=3600
 autostart=true
 autorestart=true
 stopasgroup=true
 killasgroup=true
 user=www-data
 numprocs=4
 redirect_stderr=true
 stdout_logfile=/var/www/analytics-collector/storage/logs/worker.log
 stopwaitsecs=3600
 ```
 ---
 ## Timeline Estimate
 | Phase | Duration | Dependencies | Notes |
 |-------|----------|--------------|-------|
 | 1. Setup new app | 1 day | - | Laravel install + dependencies |
 | 2. Database setup | 0.5 days | Phase 1 | Just configure `.env` - no migrations! |
 | 3. Code extraction | 2-3 days | Phase 2 | Copy controllers, models, jobs, services |
 | 4. Routes configuration | 1 day | Phase 3 | Set up API routes |
 | 5. ~~Sync mechanism~~ | ~~Skipped~~ | - | **Not needed with shared DB!** |
 | 6. ~~Data flow~~ | ~~Skipped~~ | - | **Automatic with shared DB!** |
 | 7. Deployment prep | 2 days | Phase 4 | Infrastructure setup |
 | 8. Monitoring setup | 1 day | Phase 7 | Metrics & alerting |
 | 9. Optimizations | 2 days | Phase 7 | Caching, queue tuning |
 | 10. Security review | 1 day | All above | CORS, rate limiting, validation |
 | 11. Testing | 2-3 days | Phase 10 | Unit + integration + load tests |
 | 12. Staged rollout | 1 week | Phase 11 | Gradual traffic migration |
 **Total estimated time: 2-3 weeks** (reduced from 3-4 weeks due to no sync mechanism needed!)
 ---
 ## Success Metrics
 ### Post-Launch (30 days)
 - **Uptime:** > 99.9% for collector app
 - **Response time:** p95 < 100ms for collection endpoints
 - **Error rate:** < 0.5%
 - **Data accuracy:** 100% match vs. pre-migration baseline
 - **Cost:** Within 10% of projections
 ---
 ## Conclusion
 Extracting the data collection endpoints into a separate collector app provides significant benefits in terms of scalability, reliability, and maintainability. With the **shared database approach**, implementation is significantly simplified:
 ### Key Benefits of Shared Database Architecture
 1. **Simplicity:** No sync mechanism needed - data is immediately available to both apps
 2. **Real-time:** Zero latency between data collection and analytics availability
 3. **Reliability:** No sync failures or delays to handle
 4. **Cost-effective:** Save ~$50/month vs. separate database approach
 5. **Faster implementation:** 2-3 weeks instead of 3-4 weeks
 ### Implementation Summary
 1. **Database strategy:** ✅ Shared database (both apps connect to same DB)
 2. **Sync mechanism:** ✅ Not needed - data is instantly shared
 3. **Deployment:** Gradual rollout with feature flags
 4. **Monitoring:** Comprehensive metrics and alerting from day 1
 ### Trade-offs to Consider
 **Pros:**
 - Simplest architecture
 - No data sync complexity
 - Real-time data availability
 - Lower infrastructure costs
 **Cons:**
 - Database is shared point of failure (mitigate with replication)
 - Both apps depend on same database uptime
 - Database connection pool must accommodate both apps
 - Less flexibility for future independent scaling of databases
 ### Recommendation
 The **shared database approach is recommended** for your use case because:
 - Your traffic volume doesn't require separate databases yet
 - The simplicity significantly reduces complexity and development time
 - You can always migrate to separate databases later if needed
 - The cost savings (~$50/month) and time savings (1-2 weeks) are significant
 This plan provides a clear roadmap for the extraction while minimizing risk and maintaining data integrity throughout the migration process.