jobifygarden/SCHEDULED_TASKS.md

Scheduled Tasks Guide for JobifyGarden

This document explains the optimal scheduling intervals for all automated tasks in JobifyGarden and the reasoning behind them.

Overview

All scheduled tasks are configured in bootstrap/app.php and run automatically via the Laravel scheduler (cron job running every minute).

Scheduled Tasks

1. Job Fetching (jobs:fetch)

Interval: Every 2 hours Command: php artisan jobs:fetch --limit=50

Why this interval:

• Freshness: Most job boards update their listings every few hours
• API Rate Limits: Spacing out requests prevents hitting rate limits
• Resource Management: Prevents overwhelming your queue and database
• User Experience: 2-hour intervals ensure users see fresh content without excessive API calls

Customization Options:

// More frequent (hourly) - for high-traffic sites
->hourly()

// Less frequent (every 4 hours) - for lower traffic or to save resources
->everyFourHours()

// Peak hours only (every 2 hours from 6 AM to 6 PM)
->everyTwoHours()->between('6:00', '18:00')

Limit Parameter:

• --limit=50 fetches up to 50 jobs per source
• Adjust based on your API quotas and database capacity
• Recommended: 20-100 jobs per fetch cycle

---

2. Job Enrichment (jobs:enrich)

Interval: Every 6 hours Command: php artisan jobs:enrich

Why this interval:

• Processing Time: Enrichment can be resource-intensive (AI/ML for categorization)
• Batch Efficiency: Allows multiple fetch cycles to accumulate jobs before enriching
• Priority: Less urgent than fetching; can run in background
• Resource Balance: Doesn't compete with peak fetch times

What it does:

• Auto-categorizes jobs based on title/description
• Detects experience levels (entry, mid, senior, etc.)
• Normalizes job data
• Enhances job metadata

Customization Options:

// More frequent (every 4 hours) - if categorization is critical
->everyFourHours()

// Less frequent (twice daily) - to save resources
->twiceDaily(1, 13) // 1 AM and 1 PM

// Only during off-peak hours
->dailyAt('03:00')

---

3. Job Deduplication (jobs:deduplicate)

Interval: Daily at 3:00 AM Command: php artisan jobs:deduplicate

Why this interval:

• Daily Cleanup: Removes duplicates that accumulate from multiple sources
• Off-Peak Hours: Runs at 3 AM when traffic is lowest
• Post-Fetch: Runs after fetch cycles have completed
• Database Health: Keeps database clean without running too frequently

What it does:

• Identifies duplicate jobs based on title, company, and location
• Removes or merges duplicate entries
• Updates job fingerprints
• Maintains data integrity

Customization Options:

// More frequent (every 12 hours) - if duplicates are a major issue
->twiceDaily(3, 15) // 3 AM and 3 PM

// Less frequent (weekly) - if duplicates are rare
->weeklyOn(0, '03:00') // Sunday at 3 AM

// After major fetch cycles
->dailyAt('04:00') // After enrichment completes

---

4. Job Cleanup (jobs:clean)

Interval: Daily at 2:00 AM Command: php artisan jobs:clean --days=45

Why this interval:

• Daily Maintenance: Removes expired jobs every day
• Off-Peak Hours: 2 AM has minimal user activity
• Before Deduplication: Runs before deduplication to remove stale data first
• Database Performance: Prevents database bloat

What it does:

• Removes jobs older than 45 days
• Deletes expired job listings
• Cleans up soft-deleted jobs
• Optimizes database performance

Retention Period Options:

// Shorter retention (30 days) - for high-volume sites
'jobs:clean --days=30'

// Longer retention (60-90 days) - for SEO or archival
'jobs:clean --days=60'

// Remove only expired jobs
'jobs:clean --expired-only'

Customization Options:

// Weekly cleanup instead of daily
->weeklyOn(0, '02:00')

// Twice daily (aggressive cleanup)
->twiceDaily(2, 14)

// Monthly cleanup (for archival sites)
->monthlyOn(1, '02:00')

---

5. Cache Pruning (cache:prune-stale-tags)

Interval: Hourly Command: php artisan cache:prune-stale-tags

Why this interval:

• Cache Health: Removes stale cache entries regularly
• Memory Management: Prevents Redis/cache bloat
• Frequent Cleanup: Cache tags can accumulate quickly
• Low Impact: Fast operation, doesn't affect performance

---

Recommended Schedules by Site Size

Small Site (<1,000 jobs, <1,000 visitors/day)

$schedule->command('jobs:fetch --limit=30')
    ->everyFourHours()
    ->withoutOverlapping();

$schedule->command('jobs:enrich')
    ->twiceDaily(2, 14);

$schedule->command('jobs:deduplicate')
    ->weeklyOn(0, '03:00');

$schedule->command('jobs:clean --days=60')
    ->weekly();

Medium Site (1,000-10,000 jobs, 1,000-10,000 visitors/day)

// Current default configuration in bootstrap/app.php
$schedule->command('jobs:fetch --limit=50')
    ->everyTwoHours()
    ->withoutOverlapping();

$schedule->command('jobs:enrich')
    ->everySixHours();

$schedule->command('jobs:deduplicate')
    ->dailyAt('03:00');

$schedule->command('jobs:clean --days=45')
    ->dailyAt('02:00');

Large Site (10,000+ jobs, 10,000+ visitors/day)

$schedule->command('jobs:fetch --limit=100')
    ->hourly()
    ->withoutOverlapping()
    ->onOneServer()
    ->runInBackground();

$schedule->command('jobs:enrich')
    ->everyFourHours()
    ->runInBackground();

$schedule->command('jobs:deduplicate')
    ->twiceDaily(3, 15)
    ->runInBackground();

$schedule->command('jobs:clean --days=30')
    ->dailyAt('02:00')
    ->runInBackground();

// Add sitemap regeneration
$schedule->command('sitemap:generate')
    ->dailyAt('04:00')
    ->runInBackground();

---

Best Practices

1. Monitoring

Always monitor your scheduled tasks:

# View scheduled tasks
php artisan schedule:list

# Test schedule (simulates next minute)
php artisan schedule:test

# Run schedule manually
php artisan schedule:run

# Check logs
tail -f storage/logs/laravel.log | grep -i "schedule\|fetch\|enrich\|deduplicate\|clean"

2. Resource Management

• Use ->runInBackground() for heavy tasks to prevent blocking
• Use ->withoutOverlapping() to prevent concurrent runs
• Use ->onOneServer() if running multiple app servers

3. Error Handling

Add notifications for failures:

$schedule->command('jobs:fetch')
    ->everyTwoHours()
    ->onFailure(function () {
        // Log error, send notification, etc.
        Log::error('Job fetch failed!');
    });

4. Performance Optimization

// Run during off-peak hours only
$schedule->command('jobs:fetch')
    ->everyTwoHours()
    ->between('6:00', '22:00'); // Only 6 AM to 10 PM

// Avoid peak hours
$schedule->command('jobs:deduplicate')
    ->dailyAt('03:00')
    ->unlessBetween('08:00', '18:00'); // Not during business hours

---

Task Dependencies

The order matters! Run tasks in this sequence:

1. jobs:fetch (2 AM, 4 AM, 6 AM, ...)
   ↓
2. jobs:clean (2 AM) - Remove old/expired first
   ↓
3. jobs:deduplicate (3 AM) - Then remove duplicates
   ↓
4. jobs:enrich (6 AM, 12 PM, 6 PM, 12 AM) - Finally enrich

This ensures:

• Old jobs are removed before deduplication
• Deduplication happens before enrichment
• Enrichment works on clean, deduplicated data

---

Troubleshooting

Jobs not being fetched?

1. Check if scheduler is running:

   php artisan schedule:list
   

2. Verify cron job is active:

   crontab -l
   

3. Test manual fetch:

   php artisan jobs:fetch --limit=5
   

4. Check API credentials in .env

Duplicates still appearing?

1. Run deduplication manually:

   php artisan jobs:deduplicate --force
   

2. Check fingerprint generation logic

3. Consider running deduplication more frequently

Database growing too large?

1. Reduce retention period:

   'jobs:clean --days=30'
   

2. Run cleanup more frequently:

   ->twiceDaily(2, 14)
   

3. Check for failed deletions:

   php artisan jobs:clean --verbose
   

---

Summary

Optimal Schedule (Default):

• Fetch: Every 2 hours
• Enrich: Every 6 hours
• Deduplicate: Daily at 3 AM
• Clean: Daily at 2 AM (45-day retention)

This provides a good balance of: ✅ Fresh content ✅ Resource efficiency ✅ Database health ✅ User experience ✅ API compliance

Adjust based on your specific needs, traffic patterns, and resource constraints.