jobifygarden/SCHEDULED_TASKS.md

# Scheduled Tasks Guide for JobifyGarden

This document explains the optimal scheduling intervals for all automated tasks in JobifyGarden and the reasoning behind them.

## Overview

All scheduled tasks are configured in `bootstrap/app.php` and run automatically via the Laravel scheduler (cron job running every minute).

## Scheduled Tasks

### 1. Job Fetching (`jobs:fetch`)

**Interval:** Every 2 hours
**Command:** `php artisan jobs:fetch --limit=50`

**Why this interval:**
- **Freshness:** Most job boards update their listings every few hours
- **API Rate Limits:** Spacing out requests prevents hitting rate limits
- **Resource Management:** Prevents overwhelming your queue and database
- **User Experience:** 2-hour intervals ensure users see fresh content without excessive API calls

**Customization Options:**
```php
// More frequent (hourly) - for high-traffic sites
->hourly()

// Less frequent (every 4 hours) - for lower traffic or to save resources
->everyFourHours()

// Peak hours only (every 2 hours from 6 AM to 6 PM)
->everyTwoHours()->between('6:00', '18:00')
```

**Limit Parameter:**
- `--limit=50` fetches up to 50 jobs per source
- Adjust based on your API quotas and database capacity
- Recommended: 20-100 jobs per fetch cycle

---

### 2. Job Enrichment (`jobs:enrich`)

**Interval:** Every 6 hours
**Command:** `php artisan jobs:enrich`

**Why this interval:**
- **Processing Time:** Enrichment can be resource-intensive (AI/ML for categorization)
- **Batch Efficiency:** Allows multiple fetch cycles to accumulate jobs before enriching
- **Priority:** Less urgent than fetching; can run in background
- **Resource Balance:** Doesn't compete with peak fetch times

**What it does:**
- Auto-categorizes jobs based on title/description
- Detects experience levels (entry, mid, senior, etc.)
- Normalizes job data
- Enhances job metadata

**Customization Options:**
```php
// More frequent (every 4 hours) - if categorization is critical
->everyFourHours()

// Less frequent (twice daily) - to save resources
->twiceDaily(1, 13) // 1 AM and 1 PM

// Only during off-peak hours
->dailyAt('03:00')
```

---

### 3. Job Deduplication (`jobs:deduplicate`)

**Interval:** Daily at 3:00 AM
**Command:** `php artisan jobs:deduplicate`

**Why this interval:**
- **Daily Cleanup:** Removes duplicates that accumulate from multiple sources
- **Off-Peak Hours:** Runs at 3 AM when traffic is lowest
- **Post-Fetch:** Runs after fetch cycles have completed
- **Database Health:** Keeps database clean without running too frequently

**What it does:**
- Identifies duplicate jobs based on title, company, and location
- Removes or merges duplicate entries
- Updates job fingerprints
- Maintains data integrity

**Customization Options:**
```php
// More frequent (every 12 hours) - if duplicates are a major issue
->twiceDaily(3, 15) // 3 AM and 3 PM

// Less frequent (weekly) - if duplicates are rare
->weeklyOn(0, '03:00') // Sunday at 3 AM

// After major fetch cycles
->dailyAt('04:00') // After enrichment completes
```

---

### 4. Job Cleanup (`jobs:clean`)

**Interval:** Daily at 2:00 AM
**Command:** `php artisan jobs:clean --days=45`

**Why this interval:**
- **Daily Maintenance:** Removes expired jobs every day
- **Off-Peak Hours:** 2 AM has minimal user activity
- **Before Deduplication:** Runs before deduplication to remove stale data first
- **Database Performance:** Prevents database bloat

**What it does:**
- Removes jobs older than 45 days
- Deletes expired job listings
- Cleans up soft-deleted jobs
- Optimizes database performance

**Retention Period Options:**
```php
// Shorter retention (30 days) - for high-volume sites
'jobs:clean --days=30'

// Longer retention (60-90 days) - for SEO or archival
'jobs:clean --days=60'

// Remove only expired jobs
'jobs:clean --expired-only'
```

**Customization Options:**
```php
// Weekly cleanup instead of daily
->weeklyOn(0, '02:00')

// Twice daily (aggressive cleanup)
->twiceDaily(2, 14)

// Monthly cleanup (for archival sites)
->monthlyOn(1, '02:00')
```

---

### 5. Cache Pruning (`cache:prune-stale-tags`)

**Interval:** Hourly
**Command:** `php artisan cache:prune-stale-tags`

**Why this interval:**
- **Cache Health:** Removes stale cache entries regularly
- **Memory Management:** Prevents Redis/cache bloat
- **Frequent Cleanup:** Cache tags can accumulate quickly
- **Low Impact:** Fast operation, doesn't affect performance

---

## Recommended Schedules by Site Size

### Small Site (<1,000 jobs, <1,000 visitors/day)

```php
$schedule->command('jobs:fetch --limit=30')
    ->everyFourHours()
    ->withoutOverlapping();

$schedule->command('jobs:enrich')
    ->twiceDaily(2, 14);

$schedule->command('jobs:deduplicate')
    ->weeklyOn(0, '03:00');

$schedule->command('jobs:clean --days=60')
    ->weekly();
```

### Medium Site (1,000-10,000 jobs, 1,000-10,000 visitors/day)

```php
// Current default configuration in bootstrap/app.php
$schedule->command('jobs:fetch --limit=50')
    ->everyTwoHours()
    ->withoutOverlapping();

$schedule->command('jobs:enrich')
    ->everySixHours();

$schedule->command('jobs:deduplicate')
    ->dailyAt('03:00');

$schedule->command('jobs:clean --days=45')
    ->dailyAt('02:00');
```

### Large Site (10,000+ jobs, 10,000+ visitors/day)

```php
$schedule->command('jobs:fetch --limit=100')
    ->hourly()
    ->withoutOverlapping()
    ->onOneServer()
    ->runInBackground();

$schedule->command('jobs:enrich')
    ->everyFourHours()
    ->runInBackground();

$schedule->command('jobs:deduplicate')
    ->twiceDaily(3, 15)
    ->runInBackground();

$schedule->command('jobs:clean --days=30')
    ->dailyAt('02:00')
    ->runInBackground();

// Add sitemap regeneration
$schedule->command('sitemap:generate')
    ->dailyAt('04:00')
    ->runInBackground();
```

---

## Best Practices

### 1. Monitoring

Always monitor your scheduled tasks:

```bash
# View scheduled tasks
php artisan schedule:list

# Test schedule (simulates next minute)
php artisan schedule:test

# Run schedule manually
php artisan schedule:run

# Check logs
tail -f storage/logs/laravel.log | grep -i "schedule\|fetch\|enrich\|deduplicate\|clean"
```

### 2. Resource Management

- Use `->runInBackground()` for heavy tasks to prevent blocking
- Use `->withoutOverlapping()` to prevent concurrent runs
- Use `->onOneServer()` if running multiple app servers

### 3. Error Handling

Add notifications for failures:

```php
$schedule->command('jobs:fetch')
    ->everyTwoHours()
    ->onFailure(function () {
        // Log error, send notification, etc.
        Log::error('Job fetch failed!');
    });
```

### 4. Performance Optimization

```php
// Run during off-peak hours only
$schedule->command('jobs:fetch')
    ->everyTwoHours()
    ->between('6:00', '22:00'); // Only 6 AM to 10 PM

// Avoid peak hours
$schedule->command('jobs:deduplicate')
    ->dailyAt('03:00')
    ->unlessBetween('08:00', '18:00'); // Not during business hours
```

---

## Task Dependencies

The order matters! Run tasks in this sequence:

```
1. jobs:fetch (2 AM, 4 AM, 6 AM, ...)
   ↓
2. jobs:clean (2 AM) - Remove old/expired first
   ↓
3. jobs:deduplicate (3 AM) - Then remove duplicates
   ↓
4. jobs:enrich (6 AM, 12 PM, 6 PM, 12 AM) - Finally enrich
```

This ensures:
- Old jobs are removed before deduplication
- Deduplication happens before enrichment
- Enrichment works on clean, deduplicated data

---

## Troubleshooting

### Jobs not being fetched?

1. Check if scheduler is running:
   ```bash
   php artisan schedule:list
   ```

2. Verify cron job is active:
   ```bash
   crontab -l
   ```

3. Test manual fetch:
   ```bash
   php artisan jobs:fetch --limit=5
   ```

4. Check API credentials in `.env`

### Duplicates still appearing?

1. Run deduplication manually:
   ```bash
   php artisan jobs:deduplicate --force
   ```

2. Check fingerprint generation logic

3. Consider running deduplication more frequently

### Database growing too large?

1. Reduce retention period:
   ```php
   'jobs:clean --days=30'
   ```

2. Run cleanup more frequently:
   ```php
   ->twiceDaily(2, 14)
   ```

3. Check for failed deletions:
   ```bash
   php artisan jobs:clean --verbose
   ```

---

## Summary

**Optimal Schedule (Default):**
- **Fetch:** Every 2 hours
- **Enrich:** Every 6 hours
- **Deduplicate:** Daily at 3 AM
- **Clean:** Daily at 2 AM (45-day retention)

This provides a good balance of:
✅ Fresh content
✅ Resource efficiency
✅ Database health
✅ User experience
✅ API compliance

Adjust based on your specific needs, traffic patterns, and resource constraints.