small fixes

2026-04-04 18:52:19 +03:00
parent 7c09846bd6
commit feb964f3f6
16 changed files with 717 additions and 14 deletions
@@ -0,0 +1,367 @@
+# Scheduled Tasks Guide for JobifyGarden
+
+This document explains the optimal scheduling intervals for all automated tasks in JobifyGarden and the reasoning behind them.
+
+## Overview
+
+All scheduled tasks are configured in `bootstrap/app.php` and run automatically via the Laravel scheduler (cron job running every minute).
+
+## Scheduled Tasks
+
+### 1. Job Fetching (`jobs:fetch`)
+
+**Interval:** Every 2 hours
+**Command:** `php artisan jobs:fetch --limit=50`
+
+**Why this interval:**
+- **Freshness:** Most job boards update their listings every few hours
+- **API Rate Limits:** Spacing out requests prevents hitting rate limits
+- **Resource Management:** Prevents overwhelming your queue and database
+- **User Experience:** 2-hour intervals ensure users see fresh content without excessive API calls
+
+**Customization Options:**
+```php
+// More frequent (hourly) - for high-traffic sites
+->hourly()
+
+// Less frequent (every 4 hours) - for lower traffic or to save resources
+->everyFourHours()
+
+// Peak hours only (every 2 hours from 6 AM to 6 PM)
+->everyTwoHours()->between('6:00', '18:00')
+```
+
+**Limit Parameter:**
+- `--limit=50` fetches up to 50 jobs per source
+- Adjust based on your API quotas and database capacity
+- Recommended: 20-100 jobs per fetch cycle
+
+---
+
+### 2. Job Enrichment (`jobs:enrich`)
+
+**Interval:** Every 6 hours
+**Command:** `php artisan jobs:enrich`
+
+**Why this interval:**
+- **Processing Time:** Enrichment can be resource-intensive (AI/ML for categorization)
+- **Batch Efficiency:** Allows multiple fetch cycles to accumulate jobs before enriching
+- **Priority:** Less urgent than fetching; can run in background
+- **Resource Balance:** Doesn't compete with peak fetch times
+
+**What it does:**
+- Auto-categorizes jobs based on title/description
+- Detects experience levels (entry, mid, senior, etc.)
+- Normalizes job data
+- Enhances job metadata
+
+**Customization Options:**
+```php
+// More frequent (every 4 hours) - if categorization is critical
+->everyFourHours()
+
+// Less frequent (twice daily) - to save resources
+->twiceDaily(1, 13) // 1 AM and 1 PM
+
+// Only during off-peak hours
+->dailyAt('03:00')
+```
+
+---
+
+### 3. Job Deduplication (`jobs:deduplicate`)
+
+**Interval:** Daily at 3:00 AM
+**Command:** `php artisan jobs:deduplicate`
+
+**Why this interval:**
+- **Daily Cleanup:** Removes duplicates that accumulate from multiple sources
+- **Off-Peak Hours:** Runs at 3 AM when traffic is lowest
+- **Post-Fetch:** Runs after fetch cycles have completed
+- **Database Health:** Keeps database clean without running too frequently
+
+**What it does:**
+- Identifies duplicate jobs based on title, company, and location
+- Removes or merges duplicate entries
+- Updates job fingerprints
+- Maintains data integrity
+
+**Customization Options:**
+```php
+// More frequent (every 12 hours) - if duplicates are a major issue
+->twiceDaily(3, 15) // 3 AM and 3 PM
+
+// Less frequent (weekly) - if duplicates are rare
+->weeklyOn(0, '03:00') // Sunday at 3 AM
+
+// After major fetch cycles
+->dailyAt('04:00') // After enrichment completes
+```
+
+---
+
+### 4. Job Cleanup (`jobs:clean`)
+
+**Interval:** Daily at 2:00 AM
+**Command:** `php artisan jobs:clean --days=45`
+
+**Why this interval:**
+- **Daily Maintenance:** Removes expired jobs every day
+- **Off-Peak Hours:** 2 AM has minimal user activity
+- **Before Deduplication:** Runs before deduplication to remove stale data first
+- **Database Performance:** Prevents database bloat
+
+**What it does:**
+- Removes jobs older than 45 days
+- Deletes expired job listings
+- Cleans up soft-deleted jobs
+- Optimizes database performance
+
+**Retention Period Options:**
+```php
+// Shorter retention (30 days) - for high-volume sites
+'jobs:clean --days=30'
+
+// Longer retention (60-90 days) - for SEO or archival
+'jobs:clean --days=60'
+
+// Remove only expired jobs
+'jobs:clean --expired-only'
+```
+
+**Customization Options:**
+```php
+// Weekly cleanup instead of daily
+->weeklyOn(0, '02:00')
+
+// Twice daily (aggressive cleanup)
+->twiceDaily(2, 14)
+
+// Monthly cleanup (for archival sites)
+->monthlyOn(1, '02:00')
+```
+
+---
+
+### 5. Cache Pruning (`cache:prune-stale-tags`)
+
+**Interval:** Hourly
+**Command:** `php artisan cache:prune-stale-tags`
+
+**Why this interval:**
+- **Cache Health:** Removes stale cache entries regularly
+- **Memory Management:** Prevents Redis/cache bloat
+- **Frequent Cleanup:** Cache tags can accumulate quickly
+- **Low Impact:** Fast operation, doesn't affect performance
+
+---
+
+## Recommended Schedules by Site Size
+
+### Small Site (<1,000 jobs, <1,000 visitors/day)
+
+```php
+$schedule->command('jobs:fetch --limit=30')
+    ->everyFourHours()
+    ->withoutOverlapping();
+
+$schedule->command('jobs:enrich')
+    ->twiceDaily(2, 14);
+
+$schedule->command('jobs:deduplicate')
+    ->weeklyOn(0, '03:00');
+
+$schedule->command('jobs:clean --days=60')
+    ->weekly();
+```
+
+### Medium Site (1,000-10,000 jobs, 1,000-10,000 visitors/day)
+
+```php
+// Current default configuration in bootstrap/app.php
+$schedule->command('jobs:fetch --limit=50')
+    ->everyTwoHours()
+    ->withoutOverlapping();
+
+$schedule->command('jobs:enrich')
+    ->everySixHours();
+
+$schedule->command('jobs:deduplicate')
+    ->dailyAt('03:00');
+
+$schedule->command('jobs:clean --days=45')
+    ->dailyAt('02:00');
+```
+
+### Large Site (10,000+ jobs, 10,000+ visitors/day)
+
+```php
+$schedule->command('jobs:fetch --limit=100')
+    ->hourly()
+    ->withoutOverlapping()
+    ->onOneServer()
+    ->runInBackground();
+
+$schedule->command('jobs:enrich')
+    ->everyFourHours()
+    ->runInBackground();
+
+$schedule->command('jobs:deduplicate')
+    ->twiceDaily(3, 15)
+    ->runInBackground();
+
+$schedule->command('jobs:clean --days=30')
+    ->dailyAt('02:00')
+    ->runInBackground();
+
+// Add sitemap regeneration
+$schedule->command('sitemap:generate')
+    ->dailyAt('04:00')
+    ->runInBackground();
+```
+
+---
+
+## Best Practices
+
+### 1. Monitoring
+
+Always monitor your scheduled tasks:
+
+```bash
+# View scheduled tasks
+php artisan schedule:list
+
+# Test schedule (simulates next minute)
+php artisan schedule:test
+
+# Run schedule manually
+php artisan schedule:run
+
+# Check logs
+tail -f storage/logs/laravel.log | grep -i "schedule\|fetch\|enrich\|deduplicate\|clean"
+```
+
+### 2. Resource Management
+
+- Use `->runInBackground()` for heavy tasks to prevent blocking
+- Use `->withoutOverlapping()` to prevent concurrent runs
+- Use `->onOneServer()` if running multiple app servers
+
+### 3. Error Handling
+
+Add notifications for failures:
+
+```php
+$schedule->command('jobs:fetch')
+    ->everyTwoHours()
+    ->onFailure(function () {
+        // Log error, send notification, etc.
+        Log::error('Job fetch failed!');
+    });
+```
+
+### 4. Performance Optimization
+
+```php
+// Run during off-peak hours only
+$schedule->command('jobs:fetch')
+    ->everyTwoHours()
+    ->between('6:00', '22:00'); // Only 6 AM to 10 PM
+
+// Avoid peak hours
+$schedule->command('jobs:deduplicate')
+    ->dailyAt('03:00')
+    ->unlessBetween('08:00', '18:00'); // Not during business hours
+```
+
+---
+
+## Task Dependencies
+
+The order matters! Run tasks in this sequence:
+
+```
+1. jobs:fetch (2 AM, 4 AM, 6 AM, ...)
+   ↓
+2. jobs:clean (2 AM) - Remove old/expired first
+   ↓
+3. jobs:deduplicate (3 AM) - Then remove duplicates
+   ↓
+4. jobs:enrich (6 AM, 12 PM, 6 PM, 12 AM) - Finally enrich
+```
+
+This ensures:
+- Old jobs are removed before deduplication
+- Deduplication happens before enrichment
+- Enrichment works on clean, deduplicated data
+
+---
+
+## Troubleshooting
+
+### Jobs not being fetched?
+
+1. Check if scheduler is running:
+   ```bash
+   php artisan schedule:list
+   ```
+
+2. Verify cron job is active:
+   ```bash
+   crontab -l
+   ```
+
+3. Test manual fetch:
+   ```bash
+   php artisan jobs:fetch --limit=5
+   ```
+
+4. Check API credentials in `.env`
+
+### Duplicates still appearing?
+
+1. Run deduplication manually:
+   ```bash
+   php artisan jobs:deduplicate --force
+   ```
+
+2. Check fingerprint generation logic
+
+3. Consider running deduplication more frequently
+
+### Database growing too large?
+
+1. Reduce retention period:
+   ```php
+   'jobs:clean --days=30'
+   ```
+
+2. Run cleanup more frequently:
+   ```php
+   ->twiceDaily(2, 14)
+   ```
+
+3. Check for failed deletions:
+   ```bash
+   php artisan jobs:clean --verbose
+   ```
+
+---
+
+## Summary
+
+**Optimal Schedule (Default):**
+- **Fetch:** Every 2 hours
+- **Enrich:** Every 6 hours
+- **Deduplicate:** Daily at 3 AM
+- **Clean:** Daily at 2 AM (45-day retention)
+
+This provides a good balance of:
+✅ Fresh content
+✅ Resource efficiency
+✅ Database health
+✅ User experience
+✅ API compliance
+
+Adjust based on your specific needs, traffic patterns, and resource constraints.