ismysiteup.online/SITEMAP_MONITORING_SETUP.md

# Sitemap Monitoring Setup

This feature allows you to monitor your website sitemaps to ensure they are accessible, valid XML, and properly formatted.

## Features Implemented

### 1. Database Migration
- **Table:** `sitemaps`
- **Columns:**
  - `id` - Primary key
  - `site_id` - Foreign key to sites table
  - `url` - Full URL to the sitemap file
  - `last_checked_at` - Last validation timestamp
  - `is_valid` - Boolean indicating if sitemap is valid
  - `last_error` - Text field storing validation errors
  - `is_active` - Enable/disable monitoring
  - `url_count` - Number of URLs found in sitemap
  - `created_at`, `updated_at` - Timestamps

### 2. Models & Relationships
- **Sitemap Model** (`app/Models/Sitemap.php`)
  - Belongs to Site
  - Helper methods:
    - `hasErrors()` - Check if sitemap has validation issues
    - `getStatusBadgeClass()` - Get Bootstrap badge color for status
    - `getStatusText()` - Get human-readable status text
    - `markAsValid()` - Mark sitemap as valid with URL count
    - `markAsInvalid()` - Mark sitemap as invalid with error message

- **Site Model** updated with `sitemaps()` relationship

### 3. Background Job
- **Job:** `CheckSitemap` (`app/Jobs/CheckSitemap.php`)
  - Fetches sitemap URL with proper user agent
  - Validates XML structure using `simplexml_load_string()`
  - Checks for valid sitemap format (urlset or sitemapindex)
  - Counts URLs in the sitemap
  - Creates notifications for invalid sitemaps
  - Comprehensive error handling and logging

### 4. XML Validation
Uses PHP's built-in `simplexml_load_string()` with error handling:
- Validates well-formed XML syntax
- Checks for proper sitemap structure
- Supports both regular sitemaps and sitemap indexes
- Handles XML namespaces correctly
- Provides detailed error messages

### 5. Artisan Command
- **Command:** `monitor:sitemaps`
  - Dispatches `CheckSitemap` job for all active sitemaps
  - Shows progress bar
  - Run manually: `./vendor/bin/sail artisan monitor:sitemaps`

### 6. Scheduled Task
- **Schedule:** Daily at midnight (configurable in `routes/console.php:68`)
- Automatically runs `monitor:sitemaps` command
- Logs output to `storage/logs/sitemap-monitoring.log`

### 7. Web Interface

#### Routes
- `GET /sitemaps` - List all sitemaps
- `GET /sitemaps/create` - Add new sitemap form
- `POST /sitemaps` - Store new sitemap
- `GET /sitemaps/{sitemap}` - View sitemap details
- `GET /sitemaps/{sitemap}/edit` - Edit sitemap form
- `PUT /sitemaps/{sitemap}` - Update sitemap
- `DELETE /sitemaps/{sitemap}` - Delete sitemap
- `POST /sitemaps/{sitemap}/check` - Manually trigger validation

#### Views
- `resources/views/sitemaps/index.blade.php` - Sitemap list with status badges
- `resources/views/sitemaps/create.blade.php` - Add new sitemap
- `resources/views/sitemaps/edit.blade.php` - Edit sitemap
- `resources/views/sitemaps/show.blade.php` - Sitemap details with validation info

## Setup Instructions

### 1. Run the Migration

If not already run:
```bash
./vendor/bin/sail artisan migrate
```

### 2. Start the Queue Worker

The sitemap validation jobs run in the background queue:
```bash
./vendor/bin/sail artisan queue:work
```

Or if you're already running the queue worker, it will automatically pick up these jobs.

### 3. Set Up Scheduled Tasks

Make sure your Laravel scheduler is running. The scheduler should already be configured if you followed the domain monitoring setup.

In production, ensure this is in your crontab:
```
* * * * * cd /path-to-your-project && php artisan schedule:run >> /dev/null 2>&1
```

In development with Sail, the scheduler runs automatically if started with `composer dev`.

### 4. Test the System

1. **Add a sitemap:**
   - Navigate to `/sitemaps`
   - Click "Add Sitemap"
   - Select a site and enter your sitemap URL
   - Save

2. **Manual validation:**
   - View the sitemap details
   - Click "Validate Now"
   - The job will queue and process in the background

3. **Test the command:**
   ```bash
   ./vendor/bin/sail artisan monitor:sitemaps
   ```

## Usage

### Adding Sitemaps
1. Go to "Sitemap Monitoring" in your application
2. Click "Add Sitemap"
3. Select the associated site
4. Enter the full sitemap URL (e.g., https://example.com/sitemap.xml)
5. Save - the system will automatically validate it

### Monitoring
- **Automatic:** Runs daily via scheduled task
- **Manual:** Click "Validate Now" on any sitemap
- **Command:** Run `./vendor/bin/sail artisan monitor:sitemaps` anytime

### Notifications
When a sitemap becomes invalid or inaccessible:
- A notification is created in the system
- Shows in your alerts/notifications section
- Contains sitemap URL and error details

### Status Badges
- **Valid** (Green) - Sitemap is accessible and well-formed
- **Invalid** (Red) - Sitemap has errors or is inaccessible
- **Not Checked** (Gray) - Validation not yet performed

## Validation Checks

The system validates the following:

### 1. HTTP Accessibility
- Sitemap URL returns HTTP 200 status
- Content can be fetched successfully
- Proper timeout handling (30 seconds)

### 2. XML Well-Formedness
- Valid XML syntax
- No malformed tags or attributes
- Proper encoding
- Detailed XML error messages if invalid

### 3. Sitemap Structure
- Contains `<urlset>` root element (for regular sitemaps)
- OR contains `<sitemapindex>` root element (for sitemap indexes)
- Proper namespace declarations
- Valid sitemap format according to sitemaps.org schema

### 4. URL Counting
- Counts `<url>` entries in regular sitemaps
- Counts `<sitemap>` entries in sitemap indexes
- Handles namespaced and non-namespaced XML

## Supported Sitemap Formats

### Regular Sitemap
```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
      <loc>https://example.com/page1</loc>
      <lastmod>2026-01-18</lastmod>
      <priority>1.0</priority>
   </url>
   <!-- More URLs -->
</urlset>
```

### Sitemap Index
```xml
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <sitemap>
      <loc>https://example.com/sitemap1.xml</loc>
      <lastmod>2026-01-18</lastmod>
   </sitemap>
   <!-- More sitemaps -->
</sitemapindex>
```

## Troubleshooting

### Jobs not processing
- Check queue worker is running: `ps aux | grep queue`
- Check failed jobs: `./vendor/bin/sail artisan queue:failed`
- Restart queue: `./vendor/bin/sail artisan queue:restart`

### Sitemap validation errors
- Verify sitemap URL is publicly accessible
- Check sitemap is valid XML (use online XML validator)
- Ensure proper XML encoding (UTF-8 recommended)
- Review logs: `storage/logs/laravel.log`

### False positive errors
- Some servers may block automated requests
- Try accessing sitemap URL manually in browser
- Check if server requires specific User-Agent headers
- Verify no firewall or security rules blocking requests

### Schedule not running
- Development: Ensure scheduler is part of your dev command
- Production: Verify cron job is set up
- Test manually: `./vendor/bin/sail artisan schedule:run`

## File Structure

```
app/
├── Console/Commands/
│   └── MonitorSitemaps.php         # Artisan command
├── Http/Controllers/
│   └── SitemapController.php       # Web controller
├── Jobs/
│   └── CheckSitemap.php            # Background job
└── Models/
    ├── Sitemap.php                 # Sitemap model
    └── Site.php                    # Updated with sitemaps relationship

database/migrations/
└── 2026_01_18_123118_create_sitemaps_table.php

resources/views/sitemaps/
├── index.blade.php                 # List view
├── create.blade.php                # Create form
├── edit.blade.php                  # Edit form
└── show.blade.php                  # Detail view

routes/
├── console.php                     # Scheduled tasks (line 68)
└── web.php                         # Web routes (lines 32-33)
```

## Common Sitemap Issues

### 1. 404 Not Found
- Sitemap file doesn't exist at specified URL
- Check URL spelling and path
- Verify sitemap is published to server

### 2. XML Parsing Errors
- Malformed XML syntax
- Missing closing tags
- Invalid characters in URLs
- Use XML validator to identify specific issues

### 3. Incorrect Namespace
- Ensure using correct namespace: `http://www.sitemaps.org/schemas/sitemap/0.9`
- Namespace must match sitemap protocol specification

### 4. Access Denied (403)
- Server blocking automated requests
- May need to whitelist monitoring server IP
- Check robots.txt isn't blocking sitemap

### 5. Timeout Errors
- Sitemap file too large (>50MB)
- Server response too slow
- Consider splitting into sitemap index

## Best Practices

1. **Keep Sitemaps Updated:** Regenerate sitemaps when content changes
2. **Optimize Size:** Large sitemaps (>50,000 URLs) should use sitemap index
3. **Compress Large Files:** Use gzip compression for large sitemaps
4. **Valid XML:** Always validate XML before publishing
5. **Monitor Regularly:** Enable active monitoring for all production sitemaps
6. **Fix Issues Promptly:** Address validation errors as soon as they appear

## Integration with Search Engines

While this tool validates sitemaps, remember to:
- Submit sitemaps to Google Search Console
- Submit sitemaps to Bing Webmaster Tools
- Update robots.txt with sitemap location
- Ping search engines after sitemap updates

## Future Enhancements

Consider adding:
- Email notifications for invalid sitemaps
- Slack/Discord webhooks for alerts
- Detailed XML validation reporting
- Automatic sitemap ping to search engines
- Historical validation tracking
- Sitemap size and compression detection
- URL validation within sitemaps
- Comparison with previous versions