some progress

2026-01-24 12:16:10 +02:00
parent 3665a1d5a3
commit 612cc097f8
51 changed files with 8089 additions and 470 deletions
@@ -0,0 +1,309 @@
+# Sitemap Monitoring Setup
+
+This feature allows you to monitor your website sitemaps to ensure they are accessible, valid XML, and properly formatted.
+
+## Features Implemented
+
+### 1. Database Migration
+- **Table:** `sitemaps`
+- **Columns:**
+  - `id` - Primary key
+  - `site_id` - Foreign key to sites table
+  - `url` - Full URL to the sitemap file
+  - `last_checked_at` - Last validation timestamp
+  - `is_valid` - Boolean indicating if sitemap is valid
+  - `last_error` - Text field storing validation errors
+  - `is_active` - Enable/disable monitoring
+  - `url_count` - Number of URLs found in sitemap
+  - `created_at`, `updated_at` - Timestamps
+
+### 2. Models & Relationships
+- **Sitemap Model** (`app/Models/Sitemap.php`)
+  - Belongs to Site
+  - Helper methods:
+    - `hasErrors()` - Check if sitemap has validation issues
+    - `getStatusBadgeClass()` - Get Bootstrap badge color for status
+    - `getStatusText()` - Get human-readable status text
+    - `markAsValid()` - Mark sitemap as valid with URL count
+    - `markAsInvalid()` - Mark sitemap as invalid with error message
+
+- **Site Model** updated with `sitemaps()` relationship
+
+### 3. Background Job
+- **Job:** `CheckSitemap` (`app/Jobs/CheckSitemap.php`)
+  - Fetches sitemap URL with proper user agent
+  - Validates XML structure using `simplexml_load_string()`
+  - Checks for valid sitemap format (urlset or sitemapindex)
+  - Counts URLs in the sitemap
+  - Creates notifications for invalid sitemaps
+  - Comprehensive error handling and logging
+
+### 4. XML Validation
+Uses PHP's built-in `simplexml_load_string()` with error handling:
+- Validates well-formed XML syntax
+- Checks for proper sitemap structure
+- Supports both regular sitemaps and sitemap indexes
+- Handles XML namespaces correctly
+- Provides detailed error messages
+
+### 5. Artisan Command
+- **Command:** `monitor:sitemaps`
+  - Dispatches `CheckSitemap` job for all active sitemaps
+  - Shows progress bar
+  - Run manually: `./vendor/bin/sail artisan monitor:sitemaps`
+
+### 6. Scheduled Task
+- **Schedule:** Daily at midnight (configurable in `routes/console.php:68`)
+- Automatically runs `monitor:sitemaps` command
+- Logs output to `storage/logs/sitemap-monitoring.log`
+
+### 7. Web Interface
+
+#### Routes
+- `GET /sitemaps` - List all sitemaps
+- `GET /sitemaps/create` - Add new sitemap form
+- `POST /sitemaps` - Store new sitemap
+- `GET /sitemaps/{sitemap}` - View sitemap details
+- `GET /sitemaps/{sitemap}/edit` - Edit sitemap form
+- `PUT /sitemaps/{sitemap}` - Update sitemap
+- `DELETE /sitemaps/{sitemap}` - Delete sitemap
+- `POST /sitemaps/{sitemap}/check` - Manually trigger validation
+
+#### Views
+- `resources/views/sitemaps/index.blade.php` - Sitemap list with status badges
+- `resources/views/sitemaps/create.blade.php` - Add new sitemap
+- `resources/views/sitemaps/edit.blade.php` - Edit sitemap
+- `resources/views/sitemaps/show.blade.php` - Sitemap details with validation info
+
+## Setup Instructions
+
+### 1. Run the Migration
+
+If not already run:
+```bash
+./vendor/bin/sail artisan migrate
+```
+
+### 2. Start the Queue Worker
+
+The sitemap validation jobs run in the background queue:
+```bash
+./vendor/bin/sail artisan queue:work
+```
+
+Or if you're already running the queue worker, it will automatically pick up these jobs.
+
+### 3. Set Up Scheduled Tasks
+
+Make sure your Laravel scheduler is running. The scheduler should already be configured if you followed the domain monitoring setup.
+
+In production, ensure this is in your crontab:
+```
+* * * * * cd /path-to-your-project && php artisan schedule:run >> /dev/null 2>&1
+```
+
+In development with Sail, the scheduler runs automatically if started with `composer dev`.
+
+### 4. Test the System
+
+1. **Add a sitemap:**
+   - Navigate to `/sitemaps`
+   - Click "Add Sitemap"
+   - Select a site and enter your sitemap URL
+   - Save
+
+2. **Manual validation:**
+   - View the sitemap details
+   - Click "Validate Now"
+   - The job will queue and process in the background
+
+3. **Test the command:**
+   ```bash
+   ./vendor/bin/sail artisan monitor:sitemaps
+   ```
+
+## Usage
+
+### Adding Sitemaps
+1. Go to "Sitemap Monitoring" in your application
+2. Click "Add Sitemap"
+3. Select the associated site
+4. Enter the full sitemap URL (e.g., https://example.com/sitemap.xml)
+5. Save - the system will automatically validate it
+
+### Monitoring
+- **Automatic:** Runs daily via scheduled task
+- **Manual:** Click "Validate Now" on any sitemap
+- **Command:** Run `./vendor/bin/sail artisan monitor:sitemaps` anytime
+
+### Notifications
+When a sitemap becomes invalid or inaccessible:
+- A notification is created in the system
+- Shows in your alerts/notifications section
+- Contains sitemap URL and error details
+
+### Status Badges
+- **Valid** (Green) - Sitemap is accessible and well-formed
+- **Invalid** (Red) - Sitemap has errors or is inaccessible
+- **Not Checked** (Gray) - Validation not yet performed
+
+## Validation Checks
+
+The system validates the following:
+
+### 1. HTTP Accessibility
+- Sitemap URL returns HTTP 200 status
+- Content can be fetched successfully
+- Proper timeout handling (30 seconds)
+
+### 2. XML Well-Formedness
+- Valid XML syntax
+- No malformed tags or attributes
+- Proper encoding
+- Detailed XML error messages if invalid
+
+### 3. Sitemap Structure
+- Contains `<urlset>` root element (for regular sitemaps)
+- OR contains `<sitemapindex>` root element (for sitemap indexes)
+- Proper namespace declarations
+- Valid sitemap format according to sitemaps.org schema
+
+### 4. URL Counting
+- Counts `<url>` entries in regular sitemaps
+- Counts `<sitemap>` entries in sitemap indexes
+- Handles namespaced and non-namespaced XML
+
+## Supported Sitemap Formats
+
+### Regular Sitemap
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
+   <url>
+      <loc>https://example.com/page1</loc>
+      <lastmod>2026-01-18</lastmod>
+      <priority>1.0</priority>
+   </url>
+   <!-- More URLs -->
+</urlset>
+```
+
+### Sitemap Index
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
+   <sitemap>
+      <loc>https://example.com/sitemap1.xml</loc>
+      <lastmod>2026-01-18</lastmod>
+   </sitemap>
+   <!-- More sitemaps -->
+</sitemapindex>
+```
+
+## Troubleshooting
+
+### Jobs not processing
+- Check queue worker is running: `ps aux | grep queue`
+- Check failed jobs: `./vendor/bin/sail artisan queue:failed`
+- Restart queue: `./vendor/bin/sail artisan queue:restart`
+
+### Sitemap validation errors
+- Verify sitemap URL is publicly accessible
+- Check sitemap is valid XML (use online XML validator)
+- Ensure proper XML encoding (UTF-8 recommended)
+- Review logs: `storage/logs/laravel.log`
+
+### False positive errors
+- Some servers may block automated requests
+- Try accessing sitemap URL manually in browser
+- Check if server requires specific User-Agent headers
+- Verify no firewall or security rules blocking requests
+
+### Schedule not running
+- Development: Ensure scheduler is part of your dev command
+- Production: Verify cron job is set up
+- Test manually: `./vendor/bin/sail artisan schedule:run`
+
+## File Structure
+
+```
+app/
+├── Console/Commands/
+│   └── MonitorSitemaps.php         # Artisan command
+├── Http/Controllers/
+│   └── SitemapController.php       # Web controller
+├── Jobs/
+│   └── CheckSitemap.php            # Background job
+└── Models/
+    ├── Sitemap.php                 # Sitemap model
+    └── Site.php                    # Updated with sitemaps relationship
+
+database/migrations/
+└── 2026_01_18_123118_create_sitemaps_table.php
+
+resources/views/sitemaps/
+├── index.blade.php                 # List view
+├── create.blade.php                # Create form
+├── edit.blade.php                  # Edit form
+└── show.blade.php                  # Detail view
+
+routes/
+├── console.php                     # Scheduled tasks (line 68)
+└── web.php                         # Web routes (lines 32-33)
+```
+
+## Common Sitemap Issues
+
+### 1. 404 Not Found
+- Sitemap file doesn't exist at specified URL
+- Check URL spelling and path
+- Verify sitemap is published to server
+
+### 2. XML Parsing Errors
+- Malformed XML syntax
+- Missing closing tags
+- Invalid characters in URLs
+- Use XML validator to identify specific issues
+
+### 3. Incorrect Namespace
+- Ensure using correct namespace: `http://www.sitemaps.org/schemas/sitemap/0.9`
+- Namespace must match sitemap protocol specification
+
+### 4. Access Denied (403)
+- Server blocking automated requests
+- May need to whitelist monitoring server IP
+- Check robots.txt isn't blocking sitemap
+
+### 5. Timeout Errors
+- Sitemap file too large (>50MB)
+- Server response too slow
+- Consider splitting into sitemap index
+
+## Best Practices
+
+1. **Keep Sitemaps Updated:** Regenerate sitemaps when content changes
+2. **Optimize Size:** Large sitemaps (>50,000 URLs) should use sitemap index
+3. **Compress Large Files:** Use gzip compression for large sitemaps
+4. **Valid XML:** Always validate XML before publishing
+5. **Monitor Regularly:** Enable active monitoring for all production sitemaps
+6. **Fix Issues Promptly:** Address validation errors as soon as they appear
+
+## Integration with Search Engines
+
+While this tool validates sitemaps, remember to:
+- Submit sitemaps to Google Search Console
+- Submit sitemaps to Bing Webmaster Tools
+- Update robots.txt with sitemap location
+- Ping search engines after sitemap updates
+
+## Future Enhancements
+
+Consider adding:
+- Email notifications for invalid sitemaps
+- Slack/Discord webhooks for alerts
+- Detailed XML validation reporting
+- Automatic sitemap ping to search engines
+- Historical validation tracking
+- Sitemap size and compression detection
+- URL validation within sitemaps
+- Comparison with previous versions