462 lines
15 KiB
Markdown
462 lines
15 KiB
Markdown
# Gutendex API Integration Plan
|
|
|
|
## Overview
|
|
This document outlines the phased implementation plan for integrating the Gutendex API (https://gutendex.com/) to fetch public domain books, process their summaries using a local Ollama API, and store them in the database.
|
|
|
|
## System Architecture
|
|
|
|
```
|
|
┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
|
|
│ Gutendex API │─────▶│ Laravel │─────▶│ Ollama API │
|
|
│ (Books Data) │ │ Queue Jobs │ │ (AI Summary) │
|
|
└─────────────────┘ └──────────────┘ └─────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────┐
|
|
│ Database │
|
|
│ (Books/Summary) │
|
|
└─────────────────┘
|
|
```
|
|
|
|
## Database Schema Considerations
|
|
|
|
### Current Schema
|
|
- **books**: title, slug, author, cover_image, publication_date, description, category_id
|
|
- **summaries**: content, key_insights, book_id
|
|
- **authors**: Separate table with many-to-many relationship
|
|
|
|
### Additional Fields Needed
|
|
We should track Gutenberg book metadata:
|
|
|
|
```sql
|
|
ALTER TABLE books ADD COLUMN gutenberg_id INTEGER UNIQUE NULLABLE;
|
|
ALTER TABLE books ADD COLUMN source VARCHAR(50) DEFAULT 'manual';
|
|
ALTER TABLE books ADD COLUMN full_text_url TEXT NULLABLE;
|
|
ALTER TABLE summaries ADD COLUMN processing_status ENUM('pending', 'processing', 'completed', 'failed') DEFAULT 'pending';
|
|
ALTER TABLE summaries ADD COLUMN processing_error TEXT NULLABLE;
|
|
ALTER TABLE summaries ADD COLUMN processed_at TIMESTAMP NULLABLE;
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 1: Foundation & Data Model
|
|
|
|
**Goal**: Set up database schema and basic configuration for Gutendex integration
|
|
|
|
### Tasks:
|
|
1. **Migration: Extend Books Table**
|
|
- Add `gutenberg_id` (integer, unique, nullable) - To track Project Gutenberg ID
|
|
- Add `source` (string, default: 'manual') - To differentiate between manually added and auto-imported books
|
|
- Add `full_text_url` (text, nullable) - Store URL to full book text for future reference
|
|
- Add `language` (string, nullable) - Book language code
|
|
|
|
2. **Migration: Extend Summaries Table**
|
|
- Add `processing_status` (enum: pending/processing/completed/failed)
|
|
- Add `processing_error` (text, nullable) - Store error messages for debugging
|
|
- Add `processed_at` (timestamp, nullable) - Track when summary was generated
|
|
- Add `model_used` (string, nullable) - Track which Ollama model was used
|
|
|
|
3. **Configuration File**
|
|
- Create `config/gutendex.php`:
|
|
- Base API URL
|
|
- Default language filter (e.g., 'en')
|
|
- Batch size for imports
|
|
- Rate limiting settings
|
|
- Create `config/ollama.php`:
|
|
- API endpoint URL
|
|
- Model name (e.g., 'llama3', 'mistral')
|
|
- Timeout settings
|
|
- Prompt templates
|
|
|
|
4. **Update Models**
|
|
- Update `Book` model with new fillable fields and casts
|
|
- Update `Summary` model with new fillable fields and casts
|
|
- Add scopes for filtering by source, status
|
|
|
|
### Deliverables:
|
|
- [ ] Two migration files created and run
|
|
- [ ] Two config files created
|
|
- [ ] Models updated with new fields
|
|
- [ ] `.env.example` updated with new config values
|
|
|
|
---
|
|
|
|
## Phase 2: Gutendex API Client
|
|
|
|
**Goal**: Build a robust service class to interact with Gutendex API
|
|
|
|
### Tasks:
|
|
1. **Service Class: `GutendexClient`**
|
|
- Location: `app/Services/GutendexClient.php`
|
|
- Methods:
|
|
- `searchBooks($query, $page = 1, $filters = [])`
|
|
- `getBook($gutenbergId)`
|
|
- `getBooksByLanguage($language = 'en', $page = 1)`
|
|
- `getBooksByCopyright($copyright = 'false', $page = 1)` - Public domain only
|
|
- `fetchFullText($textUrl)` - Download book text from provided URL
|
|
- Features:
|
|
- Use Laravel HTTP client
|
|
- Implement retry logic with exponential backoff
|
|
- Handle rate limiting (respect 429 responses)
|
|
- Log all API interactions
|
|
- Cache frequently accessed data
|
|
|
|
2. **Data Transfer Objects (DTOs)**
|
|
- Create `app/DTOs/GutendexBook.php`
|
|
- Map Gutendex API response to internal structure
|
|
- Handle author array parsing
|
|
- Extract subject/category information
|
|
- Normalize data formats
|
|
|
|
3. **Service Provider**
|
|
- Register `GutendexClient` as singleton in `AppServiceProvider`
|
|
|
|
### Deliverables:
|
|
- [ ] `GutendexClient` service class with all methods
|
|
- [ ] `GutendexBook` DTO class
|
|
- [ ] Unit tests for API client
|
|
- [ ] Error handling and logging
|
|
|
|
---
|
|
|
|
## Phase 3: Ollama Integration
|
|
|
|
**Goal**: Create service to generate summaries using local Ollama API
|
|
|
|
### Tasks:
|
|
1. **Service Class: `OllamaService`**
|
|
- Location: `app/Services/OllamaService.php`
|
|
- Methods:
|
|
- `generateSummary($bookText, $bookTitle, $bookAuthor)`
|
|
- `generateKeyInsights($bookText)`
|
|
- `checkConnection()` - Verify Ollama is accessible
|
|
- `listModels()` - Get available models
|
|
- Features:
|
|
- Configurable prompts via config file
|
|
- Handle long texts (chunking if needed)
|
|
- Timeout handling
|
|
- Token limit awareness
|
|
- Retry logic
|
|
|
|
2. **Prompt Engineering**
|
|
- Create optimized prompts in `config/ollama.php`:
|
|
```php
|
|
'prompts' => [
|
|
'summary' => 'You are a literary expert. Provide a comprehensive summary of the following book titled "{title}" by {author}. Focus on main themes, plot, and significance. Limit to 500 words.\n\nBook text:\n{text}',
|
|
'key_insights' => 'Extract 5-7 key insights or lessons from the following book summary. Format as a bullet list.\n\nSummary:\n{summary}'
|
|
]
|
|
```
|
|
|
|
3. **Text Processing Helper**
|
|
- Create `app/Helpers/TextProcessor.php`
|
|
- Methods:
|
|
- `extractExcerpt($fullText, $maxLength = 50000)` - Get representative sample
|
|
- `cleanText($text)` - Remove formatting artifacts
|
|
- `truncateToTokenLimit($text, $limit = 4000)`
|
|
|
|
### Deliverables:
|
|
- [ ] `OllamaService` class with all methods
|
|
- [ ] `TextProcessor` helper class
|
|
- [ ] Prompt templates in config
|
|
- [ ] Connection testing command
|
|
- [ ] Unit tests for text processing
|
|
|
|
---
|
|
|
|
## Phase 4: Queue Jobs & Processing Pipeline
|
|
|
|
**Goal**: Implement background jobs for fetching and processing books
|
|
|
|
### Tasks:
|
|
1. **Job: `ImportBookFromGutendex`**
|
|
- Location: `app/Jobs/ImportBookFromGutendex.php`
|
|
- Inputs: Gutenberg ID
|
|
- Process:
|
|
1. Fetch book metadata from Gutendex
|
|
2. Map authors to `authors` table
|
|
3. Determine category from subjects
|
|
4. Create/update book record
|
|
5. Dispatch `GenerateSummary` job
|
|
- Error handling: Log failures, mark as failed in database
|
|
|
|
2. **Job: `GenerateSummary`**
|
|
- Location: `app/Jobs/GenerateSummary.php`
|
|
- Inputs: Book ID
|
|
- Process:
|
|
1. Fetch full text from URL
|
|
2. Extract/clean excerpt
|
|
3. Call Ollama API for summary
|
|
4. Call Ollama API for key insights
|
|
5. Store in `summaries` table
|
|
6. Update processing_status to 'completed'
|
|
- Features:
|
|
- Retry on failure (3 attempts)
|
|
- Update status to 'processing' at start
|
|
- Handle timeouts gracefully
|
|
- Store errors in `processing_error` field
|
|
|
|
3. **Job: `BatchImportFromGutendex`**
|
|
- Location: `app/Jobs/BatchImportFromGutendex.php`
|
|
- Inputs: Search parameters (language, page range, limit)
|
|
- Process:
|
|
1. Fetch books list from Gutendex
|
|
2. Filter out already imported books
|
|
3. Dispatch individual `ImportBookFromGutendex` jobs
|
|
4. Rate limit dispatching
|
|
|
|
4. **Queue Configuration**
|
|
- Configure queue driver in `.env` (database or Redis)
|
|
- Create queues: `gutendex-import`, `summary-generation`
|
|
- Set up queue priorities
|
|
|
|
### Deliverables:
|
|
- [ ] Three job classes implemented
|
|
- [ ] Queue tables migrated (if using database driver)
|
|
- [ ] Job failure handling
|
|
- [ ] Queue monitoring setup
|
|
|
|
---
|
|
|
|
## Phase 5: Artisan Commands
|
|
|
|
**Goal**: Provide CLI interface for managing imports
|
|
|
|
### Tasks:
|
|
1. **Command: `gutendex:import`**
|
|
- Location: `app/Console/Commands/ImportFromGutendex.php`
|
|
- Signature: `gutendex:import {--language=en} {--limit=100} {--page=1} {--queue}`
|
|
- Functionality:
|
|
- Import books matching criteria
|
|
- Option to queue jobs or run immediately
|
|
- Progress bar showing import status
|
|
- Summary of imported books
|
|
|
|
2. **Command: `gutendex:sync-book`**
|
|
- Signature: `gutendex:sync-book {gutenberg_id} {--force}`
|
|
- Functionality:
|
|
- Import single book by Gutenberg ID
|
|
- `--force` flag to re-import existing book
|
|
- Show processing status
|
|
|
|
3. **Command: `summary:generate`**
|
|
- Signature: `summary:generate {book_id?} {--all} {--pending} {--failed}`
|
|
- Functionality:
|
|
- Generate summary for specific book
|
|
- `--all` to process all books without summaries
|
|
- `--pending` to retry pending summaries
|
|
- `--failed` to retry failed summaries
|
|
|
|
4. **Command: `summary:status`**
|
|
- Signature: `summary:status`
|
|
- Functionality:
|
|
- Display statistics: total books, summaries completed, pending, failed
|
|
- Show recent processing activity
|
|
- Estimate time remaining
|
|
|
|
5. **Command: `ollama:test`**
|
|
- Signature: `ollama:test`
|
|
- Functionality:
|
|
- Test connection to Ollama API
|
|
- Show available models
|
|
- Run test summary generation
|
|
|
|
### Deliverables:
|
|
- [ ] Five command classes implemented
|
|
- [ ] Commands registered in `Kernel.php`
|
|
- [ ] Help text and documentation
|
|
- [ ] Error handling for all commands
|
|
|
|
---
|
|
|
|
## Phase 6: Admin Interface (Optional)
|
|
|
|
**Goal**: Provide web interface for monitoring and managing imports
|
|
|
|
### Tasks:
|
|
1. **Controller: `GutendexController`**
|
|
- Routes:
|
|
- `GET /admin/gutendex` - Dashboard
|
|
- `POST /admin/gutendex/import` - Trigger import
|
|
- `GET /admin/gutendex/status` - Processing status
|
|
- `POST /admin/gutendex/retry/{book}` - Retry failed book
|
|
|
|
2. **Views**
|
|
- `resources/views/admin/gutendex/dashboard.blade.php`
|
|
- Statistics cards
|
|
- Recent imports table
|
|
- Import form
|
|
- `resources/views/admin/gutendex/status.blade.php`
|
|
- Processing queue status
|
|
- Failed jobs list with retry buttons
|
|
|
|
3. **Middleware**
|
|
- Add authentication/authorization
|
|
- Only admin users can access
|
|
|
|
### Deliverables:
|
|
- [ ] Controller with routes
|
|
- [ ] Blade views for dashboard
|
|
- [ ] Authentication middleware
|
|
- [ ] AJAX progress updates
|
|
|
|
---
|
|
|
|
## Phase 7: Monitoring & Optimization
|
|
|
|
**Goal**: Ensure system runs reliably and efficiently
|
|
|
|
### Tasks:
|
|
1. **Logging & Monitoring**
|
|
- Create dedicated log channel for Gutendex operations
|
|
- Log all API calls with duration
|
|
- Track success/failure rates
|
|
- Alert on high failure rates
|
|
|
|
2. **Performance Optimization**
|
|
- Add database indexes:
|
|
- `books.gutenberg_id`
|
|
- `summaries.processing_status`
|
|
- `books.source`
|
|
- Implement caching for Gutendex API responses
|
|
- Optimize text processing for memory usage
|
|
|
|
3. **Scheduled Tasks**
|
|
- Add to `app/Console/Kernel.php`:
|
|
- Daily import of new books: `schedule->command('gutendex:import --limit=50')->daily()`
|
|
- Retry failed summaries: `schedule->command('summary:generate --failed')->weekly()`
|
|
- Clean up old processing jobs
|
|
|
|
4. **Documentation**
|
|
- Update README.md with:
|
|
- Setup instructions for Ollama
|
|
- Environment variables documentation
|
|
- Command usage examples
|
|
- Create troubleshooting guide
|
|
|
|
### Deliverables:
|
|
- [ ] Logging configuration
|
|
- [ ] Database indexes added
|
|
- [ ] Scheduled tasks configured
|
|
- [ ] Documentation updated
|
|
|
|
---
|
|
|
|
## Phase 8: Testing & Quality Assurance
|
|
|
|
**Goal**: Ensure reliability and correctness
|
|
|
|
### Tasks:
|
|
1. **Unit Tests**
|
|
- Test `GutendexClient` with mocked HTTP responses
|
|
- Test `OllamaService` with mocked API
|
|
- Test `TextProcessor` helper methods
|
|
- Test DTOs and data transformation
|
|
|
|
2. **Feature Tests**
|
|
- Test complete import workflow
|
|
- Test summary generation pipeline
|
|
- Test error handling scenarios
|
|
- Test queue job processing
|
|
|
|
3. **Integration Tests**
|
|
- Test with real Gutendex API (limited)
|
|
- Test with real Ollama API (local)
|
|
- Test end-to-end workflow
|
|
|
|
4. **Error Scenarios**
|
|
- Gutendex API down/rate limited
|
|
- Ollama API timeout
|
|
- Invalid book data
|
|
- Network failures
|
|
|
|
### Deliverables:
|
|
- [ ] Unit test suite with >80% coverage
|
|
- [ ] Feature tests for all workflows
|
|
- [ ] Integration tests documented
|
|
- [ ] Error handling verified
|
|
|
|
---
|
|
|
|
## Implementation Timeline
|
|
|
|
### Week 1: Foundation
|
|
- Phase 1: Database & Configuration (Days 1-2)
|
|
- Phase 2: Gutendex Client (Days 3-5)
|
|
|
|
### Week 2: Processing
|
|
- Phase 3: Ollama Integration (Days 1-3)
|
|
- Phase 4: Queue Jobs (Days 4-5)
|
|
|
|
### Week 3: Interface & Tools
|
|
- Phase 5: Artisan Commands (Days 1-3)
|
|
- Phase 6: Admin Interface (Days 4-5) - Optional
|
|
|
|
### Week 4: Polish
|
|
- Phase 7: Monitoring & Optimization (Days 1-2)
|
|
- Phase 8: Testing & QA (Days 3-5)
|
|
|
|
---
|
|
|
|
## Environment Variables Required
|
|
|
|
```env
|
|
# Gutendex Configuration
|
|
GUTENDEX_API_URL=https://gutendex.com
|
|
GUTENDEX_DEFAULT_LANGUAGE=en
|
|
GUTENDEX_BATCH_SIZE=50
|
|
GUTENDEX_RATE_LIMIT_DELAY=1000
|
|
|
|
# Ollama Configuration
|
|
OLLAMA_API_URL=http://localhost:11434
|
|
OLLAMA_MODEL=llama3
|
|
OLLAMA_TIMEOUT=300
|
|
OLLAMA_MAX_TOKENS=4000
|
|
|
|
# Queue Configuration
|
|
QUEUE_CONNECTION=database
|
|
```
|
|
|
|
---
|
|
|
|
## Success Metrics
|
|
|
|
- Successfully import 100+ books from Gutendex
|
|
- Generate summaries with >90% success rate
|
|
- Average processing time per book < 5 minutes
|
|
- Zero data loss or corruption
|
|
- Comprehensive error logging and recovery
|
|
|
|
---
|
|
|
|
## Future Enhancements
|
|
|
|
1. **Multi-language Support**: Process books in multiple languages
|
|
2. **Advanced Categorization**: Use AI to auto-categorize books
|
|
3. **Quality Scoring**: Rate summary quality and retry low-quality ones
|
|
4. **User Requests**: Allow users to request specific Gutenberg books
|
|
5. **Incremental Updates**: Sync with Gutendex daily for new additions
|
|
6. **Summary Variations**: Generate different summary lengths (short/medium/long)
|
|
7. **Related Books**: Use AI to suggest related books based on content
|
|
|
|
---
|
|
|
|
## Risk Mitigation
|
|
|
|
| Risk | Impact | Mitigation |
|
|
|------|--------|------------|
|
|
| Ollama API slow/unavailable | High | Implement timeout, retry logic, queue system |
|
|
| Rate limiting from Gutendex | Medium | Respect rate limits, implement backoff, cache responses |
|
|
| Poor summary quality | Medium | Fine-tune prompts, validate output, human review process |
|
|
| Memory issues with large texts | Medium | Chunk processing, optimize text extraction |
|
|
| Database growth | Low | Regular cleanup, archival strategy |
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
- Start with small batch (10-20 books) to test pipeline
|
|
- Monitor Ollama performance and adjust model/prompts as needed
|
|
- Consider using a faster model for initial testing
|
|
- Keep Gutenberg IDs to avoid duplicate imports
|
|
- Implement proper error notifications for production
|