some progress

sidecars/scraper/README.md

@@ -0,0 +1,82 @@
+# Adzuna Scraper
+
+A Node.js scraper for extracting job descriptions from Adzuna job listings.
+
+## Requirements
+
+- Node.js 14.21.3 or higher
+- The entire `node_modules` folder (do not run `npm install` on the server)
+
+## Installation on Server
+
+Since the server has limited disk space and write permissions, **do not run `npm install` on the server**.
+
+Instead, deploy the entire `sidecars` directory from your local machine:
+
+### Method 1: Using rsync (recommended)
+
+```bash
+# From your project root directory
+rsync -avz --progress sidecars/ user@yourserver.com:/path/to/jobifygarden.com/engine/sidecars/
+```
+
+### Method 2: Using FTP/SFTP
+
+Upload the entire `sidecars` folder including:
+- `adzuna_scraper.js`
+- `package.json`
+- `node_modules/` (entire folder)
+- `page_example.html`
+
+### Method 3: Create a tar archive
+
+```bash
+# On your local machine
+cd /path/to/jobifygarden.com
+tar -czf sidecars.tar.gz sidecars/
+
+# Upload sidecars.tar.gz to server, then on server:
+cd /path/to/jobifygarden.com/engine
+tar -xzf sidecars.tar.gz
+```
+
+## Usage
+
+```bash
+node adzuna_scraper.js <adzuna-url>
+```
+
+### Example
+
+```bash
+node adzuna_scraper.js https://www.adzuna.com/details/5694360484
+```
+
+## Output
+
+The scraper outputs JSON to stdout:
+
+```json
+{
+  "url": "https://www.adzuna.com/details/5694360484",
+  "jobDescription": {
+    "html": "...",
+    "text": "..."
+  },
+  "timestamp": "2026-04-11T12:00:00.000Z"
+}
+```
+
+## Technical Details
+
+- Uses `puppeteer-core` with `@sparticuz/chromium`
+- Compatible with Node.js 14.21.3
+- Includes self-contained Chromium binary
+- No system dependencies required (no libdrm.so.2, etc.)
+- Optimized for shared hosting environments
+
+## Troubleshooting
+
+If you get "Too Many Requests" errors, wait a few minutes before trying again (Adzuna rate limiting).
+
+If the scraper fails to find job descriptions, check that the URL is valid and the job posting still exists.