82 lines
1.8 KiB
Markdown
82 lines
1.8 KiB
Markdown
|
# Adzuna Scraper
|
||
|
|
|
||
|
|
A Node.js scraper for extracting job descriptions from Adzuna job listings.
|
||
|
|
|
||
|
|
## Requirements
|
||
|
|
|
||
|
|
- Node.js 14.21.3 or higher
|
||
|
|
- The entire `node_modules` folder (do not run `npm install` on the server)
|
||
|
|
|
||
|
|
## Installation on Server
|
||
|
|
|
||
|
|
Since the server has limited disk space and write permissions, **do not run `npm install` on the server**.
|
||
|
|
|
||
|
|
Instead, deploy the entire `sidecars` directory from your local machine:
|
||
|
|
|
||
|
|
### Method 1: Using rsync (recommended)
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# From your project root directory
|
||
|
|
rsync -avz --progress sidecars/ user@yourserver.com:/path/to/jobifygarden.com/engine/sidecars/
|
||
|
|
```
|
||
|
|
|
||
|
|
### Method 2: Using FTP/SFTP
|
||
|
|
|
||
|
|
Upload the entire `sidecars` folder including:
|
||
|
|
- `adzuna_scraper.js`
|
||
|
|
- `package.json`
|
||
|
|
- `node_modules/` (entire folder)
|
||
|
|
- `page_example.html`
|
||
|
|
|
||
|
|
### Method 3: Create a tar archive
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# On your local machine
|
||
|
|
cd /path/to/jobifygarden.com
|
||
|
|
tar -czf sidecars.tar.gz sidecars/
|
||
|
|
|
||
|
|
# Upload sidecars.tar.gz to server, then on server:
|
||
|
|
cd /path/to/jobifygarden.com/engine
|
||
|
|
tar -xzf sidecars.tar.gz
|
||
|
|
```
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```bash
|
||
|
|
node adzuna_scraper.js <adzuna-url>
|
||
|
|
```
|
||
|
|
|
||
|
|
### Example
|
||
|
|
|
||
|
|
```bash
|
||
|
|
node adzuna_scraper.js https://www.adzuna.com/details/5694360484
|
||
|
|
```
|
||
|
|
|
||
|
|
## Output
|
||
|
|
|
||
|
|
The scraper outputs JSON to stdout:
|
||
|
|
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
"url": "https://www.adzuna.com/details/5694360484",
|
||
|
|
"jobDescription": {
|
||
|
|
"html": "...",
|
||
|
|
"text": "..."
|
||
|
|
},
|
||
|
|
"timestamp": "2026-04-11T12:00:00.000Z"
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
## Technical Details
|
||
|
|
|
||
|
|
- Uses `puppeteer-core` with `@sparticuz/chromium`
|
||
|
|
- Compatible with Node.js 14.21.3
|
||
|
|
- Includes self-contained Chromium binary
|
||
|
|
- No system dependencies required (no libdrm.so.2, etc.)
|
||
|
|
- Optimized for shared hosting environments
|
||
|
|
|
||
|
|
## Troubleshooting
|
||
|
|
|
||
|
|
If you get "Too Many Requests" errors, wait a few minutes before trying again (Adzuna rate limiting).
|
||
|
|
|
||
|
|
If the scraper fails to find job descriptions, check that the URL is valid and the job posting still exists.
|