Add supervised systemd queue worker (phase 8)

Runs php artisan queue:work as a systemd --user service (Restart=always,
hourly recycling) so OSINT lookups process automatically instead of
requiring a manually-run queue:work. All 8 planned phases now done.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
cedry2k1992
2026-07-20 10:40:37 +03:00
co-authored by Claude Sonnet 5
parent 873699e23e
commit 853686f3f3
3 changed files with 70 additions and 7 deletions
+18 -7
View File
@@ -145,9 +145,21 @@ commands).
and by target value, opened the printable report, downloaded and and by target value, opened the printable report, downloaded and
inspected both CSV and JSON exports. All test data cleaned up inspected both CSV and JSON exports. All test data cleaned up
afterward. afterward.
8. **Queue worker process management** — `QUEUE_CONNECTION=database` 8. **Queue worker process management** (done) — `deploy/systemd/
needs a supervised `queue:work` process (supervisor/systemd) before osint-queue-worker.service`, a `systemd --user` unit (no sudo
this is usable outside manual dev testing (see Environment notes) needed; this box already has lingering enabled so it survives
logout) running `queue:work --timeout=180 --max-time=3600
--tries=1` with `Restart=always`. `--timeout` is set above
`config('osint.run_timeout')` (120s) — the worker's timeout must
exceed a runner's own timeout or the worker kills the job first.
`--max-time` restarts the worker hourly to pick up code/`.env`
changes and avoid long-lived-process memory growth. Installed and
verified live: dispatched a real job via the web app *without*
manually running `queue:work`, confirmed the persistent service
picked it up and completed it (ip-api lookup, ~200ms); killed the
worker process directly and confirmed systemd auto-restarted it
within seconds. See `deploy/systemd/README.md` for install/operate
commands.
## Open questions ## Open questions
- Any legal/ToS constraints on the sources to be queried? - Any legal/ToS constraints on the sources to be queried?
@@ -164,10 +176,9 @@ commands).
`--env=testing` directly against migrate/seed `--env=testing` directly against migrate/seed
- No sudo access in the dev sandbox that created this project; SQLite - No sudo access in the dev sandbox that created this project; SQLite
extension isn't installed for PHP 8.4 here, so MySQL was used instead extension isn't installed for PHP 8.4 here, so MySQL was used instead
- `QUEUE_CONNECTION=database`, so lookups won't actually run unless a - `QUEUE_CONNECTION=database`; the queue worker is supervised by
queue worker is running (`php artisan queue:work`). No supervisor/ `systemd --user` (see `deploy/systemd/`) and running continuously —
systemd unit set up for this yet — needed before this is usable no more need to manually run `php artisan queue:work` in dev
outside of manual dev testing
- The two Python venvs are ~390MB combined (`.venv` 166MB, - The two Python venvs are ~390MB combined (`.venv` 166MB,
`.venv-theharvester` 223MB, mostly playwright/censys/shodan/fastapi `.venv-theharvester` 223MB, mostly playwright/censys/shodan/fastapi
pulled in by theHarvester) — worth knowing before assuming this is pulled in by theHarvester) — worth knowing before assuming this is
+31
View File
@@ -0,0 +1,31 @@
# Queue worker (systemd --user)
`QUEUE_CONNECTION=database`, so OSINT lookups only actually run while a
`queue:work` process is alive. This unit supervises one as a `systemd --user`
service — no root/sudo needed, survives crashes (`Restart=always`), and
restarts hourly (`--max-time=3600`) to pick up code/config changes and avoid
long-lived-process memory growth.
`--timeout=180` is intentionally above `config('osint.run_timeout')` (120s,
in `config/osint.php`) — the worker's timeout must exceed a runner's own
timeout, or the worker kills the job before the runner gets a chance to.
## Install
```
cp deploy/systemd/osint-queue-worker.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now osint-queue-worker.service
```
Requires lingering enabled so the service keeps running without an active
login session: `loginctl enable-linger $USER` (needs sudo once, or ask your
sysadmin — already enabled on this box).
## Operate
```
systemctl --user status osint-queue-worker.service
journalctl --user -u osint-queue-worker.service -f
systemctl --user restart osint-queue-worker.service # after deploying code/.env changes
```
+21
View File
@@ -0,0 +1,21 @@
[Unit]
Description=OSINT Laravel Queue Worker
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=0
[Service]
Type=simple
ExecStart=/usr/bin/php /home/m105/projects/osint/artisan queue:work --queue=default --sleep=3 --tries=1 --timeout=180 --max-time=3600
WorkingDirectory=/home/m105/projects/osint
Environment="PATH=/home/m105/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Restart=always
RestartSec=5
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=30
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=default.target