Add supervised systemd queue worker (phase 8)
Runs php artisan queue:work as a systemd --user service (Restart=always, hourly recycling) so OSINT lookups process automatically instead of requiring a manually-run queue:work. All 8 planned phases now done. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
873699e23e
commit
853686f3f3
@@ -145,9 +145,21 @@ commands).
|
|||||||
and by target value, opened the printable report, downloaded and
|
and by target value, opened the printable report, downloaded and
|
||||||
inspected both CSV and JSON exports. All test data cleaned up
|
inspected both CSV and JSON exports. All test data cleaned up
|
||||||
afterward.
|
afterward.
|
||||||
8. **Queue worker process management** — `QUEUE_CONNECTION=database`
|
8. **Queue worker process management** (done) — `deploy/systemd/
|
||||||
needs a supervised `queue:work` process (supervisor/systemd) before
|
osint-queue-worker.service`, a `systemd --user` unit (no sudo
|
||||||
this is usable outside manual dev testing (see Environment notes)
|
needed; this box already has lingering enabled so it survives
|
||||||
|
logout) running `queue:work --timeout=180 --max-time=3600
|
||||||
|
--tries=1` with `Restart=always`. `--timeout` is set above
|
||||||
|
`config('osint.run_timeout')` (120s) — the worker's timeout must
|
||||||
|
exceed a runner's own timeout or the worker kills the job first.
|
||||||
|
`--max-time` restarts the worker hourly to pick up code/`.env`
|
||||||
|
changes and avoid long-lived-process memory growth. Installed and
|
||||||
|
verified live: dispatched a real job via the web app *without*
|
||||||
|
manually running `queue:work`, confirmed the persistent service
|
||||||
|
picked it up and completed it (ip-api lookup, ~200ms); killed the
|
||||||
|
worker process directly and confirmed systemd auto-restarted it
|
||||||
|
within seconds. See `deploy/systemd/README.md` for install/operate
|
||||||
|
commands.
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
- Any legal/ToS constraints on the sources to be queried?
|
- Any legal/ToS constraints on the sources to be queried?
|
||||||
@@ -164,10 +176,9 @@ commands).
|
|||||||
`--env=testing` directly against migrate/seed
|
`--env=testing` directly against migrate/seed
|
||||||
- No sudo access in the dev sandbox that created this project; SQLite
|
- No sudo access in the dev sandbox that created this project; SQLite
|
||||||
extension isn't installed for PHP 8.4 here, so MySQL was used instead
|
extension isn't installed for PHP 8.4 here, so MySQL was used instead
|
||||||
- `QUEUE_CONNECTION=database`, so lookups won't actually run unless a
|
- `QUEUE_CONNECTION=database`; the queue worker is supervised by
|
||||||
queue worker is running (`php artisan queue:work`). No supervisor/
|
`systemd --user` (see `deploy/systemd/`) and running continuously —
|
||||||
systemd unit set up for this yet — needed before this is usable
|
no more need to manually run `php artisan queue:work` in dev
|
||||||
outside of manual dev testing
|
|
||||||
- The two Python venvs are ~390MB combined (`.venv` 166MB,
|
- The two Python venvs are ~390MB combined (`.venv` 166MB,
|
||||||
`.venv-theharvester` 223MB, mostly playwright/censys/shodan/fastapi
|
`.venv-theharvester` 223MB, mostly playwright/censys/shodan/fastapi
|
||||||
pulled in by theHarvester) — worth knowing before assuming this is
|
pulled in by theHarvester) — worth knowing before assuming this is
|
||||||
|
|||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Queue worker (systemd --user)
|
||||||
|
|
||||||
|
`QUEUE_CONNECTION=database`, so OSINT lookups only actually run while a
|
||||||
|
`queue:work` process is alive. This unit supervises one as a `systemd --user`
|
||||||
|
service — no root/sudo needed, survives crashes (`Restart=always`), and
|
||||||
|
restarts hourly (`--max-time=3600`) to pick up code/config changes and avoid
|
||||||
|
long-lived-process memory growth.
|
||||||
|
|
||||||
|
`--timeout=180` is intentionally above `config('osint.run_timeout')` (120s,
|
||||||
|
in `config/osint.php`) — the worker's timeout must exceed a runner's own
|
||||||
|
timeout, or the worker kills the job before the runner gets a chance to.
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```
|
||||||
|
cp deploy/systemd/osint-queue-worker.service ~/.config/systemd/user/
|
||||||
|
systemctl --user daemon-reload
|
||||||
|
systemctl --user enable --now osint-queue-worker.service
|
||||||
|
```
|
||||||
|
|
||||||
|
Requires lingering enabled so the service keeps running without an active
|
||||||
|
login session: `loginctl enable-linger $USER` (needs sudo once, or ask your
|
||||||
|
sysadmin — already enabled on this box).
|
||||||
|
|
||||||
|
## Operate
|
||||||
|
|
||||||
|
```
|
||||||
|
systemctl --user status osint-queue-worker.service
|
||||||
|
journalctl --user -u osint-queue-worker.service -f
|
||||||
|
systemctl --user restart osint-queue-worker.service # after deploying code/.env changes
|
||||||
|
```
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=OSINT Laravel Queue Worker
|
||||||
|
After=network-online.target
|
||||||
|
Wants=network-online.target
|
||||||
|
StartLimitIntervalSec=0
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
ExecStart=/usr/bin/php /home/m105/projects/osint/artisan queue:work --queue=default --sleep=3 --tries=1 --timeout=180 --max-time=3600
|
||||||
|
WorkingDirectory=/home/m105/projects/osint
|
||||||
|
Environment="PATH=/home/m105/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||||
|
Restart=always
|
||||||
|
RestartSec=5
|
||||||
|
KillMode=mixed
|
||||||
|
KillSignal=SIGTERM
|
||||||
|
TimeoutStopSec=30
|
||||||
|
StandardOutput=journal
|
||||||
|
StandardError=journal
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
Reference in New Issue
Block a user