Operations

Stafett keeps no state of its own beyond pgqueuer's tables in schema stafett. Inspecting, replaying, and cleaning up jobs uses pgqueuer's pgq command, which ships in both Stafett images.

Running pgq

pgq reads its connection and schema from the environment. Inside a Stafett container:

kubectl -n stafett exec deploy/stafett-worker-deliver -- sh -c \
  'PGDSN=$DATABASE_URL PGQUEUER_SCHEMA=stafett PGQUEUER_CHANNEL=stafett_ch_pgqueuer pgq <command>'

With Docker Compose, swap the first line for docker compose exec deliver sh -c. The examples below show only pgq <command>.

Always set PGQUEUER_SCHEMA=stafett. Without it pgq uses the connection's search_path, usually schema public, which is empty or holds a different pgqueuer install.

Common tasks

Run pgq <command> --help for the options.

pgq failed shows pgqueuer's own attempt counter, which counts retries in the current cycle and resets on replay. It reads one lower than the attempts from GET /jobs/{id}, which counts every attempt over the job's life.

Each failed attempt records its reason, such as HTTP 404 or connect error, in pgqueuer's log. pgq failed does not show it; query it:

SELECT created, status, traceback->>'exception_message' AS reason
FROM stafett.pgqueuer_log
WHERE job_id = 1234 AND traceback IS NOT NULL
ORDER BY id;

pgq web, pgqueuer's browser dashboard, is not in the image: it needs pgqueuer's web extra. If you install it elsewhere, set both PGQUEUER_WEB_USER and PGQUEUER_WEB_PASSWORD, because it shows every job's URL, headers, and payload, and keep it internal.

Replay

  [ dead ] ── pgq requeue <id> ──► [ retrying ] ──► deliver picks it up
                                                      within 5 seconds

A replay gives the job a fresh retry budget. Until its next attempt, GET /jobs/{id} reports it as retrying, because it has attempts behind it. The receiver sees the same webhook-id as before, and the attempt count keeps climbing across replays.

A replay can fail with a unique-key error if the job had a Stafett-Dedupe-Key and a newer job with the same key is queued or in flight.

Fix the cause first: replaying into a destination that still answers 404 kills the job again on its first attempt.

What to watch

Stafett logs to stderr. It never logs payloads, destination URLs, headers, or the signing secret. A deliver pod whose delivery loop stopped logs the error only when the process exits.

Postgres housekeeping

The queue table churns: every delivered job is a deleted row. pgqueuer applies autovacuum settings tuned for its tables:

pgq autovac              # apply them to schema stafett
pgq autovac --rollback   # back to Postgres defaults

Two tables grow without bound, and Stafett never deletes from them:

• stafett.pgqueuer_log keeps a row for every enqueue, attempt, and cancel. GET /jobs/{id} reads attempt counts and the status of finished jobs from it.
• stafett.pgqueuer_statistics holds per-second counts that pgqueuer rolls up from the log.

To bound them, delete old rows yourself, for example older than 30 days, from a scheduled job such as a Kubernetes CronJob or pg_cron:

DELETE FROM stafett.pgqueuer_log WHERE created < now() - INTERVAL '30 days';
DELETE FROM stafett.pgqueuer_statistics WHERE created < now() - INTERVAL '30 days';

After that, GET /jobs/{id} answers 404 for jobs that finished before the cut-off.

Signing secret rotation

There is one signing secret per install, and no overlap window yet. Rotating it means:

1. Update STAFETT_SIGNING_SECRET in the Secret.
2. Restart deliver (kubectl rollout restart deploy/stafett-worker-deliver).
3. Update every receiver.

Deliveries between steps 2 and 3 fail verification at the receiver. If the receiver answers 4xx, those jobs are dead; replay them once the receivers have the new secret.

Upgrades

New pods run setup before they start, which upgrades the schema in place. Jobs written by a newer version that an older deliver does not understand are dead with reason unknown envelope version; replay them once every deliver pod is upgraded.