Troubleshooting

Pods fail to start: "missing stafett.pgqueuer"

missing stafett.pgqueuer, stafett.pgqueuer_log; run either image with `setup` against this database first

The schema is not installed. Run docker run <image> setup with either image, or let the chart's init container do it. If the init container itself fails, read its log:

kubectl -n stafett logs deploy/stafett-worker-enqueue -c setup

A permission error there means the role in DATABASE_URL can neither create schema stafett nor owns it. See Configuration.

Jobs are slow to start, or only start every few seconds

deliver hears about new jobs through Postgres LISTEN, and polls every 5 seconds as a fallback. A retry that starts up to 5 seconds after its backoff ends is expected; waits under 5 seconds often stretch to about 5.

If every new job waits up to 5 seconds, LISTEN is not reaching deliver.

A transaction pooler between Stafett and Postgres (PgBouncer in transaction mode, a managed provider's pooled port) breaks LISTEN, and can break Stafett's queries outright. Use the direct connection string. Pooled connections are not supported.

A job is dead

List dead jobs with pgq failed, then read why one failed with the query in Operations.

The receiver rejects the signature

• The receiver must use the exact STAFETT_SIGNING_SECRET deliver has, including the whsec_ prefix.
• Verify the raw body bytes. Parsing the JSON and serialising it again changes the bytes and breaks the signature.
• Standard Webhooks libraries reject timestamps too far from their own clock (5 minutes in most). Check the clocks on both sides.
• The Python standardwebhooks library's verify() also parses the body as JSON by default, so it raises on a valid signature over a non-JSON body. Pass json_parse=False for those.
• Signatures you set yourself, with deliver's signing off, are sent unchanged on every attempt. If they include a timestamp, a late delivery (a delay, a backlog, retries, a replay) can arrive outside the receiver's window. Let deliver sign instead, or use a credential that does not expire.

The same job arrived twice

Expected occasionally: delivery is at-least-once. A timeout, a deliver pod that stopped mid-attempt, or a replay can each send a job again. Dedupe on webhook-id, or, with signing off, on an id header of your own.

Jobs stay queued and nothing is delivered

deliver's delivery loop stops when it loses its database connection, for example when Postgres restarts or fails over. It does not reconnect by itself, and it logs the error (connection is closed) only when the process exits. Its /healthz answers 503 meanwhile.

• Kubernetes: the liveness probe restarts deliver, and delivery resumes about 30 seconds later. Frequent restarts mean something cuts the connection: a pooler, a firewall idle timeout, or an unstable database.
• Docker Compose: nothing restarts it. Restart deliver by hand (docker compose restart deliver), or give it a healthcheck on /healthz and a restart policy.

enqueue keeps accepting jobs meanwhile, so nothing is lost: they are delivered once deliver is running again.

pgq shows nothing

Set PGQUEUER_SCHEMA=stafett and PGDSN. Without the schema, pgq uses the connection's search_path, usually schema public. See Operations.

POST /enqueue answers 503

enqueue cannot reach Postgres. Right after a database restart or failover, the first requests can also answer 503 while enqueue replaces its closed connections. Nothing was written, with one rare exception covered in the API reference: retry with a Stafett-Dedupe-Key.