Deployment

Stafett runs next to your services, against a Postgres you already operate. This page covers Kubernetes with Helm, plain Kubernetes manifests, and Docker Compose.

What you need

Postgres 14 or later, reachable over a direct connection. pgqueuer, the queue engine, is tested on 14 through 18. No transaction pooler; see Configuration.
A database role that can create schema stafett, or owns it.
Optionally, a signing secret, shared with the receivers that verify deliveries. Without one, deliveries go out unsigned.
Outbound network access from deliver to your destinations.

Stafett needs no inbound access from the internet and never calls stafett.dev.

Downloads

What	Where
Helm chart repository	`https://stafett.dev/charts`, chart `stafett-worker`
Chart values, with comments	values.yaml
Values schema	values.schema.json
Plain Kubernetes manifest	install.yaml
Images	`stafett-enqueue`, `stafett-deliver`; not published yet, see below

Install with Helm:

helm repo add stafett https://stafett.dev/charts
helm install stafett-worker stafett/stafett-worker -n stafett \
  --set secret.existingSecret=stafett-worker

The full steps, the Secret included, are under "Install with Helm" below.

Editors that read a $schema comment, such as VS Code with a YAML extension, check and complete your own values file against the schema:

# yaml-language-server: $schema=https://stafett.dev/charts/values.schema.json
enqueue:
  replicas: 2

The images

Stafett is two images, one per role, built from one Dockerfile. They share every layer and one tag:

Image	Runs
`stafett-enqueue`	The HTTP API: `POST /enqueue` and the job routes.
`stafett-deliver`	The delivery loop: signs and sends each job, with retries.

They are not published to a registry yet. Build them from the repository root and push them to a registry your cluster can pull from:

for role in enqueue deliver; do
  docker build -f deploy/Dockerfile --target $role -t <registry>/stafett-$role:edge .
  docker push <registry>/stafett-$role:edge
done

For a local kind cluster, load them instead: kind load docker-image <image>.

Both run as a non-root user (uid 10001) and write nothing to disk, so they work with a read-only root filesystem; the chart and the plain manifest set one. Each takes one optional command:

Command	Does
`<image>`	Runs the image's role.
`<image> setup`	Installs or upgrades schema `stafett`, then exits. Safe to run again and from several pods at once.

Kubernetes topology

 namespace: stafett
 ┌────────────────────────────────────────────────────────────────┐
 │  your services                                                 │
 │      │ http://stafett-worker-enqueue:8000/enqueue              │
 │      ▼                                                         │
 │  Service stafett-worker-enqueue (ClusterIP)                    │
 │      │                                                         │
 │      ▼                                                         │
 │  Deployment ...-enqueue (N)       Deployment ...-deliver (M)   │
 │  ┌────────────────────────┐       ┌─────────────────────────┐  │
 │  │ init: setup            │       │ init: setup             │  │
 │  │ enqueue :8000          │       │ deliver :8000 /healthz  │  │
 │  │ ready + live: /healthz │       │ live: /healthz          │  │
 │  │ env: DATABASE_URL      │       │ grace period: 60 s      │  │
 │  │                        │       │ env: DATABASE_URL,      │  │
 │  │                        │       │   STAFETT_SIGNING_SECRET│  │
 │  └────────────┬───────────┘       └───────┬──────────┬──────┘  │
 │               │                           │          │         │
 └───────────────┼───────────────────────────┼──────────┼─────────┘
                 │                           │          │
                 └───────────────────────────┤          ▼
                                             ▼    destinations
                               your Postgres (direct)

No Ingress. enqueue has no authentication, so it must stay inside the cluster. Your services call http://<service>.<namespace>:8000/enqueue.
Every pod runs setup as an init container, so no pod starts against an older schema than its image expects. pgqueuer's schema lock serialises concurrent runs.
deliver's 60-second termination grace period gives deliveries in flight time to finish on a rollout. An attempt still running when it ends is cut off and retried, so the destination may see it twice.
A deliver pod whose delivery loop stopped answers 503 on /healthz, and the kubelet restarts it. The loop stops when the pod loses its database connection, for example on a Postgres restart or failover; delivery resumes after the restart, about 30 seconds later.
enqueue's /healthz does not check the database. During a database outage enqueue stays Ready and answers POST /enqueue with 503.
Both pods meet the restricted Pod Security Standard.

Install with Helm

stafett.dev serves the chart as a Helm repository:

helm repo add stafett https://stafett.dev/charts
helm repo update

To read the chart before you install it:

helm show readme stafett/stafett-worker     # what it installs, in short
helm show values stafett/stafett-worker     # every value, with comments
helm template stafett-worker stafett/stafett-worker \
  --set secret.existingSecret=stafett-worker  # the exact objects it applies
helm pull stafett/stafett-worker --untar    # the chart's files

A browser can read values.yaml and values.schema.json too.

Create the Secret yourself, so the database password and signing secret never appear in Helm values or release history:

kubectl create namespace stafett
kubectl -n stafett create secret generic stafett-worker \
  --from-literal=DATABASE_URL='postgresql://user:password@host:5432/db' \
  --from-literal=STAFETT_SIGNING_SECRET="whsec_$(openssl rand -base64 24)"
# Leave out the STAFETT_SIGNING_SECRET line to deliver unsigned.

helm install stafett-worker stafett/stafett-worker -n stafett \
  --set secret.existingSecret=stafett-worker \
  --set image.repository=<registry>/stafett \
  --set image.tag=edge \
  --wait

Release name stafett-worker gives the same object names as the plain manifest: Service stafett-worker-enqueue, Deployments stafett-worker-enqueue and stafett-worker-deliver. --wait fails the install if setup fails, for example on a missing privilege. The notes printed after install show the in-cluster enqueue URL.

Check it from inside the cluster. A namespace that enforces the restricted Pod Security Standard refuses an ad-hoc kubectl run curl pod, so this runs in the enqueue container, which has Python:

kubectl -n stafett exec deploy/stafett-worker-enqueue -c enqueue -- python -c "
import urllib.request as u
request = u.Request('http://localhost:8000/enqueue', data=b'{\"hello\": \"world\"}',
                    headers={'content-type': 'application/json',
                             'stafett-url': 'https://example.com/'})
print(u.urlopen(request).read().decode())"

Values

The chart's values.yaml, with its defaults and comments:

image:
  # One image per role: <repository>-enqueue and <repository>-deliver. Not
  # published yet: point it at a registry your cluster can pull from.
  repository: ghcr.io/janbjorge/stafett
  # Both images share it. Defaults to the chart's appVersion.
  tag: ""
  pullPolicy: IfNotPresent

# For a private registry, e.g. [{name: regcred}].
imagePullSecrets: []

# DATABASE_URL, and STAFETT_SIGNING_SECRET to sign deliveries. Prefer an
# existing Secret with those keys; inline values end up in the release,
# base64 only. DATABASE_URL must be a direct connection, not a pooler.
# Without a signing secret, deliver sends no signature headers.
secret:
  existingSecret: ""
  databaseUrl: ""
  # Changing an inline secret rolls the pods.
  signingSecret: ""

enqueue:
  replicas: 1
  # Extra environment variables, name: value. Any setting from
  # https://stafett.dev/docs/configuration except STAFETT_PORT: the probes
  # expect port 8000.
  env: {}
  # A namespace with a ResourceQuota on CPU limits also needs limits.cpu,
  # here and under deliver and setup.
  resources:
    requests: {cpu: 50m, memory: 128Mi}
    limits: {memory: 256Mi}
  # No Ingress: the API has no auth (ADR 0004). Keep it in-cluster.
  service:
    type: ClusterIP
    port: 8000

deliver:
  replicas: 1
  # Extra environment, e.g. STAFETT_CONCURRENCY or the retry settings.
  env: {}
  resources:
    requests: {cpu: 50m, memory: 128Mi}
    limits: {memory: 256Mi}
  # Keep it above STAFETT_DELIVERY_TIMEOUT_SECONDS (30s default), so
  # deliveries in flight finish on SIGTERM instead of being sent twice.
  terminationGracePeriodSeconds: 60

# The init container that installs or upgrades schema "stafett" before each
# pod starts. Namespaces with a ResourceQuota refuse pods whose containers,
# init containers included, set no requests and limits.
setup:
  resources:
    requests: {cpu: 50m, memory: 128Mi}
    limits: {memory: 256Mi}

# Both apply to enqueue and deliver.
nodeSelector: {}
tolerations: []

values.schema.json checks every value on install and upgrade. A misspelt key, such as enqueue.replica, fails with an error naming it before anything reaches the cluster.

Upgrade

helm repo update
helm upgrade stafett-worker stafett/stafett-worker -n stafett \
  --reset-then-reuse-values --set image.tag=<new tag> --wait

--reset-then-reuse-values (Helm 3.14 or later) keeps your overrides and picks up defaults the new chart adds. Plain --reuse-values skips those new defaults. Keeping your values in a file and passing -f values.yaml works too.

New pods run setup before they start, which upgrades the schema in place. Old and new deliver pods can share the queue during the rollout.

Chart 0.2.0 renamed the values api and worker to enqueue and deliver, and the Deployments and Service with them. From 0.1.x, move your overrides to the new keys and upgrade with -f values.yaml rather than reusing values. Reused 0.1.x values fail the schema check: additional properties 'api' not allowed.

Scale

More enqueue throughput: raise enqueue.replicas.
More delivery throughput: raise deliver.replicas or STAFETT_CONCURRENCY. Each deliver pod holds one database connection; each enqueue pod holds a pool of up to 10. Budget Postgres max_connections for deliver.replicas + 10 × enqueue.replicas, plus one per pod while setup runs.
Watch the destination: it can receive deliver.replicas × STAFETT_CONCURRENCY requests at once.

Plain Kubernetes manifest

install.yaml holds the same objects the chart renders by default, with no templating and with comments. Edit its image lines to point at your registry, create the Secret as above, then:

kubectl -n stafett apply -f install.yaml

The file's header has the full steps.

Docker Compose

examples/quickstart/docker-compose.yml runs Postgres, setup, enqueue, deliver, and a signature-verifying receiver. It is the Getting started stack. For a single host against your own Postgres, keep its setup, enqueue, and deliver services, point DATABASE_URL at your database, and generate your own signing secret. Publish the enqueue port on loopback or a private interface only.

The example's Stafett services have no healthchecks or restart policies. A deliver that lost its database connection stays down until restarted; see Troubleshooting.

Uninstall

helm uninstall stafett-worker -n stafett

This removes the Deployments, the Service, and a Secret the chart created from inline values. It leaves a Secret you created yourself, and schema stafett with every job in it. To remove the schema as well:

DROP SCHEMA stafett CASCADE;

That deletes every queued and dead job, their history, and pgqueuer's statistics. It cannot be undone.