Helm

The icoretech/codex-pooler Helm chart deploys Codex Pooler into Kubernetes from the iCoreTech Helm repository. It separates web traffic, background work, scheduled work, and migration work so each role can scale and restart on its own.

Use this page as the public shape of the chart. Keep environment-specific secrets, private image repository names, internal hostnames, and operational evidence out of values files and public docs.

Install the chart

helm repo add icoretech https://icoretech.github.io/helm
helm repo update
helm upgrade --install codex-pooler icoretech/codex-pooler \
  --namespace codex-pooler \
  --create-namespace \
  --version 0.9.4 \
  --values ./values.production.yaml

The same chart is also published as an OCI artifact:

helm upgrade --install codex-pooler oci://ghcr.io/icoretech/charts/codex-pooler \
  --namespace codex-pooler \
  --create-namespace \
  --version 0.9.4 \
  --values ./values.production.yaml

Chart shape

The chart has four release roles:

• app, serves HTTP with OBAN_MODE=web and PHX_SERVER=true
• oban.worker, runs background jobs with OBAN_MODE=worker
• oban.scheduler, runs scheduled jobs with OBAN_MODE=scheduler
• migrations, runs release migrations before app rollout when enabled

All roles use the same image repository and tag. Only the app role serves HTTP. Worker, scheduler, and migration pods don’t expose the HTTP endpoint.

Image expectations

image:
  repository: ghcr.io/icoretech/codex-pooler
  tag: ""
  pullPolicy: IfNotPresent

The chart defaults image.tag to its appVersion, so a pinned chart version normally selects the matching Codex Pooler image. Set image.tag only when you intentionally run a custom image or a different immutable release tag.

Required secrets

By default, the chart expects an existing Kubernetes Secret:

secrets:
  create: false
  existingSecret: codex-pooler-secrets

That secret must provide the release values needed before Codex Pooler can read database-managed settings:

• database-url
• secret-key-base
• totp-encryption-key
• totp-key-version
• upstream-secret-key
• upstream-secret-key-version
• release-cookie, when app clustering is enabled and you use the default clustering cookie key

Don’t put upstream access tokens, Pool API keys, MCP tokens, cookies, auth.json, SMTP passwords, or raw client payloads into chart values.

Host and ingress

Set the public host through chart values:

config:
  host: codex-pooler.example.com

ingress:
  enabled: true
  hosts:
    - host: codex-pooler.example.com
      paths:
        - path: /
          pathType: Prefix

Public client examples should use https://codex-pooler.example.com as the deployed base URL.

Role topology

The default topology is conservative:

app:
  enabled: true
  replicaCount: 1

oban:
  worker:
    enabled: true
    replicaCount: 1
  scheduler:
    enabled: true
    replicaCount: 1

migrations:
  enabled: true

The migration hook runs database migrations and imports the vendored OpenAI pricing feed. The scheduler keeps scheduled background work separate from request-serving pods.

Complete migrations before starting app pods on the new image. Execution recovery requires the attempt execution-identity columns, their index, and the execution_terminal_proofs table; disabling the hook requires completing the same release migrations separately before app startup.

The 0.8.0 upgrade and rollback procedures target PostgreSQL 18. API-key history migrations commit short NOT VALID constraint swaps before validating existing rows separately. Migration runners serialize through PostgreSQL advisory locks; interrupted concurrent index builds are checked and repaired on retry when their definitions match. A conflicting index definition requires investigation. These migrations do not make an arbitrary older image compatible: after API-key deletion leaves null-key history, the 0.7.8 image is incompatible with that retained history and the ledger migration refuses to restore required key ownership. Preserve the history and use compatible code; do not treat an image-only rollback as recovery.

Index migrations build and drop their indexes concurrently, so application reads and writes continue during the upgrade. A concurrent index build still has to wait for every transaction on the same database that uses the table or holds an older snapshot, such as a running pg_dump backup, a long reporting query or a psql session left idle inside a transaction. The migration waits up to five minutes in total for those transactions and logs the blocking sessions (process id, application name, backend type, state and transaction age, never the query text) while it waits; a wait for a table lock held by another session stops after ten seconds. When either budget runs out the migration fails with a message that names the blocking sessions: end them or let them finish, then let the migration run again. PostgreSQL shows the state, backend type and transaction start of a session owned by another database role only to superusers, members of pg_read_all_stats and that role, so when the migration role is an ordinary application role, a backup or psql session run as another role is listed with those fields as hidden and its role name; grant pg_read_all_stats to the migration role if you want them in the message. An interrupted index build is dropped and rebuilt on the next run. The migration Job retries up to migrations.backoffLimit times within migrations.activeDeadlineSeconds, so with the defaults (3 retries, 900 seconds) one long-lived transaction can hold an upgrade for at most 15 minutes before the Job fails.

A pg_dump backup leaves out planner statistics unless it was taken with --statistics, so a database restored with psql or pg_restore has no column statistics until autovacuum analyzes each table. Run ANALYZE on it (or vacuumdb --analyze-in-stages) before the app pods serve traffic again.

Metrics collection

The app role exposes Prometheus metrics on /metrics through the same HTTP service as /healthz and /readyz. Metrics authentication is separate from the runtime firewall: the endpoint is open with no configured bearer, bearer-protected when configured, and unavailable fail-closed when metrics settings cannot be read. Worker, scheduler, and migration pods don’t expose the HTTP endpoint, so app-level metrics such as BEAM memory, request rate, route latency, gateway admission, stream-buffer pressure, and Ecto query pressure are scraped from app pods only. Worker and scheduler roles still run the memory sampler and VM telemetry poller, but they don’t start the Prometheus reporter because unswept histogram samples are drained only by a /metrics scrape.

If you run Prometheus Operator, enable the chart ServiceMonitor:

monitoring:
  serviceMonitor:
    enabled: true
    labels:
      release: kube-prometheus-stack
    interval: 10s
    scrapeTimeout: 5s
    path: /metrics

Set labels to match your Prometheus Operator selector. The common release: kube-prometheus-stack label is only an example; use the label your Prometheus installation selects.

If you configure a metrics bearer token from /admin/system, create a Kubernetes Secret containing the same raw token and point the ServiceMonitor at it:

monitoring:
  serviceMonitor:
    enabled: true
    bearerTokenSecret:
      name: codex-pooler-metrics
      key: token

Don’t put the metrics credential in Helm values. The app stores only a keyed digest, so retain the credential separately at creation time if Prometheus will authenticate with it. To roll back a metrics exposure change, restore the intended bearer state in /admin/system and keep the ServiceMonitor reference aligned.

Render the monitoring resource before installing:

helm template codex-pooler icoretech/codex-pooler \
  --namespace codex-pooler \
  --version 0.9.4 \
  --set monitoring.serviceMonitor.enabled=true \
  --set monitoring.serviceMonitor.labels.release=kube-prometheus-stack \
  --set monitoring.serviceMonitor.interval=10s \
  --set monitoring.serviceMonitor.scrapeTimeout=5s \
  --show-only templates/servicemonitor.yaml

After installation, verify that Kubernetes rendered the ServiceMonitor and that Prometheus has active targets:

kubectl -n codex-pooler get servicemonitor codex-pooler-app
kubectl -n codex-pooler get endpoints codex-pooler-app

When you can reach the Prometheus HTTP API, check scrape health:

# If Prometheus is only reachable inside the cluster, port-forward your Prometheus service first.
kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090

curl -fsS --get http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=up{namespace="codex-pooler",job="codex-pooler-app"}'

Useful first queries:

vm_memory_total_bytes{namespace="codex-pooler", job="codex-pooler-app"}
sum by (pod) (rate(phoenix_endpoint_stop_count{namespace="codex-pooler", job="codex-pooler-app"}[5m]))
sum by (method, status_class) (rate(codex_pooler_http_request_count{namespace="codex-pooler", job="codex-pooler-app"}[5m]))
topk(10, sum by (source, command) (rate(codex_pooler_repo_query_count{namespace="codex-pooler", job="codex-pooler-app"}[5m])))

For memory incidents, also collect Kubernetes cgroup metrics such as container_memory_working_set_bytes, container_memory_rss, restart counters, and OOM counters from your cluster collectors. The app metrics explain BEAM and gateway behavior; cgroup and kube-state-metrics data explain container limits, restarts, and killed pods.

The Monitoring section covers metric collection, the starter Grafana dashboard, PromQL recipes and runtime troubleshooting.

Websocket replica caveat

The chart defaults the web app to one replica. If app.replicaCount is 2 or higher, the chart treats websocket owner forwarding as required.

The current chart guard rejects multi-replica app rendering unless app clustering is enabled and app pods participate in that cluster. App participation defaults to enabled, so the minimal multi-replica values shape is:

clustering:
  enabled: true

app:
  replicaCount: 2

With that shape, the chart renders CODEX_POOLER_WEBSOCKET_OWNER_FORWARDING=true on app pods automatically. Render the chart and verify the topology before raising app.replicaCount.

Flux multi-replica example

This example shows a Flux HelmRepository and HelmRelease for two app replicas. It assumes:

• a Kubernetes Secret named codex-pooler-secrets exists in that namespace
• the Secret contains the required release keys plus release-cookie
• Postgres is reachable through the Secret’s database-url
• https://codex-pooler.example.com is the public app URL

The example leaves chart defaults out of values. In particular, it does not set image values, secrets.create, secrets.existingSecret, app probes, rollout strategy, termination grace, worker or scheduler replica counts, or explicit websocket owner-forwarding values.

apiVersion: v1
kind: Namespace
metadata:
  name: codex-pooler
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: icoretech
  namespace: flux-system
spec:
  interval: 1h
  url: https://icoretech.github.io/helm
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: codex-pooler
  namespace: codex-pooler
spec:
  releaseName: codex-pooler
  chart:
    spec:
      chart: codex-pooler
      version: "0.6.22"
      sourceRef:
        kind: HelmRepository
        name: icoretech
        namespace: flux-system
  interval: 5m
  # Allow time for migrations, pod draining, and readiness checks.
  timeout: 15m
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
      # Restore the previous release if an upgrade fails.
      strategy: rollback
      remediateLastFailure: true
  values:
    config:
      host: codex-pooler.example.com
    ingress:
      enabled: true
      className: nginx
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt
      hosts:
        - host: codex-pooler.example.com
          paths:
            - path: /
              pathType: Prefix
      tls:
        - secretName: codex-pooler-tls
          hosts:
            - codex-pooler.example.com
    monitoring:
      serviceMonitor:
        enabled: true
        labels:
          release: kube-prometheus-stack
        interval: 10s
        scrapeTimeout: 5s
    clustering:
      enabled: true
      participants:
        worker: false
        scheduler: false
    app:
      replicaCount: 2
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: codex-pooler
                    app.kubernetes.io/instance: codex-pooler
                    app.kubernetes.io/component: app
                topologyKey: kubernetes.io/hostname

When app.replicaCount is 2 or higher, the chart enables websocket owner forwarding automatically. Worker and scheduler cluster participation is optional for dead-execution recovery: keep clustering.participants.worker and clustering.participants.scheduler disabled as in this example unless another feature needs their BEAM membership. These explicit overrides differ from the chart’s participant defaults.

Dead-execution recovery reads exact owner-attested PostgreSQL terminal proofs, so workers do not need an eRPC connection to the app owner. The app publishes those proofs asynchronously in bounded batches; database publication time starts their six-hour retention window. A lost or unpublished proof leaves execution death unknown and cannot authorize settlement. App-to-app clustering for websocket owner forwarding remains required in the multi-replica topology above.

Render safely before install

Render manifests locally before applying them:

helm template codex-pooler icoretech/codex-pooler \
  --version 0.9.4 \
  --set config.host=codex-pooler.example.com \
  --set ingress.enabled=true \
  --set ingress.hosts[0].host=codex-pooler.example.com

Inspect the output for the expected app, worker, scheduler, migration, Secret reference, Service, and Ingress resources. The rendered app Deployment should serve HTTP. Worker, scheduler, and migration resources should not.

For install or upgrade, provide your production values file from a private path:

helm upgrade --install codex-pooler icoretech/codex-pooler \
  --namespace codex-pooler \
  --create-namespace \
  --version 0.9.4 \
  --values ./values.production.yaml

Keep values.production.yaml private if it contains secret names, internal DNS names, ingress details, or environment-specific settings.