# Deploy Codex Pooler on Kubernetes with Helm The `icoretech/codex-pooler` Helm chart deploys Codex Pooler into Kubernetes from the iCoreTech Helm repository. It separates web traffic, background work, scheduled work, and migration work so each role can scale and restart on its own. Use this page as the public shape of the chart. Keep environment-specific secrets, private image repository names, internal hostnames, and operational evidence out of values files and public docs. ## Install the chart ```bash helm repo add icoretech https://icoretech.github.io/helm helm repo update helm upgrade --install codex-pooler icoretech/codex-pooler \ --namespace codex-pooler \ --create-namespace \ --version 0.9.4 \ --values ./values.production.yaml ``` The same chart is also published as an OCI artifact: ```bash helm upgrade --install codex-pooler oci://ghcr.io/icoretech/charts/codex-pooler \ --namespace codex-pooler \ --create-namespace \ --version 0.9.4 \ --values ./values.production.yaml ``` ## Chart shape The chart has four release roles: - `app`, serves HTTP with `OBAN_MODE=web` and `PHX_SERVER=true` - `oban.worker`, runs background jobs with `OBAN_MODE=worker` - `oban.scheduler`, runs scheduled jobs with `OBAN_MODE=scheduler` - `migrations`, runs release migrations before app rollout when enabled All roles use the same image repository and tag. Only the app role serves HTTP. Worker, scheduler, and migration pods don't expose the HTTP endpoint. ## Image expectations ```yaml image: repository: ghcr.io/icoretech/codex-pooler tag: "" pullPolicy: IfNotPresent ``` The chart defaults `image.tag` to its `appVersion`, so a pinned chart version normally selects the matching Codex Pooler image. Set `image.tag` only when you intentionally run a custom image or a different immutable release tag. ## Required secrets By default, the chart expects an existing Kubernetes Secret: ```yaml secrets: create: false existingSecret: codex-pooler-secrets ``` That secret must provide the release values needed before Codex Pooler can read database-managed settings: - `database-url` - `secret-key-base` - `totp-encryption-key` - `totp-key-version` - `upstream-secret-key` - `upstream-secret-key-version` - `release-cookie`, when app clustering is enabled and you use the default clustering cookie key Don't put upstream access tokens, Pool API keys, MCP tokens, cookies, `auth.json`, SMTP passwords, or raw client payloads into chart values. ## Host and ingress Set the public host through chart values: ```yaml config: host: codex-pooler.example.com ingress: enabled: true hosts: - host: codex-pooler.example.com paths: - path: / pathType: Prefix ``` Public client examples should use `https://codex-pooler.example.com` as the deployed base URL. ## Role topology The default topology is conservative: ```yaml app: enabled: true replicaCount: 1 oban: worker: enabled: true replicaCount: 1 scheduler: enabled: true replicaCount: 1 migrations: enabled: true ``` The migration hook runs database migrations and imports the vendored OpenAI pricing feed. The scheduler keeps scheduled background work separate from request-serving pods. Complete migrations before starting app pods on the new image. Execution recovery requires the attempt execution-identity columns, their index, and the `execution_terminal_proofs` table; disabling the hook requires completing the same release migrations separately before app startup. The 0.8.0 upgrade and rollback procedures target PostgreSQL 18. API-key history migrations commit short `NOT VALID` constraint swaps before validating existing rows separately. Migration runners serialize through PostgreSQL advisory locks; interrupted concurrent index builds are checked and repaired on retry when their definitions match. A conflicting index definition requires investigation. These migrations do not make an arbitrary older image compatible: after API-key deletion leaves null-key history, the 0.7.8 image is incompatible with that retained history and the ledger migration refuses to restore required key ownership. Preserve the history and use compatible code; do not treat an image-only rollback as recovery. Index migrations build and drop their indexes concurrently, so application reads and writes continue during the upgrade. A concurrent index build still has to wait for every transaction on the same database that uses the table or holds an older snapshot, such as a running `pg_dump` backup, a long reporting query or a `psql` session left idle inside a transaction. The migration waits up to five minutes in total for those transactions and logs the blocking sessions (process id, application name, backend type, state and transaction age, never the query text) while it waits; a wait for a table lock held by another session stops after ten seconds. When either budget runs out the migration fails with a message that names the blocking sessions: end them or let them finish, then let the migration run again. PostgreSQL shows the state, backend type and transaction start of a session owned by another database role only to superusers, members of `pg_read_all_stats` and that role, so when the migration role is an ordinary application role, a backup or `psql` session run as another role is listed with those fields as `hidden` and its role name; grant `pg_read_all_stats` to the migration role if you want them in the message. An interrupted index build is dropped and rebuilt on the next run. The migration Job retries up to `migrations.backoffLimit` times within `migrations.activeDeadlineSeconds`, so with the defaults (3 retries, 900 seconds) one long-lived transaction can hold an upgrade for at most 15 minutes before the Job fails. A `pg_dump` backup leaves out planner statistics unless it was taken with `--statistics`, so a database restored with `psql` or `pg_restore` has no column statistics until autovacuum analyzes each table. Run `ANALYZE` on it (or `vacuumdb --analyze-in-stages`) before the app pods serve traffic again. ## Metrics collection The app role exposes Prometheus metrics on `/metrics` through the same HTTP service as `/healthz` and `/readyz`. Metrics authentication is separate from the runtime firewall: the endpoint is open with no configured bearer, bearer-protected when configured, and unavailable fail-closed when metrics settings cannot be read. Worker, scheduler, and migration pods don't expose the HTTP endpoint, so app-level metrics such as BEAM memory, request rate, route latency, gateway admission, stream-buffer pressure, and Ecto query pressure are scraped from app pods only. Worker and scheduler roles still run the memory sampler and VM telemetry poller, but they don't start the Prometheus reporter because unswept histogram samples are drained only by a `/metrics` scrape. If you run Prometheus Operator, enable the chart `ServiceMonitor`: ```yaml monitoring: serviceMonitor: enabled: true labels: release: kube-prometheus-stack interval: 10s scrapeTimeout: 5s path: /metrics ``` Set `labels` to match your Prometheus Operator selector. The common `release: kube-prometheus-stack` label is only an example; use the label your Prometheus installation selects. If you configure a metrics bearer token from `/admin/system`, create a Kubernetes Secret containing the same raw token and point the `ServiceMonitor` at it: ```yaml monitoring: serviceMonitor: enabled: true bearerTokenSecret: name: codex-pooler-metrics key: token ``` Don't put the metrics credential in Helm values. The app stores only a keyed digest, so retain the credential separately at creation time if Prometheus will authenticate with it. To roll back a metrics exposure change, restore the intended bearer state in `/admin/system` and keep the ServiceMonitor reference aligned. Render the monitoring resource before installing: ```bash helm template codex-pooler icoretech/codex-pooler \ --namespace codex-pooler \ --version 0.9.4 \ --set monitoring.serviceMonitor.enabled=true \ --set monitoring.serviceMonitor.labels.release=kube-prometheus-stack \ --set monitoring.serviceMonitor.interval=10s \ --set monitoring.serviceMonitor.scrapeTimeout=5s \ --show-only templates/servicemonitor.yaml ``` After installation, verify that Kubernetes rendered the ServiceMonitor and that Prometheus has active targets: ```bash kubectl -n codex-pooler get servicemonitor codex-pooler-app kubectl -n codex-pooler get endpoints codex-pooler-app ``` When you can reach the Prometheus HTTP API, check scrape health: ```bash # If Prometheus is only reachable inside the cluster, port-forward your Prometheus service first. kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090 curl -fsS --get http://127.0.0.1:9090/api/v1/query \ --data-urlencode 'query=up{namespace="codex-pooler",job="codex-pooler-app"}' ``` Useful first queries: ```promql vm_memory_total_bytes{namespace="codex-pooler", job="codex-pooler-app"} sum by (pod) (rate(phoenix_endpoint_stop_count{namespace="codex-pooler", job="codex-pooler-app"}[5m])) sum by (method, status_class) (rate(codex_pooler_http_request_count{namespace="codex-pooler", job="codex-pooler-app"}[5m])) topk(10, sum by (source, command) (rate(codex_pooler_repo_query_count{namespace="codex-pooler", job="codex-pooler-app"}[5m]))) ``` For memory incidents, also collect Kubernetes cgroup metrics such as `container_memory_working_set_bytes`, `container_memory_rss`, restart counters, and OOM counters from your cluster collectors. The app metrics explain BEAM and gateway behavior; cgroup and kube-state-metrics data explain container limits, restarts, and killed pods. The [Monitoring section](/monitoring/overview/) covers metric collection, the starter Grafana dashboard, PromQL recipes and runtime troubleshooting. ## Websocket replica caveat The chart defaults the web app to one replica. If `app.replicaCount` is `2` or higher, the chart treats websocket owner forwarding as required. The current chart guard rejects multi-replica app rendering unless app clustering is enabled and app pods participate in that cluster. App participation defaults to enabled, so the minimal multi-replica values shape is: ```yaml clustering: enabled: true app: replicaCount: 2 ``` With that shape, the chart renders `CODEX_POOLER_WEBSOCKET_OWNER_FORWARDING=true` on app pods automatically. Render the chart and verify the topology before raising `app.replicaCount`. ## Flux multi-replica example This example shows a Flux `HelmRepository` and `HelmRelease` for two app replicas. It assumes: - a Kubernetes Secret named `codex-pooler-secrets` exists in that namespace - the Secret contains the required release keys plus `release-cookie` - Postgres is reachable through the Secret's `database-url` - `https://codex-pooler.example.com` is the public app URL The example leaves chart defaults out of `values`. In particular, it does not set image values, `secrets.create`, `secrets.existingSecret`, app probes, rollout strategy, termination grace, worker or scheduler replica counts, or explicit websocket owner-forwarding values. ```yaml apiVersion: v1 kind: Namespace metadata: name: codex-pooler --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: name: icoretech namespace: flux-system spec: interval: 1h url: https://icoretech.github.io/helm --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: codex-pooler namespace: codex-pooler spec: releaseName: codex-pooler chart: spec: chart: codex-pooler version: "0.6.22" sourceRef: kind: HelmRepository name: icoretech namespace: flux-system interval: 5m # Allow time for migrations, pod draining, and readiness checks. timeout: 15m install: remediation: retries: 3 upgrade: remediation: retries: 3 # Restore the previous release if an upgrade fails. strategy: rollback remediateLastFailure: true values: config: host: codex-pooler.example.com ingress: enabled: true className: nginx annotations: cert-manager.io/cluster-issuer: letsencrypt hosts: - host: codex-pooler.example.com paths: - path: / pathType: Prefix tls: - secretName: codex-pooler-tls hosts: - codex-pooler.example.com monitoring: serviceMonitor: enabled: true labels: release: kube-prometheus-stack interval: 10s scrapeTimeout: 5s clustering: enabled: true participants: worker: false scheduler: false app: replicaCount: 2 affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/name: codex-pooler app.kubernetes.io/instance: codex-pooler app.kubernetes.io/component: app topologyKey: kubernetes.io/hostname ``` When `app.replicaCount` is `2` or higher, the chart enables websocket owner forwarding automatically. Worker and scheduler cluster participation is optional for dead-execution recovery: keep `clustering.participants.worker` and `clustering.participants.scheduler` disabled as in this example unless another feature needs their BEAM membership. These explicit overrides differ from the chart's participant defaults. Dead-execution recovery reads exact owner-attested PostgreSQL terminal proofs, so workers do not need an eRPC connection to the app owner. The app publishes those proofs asynchronously in bounded batches; database publication time starts their six-hour retention window. A lost or unpublished proof leaves execution death unknown and cannot authorize settlement. App-to-app clustering for websocket owner forwarding remains required in the multi-replica topology above. ## Render safely before install Render manifests locally before applying them: ```bash helm template codex-pooler icoretech/codex-pooler \ --version 0.9.4 \ --set config.host=codex-pooler.example.com \ --set ingress.enabled=true \ --set ingress.hosts[0].host=codex-pooler.example.com ``` Inspect the output for the expected app, worker, scheduler, migration, Secret reference, Service, and Ingress resources. The rendered app Deployment should serve HTTP. Worker, scheduler, and migration resources should not. For install or upgrade, provide your production values file from a private path: ```bash helm upgrade --install codex-pooler icoretech/codex-pooler \ --namespace codex-pooler \ --create-namespace \ --version 0.9.4 \ --values ./values.production.yaml ``` Keep `values.production.yaml` private if it contains secret names, internal DNS names, ingress details, or environment-specific settings.