Deploy with Helm on Kubernetes

What it installs

The ouo11y collector (the only ingest), rum-api (ClusterIP only), Grafana with the ouo11y RUM app inside the image, and either a bundled single-binary Loki, Tempo and Mimir or your own LGTM stack.

Outside

Appshttps://otel.example.com

Your teamhttps://grafana.example.com

HTTPS, Ingress controller

Ingresses

ouo-collector/v1/replay → :4319, / → :4318

ouo-grafana→ :3000

ClusterIP Services

Namespace ouo

Deployments

ouo-collectorSecret: projects htpasswd

ouo-rum-apiPVC ouo-rum-api-replays, or S3

ouo-grafanaPVC ouo-grafana

StatefulSets (bundled)

ouo-loki

ouo-tempo

ouo-mimir

Internal only

ouo-collector-internal:4317 gRPC · :13133 health · :8888 metrics

Release ouo in namespace ouo, bundled mode. Only the two Ingresses are reachable from outside the cluster.

Requirement
Kubernetes	1.25 or later
Helm	3 or 4
An Ingress controller and certificates	for example ingress-nginx and cert-manager; or a `LoadBalancer` Service
A default StorageClass	for the replay, Grafana and bundled LGTM volumes
Pull access to `ghcr.io/ouo11y/*`	during early access

Install

Create the namespace and the pull secret. The token is a classic GitHub token with only read:packages, from your shell.

kubectl create namespace ouo

kubectl create secret docker-registry ghcr -n ouo --docker-server=ghcr.io --docker-username=<github-user> --docker-password="$GHCR_TOKEN"

Create the secrets, so no credential ever sits in a values file. First a project token; keep it, your apps need it:

PT="pt_$(openssl rand -hex 16)" && echo "$PT" > web-store.token

The project registry, one {SHA} line per project:

kubectl -n ouo create secret generic ouo-projects --from-literal=projects.htpasswd="web-store:{SHA}$(printf '%s' "$PT" | openssl dgst -binary -sha1 | openssl base64)"

rum-api’s tokens, and the credential for rum-api’s own telemetry (otlp-headers; without it, rum-api’s telemetry is refused with 401):

kubectl -n ouo create secret generic ouo-rum-api-tokens --from-literal=ingest-token="it_$(openssl rand -hex 16)" --from-literal=read-token="rt_$(openssl rand -hex 16)" --from-literal=admin-token="at_$(openssl rand -hex 16)" --from-literal=otlp-headers="Authorization=Basic%20$(printf 'web-store:%s' "$PT" | base64)"

Grafana’s admin login:

kubectl -n ouo create secret generic ouo-grafana-admin --from-literal=admin-user=admin --from-literal=admin-password="$(openssl rand -base64 18)"

Save your values as my-values.yaml. Pin the image versions here, so an upgrade never moves them by surprise:

# ouo11y on Kubernetes: bundled single-binary Loki, Tempo and Mimir.
imagePullSecrets:
  - name: ghcr                      # while the ouo11y images are private
collector:
  image: { tag: "0.4.0" }
  existingProjectsSecret: ouo-projects
  allowedOrigins:
    - https://shop.example.com
  ingress:
    enabled: true
    className: nginx
    host: otel.example.com
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt
      nginx.ingress.kubernetes.io/proxy-body-size: 16m   # replay chunks are up to 8 MiB
    tls: { enabled: true, secretName: ouo-collector-tls }
rumApi:
  image: { tag: "0.5.0" }
  existingSecret: ouo-rum-api-tokens
grafana:
  image: { tag: "0.5.0" }
  existingSecret: ouo-grafana-admin
  ingress:
    enabled: true
    className: nginx
    host: grafana.example.com
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt
    tls: { enabled: true, secretName: ouo-grafana-tls }
retention:
  logs: 720h
  traces: 336h
  metrics: 2160h
  replays: 720h

Install:
Terminal window
```
helm install ouo ./ouo -n ouo -f my-values.yaml
```
helm install prints the port-forward commands, the projects, the SDK snippet and the plugin URL.
Watch the pods come up:
Terminal window
```
kubectl -n ouo get pods
```
Three Deployments and three StatefulSets, each 1/1 Running. A pod in ImagePullBackOff with 403 Forbidden in kubectl describe pod means the pull secret is missing or has no access.

Check the edge from outside: 401 without a credential, 200 with one.

curl -s -o /dev/null -w '%{http_code}\n' -X POST https://otel.example.com/v1/traces -H 'Content-Type: application/json' -d '{"resourceSpans":[]}'

curl -s -o /dev/null -w '%{http_code}\n' -X POST https://otel.example.com/v1/traces -u "web-store:$(cat web-store.token)" -H 'Content-Type: application/json' -d '{"resourceSpans":[]}'

Then open https://grafana.example.com/a/ouo11y-rum-app/overview.

The chart refuses to render without a project source (projects or collector.existingProjectsSecret) and without a Grafana admin password: there is no auth-off mode.

Exposing the collector

One host. /v1/replay goes to port 4319, everything else to 4318. The SDK’s endpoint is https://otel.example.com and replayEndpoint stays unset. Raise the controller’s body limit to at least 16 MiB for replay chunks, as in the values above.

With the Ingress on, the per-client rate limit keys on the first X-Forwarded-For hop, because every request now comes from the controller. ingress-nginx overwrites that header by default (use-forwarded-headers: false); keep it that way, or clients can pick their own rate-limit key.

collector.service.type: LoadBalancer exposes 4318 and 4319 on the address your cloud hands out. The SDK guesses the replay port when the OTLP port is 4318. There is no TLS unless the load balancer terminates it.

For a laptop or CI:

kubectl -n ouo port-forward svc/ouo-collector 4318:4318 4319:4319

kubectl -n ouo port-forward svc/ouo-grafana 3000:3000

The gRPC receiver (4317, with the same Basic credential), the health port and the self-metrics port are only on the ClusterIP Service ouo-collector-internal, never behind the Ingress. rum-api has no Ingress and no Service type option: it is always ClusterIP.

Your own LGTM stack

Set lgtm.mode: external and nothing LGTM is deployed. The collector, rum-api and Grafana’s data sources point at your Services:

lgtm:
  mode: external
  external:
    lokiUrl: http://loki-gateway.observability.svc:80
    tempoUrl: http://tempo-query-frontend.observability.svc:3200
    tempoOtlpGrpcEndpoint: tempo-distributor.observability.svc:4317
    mimirUrl: http://mimir-nginx.observability.svc:80
    # lokiOtlpUrl, mimirOtlpUrl, mimirPrometheusUrl: when write and read paths differ
    # tenantId: ouo   # multi-tenant stacks: X-Scope-OrgID on exports and data sources

Your stack needs:

Loki with allow_structured_metadata: true and these resource attributes promoted to labels (limits_config.otlp_config.resource_attributes, ignore_defaults: true): service.name, service.namespace, deployment.environment.name, ouo.platform, ouo.project.id. Without ignore_defaults, Loki’s default list promotes high-cardinality attributes from browsers.
Tempo with stream_over_http_enabled: true and an OTLP gRPC receiver.
Mimir with OTLP ingestion.
Retention set in your own charts: retention in this chart then only drives rum-api’s replays.

Already run Grafana? Set grafana.enabled: false, install the ouo11y RUM app in your Grafana 13 (it is unsigned: allow it with GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=ouo11y-rum-app), and provision it with jsonData.rumApiUrl: http://ouo-rum-api.ouo.svc:8080 and secureJsonData.readToken.

Replays on S3

rumApi:
  replicas: 2                        # more than one needs S3 (or a ReadWriteMany volume)
  storage:
    backend: s3
    s3:
      bucket: ouo-replays            # must exist; rum-api never creates it
      region: eu-west-1
      prefix: prod                   # optional
      # endpoint: minio.storage.svc:9000, forcePathStyle: true, insecure: true  for MinIO
      existingSecret: ouo-rum-api-s3 # keys access-key-id, secret-access-key; omit for pod identity
      lifecycleRules: true           # rum-api writes the bucket's expiration rules from retention.replays
  serviceAccountName: ouo-rum-api    # IRSA or workload identity, instead of keys

On AWS leave endpoint empty. With the S3 store rum-api has no volume and rolls with a RollingUpdate; on the filesystem store it uses Recreate, a few seconds of downtime per upgrade.

Values a platform team sets

Key	Default	What it does
`projects`	`[]`	`{id, token}` list rendered into the htpasswd Secret; or `collector.existingProjectsSecret`
`ingestToken`, `readToken`, `adminToken`	`""`	rum-api’s secrets; or `rumApi.existingSecret`
`collector.allowedOrigins`	`[]` (localhost only)	browser origins, for all projects; wildcards like `https://*.preview.example.com` work
`collector.replicas`	`1`	rate-limit buckets are per pod: N replicas allow N times the limits
`collector.rateLimit.project.{requestsPerSecond,burst}`	`1000`, `5000`	per project
`collector.rateLimit.client.{requestsPerSecond,burst,keyFrom}`	`100`, `300`, auto	per client; `x_forwarded_for` automatically with the Ingress
`collector.limits.{otlpMaxBodyBytes,replayMaxBodyBytes}`	4 MiB, 8 MiB	body size per receiver
`collector.memoryLimiter.{limitMib,spikeLimitMib}`	`512`, `128`	backpressure
`collector.configOverride`	`""`	a whole collector config, replacing the generated one (Customize)
`rumApi.config.{sessionInactivity,sessionMaxDuration}`	`30m`, `4h`	must match the SDKs’ session options
`rumApi.structuredConfig`	`{}`	merged over rum-api’s config file: any rum-api key
`rumApi.sampling`	`{}`	server-side replay sampling per project
`rumApi.persistence.size`	`20Gi`	the replay volume
`retention.{logs,traces,metrics,replays}`	`720h`, `336h`, `2160h`, `720h`	one entry for Loki, Tempo, Mimir (bundled) and rum-api
`retention.projects.<id>.{logs,replays}`		per project: a Loki `retention_stream` and rum-api’s per-project retention
`grafana.env`, `grafana.extraEnv`		any `GF_*` setting: SSO, an external database, SMTP
`lgtm.bundled.<loki,tempo,mimir>.persistence.size`	`20Gi`
`lgtm.bundled.<x>.configOverride`	`""`	a whole Loki, Tempo or Mimir config
`imagePullSecrets`	`[]`	`[{name: ghcr}]` during early access

Every component also takes resources, nodeSelector, tolerations, affinity, security contexts and pod annotations.

Projects and tokens

Every Deployment carries a checksum of its ConfigMaps and Secrets, so a change to a project, a token or the config rolls its pods. The collector reads the htpasswd once at start; the roll is what applies a change. See Project tokens for adding and rotating.

Uninstall

helm uninstall ouo -n ouo

Data stays on purpose: the replay and Grafana claims carry helm.sh/resource-policy: keep, and Helm never deletes StatefulSet claims. Reinstalling with the same release name reattaches them. To wipe them, list them first, then delete:

kubectl -n ouo get pvc -l app.kubernetes.io/instance=ouo

kubectl -n ouo delete pvc -l app.kubernetes.io/instance=ouo

High availability

The bundled Loki, Tempo and Mimir are single replicas on local volumes: fine for small installs, not redundant. For redundancy:

run Loki, Tempo and Mimir from the upstream grafana/loki, grafana/tempo-distributed and grafana/mimir-distributed charts, replicated, on object storage, and this chart with lgtm.mode: external;
two or more collectors (collector.replicas) behind the Ingress, with pod anti-affinity, sized for one pod’s share of the rate limits;
two or more rum-api replicas on an S3 bucket;
Grafana on an external database (grafana.env).

The Sentry bridge is the exception: it must run as exactly one replica, in its own Deployment.