Troubleshoot a Kubernetes Installation | NVIDIA Minerva User Guide

Use this page to diagnose a failed or unhealthy Kubernetes installation and to collect logs for NVIDIA Support.

Where to Start

Begin with the failing resource's conditions and the recent warnings. Search all namespaces: the warnings that matter are often not in the namespace you're looking at.

kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp | tail -30
kubectl -n minerva-infra describe minervainfra minerva-infra | tail -40
kubectl -n minerva-system logs deploy/minerva-operator-controller-manager --tail=100

Expected warnings. A successful installation leaves about 20 Warning events behind, from startup probes on Kafka, MySQL, Neo4j, the object store, Druid, Keycloak, and several application services. They're historical. What matters is whether the object is Ready now, not whether it ever logged a warning.

Common Problems

Symptom	Cause and fix
`namespaces "minerva-*" not found`, but the namespaces exist	Wrong cluster. Re-export `KUBECONFIG` and check `kubectl config current-context`
`ImagePullBackOff` with `unauthorized`	The pull secret is missing or wrong in that namespace (see Create the Namespaces and Pull Secrets), the credential expired, or the node runtime doesn't trust the registry TLS
`ImagePullBackOff` with `404 not found`	The image is missing from your registry (see Import the Images into Your Registry), or a required setting was removed from the custom resource: `spec.version` unset, `mysql.imageRepository` removed, or `flatten` changed
The operator pod exits at startup, mentioning cert-manager	The cert-manager API isn't served. cert-manager is a hard prerequisite. Also confirm that its controller and your issuer are healthy
Helm rejects the CRDs on annotation size	Apply the CRDs first with `kubectl apply --server-side -f work/chart/minerva-operator/files/crds/`, then install with `crds.install: false`, as in Install the NVIDIA Minerva Operator
`failed to download openapi: the server could not find the requested resource`	A cluster APIService is unavailable (`kubectl get apiservices \| grep -v True`). Run the command again with `--validate=false`
`keycloak-operator` in CrashLoopBackOff with `Couldn't start informer … Not Found`	One of the four Keycloak CRDs is missing. Apply the overlay again (see Install the Keycloak Operator)
`keycloak-operator` runs, but every reconcile is forbidden	The ClusterRoleBinding subject still names the upstream namespace: the third patch in `kustomization.yaml` is missing (see Install the Keycloak Operator)
Identity reports `KeycloakCRDsNotRegistered`	Normal for about 60 seconds after apply. If it persists, check the Keycloak Operator (see Install the Keycloak Operator)
Identity reports `TrustedCABundleMissing`, or pods wait on a `ca-certs` mount	The `ca-certs` Secret is missing in that namespace (see Create the Certificates, Trust Bundle, and License)
TLS errors inside application pods	`ca-certs` is empty, or the UI certificate lacks the in-cluster names (see Create the Certificates, Trust Bundle, and License)
`Could not read certificate` right after you create the UI certificate	With cert-manager, most likely the certificate isn't issued yet. Wait for the Certificate to become `Ready`, and if it doesn't, check its events and your issuer. With your own certificates, check the Secret's `tls.crt` content
The application reports `waiting for MinervaIdentity Ready=True`	The ordering gate. Diagnose the identity tier. Don't disable the gate
Sign-in sends the browser to an internal or wrong address	A gateway in front of the UI rewrote the `Host` header or dropped the `X-Forwarded-*` headers. Keycloak builds the sign-in addresses from them (see Prepare the External Addresses)
Identity is `Ready`, but login fails after the password prompt	Check the redirect URIs and web origins of `ui-react` (see Sign In and Set the Administrator Password), and that the user belongs to a tenant with a role, or has the Platform Admin role
`BadConfig: Certificate will be issued with an empty Issuer DN` in `minerva-system`	From the operator chart's own bootstrap issuer. The certificates are issued and the operator is `Ready`. No action is needed
The license state is `invalid`	No usable license was found. Check that the ConfigMap is `minerva-license` in `minerva-app` with the key `license.lic`, then restart `deploy/license` and repeat the license check in Validate the Installation
Druid coordinator or router stuck on a wrong image while the broker restarts waiting for them	A rollout deadlock: the Druid operator waits for the broker, and the broker needs the coordinator. Correct the setting in the custom resource first, then set the correct image directly on the affected StatefulSets and let them roll
nginx OOMKilled (exit code 137) with clean logs on nodes with many CPU cores	nginx sizes its workers from the node's CPU count. Raise the nginx memory limits in the application file
PVC `Pending`	The StorageClass can't provision. Check that your class is the cluster default and, on node-local storage, the free disk on each node
LoadBalancer address `<pending>`	No address pool, or a pool whose name doesn't match the annotation in the application file (see Prepare the External Addresses)
Address assigned but unreachable	The network path or `loadBalancerSourceRanges`. See Reaching the UI and the Collectors below
A telemetry source sends data, but nothing appears in the UI	The collector accepts well-formed OTLP with `HTTP 200` even when it then discards the data: it processes only data from supported sources. The first sample of a cumulative counter is also dropped, and data timestamped outside the queried time window doesn't show
Trends and dashboards show nothing for the last few minutes of a steady metric stream. Those minutes fill in later with their original timestamps, or stay empty	Druid ingestion for that table is behind its metric stream. By default, each table is ingested by one task, which is limited to one CPU core, and the platform raises no event when it falls behind. Data older than the Kafka retention window (24 hours or 1 GiB per partition in the reference configuration) isn't recovered, so act early. Raise that table's ingestion task count as described under "Druid ingestion" in the install guide (`README.md` in the release archive)

Reaching the UI and the Collectors

Depending on your network, a load-balancer address may be reachable only from some networks. For example, a layer-2 address on the cluster's subnet is reachable only from that subnet unless your network routes to it from elsewhere. Clients on other networks then need a site-specific path: routing, NAT, or an ingress route. This is network work outside the platform, and it's easy to mistake for a broken installation.

Confirm the platform side first with the front-door check in Validate the Installation, which bypasses the address entirely. HTTP 200 there means the front door is serving. If clients still can't reach the address, look at the network path. As a control, test an address you already reach successfully from the same place. If that one is unreachable too, you're testing from the wrong place rather than looking at a fault.

Check the allowed source ranges. Both Services silently drop traffic from outside their ranges, and the symptom looks exactly like a routing failure:

kubectl -n minerva-app get svc minerva-app-nginx minerva-app-otel-data-lb \
  -o custom-columns=NAME:.metadata.name,IP:.status.loadBalancer.ingress[0].ip,HOSTNAME:.status.loadBalancer.ingress[0].hostname,ALLOWED:.spec.loadBalancerSourceRanges

Mind NAT: the address your clients present after NAT, not their own address, must fall inside the range.

Collect Logs for NVIDIA Support

kubectl logs reaches back only a short time on a busy pod. NVIDIA Minerva services also forward their logs to the platform's log collector, which keeps a longer history, and NVIDIA Support usually needs that archive. To collect it:

kubectl -n minerva-infra exec deploy/minerva-fluentd -- \
  tar cz -C /fluentd/log --exclude='*/buffer.*' . > minerva-logs.tgz

Some infrastructure components, such as Kafka, MySQL, Neo4j, and nginx, write only to their container output, which this archive doesn't hold. Send the archive together with:

the output of the Where to Start commands above
kubectl logs for the affected pods
kubectl logs --previous for a pod that has just crashed

Last updated: October 29, 2026