domino_admin_toolkit.checks.test_k8s_platform_sizing module
- class domino_admin_toolkit.checks.test_k8s_platform_sizing.TestK8sPlatformSizing
Bases:
object- test_k8s_platform_pod_sizing(container_memory_df, container_cpu_df, runner, df_key, analyzers, df_analyzers, sort_col, column_order, top_n, display_filter)
- Description: Validates per-pod memory and CPU usage on platform nodes over the last hour.
Uses node-pool-scoped queries (pods on platform nodes regardless of namespace) when node pool labels are present; falls back to platform namespace scope otherwise. The memory-oom variant reads two independent OOM signals — cAdvisor’s container_oom_events_total and kube-state-metrics’ kube_pod_container_status_last_terminated_reason{reason=”OOMKilled”} — because some container runtimes export no cAdvisor OOM counter at all. When neither signal is available the variant ERRORs instead of passing: an empty OOM table without a signal behind it is a monitoring gap, not an all-clear.
- Failure Conditions:
memory-excess: Any pod’s avg memory usage exceeds its requests by more than 2 GiB. memory-oom: Any container has cAdvisor OOM events for its current instance, or
kube-state-metrics reports its last termination reason as OOMKilled. The KSM gauge is sticky for the container’s lifetime, so the table’s “Restarts 24h” column distinguishes an active OOM loop from a single historical kill.
- memory-oom (ERROR): Neither OOM signal returned any series — OOM detection is
unavailable on this cluster and the result cannot be read as “no OOM kills”.
cpu-throttling: Any pod’s CPU throttle percentage exceeds 50%. cpu-excess: Any pod’s avg CPU usage exceeds its requests by more than 2 cores.
- Troubleshooting Steps:
Check pod resource usage: kubectl top pods -n <namespace>
- Confirm the OOM kill directly:
kubectl get pod <pod> -n <namespace> -o jsonpath=’{.status.containerStatuses[*].lastState.terminated.reason}{” “}{.status.containerStatuses[*].lastState.terminated.exitCode}’
An exit code of 137 with reason OOMKilled confirms a memory kill.
- Check restart cadence:
kubectl get pods -n <namespace> –sort-by=’.status.containerStatuses[0].restartCount’
Review container resource requests/limits in the deployment or StatefulSet spec.
Check for memory leaks or unexpected load patterns via the Grafana sizing dashboard.
On an ERROR for memory-oom: verify kube-state-metrics is up and scraped, and check whether the cluster’s container runtime exposes cAdvisor OOM counters at all.
- Resolution Steps:
Adjust container memory/CPU requests and limits in the helm values.
For OOM kills: raise the memory limit, or investigate the leak if usage grows without bound.
For CPU throttling: increase CPU limits or reduce workload.
For an OOM-signal ERROR: restore the missing exporter — until then, rely on test_kubernetes.py::test_container_restarts and exit-code 137 evidence.
Required Permissions: Platform admin access, Prometheus read access (in-cluster), kubectl
- See also:
test_kubernetes.py::test_container_restarts — cumulative restart counters; the only signal that surfaced the two-month Vault OOM loop this check used to miss
test_node_oom_kills.py — kernel-level (host) OOM kills from node-exporter; a pod OOM kill with no host OOM kill means the container hit its own limit, not node pressure
test_nodes_with_container_errors.py — node-localized clusters of sick pods, sharing the same kube-state-metrics terminated-reason signal
info/test_karpenter_nodepool_memory.py — node-pool level memory headroom, for whether raising a limit will even schedule
test_hardware_tier_sizing.py — the compute-side counterpart to this platform-side check
- domino_admin_toolkit.checks.test_k8s_platform_sizing.container_cpu_df(prometheus_client_v2, platform_namespace)
Per-pod CPU metrics for platform nodes, collected once per session.
- Return type:
- domino_admin_toolkit.checks.test_k8s_platform_sizing.container_memory_df(prometheus_client_v2, platform_namespace)
Per-pod memory metrics for platform nodes, collected once per session.
- Return type: