domino_admin_toolkit.checks.test_node_oom_kills module

pydantic model domino_admin_toolkit.checks.test_node_oom_kills.NodeOOMKillAnalyzer

Bases: AnalyzerBase

Detects kernel-level OOM kills on cluster nodes.

Fields:

• oom_kill_threshold (int)

field oom_kill_threshold: int = 0

analyze(data)

Evaluate a single node’s OOM kill count against the configured threshold.

Return type:

list[CheckResult]

name: ClassVar[str] = 'NodeOOMKillAnalyzer'

domino_admin_toolkit.checks.test_node_oom_kills.node_oom_data(prometheus_client_v2)

Collect node OOM kill metrics from Prometheus.

Return type:

DataFrame

domino_admin_toolkit.checks.test_node_oom_kills.test_node_oom_kills(node_oom_data, ksm_node_inventory, runner)

Description: Detects kernel-level OOM kills on cluster nodes. Failure Conditions: Any node had host-level OOM kills in the last hour; node-exporter

covers too little of the cluster for the verdict to be cluster-wide.

Troubleshooting Steps:

1. Identify which node(s) had OOM kills from the output table

2. Check node memory usage: kubectl top node <node>

3. Check system process memory: ssh to node, run ps aux –sort=-rss | head

4. Check for memory leaks in system processes (containerd, kubelet)

Resolution Steps:

1. Cordon the affected node: kubectl cordon <node>

2. Drain workloads: kubectl drain <node> –ignore-daemonsets

3. Investigate root cause (containerd leak, kubelet memory, etc.)

4. Restart the affected system process or replace the node

5. If the coverage result reports missing nodes, check that the prometheus-node-exporter DaemonSet tolerates every node pool’s taints

Required Permissions: Cluster admin, node SSH access for investigation See also:

• test_node_ephemeral_storage.py — same node-exporter source, so the same coverage limits

• test_node_time_drift.py — same node-exporter source, so the same coverage limits

• info/test_node_roles.py — full node list from the Kubernetes API, for comparison