domino_admin_toolkit.checks.test_nodes_with_container_errors module

class domino_admin_toolkit.checks.test_nodes_with_container_errors.CollectResult(data, metrics_available, missing_metric=None)

Bases: object

data: DataFrame
metrics_available: bool
missing_metric: str | None = None
pydantic model domino_admin_toolkit.checks.test_nodes_with_container_errors.NodeContainerErrorAnalyzer

Bases: AnalyzerBase[NodeContainerErrorRow]

Flags nodes hosting more than the threshold of sick pods simultaneously.

Fields:
field min_node_age_hours: float = 1.0
field pod_names_to_show: int = 5
field threshold: int = 2
analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'NodeContainerErrorAnalyzer'
class domino_admin_toolkit.checks.test_nodes_with_container_errors.NodeContainerErrorRow

Bases: TypedDict

failed_pod_count: int
failed_pods: str
failure_reasons: str
namespace: str
node_age_hours: float
node_name: str
domino_admin_toolkit.checks.test_nodes_with_container_errors.node_container_errors_data(request, prometheus_client_v2)

Collect per-node sick pod clusters for a parametrized namespace.

Return type:

CollectResult

domino_admin_toolkit.checks.test_nodes_with_container_errors.test_nodes_with_container_errors(node_container_errors_data, runner)

Description: Detects node-localized clustering of sick pods — multiple containers in Waiting or recently terminated with error reasons on a single node at the same time. Surfaces node-level hotspots that per-pod checks miss when individual failures look benign in isolation.

Failure Conditions:

  • A node older than 1 hour hosts more than 2 sick pods in the namespace under test

  • Missing failed_pod_count data for a row

Troubleshooting Steps:

  1. Identify the worst-offending node from the output table (sorted by failed_pod_count)

  2. Correlate with the kubernetes-cni-pod-lifecycle-stability Grafana dashboard (Container Waiting Reasons panel)

  3. kubectl describe node <node> — check MemoryPressure, DiskPressure, PIDPressure, Ready

  4. kubectl get pods -n <namespace> -o wide –field-selector spec.nodeName=<node>

  5. Check node-level OOM kills and ephemeral storage: test_node_oom_kills, test_node_ephemeral_storage

Resolution Steps:

  1. Cordon the affected node: kubectl cordon <node>

  2. Drain workloads: kubectl drain <node> –ignore-daemonsets –delete-emptydir-data

  3. Investigate root cause (memory pressure, disk, kubelet, hardware)

  4. Replace or reboot the node if infrastructure issue confirmed

Required Permissions: Platform admin access, Prometheus read access (in-cluster), kubectl

See also:

  • test_kubernetes.py::test_container_restarts — per-container restart cadence for flapping workloads

  • test_kubernetes.py::test_pod_health — per-pod Failed/Pending phase failures without node aggregation

  • test_kubernetes.py::test_node_health — node Ready condition and eviction ratio

  • test_flapping_nodes.py — node Ready condition toggling (flapping) in the last 15 minutes