domino_admin_toolkit.checks.test_pod_health module

Pod Health Check — Owner-Aware Classification

Data source: Kubernetes API (cluster-wide pod listing, batch ReplicaSet/Job lookups for owner resolution).

Question answered: “Are there pods in the cluster with current platform degradation that requires operator attention?”

Key design decisions (RE-3490):

  • Resolves pod → owner chain (RS→Deployment, Job→CronJob) so retained historical pods are not confused with active failures.

  • FAILs only on evidence of current platform/system degradation.

  • Retained terminal pods whose controller recovered are WARN.

  • Customer compute failures are WARN (not platform degradation).

  • Every result row carries a concrete message — never the blanket “Failed pods have been detected.”

See also:

  • test_dmm_pods_list.py — DMM-specific pod presence check

  • test_kube_events.py — event-level analysis (ImagePullBackOff, registry auth)

  • test_kubernetes.py:test_node_health — node-level conditions

  • test_kubernetes.py:test_container_restarts — restart frequency analysis

pydantic model domino_admin_toolkit.checks.test_pod_health.PodHealthAnalyzer

Bases: DataFrameAnalyzerBase

Owner-aware pod health classification.

The fixture only contains unhealthy pods (Running+Ready are dropped), so this analyzer never sees healthy siblings. For controller-managed Failed pods it uses a count heuristic: a single Failed pod with no waiting reason is treated as retained history (WARN); two or more Failed pods under the same owner are treated as current degradation.

Severity rules:
FAIL — current platform/system degradation:

  • Platform pod Pending >30 min.

  • Platform pod with an actionable waiting reason (CrashLoopBackOff, etc.).

  • Platform pod Failed with no controller (nothing will replace it).

  • Two or more Failed pods under the same controller.

  • Job pod whose owning Job is currently Failed.

  • Platform pod Running-not-Ready >30 min.

WARN — retained history, startup, or non-platform:

  • Single Failed pod under a controller (likely retained).

  • Compute-namespace failures.

  • Evicted / NodeShutdown pods.

  • Terminating pods (deletion_timestamp set).

  • Running-not-Ready within the 30 min startup window.

PASS:

  • Pending within the 30 min startup window.

ERROR — unknown state (not false PASS):

  • Unclassified phase / unexpected data.

  • Job pod whose Job list call failed (status could not be determined).

Fields:

analyze(data)

Analyzes the full DataFrame and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: The full DataFrame. Called once per check_df() invocation.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'PodHealthAnalyzer'
class domino_admin_toolkit.checks.test_pod_health.PodHealthRow

Bases: TypedDict

age_minutes: int
condition_message: str | None
deletion_timestamp: str | None
is_platform: bool
namespace: str
owner_kind: str
owner_name: str
phase: str
pod_name: str
pod_reason: str | None
ready: bool
severity: str
terminated_reason: str | None
waiting_reason: str | None
domino_admin_toolkit.checks.test_pod_health.pod_health_data(k8s_client, platform_namespace, kube_system_namespace, domino_system_namespace)

Cluster-wide pod health DataFrame with resolved owners.

domino_admin_toolkit.checks.test_pod_health.test_pod_health(pod_health_data, runner)
Description:

Validates pod health across the cluster with owner-aware classification. Resolves pod ownership chains (ReplicaSet→Deployment, Job→CronJob) to distinguish current degradation from retained historical pods.

Failure Conditions:

  • Kubernetes API list_pods failure (ERROR — not treated as a healthy cluster)

  • Platform pod in CrashLoopBackOff / ImagePullBackOff

  • Platform pod Pending >30 minutes with scheduling failure

  • Platform pod Failed with no controller (e.g. helm hook) or no healthy replacement

  • Platform Job pod whose owning Job is currently Failed

  • Job pod whose Job list call failed (ERROR — status could not be determined)

  • Platform pod Running but not Ready for >30 minutes

Troubleshooting Steps:

  1. Check the output table for FAIL rows — each row has namespace, pod, owner, and reason

  2. kubectl describe pod -n <namespace> <pod-name>

  3. kubectl get events -n <namespace> –field-selector involvedObject.name=<pod-name>

  4. For scheduling failures: kubectl describe node <node-name> (check taints, capacity)

Resolution Steps:

  1. CrashLoopBackOff: inspect logs — kubectl logs -n <namespace> <pod-name> –tail=200

  2. Scheduling failures: check node capacity, taints, resource requests

  3. Failed Jobs: inspect job spec, re-trigger — kubectl get job -n <namespace>

  4. ImagePullBackOff: verify image exists and registry credentials are configured

Required Permissions:

Platform admin (kubectl read on all namespaces)

See also:

  • test_dmm_pods_list.py — DMM-specific pod presence check

  • test_kube_events.py — event-level analysis (ImagePullBackOff, registry auth)

  • test_kubernetes.py:test_node_health — node-level conditions

  • test_kubernetes.py:test_container_restarts — restart frequency analysis