domino_admin_toolkit.checks.test_kubernetes module

Description: Checks a number of kubernetes components health Result: Please see individual check for description and result

class domino_admin_toolkit.checks.test_kubernetes.RestartCollection(data, failed_namespaces=())

Bases: object

Collected rows plus the namespaces that could not be listed.

Separate because an empty frame otherwise reads as “nothing restarted”, making an RBAC-blocked namespace indistinguishable from a healthy one.

data: DataFrame
failed_namespaces: tuple[str, ...] = ()
class domino_admin_toolkit.checks.test_kubernetes.RestartRateMaps(regular, init)

Bases: object

Per-kind Prom results. None on a field means that query failed.

init: dict[tuple[str, str, str], float] | None
regular: dict[tuple[str, str, str], float] | None
property unreachable: bool
domino_admin_toolkit.checks.test_kubernetes.container_restart_data(request, k8s_client, platform_namespace, kube_system_namespace, compute_namespace)

Restart history for every namespace the check reports on, collected once.

Session-scoped: all four parametrized instances share this pass, so only the first pays. Scope comes from _restart_namespaces, not the skipping field_namespace fixture, so a missing optional namespace cannot suppress the other three.

Prometheus is fetched defensively, not as a normal fixture argument: prometheus_client_v2 raises at construction when Prometheus is unreachable, and that would fail this fixture before _collect_container_restart_data ever got a chance to fall back to Kubernetes-only classification — exactly the situation the fallback exists for.

domino_admin_toolkit.checks.test_kubernetes.test_component_status(component, k8s_client)
Description:

Validate the scheduler, controller manager and etcd are healthy

Result:

If any component is marked as failed then the test will fail

domino_admin_toolkit.checks.test_kubernetes.test_container_restarts(ns, request, container_restart_data, runner)
Description:

Classifies container restarts in this namespace over the last 24 hours. Severity is the product of how often a container restarted inside that window, why it last terminated, whether it is passing its readiness probe now, and whether it belongs to the platform or to a customer workload. A lifetime restart count never drives a verdict on its own.

Failure Conditions:

  • A platform container in CrashLoopBackOff

  • A platform container that is not Ready after repeated restarts in the window

  • A platform container repeatedly ending in forcible termination (OOMKilled, 137, 139)

  • Several independent compute workloads forcibly terminated on the same node, which points at the node rather than at the workloads

  • A restarting container older than 24h with no 24h restart count (Prometheus unreachable or missing its data), since that row is judged only on when it last terminated and a missing FAIL is not an all-clear

Troubleshooting Steps:

  1. Read the table: each row names the namespace, pod, container, owning workload, node, restarts in the window, and how the container last died.

  2. kubectl describe pod -n <namespace> <pod> — confirm the current waiting reason.

  3. kubectl logs -n <namespace> <pod> -c <container> –previous — read the crash.

  4. In Grafana, plot container_memory_working_set_bytes for the pod when the exit code is 137, to confirm whether it hit its memory limit.

  5. kubectl get events -n <namespace> –field-selector involvedObject.name=<pod>

  6. If 24h restart counts are unavailable: confirm prometheus-server in the platform namespace is running, then query kube_pod_container_status_restarts_total for the namespace in Prometheus or Grafana to confirm kube-state-metrics is being scraped. For a pod younger than 24h the table uses the higher of Prometheus’ count and Kubernetes’ own restart count, so those rows still have a count when Prometheus missed restarts or is down. Prometheus and Grafana may show fewer restarts than the table for such a pod if Prometheus was not scraping when they happened (for example, the whole platform restarting at once). Kubernetes’ count can reset to 0 when a node restarts, so the table can show more restarts than kubectl get pods for that pod.

Resolution Steps:

  1. OOMKilled / exit 137: raise the container memory limit, or reduce its workload.

  2. CrashLoopBackOff: fix the startup failure the previous-container logs show — missing config, unreachable dependency, failed migration.

  3. Exit 139 (segfault): capture the stack trace and escalate with the image version.

  4. Repeated forcible terminations across workloads on one node: cordon the node and check kernel OOM kills and memory pressure on it.

  5. Compute-namespace WARN rows are customer workload failures — report them to the project owner rather than treating them as a platform defect.

Required Permissions:

Platform admin (kubectl read across namespaces, Grafana access)

See also:

  • test_pod_health.py — current pod state (Pending/Failed/not-Ready), owner-aware

  • test_k8s_platform_sizing.py — memory headroom and the OOM signal behind exit 137

  • test_node_oom_kills.py — kernel-level OOM kills, when restarts cluster on one node

  • test_nodes_with_container_errors.py — per-node aggregation of container failures

  • test_kube_events.py — ImagePullBackOff and registry auth, for restarts that never start

  • test_dmm_pods_list.py — DMM pod presence and restart context

domino_admin_toolkit.checks.test_kubernetes.test_get_version(k8s_client)
Description:

Checks we have a valid kubernetes version

Result:

Shows current kubernetes version, fails if non returned

domino_admin_toolkit.checks.test_kubernetes.test_k8s_api_server_responding(k8s_client)
Description:

Tries to query the Kubernetes API server version to ensure the API server is responding to requests.

Results:

Fails if there is a problem connecting to or querying the API server

domino_admin_toolkit.checks.test_kubernetes.test_kube_dns_health(k8s_client)
Description:

Retrieves kube-dns pods in running state

Results:

Fails if there are no kube-dns pods in running phase

domino_admin_toolkit.checks.test_kubernetes.test_node_health(k8s_client)
Description:

Checks node health and is considered “failed” if met with any of these conditions: - The node is ready but status is not “True”. Can be stuck ‘pending’ in scaling, But not more than 30 miuntes - The total number of evicted pods is more than 10% of the total pods for a single node

Result:

Table prints failed nodes: - Node name - Internal node IP - Node condition type - Node condition status - Node Label - All the node condition statuses

domino_admin_toolkit.checks.test_kubernetes.test_os_version(k8s_client)
Description:

Checks for node operating system version

Result:

Shows current OS version, fails if nothing is returned