domino_admin_toolkit.analyzers.pod_memory module

pydantic model domino_admin_toolkit.analyzers.pod_memory.K8sPodMemoryOOMDetected

Bases: AnalyzerBase

Detects OOM-killed containers from two independent signals.

  • container_oom_events_total — cAdvisor counter, scoped to the current container instance (it resets when the OOM-killed container restarts) and absent entirely on some container runtimes. A value > 0 means this container instance was OOM-killed.

  • kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} — kube-state-metrics gauge, 1 while the container’s last termination reason was an OOM kill. Read with max() and never rate(): an earlier attempt used rate() on this gauge and always got ~0, which is why the metric was previously dismissed.

FAILs when either signal is positive, with one exception: the KSM gauge is sticky for the container’s lifetime, so a container OOM-killed once and healthy ever since would stay red in every report run. When the 24h restart count is a known zero that kill is historical, and the KSM branch is downgraded to WARN (which does not fail the check). An unknown restart count stays a FAIL.

Emits no result when both signals are unknown (NaN). That blackout is reported once, cluster-wide, by K8sPodOOMSignalAvailability — a per-pod PASS would be an unfounded all-clear and a per-pod ERROR would produce hundreds of duplicates.

Fields:

analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'K8sPodMemoryOOMDetected'
pydantic model domino_admin_toolkit.analyzers.pod_memory.K8sPodMemoryRequestsVsUsed

Bases: AnalyzerBase

Analyzes per-pod memory requests versus actual sustained usage (avg over 1h).

FAILs when a pod’s memory usage exceeds its requests by more than critical_diff_bytes (default 2 GiB). Uses avg (not peak) so transient spikes don’t trigger false failures.

Fields:
field critical_diff_bytes: int = 2147483648
analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'K8sPodMemoryRequestsVsUsed'
pydantic model domino_admin_toolkit.analyzers.pod_memory.K8sPodOOMSignalAvailability

Bases: DataFrameAnalyzerBase

Cluster-wide guard: confirms at least one OOM signal actually reached the collector.

Signal availability is a property of the whole collection, not of any single pod, so it is checked once here rather than once per row — on a large cluster a per-row ERROR would produce hundreds of identical results. Without this guard, a cluster whose container runtime never exports container_oom_events_total reports an empty, green memory-oom table, which a reader takes as positive evidence that nothing is OOMing.

Fields:

analyze(data)

Analyzes the full DataFrame and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: The full DataFrame. Called once per check_df() invocation.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'K8sPodOOMSignalAvailability'