domino_admin_toolkit.analyzers.pod_memory module

pydantic model domino_admin_toolkit.analyzers.pod_memory.K8sPodMemoryLimitHeadroom

Bases: AnalyzerBase[MemoryLimitHeadroomRow]

Analyzes peak memory usage against the container’s memory limit.

The kernel OOM-kills on the limit, so headroom-to-limit is the axis that predicts a kill. K8sPodMemoryRequestsVsUsed measures the other axis — excess over request — and structurally cannot surface a container whose request is modest but whose limit is tight: on one production cluster vault-0/vault-1 were OOMKilled ~4x/day for two months without ever entering the excess-ranked top 20, so the sizing table showed no Vault row at all.

Reads peak and not the average: OOM kills are driven by transient spikes that a healthy-looking average hides. Complements K8sPodMemoryOOMDetected (containers already dying) rather than replacing it — working set is only a proxy for what the kernel charges against the limit, and a 1h peak has scrape-interval resolution.

A container with no limit, or with no measured peak, PASSes: neither is evidence of a problem, and a per-row ERROR would redden the check once per limitless sidecar. Node-pressure kills are test_node_oom_kills.py’s job.

Fields:
field critical_limit_pct: float = 90.0

Peak memory as a percentage of the container limit at which an OOM kill is likely

analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'K8sPodMemoryLimitHeadroom'
pydantic model domino_admin_toolkit.analyzers.pod_memory.K8sPodMemoryOOMDetected

Bases: AnalyzerBase

Detects OOM-killed containers from two independent signals.

  • container_oom_events_total — cAdvisor counter, scoped to the current container instance (it resets when the OOM-killed container restarts) and absent entirely on some container runtimes. A value > 0 means this container instance was OOM-killed.

  • kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} — kube-state-metrics gauge, 1 while the container’s last termination reason was an OOM kill. Read with max() and never rate(): an earlier attempt used rate() on this gauge and always got ~0, which is why the metric was previously dismissed.

FAILs when either signal is positive, with one exception: the KSM gauge is sticky for the container’s lifetime, so a container OOM-killed once and healthy ever since would stay red in every report run. When the 24h restart count is a known zero that kill is historical, and the KSM branch is downgraded to WARN (which does not fail the check). An unknown restart count stays a FAIL.

Emits no result when both signals are unknown (NaN). That blackout is reported once, cluster-wide, by K8sPodOOMSignalAvailability — a per-pod PASS would be an unfounded all-clear and a per-pod ERROR would produce hundreds of duplicates.

Fields:

analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'K8sPodMemoryOOMDetected'
pydantic model domino_admin_toolkit.analyzers.pod_memory.K8sPodMemoryRequestsVsUsed

Bases: AnalyzerBase

Analyzes per-pod memory requests versus actual sustained usage (avg over 1h).

FAILs when a pod’s memory usage exceeds its requests by more than critical_diff_bytes (default 2 GiB). Uses avg (not peak) so transient spikes don’t trigger false failures.

Fields:
field critical_diff_bytes: int = 2147483648
analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'K8sPodMemoryRequestsVsUsed'
pydantic model domino_admin_toolkit.analyzers.pod_memory.K8sPodOOMSignalAvailability

Bases: DataFrameAnalyzerBase

Cluster-wide guard: confirms at least one OOM signal actually reached the collector.

Signal availability is a property of the whole collection, not of any single pod, so it is checked once here rather than once per row — on a large cluster a per-row ERROR would produce hundreds of identical results. Without this guard, a cluster whose container runtime never exports container_oom_events_total reports an empty, green memory-oom table, which a reader takes as positive evidence that nothing is OOMing.

Fields:

analyze(data)

Analyzes the full DataFrame and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: The full DataFrame. Called once per check_df() invocation.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'K8sPodOOMSignalAvailability'
class domino_admin_toolkit.analyzers.pod_memory.MemoryLimitHeadroomRow

Bases: TypedDict

Container: str
Pod: str