domino_admin_toolkit.analyzers.container_restarts module

Container restart severity classification.

Restart count is not a verdict: the same counter increments for a rollout SIGTERM, a recovered app error, an OOM kill, and a container that never started. Severity is instead recent frequency x termination reason x current readiness x namespace class.

Deliberately absent: any threshold on the lifetime restart count. Treating an old cumulative count as an incident is what made the previous generation of this check fail everywhere.

pydantic model domino_admin_toolkit.analyzers.container_restarts.ComputeRestartConcentrationAnalyzer

Bases: DataFrameAnalyzerBase

Escalates compute restarts to FAIL when they look like a shared fault.

One workload dying repeatedly is that customer’s bug. Several independent workloads dying the same forcible death on one node is a property of the node. Deeper node analysis belongs to test_node_oom_kills / test_nodes_with_container_errors.

Fields:

• min_owners (int)

• min_pods (int)

field min_owners: int = 2

field min_pods: int = 3

analyze(data)

Analyzes the full DataFrame and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: The full DataFrame. Called once per check_df() invocation.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'ComputeRestartConcentrationAnalyzer'

pydantic model domino_admin_toolkit.analyzers.container_restarts.ContainerRestartAnalyzer

Bases: AnalyzerBase[ContainerRestartRow]

Classifies one container’s restart history into a severity tier.

Compute rows never FAIL here — a customer workload crashing is not platform degradation. Escalation back to FAIL belongs to ComputeRestartConcentrationAnalyzer.

Fields:

• repeated_restarts (int)

field repeated_restarts: int = 2

analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'ContainerRestartAnalyzer'

class domino_admin_toolkit.analyzers.container_restarts.ContainerRestartRow

Bases: TypedDict

One container’s restart history, post-pandas coercion.

restarts_24h is NaN — never 0 — when Prometheus reported no rate. last_finished_within_lookback is derived in the collector to keep this module free of wall-clock dependencies.

age: str | None

container: str

is_platform: bool

last_exit_code: int | None

last_finished_at: str | None

last_finished_within_lookback: bool | None

last_terminated_reason: str | None

namespace: str

node_name: str

owner_kind: str

owner_name: str

pod_name: str

prometheus_reachable: bool

ready: bool

restarts_24h: float | None

restarts_lifetime: int

signal_source: str

started_within_lookback: bool

waiting_reason: str | None

pydantic model domino_admin_toolkit.analyzers.container_restarts.ContainerRestartSignalAvailability

Bases: DataFrameAnalyzerBase

Namespace-wide guard: does every container that restarted have a 24h restart count?

A row without one is judged on recency and readiness alone, so its missing FAIL is not an all-clear. That is the only case worth an admin’s attention. Where the count came from (Prometheus, or the pod’s own restart count when it is younger than the window) does not change what the admin should do, so it is not surfaced here. Prometheus’ own health, and whether it scrapes kube-state-metrics, is for the Prometheus checks to report.

Checked once, not per row — a per-row ERROR would produce hundreds of identical results.

Fields:

analyze(data)

Analyzes the full DataFrame and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: The full DataFrame. Called once per check_df() invocation.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'ContainerRestartSignalAvailability'