domino_admin_toolkit.checks.test_node_time_drift module

pydantic model domino_admin_toolkit.checks.test_node_time_drift.NodeTimeDriftAnalyzer

Bases: AnalyzerBase

Fields:

analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'NodeTimeDriftAnalyzer'
domino_admin_toolkit.checks.test_node_time_drift.format_time_columns(df)
domino_admin_toolkit.checks.test_node_time_drift.node_time_df(prometheus_client_v2)
Return type:

DataFrame

domino_admin_toolkit.checks.test_node_time_drift.test_node_time_drift(node_time_df, ksm_node_inventory, runner)
Description: Compares each node’s clock against the Prometheus scrape timestamp to detect

clock drift, which breaks certificate validation, token expiry and log correlation.

Failure Conditions: Any node’s clock differs from scrape time by 1 second or more;

node-exporter covers too little of the cluster for the verdict to be cluster-wide.

Troubleshooting Steps:

  1. Identify the drifting node(s) from the DIFF column in the table

  2. Confirm the node’s time service is running: ssh to the node, run timedatectl status

  3. Check NTP/chrony reachability from the node: chronyc tracking or ntpq -p

Resolution Steps:

  1. Restart the time service on the affected node (systemctl restart chronyd)

  2. Verify the node can reach its NTP servers (egress rules, security groups)

  3. If the coverage result reports missing nodes, check that the prometheus-node-exporter DaemonSet tolerates every node pool’s taints

Required Permissions: Node SSH access, cluster admin See also:

  • test_node_oom_kills.py — same node-exporter source, so the same coverage limits

  • test_node_ephemeral_storage.py — same node-exporter source, so the same coverage limits

  • info/test_node_roles.py — full node list from the Kubernetes API, for comparison