domino_admin_toolkit.checks.info.test_cluster_autoscaler_cm module

pydantic model domino_admin_toolkit.checks.info.test_cluster_autoscaler_cm.AutoscalerCmAnalyzer

Bases: AnalyzerBase[AutoscalerScopeRow]

Fields:
field staleness_threshold_s: float = 60.0

ConfigMap age (seconds) above which the autoscaler is treated as stale/hung.

field strict_yaml: bool = False

When True, a raw-text fallback row (unparsable YAML) is an ERROR instead of a PASS.

analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'AutoscalerCmAnalyzer'
class domino_admin_toolkit.checks.info.test_cluster_autoscaler_cm.AutoscalerScopeRow

Bases: TypedDict

cm_age_s: float | None
cm_time: str | None
health_status: str | None
max_size: float | None
min_size: float | None
raw_text: str | None
ready: float | None
registered_total: float | None
scale_down_state: str | None
scale_up_state: str | None
scope: str
sort_key: str
unready: float | None
domino_admin_toolkit.checks.info.test_cluster_autoscaler_cm.autoscaler_cm_data(k8s_client, platform_namespace, kube_system_namespace)

Collect and normalize the cluster-autoscaler-status ConfigMap.

domino_admin_toolkit.checks.info.test_cluster_autoscaler_cm.test_cluster_autoscaler_cm(skip_karpenter_enabled, k8s_client, autoscaler_cm_data, runner)
Description:

Renders the cluster-autoscaler-status ConfigMap as a structured per-scope table — one row for the cluster-wide view and one per nodegroup — with WARN markers for a stale ConfigMap, an unhealthy status, or a saturated nodegroup that isn’t scaling up. This is an informational check: it surfaces the autoscaler’s self-reported state but never FAILs. Not applicable to Karpenter-enabled or on-prem deployments.

Failure Conditions:

This info check emits WARN, never FAIL. WARN is raised when: - The ConfigMap is older than the staleness threshold (default 60s), implying a hung autoscaler. - A cluster-wide or per-nodegroup health status is not “Healthy”. - A nodegroup is saturated (ready == maxSize) while scaleUp is not InProgress — a strong

“workspaces/jobs won’t start” signal.

ERROR is raised only when the status body can’t be parsed as YAML and strict_yaml is enabled.

Troubleshooting Steps:
  1. Inspect the raw ConfigMap on the cluster:

    kubectl -n <platform-namespace> get cm cluster-autoscaler-status -o jsonpath=’{.data.status}’

    (falls back to kube-system on some deploys).

  2. For a stale ConfigMap, check the autoscaler pod is running and not crash-looping:

    kubectl get pods -n <platform-namespace> -l app.kubernetes.io/name=cluster-autoscaler

  3. For an unhealthy or saturated nodegroup, review node group capacity limits in the cloud provider console (EC2 Auto Scaling Groups for AWS, Node Pools for GCP/Azure).

  4. Review the Cluster Autoscaler Runbook (Confluence page 1090027657) for known patterns.

Resolution Steps:

  1. For saturation at maxSize: raise the node group max size or request a cloud quota increase.

  2. For a hung/stale autoscaler: restart the cluster-autoscaler deployment and review its logs.

  3. For persistent unhealthy nodegroups: reconcile the autoscaler Helm values (expanders, node group labels) and consult Domino support.

Required Permissions:

  • kubectl read access to the platform / kube-system namespace to view the ConfigMap and pods.

  • Cloud provider console access to view and modify node group / Auto Scaling Group settings.

See also:

  • test_autoscaler_errors.py — structured PASS/FAIL on CA error and failed-scale-up counters from Prometheus; the actionable companion to this snapshot of self-reported state.

  • test_autoscaler_health.py (planned — RE-3160 Ticket 2) — current-state gauges from Prometheus (unschedulable pods, safe_to_autoscale, cap ratios); same questions, different data source.

  • info/test_autoscaler_scaledown.py — scaledown cooldown / unneeded-node duration for the same CA.

  • test_karpenter_capacity.py — Karpenter equivalent; this check is a no-op on those clusters.

  • alerts/cluster-autoscaler.yaml — overlapping Grafana alert rules for the same autoscaler signals.

  • Confluence CA Runbook page 1090027657 — status format, known issues, metrics flow.