domino_admin_toolkit.checks.test_autoscaler_errors module

pydantic model domino_admin_toolkit.checks.test_autoscaler_errors.AutoscalerErrorAnalyzer

Bases: DataFrameAnalyzerBase

Fields:
field error_threshold: int = 0

Errors above this count in any window trigger FAIL

analyze(data)

Analyzes the full DataFrame and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: The full DataFrame. Called once per check_df() invocation.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'AutoscalerErrorAnalyzer'
domino_admin_toolkit.checks.test_autoscaler_errors.autoscaler_error_data(prometheus_client_v2, autoscaler_pod_name, platform_namespace)

Collect cluster autoscaler error counts from Prometheus.

domino_admin_toolkit.checks.test_autoscaler_errors.test_cluster_autoscaler_errors(skip_karpenter_enabled, autoscaler_error_data, runner)
Description:

Checks for cluster autoscaler errors and failed scale-up events over the past 1 hour, 12 hours, and 24 hours. Non-zero error counts for either metric indicate the autoscaler is struggling to provision nodes, which can prevent workspaces and jobs from starting. Not applicable to Karpenter-enabled deployments.

Failure Conditions:

  • cluster_autoscaler_errors_total has non-zero counts in any time window

  • cluster_autoscaler_failed_scale_ups_total has non-zero counts in any time window

  • Relevant error types: cloudProviderError, otherError (see OT-3454 for credential rotation impact)

Troubleshooting Steps:
  1. Review cluster autoscaler logs for the specific error type:

    kubectl logs -n <platform-namespace> -l app.kubernetes.io/name=cluster-autoscaler –tail=200

  2. Check node group status and capacity limits in the cloud provider console (EC2 Auto Scaling Groups for AWS, Node Pools for GCP/Azure).

  3. Inspect Kubernetes events for scale-up failures:

    kubectl get events -n <platform-namespace> –sort-by=’.lastTimestamp’ | grep -i autoscal

  4. Verify autoscaler IAM role or service account permissions in the cloud provider console.

  5. Review the Cluster Autoscaler Runbook (Confluence page 1090027657) for known error patterns.

Resolution Steps:

  1. For failed scale-ups due to capacity limits: increase the node group max size or request a quota increase via your cloud provider console.

  2. For IAM/permissions errors (e.g. after credentials rotation, as in OT-3454): ensure the autoscaler service account has the required permissions to describe and modify Auto Scaling Groups (AWS) or Node Pools (GCP/Azure).

  3. For persistent errors: review the autoscaler Helm values for misconfigured expanders or conflicting node group labels and consult Domino support.

Required Permissions:

  • kubectl access to the platform namespace to read logs and events

  • Cloud provider console access to view and modify node group / Auto Scaling Group settings

See also:

  • info/test_autoscaler_scaledown.py — scaledown cooldown and unneeded-node duration checks for the same cluster autoscaler; run alongside this check when diagnosing scaling issues

  • test_autoscaler_health.py (planned — RE-3160 Ticket 2) — current-state gauges (unschedulable pods, safe_to_autoscale, last_activity, cap ratios); will complement this check’s historical error counters once it lands

  • info/test_cluster_autoscaler_cm.py — ConfigMap snapshot of cluster-autoscaler-status; useful when Prometheus is unavailable or for ad-hoc state inspection

  • test_karpenter_capacity.py — Karpenter equivalent; this check is a no-op on those clusters

  • alerts/cluster-autoscaler.yaml — Grafana alert rules for the same autoscaler signals

  • Confluence CA Runbook page 1090027657 — metrics flow, known issues, pending-pods-without-scale-up

  • OT-3454 (Essence, 2025-08-18, P1) — credential rotation broke cloud provider auth; cloudProviderError is the exact dimension this check surfaces