domino_admin_toolkit.checks.test_k8s_apiserver_response_codes module

Kubernetes API-server response-code breakdown check.

Replaces the old single-percentage test_k8s_api_server_request_rate (which reduced a richly-labeled metric to one cluster-wide error rate and FAILed at 5%). When that fired, the support bundle carried just that number — an engineer could see the apiserver was unhappy but not which resource type was noisy, forcing a manual Grafana topk pivot to make progress.

This check emits a per-(resource, verb, subresource) response-code table over a recent window, the same signal an SRE reads off the “Kubernetes / API server” Grafana dashboard, captured into the bundle. Same shape and architecture as the nginx-ingress response-code check (test_nginx_ingress.py::test_nginx_response_codes) — server-side topk cap plus a synthetic (other) rollup row for the suppressed tail — but sourced from apiserver_request_total.

Gate semantics (validated against a live managed control plane, see RE-3223):

The old cluster-wide gate counted all 4xx+5xx. That works in aggregate because the huge 2xx volume dilutes it, but it does NOT survive the move to per-row resolution: 404 (controller get-before-create) and 409 (optimistic-concurrency / leader-election conflict) are baseline Kubernetes control-plane chatter and dominate per-(resource, verb) 4xx rates on a perfectly healthy cluster. Tiering on raw 4xx+5xx produced ~10 false FAILs on a healthy cluster.

So the WARN/FAIL gate counts 5xx plus only the actionable 4xx codes — 401 (unauth), 403 (forbidden / misconfigured admission webhook), 422 (validation), 429 (throttling) — and deliberately excludes the benign 404 / 409 / 410. All raw 2xx/3xx/4xx/5xx counts are still shown as columns so an engineer can eyeball the full mix; only the gate is opinionated. Confirmed across three independent clusters: the raw-4xx+5xx basis false-FAILed 3-10 rows per healthy cluster (all 404/409); this basis FAILed none.

Known blind spot: because the gate is rate-based and excludes 404/409, a high-volume 404 flood (a broken operator hammering a deleted resource) or 409 storm (lease/leader-election thrash) will not FAIL even though it is a real problem. It is still visible in the 4xx column and the “(other)” rollup — a reader scanning the table will see it. A dedicated volume-based guard was deliberately not added: no observed cluster exhibited such a storm, and a speculative second threshold would be untuned. Revisit if a 404/409-storm incident is ever seen in the field.

apiserver_request_total is a core kube-prometheus-stack counter scraped on every standard Domino install. Status code 0 is the apiserver’s convention for streaming / long-running connections (WATCH, CONNECT exec); it is bucketed into its own column and excluded from the total and every rate so it never skews the gate. (Some managed control planes don’t emit 0 at all — the column is simply empty there.)

pydantic model domino_admin_toolkit.checks.test_k8s_apiserver_response_codes.ApiserverResponseCodeAnalyzer

Bases: AnalyzerBase[ApiserverResponseRow]

Tiers each (resource, verb, subresource) row on its gate rate — 5xx plus actionable 4xx (401/403/422/429), benign 404/409/410 excluded:

  • volume floor: rows under volume_floor total requests PASS quietly (chatter on rare resource types isn’t actionable). The rollup row is always evaluated regardless of floor.

  • PASS below warn_threshold

  • WARN between warn_threshold and fail_threshold

  • FAIL at or above fail_threshold (default 5% — preserves the old cluster-wide gate value)

Fields:
field fail_threshold: float = 0.05
field volume_floor: int = 50
field warn_threshold: float = 0.01
analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'ApiserverResponseCodeAnalyzer'
class domino_admin_toolkit.checks.test_k8s_apiserver_response_codes.ApiserverResponseRow

Bases: TypedDict

actionable_4xx: int
err_4xx_pct: float
err_5xx_pct: float
gate_pct: float
is_rollup: bool
req_2xx: int
req_3xx: int
req_4xx: int
req_5xx: int
req_streaming: int
resource: str
subresource: str
total: int
verb: str
domino_admin_toolkit.checks.test_k8s_apiserver_response_codes.apiserver_response_data(prometheus_client_v2)

Collect per-(resource, verb, subresource) apiserver response-code counts from Prometheus.

The topk cap and lookback window are overridable via the APISERVER_RESPONSE_TOPK_N and APISERVER_RESPONSE_WINDOW env vars so a cluster can be adjusted without a code change.

Return type:

DataFrame

domino_admin_toolkit.checks.test_k8s_apiserver_response_codes.test_k8s_apiserver_response_codes(apiserver_response_data, runner)
Description: Surfaces the per-(resource, verb, subresource) HTTP response-code mix

(2xx/3xx/4xx/5xx, plus a separate streaming column for status 0) of the Kubernetes API server over the last hour — the same signal an SRE reads off the “Kubernetes / API server” Grafana dashboard, captured into the support bundle. Replaces the old single cluster-wide error-percentage check, which buried which resource type was noisy. High-cardinality installs are capped server-side with a topk; the suppressed tail is rolled up into an “(other)” row so a spike hiding beyond the cap still surfaces.

Failure Conditions: A (resource, verb, subresource) row — or the suppressed-tail rollup —

exceeds the gate threshold (default 5% FAIL, 1% WARN) over the window. The gate counts 5xx plus actionable 4xx (401/403/422/429) and excludes benign 404/409/410 (normal control-plane get-before-create and optimistic-concurrency chatter). Rows under a volume floor (default 50 requests) are not evaluated. No apiserver_request_total series at all ERRORs the check — that means the apiserver scrape is broken, which is itself worth surfacing.

Troubleshooting Steps:

  1. Read the top of the table (sorted worst-first). The offending resource/verb is row one.

  2. A 5xx-heavy row points at the apiserver or its backing store (etcd) / a CONNECT-proxy target (kubelet). A 403/401 row points at RBAC or a misconfigured admission webhook; a 429 row points at apiserver flow-control (priority-and-fairness) shedding load.

  3. Open the “Kubernetes / API server” Grafana dashboard, filter to that resource/verb, and inspect the breakdown over a wider window to confirm it is sustained, not a blip.

  4. For a 5xx CONNECT/proxy row, check the target kubelet/node health.

  5. If the “(other)” rollup row fired, the offender is below the topk cap — pivot Grafana by resource to locate it.

Resolution Steps:

  1. 5xx from etcd pressure: check etcd health/latency and disk; relieve apiserver load.

  2. 403 from an admission webhook: fix or remove the misconfigured Validating/Mutating webhook (check its failurePolicy and CA bundle).

  3. 429 throttling: review APIService priority-and-fairness flow schemas, or the controller hammering the apiserver.

Required Permissions: K8s cluster access (kubectl), Grafana / Prometheus read access. See also:

  • test_nginx_ingress.py::test_nginx_response_codes — the same per-entity response-code breakdown on the ingress data plane; read it for client-facing 5xx.

  • test_kubernetes.py::test_k8s_api_server_responding — the simple apiserver up/down check; start there if this check ERRORs with no series.

  • test_container_restarts.py[platform_namespace] — a crash-looping controller often correlates with apiserver 5xx (it retries hard against a degraded apiserver).

  • info/test_platform_logs.py — raw control-plane component logs for the offending verb.