domino_admin_toolkit.checks.test_nginx_ingress module

nginx-ingress data-plane health checks.

Two golden checks sharing this file:

1. test_nginx_response_codes — per-(ingress, service) HTTP response-code mix over a recent window, the same signal an SRE reads off the operations-grafana “NGINX Ingress controller” dashboard. Survives high-cardinality customer installs via a server-side topk cap plus a synthetic (other) rollup row for the suppressed tail.

2. test_nginx_config_reload — per-controller-pod config-reload status. When nginx-ingress fails to apply a new config (broken annotation, malformed Ingress, missing Secret) the pod stays Ready and silently serves the previous good config; the only signal is this gauge.

Both metrics are stable counters/gauges from upstream kubernetes/ingress-nginx and are scraped on every standard Domino install.

class domino_admin_toolkit.checks.test_nginx_ingress.ConfigReloadRow

Bases: TypedDict

controller_pod: str

last_reload_successful: int

namespace: str

seconds_since_last_reload: float | None

pydantic model domino_admin_toolkit.checks.test_nginx_ingress.NginxConfigReloadAnalyzer

Bases: AnalyzerBase[ConfigReloadRow]

FAILs any controller pod whose last config reload failed (gauge == 0).

Fields:

analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'NginxConfigReloadAnalyzer'

pydantic model domino_admin_toolkit.checks.test_nginx_ingress.NginxResponseCodeAnalyzer

Bases: AnalyzerBase[ResponseCodeRow]

Flags (ingress, service) rows whose 5xx rate is high enough to matter (rate AND count).

Fields:

• err_rate_threshold (float)

• min_5xx_floor (int)

field err_rate_threshold: float = 0.05

field min_5xx_floor: int = 10

analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'NginxResponseCodeAnalyzer'

class domino_admin_toolkit.checks.test_nginx_ingress.ResponseCodeRow

Bases: TypedDict

err_pct: float

host: str

ingress: str

is_rollup: bool

namespace: str

req_2xx: int

req_3xx: int

req_4xx: int

req_5xx: int

service: str

total: int

domino_admin_toolkit.checks.test_nginx_ingress.nginx_reload_data(prometheus_client_v2)

Collect per-controller-pod nginx-ingress config-reload status from Prometheus.

Return type:

DataFrame

domino_admin_toolkit.checks.test_nginx_ingress.nginx_response_data(prometheus_client_v2)

Collect per-(ingress, service) nginx-ingress response-code counts from Prometheus.

Return type:

DataFrame

domino_admin_toolkit.checks.test_nginx_ingress.test_nginx_config_reload(nginx_reload_data, runner)

Description: Reports each nginx-ingress controller pod’s last config-reload status. When the

controller fails to apply a new config (broken annotation, malformed Ingress, missing referenced Secret) the pod stays Ready and silently serves the previous good config — nothing else in the toolkit or the alert set notices.

Failure Conditions: Any controller pod’s nginx_ingress_controller_config_last_reload_successful

gauge is 0 (last reload failed).

Troubleshooting Steps:

1. Identify the controller pod from the output table.

2. Inspect controller logs for the reload error: kubectl logs -n <platform-namespace> <controller-pod> | grep -i reload.

3. Look for a recently applied Ingress/annotation/Secret change that the controller rejected.

Resolution Steps:

1. Fix or revert the offending Ingress resource, annotation, or missing Secret.

2. The controller reloads automatically once the config is valid; confirm the gauge returns to 1.

Required Permissions: K8s cluster access (kubectl), Grafana read access. See also:

• test_nginx_ingress.py::test_nginx_response_codes — a stale-config controller can show up there as unexpected 4xx/5xx for routes the new config would have fixed.

• test_deployment_replicas.py — confirms the controller deployment itself is healthy.

• test_container_restarts.py[platform_namespace] — a crashlooping controller is a different failure mode from a failed reload.

• info/test_platform_logs.py[nginx-ingress-controller-controller] — raw controller logs.

domino_admin_toolkit.checks.test_nginx_ingress.test_nginx_response_codes(nginx_response_data, runner)

Description: Surfaces the per-(ingress, service) HTTP response-code mix (2xx/3xx/4xx/5xx) over

the last hour from the nginx-ingress controller — the same signal an SRE reads off the

“NGINX Ingress controller” Grafana dashboard, captured into the support bundle.

High-cardinality installs are capped server-side with a topk; the suppressed tail is rolled up into an “(other)” row so a 5xx spike hiding beyond the cap still surfaces.

Failure Conditions: An (ingress, service) — or the suppressed-tail rollup — exceeds the 5xx

rate threshold (default 5%) AND the 5xx count floor (default 10) over the window.

Troubleshooting Steps:

1. Identify the failing (ingress, service) from the output table.

2. Open the “NGINX Ingress controller” Grafana dashboard, filter to that ingress/namespace, and inspect the per-status breakdown over a wider window.

3. Check the backing service’s pods: kubectl get pods -n <namespace> -l <service-selector>.

4. Review the service’s own logs/health — nginx 5xx usually mean the upstream is failing, not nginx itself.

Resolution Steps:

1. Restore the unhealthy backing service (scale up, fix crashloop, check readiness).

2. If the rollup row fired, filter Grafana by namespace to locate the suppressed offender.

3. For 4xx-heavy rows, check client/auth misconfiguration rather than service health.

Required Permissions: K8s cluster access (kubectl), Grafana read access. See also:

• test_nginx_ingress.py::test_nginx_config_reload — controller silently serving a stale config can masquerade as upstream errors here.

• test_deployment_replicas.py — a missing/under-scaled backing deployment is the most common root cause of a 5xx row.

• test_container_restarts.py[platform_namespace] — crashlooping upstream pods produce 5xx.

• test_certificate_expiration.py — TLS-layer failures present differently; rule them out.

• info/test_platform_logs.py[nginx-ingress-controller-controller] — raw controller logs.