domino_admin_toolkit.checks.test_webhook_ca_bundle_expiration module

pydantic model domino_admin_toolkit.checks.test_webhook_ca_bundle_expiration.WebhookCaBundleExpiryAnalyzer

Bases: AnalyzerBase[WebhookCaBundleRow]

Validates webhook admission-config caBundle expiry with WARN/FAIL tiers.

Thresholds mirror RE-3163’s test_tls_secret_expiration defaults: an expiring caBundle is functionally identical to an expiring TLS secret cert for alerting purposes, even though the failure mode here (broken admission) is more severe than a single service’s TLS handshake failing.

Fields:
field fail_days: int = 7
field warn_days: int = 30
analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'WebhookCaBundleExpiryAnalyzer'
class domino_admin_toolkit.checks.test_webhook_ca_bundle_expiration.WebhookCaBundleRow

Bases: TypedDict

cert_issuer: str
cert_subject: str
collection_error: str
config_name: str
days_until_expiration: int | None
is_ca_injector_managed: bool
kind: str
not_after: str
not_before: str
webhook_name: str
domino_admin_toolkit.checks.test_webhook_ca_bundle_expiration.test_webhook_ca_bundle_expiration(webhook_ca_bundle_data, runner)
Description:

Checks caBundle expiry across every MutatingWebhookConfiguration and ValidatingWebhookConfiguration in the cluster. Each webhook entry’s clientConfig.caBundle (service-based or url-based — both are included, only a missing/empty caBundle is skipped) is parsed and every certificate it contains is evaluated individually, so a multi-cert caBundle surfaces one row per cert.

The caBundle signs the webhook’s serving cert; when it expires, every admission request the webhook fronts is rejected by the API server. This typically manifests as “nothing deploys, nothing admits, the cluster feels stuck” even though every workload already running looks healthy — a classic 3am incident. Unlike test_tls_secret_expiration.py (the storage layer) or test_cert_manager_resources.py (the declarative Certificate/Issuer layer), this check inspects the actual client config the API server uses to trust the webhook at admission time; the caBundle does not automatically rotate with the cert behind it unless cert-manager’s cainjector (or an equivalent mechanism) actively re-injects it.

Prototype incident OT-3385 (cloud-braze, 2025-06-17, P0): data-plane-workloads-webhook-server.domino-compute.svc’s caBundle expired 2025-04-28 and broke execution-start; discovery lagged expiry by 46 days. This check alerts on the same signal ~30 days in advance (see Warning Conditions).

is_ca_injector_managed is informational only — True when either cert-manager.io/inject-ca-from or cert-manager.io/inject-ca-from-secret is present on the parent webhook config. It does not affect the FAIL/WARN verdict: a cainjector-managed caBundle can still expire if the cainjector itself is broken (see test_cert_manager.py).

Failure Conditions:

  • A caBundle certificate has already expired

  • A caBundle certificate expires in fewer than 7 days

Warning Conditions:

  • A caBundle certificate expires in fewer than 30 days

Error Conditions:

  • The toolkit could not list MutatingWebhookConfiguration or ValidatingWebhookConfiguration objects (RBAC denial, API server error, etc.), or could not process an already-listed config’s webhook entries — surfaced as an ERROR naming which kind failed, whether the failure was in listing or processing, and why, rather than silently omitting that kind from the report. If only one kind fails, the other kind’s rows still render normally alongside this ERROR.

Troubleshooting Steps:

  1. Inspect a specific webhook’s caBundle expiry directly: kubectl get validatingwebhookconfiguration <name> -o jsonpath=’{.webhooks[0].clientConfig.caBundle}’

    base64 -d | openssl x509 -noout -dates

    (use mutatingwebhookconfiguration for mutating configs)

  2. Check whether the config carries a cainjector annotation (is_ca_injector_managed=True in this check’s output) — if so, the fix is upstream in cert-manager, not the webhook: kubectl get validatingwebhookconfiguration <name> -o jsonpath=’{.metadata.annotations}’

  3. If cainjector-managed, verify cert-manager (and specifically the cainjector controller) is healthy: see test_cert_manager.py

  4. If not cainjector-managed, find what is expected to re-inject the caBundle (a Job, a Helm hook, a manual process) and check whether it last ran successfully

  5. Domsed mutating-webhook runbook: Confluence page 1368555604 — opens with “It’s cert-manager. It’s always cert-manager.”

  6. On an ERROR naming a failed list call: verify the toolkit’s ServiceAccount has list/get RBAC on the named resource (see Required Permissions) and that the API server is otherwise healthy (kubectl get –raw /healthz)

Resolution Steps:

  1. For cainjector-managed webhooks: fix the underlying Certificate/Issuer (see test_cert_manager_resources.py) so cainjector can re-populate the caBundle; no direct edit is needed once cert-manager is healthy again

  2. For manually-managed webhooks: regenerate the caBundle and patch it onto the *WebhookConfiguration directly: kubectl patch validatingwebhookconfiguration <name> –type=json

    -p=’[{“op”:”replace”,”path”:”/webhooks/0/clientConfig/caBundle”,”value”:”<new-b64-bundle>”}]’

  3. Confirm admission is restored by re-running a request that exercises the webhook (e.g. creating the resource kind it validates/mutates)

Required Permissions:

  • kubectl read access to MutatingWebhookConfiguration and ValidatingWebhookConfiguration (cluster-scoped)

See also:

  • test_tls_secret_expiration.py — storage-layer cert expiry (RE-3163); this check reuses its parse_pem_bundle helper (lib/cert_parsing.py)

  • test_cert_manager_resources.py — cert-manager Certificate/Issuer readiness (Ticket 2); check this first when a cainjector-managed caBundle is expiring

  • test_cert_manager.py — cert-manager controller health; if cainjector is broken, this check (webhook caBundle expiry) is how you notice

  • test_mutations_patch.py — Domsed MutatingWebhookConfiguration detection; the same webhook configs this check inspects for caBundle expiry

  • cert-manager cainjector docs: https://cert-manager.io/docs/concepts/ca-injector/

  • Confluence: cert-manager runbook, page 2716729345

  • Confluence: Domsed mutating-webhook runbook, page 1368555604

  • Incident: OT-3385 (cloud-braze, P0, data-plane-workloads-webhook-server caBundle expiry)

domino_admin_toolkit.checks.test_webhook_ca_bundle_expiration.webhook_ca_bundle_data(k8s_client)

Collect webhook admission-config caBundle expiry data cluster-wide.