domino_admin_toolkit.checks.test_cert_manager_resources module

pydantic model domino_admin_toolkit.checks.test_cert_manager_resources.CertManagerCertificateAnalyzer

Bases: AnalyzerBase[CertificateRow]

Validates Certificate CR readiness and expiry tiers.

Fields:
field fail_days: int = 7
field warn_days: int = 30
analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'CertManagerCertificateAnalyzer'
class domino_admin_toolkit.checks.test_cert_manager_resources.CertificateRow

Bases: TypedDict

days_until_expiration: float | None
days_until_renewal: float | None
issuer_kind: str
issuer_name: str
issuing: bool | None
name: str
namespace: str
not_after: str
ready: bool | None
ready_reason: str
renewal_time: str
secret_name: str
pydantic model domino_admin_toolkit.checks.test_cert_manager_resources.IssuerReadinessAnalyzer

Bases: AnalyzerBase[IssuerRow]

Validates Issuer and ClusterIssuer readiness.

Fields:

analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'IssuerReadinessAnalyzer'
class domino_admin_toolkit.checks.test_cert_manager_resources.IssuerRow

Bases: TypedDict

kind: str
name: str
ready: bool | None
ready_reason: str
scope: str
type: str
domino_admin_toolkit.checks.test_cert_manager_resources.cert_manager_certificates_data(_cert_manager_collection)

Certificate CR readiness data (one row per Certificate CR).

domino_admin_toolkit.checks.test_cert_manager_resources.cert_manager_issuers_data(_cert_manager_collection)

Issuer and ClusterIssuer readiness data (one row per Issuer/ClusterIssuer).

domino_admin_toolkit.checks.test_cert_manager_resources.test_cert_manager_certificates(cert_manager_certificates_data, runner)
Description:

Checks the readiness and expiry of Certificate custom resources managed by cert-manager. Reads status.conditions[Ready], status.conditions[Issuing], status.notAfter, and status.renewalTime from each Certificate CR across all namespaces. This is the control-plane view — it complements test_tls_secret_expiration.py (storage layer) and catches problems that the secret-level check cannot: an unissued CR, a cert stuck mid-renewal, or a cert that cert-manager considers valid but whose renewalTime has elapsed.

Known silent-failure class (OT-3440 / DOM-71255 / cert-manager#5864): cert-manager can report Ready=True while the backing secret has a CA/cert mismatch after CA rotation. This check records not_after in the output so a future consumer can cross-check it against test_tls_secret_expiration.py data, but does not implement the cross-check itself. If this check says PASS but certs behave broken, check test_tls_secret_expiration.py and info/test_certificate_inventory.py.

Failure Conditions:

  • Certificate CR is not Ready and not mid-renewal

  • Certificate expires in fewer than 7 days

Warning Conditions:

  • Certificate is mid-renewal (Ready=False, Issuing=True) — transient during rotation

  • Certificate expires in fewer than 30 days

  • Certificate is past its renewalTime but still Ready=True (stuck renewal)

Error Conditions:

  • Certificate CR has no status.conditions — controller has not reconciled

Troubleshooting Steps:

  1. Inspect the Certificate CR directly: kubectl describe certificate -n <ns> <name>

  2. Check cert-manager controller logs for reconcile errors: kubectl logs -n cert-manager deploy/cert-manager –tail=100

  3. Verify the referenced Issuer/ClusterIssuer is Ready: kubectl describe issuer -n <ns> <issuer-name> (see test_cert_manager_issuers for a cluster-wide readiness view)

  4. Trigger manual renewal if the cert is stuck: kubectl annotate certificate -n <ns> <name> cert-manager.io/issueOnce=true

  5. Confluence cert-manager runbook: page 2716729345

Resolution Steps:

  1. For an unready Issuer: fix the Issuer config (ACME credentials, CA secret, etc.)

  2. For a stuck renewal: delete the Certificate CR’s associated CertificateRequest and Secret to force cert-manager to re-issue

  3. For imminent expiry: trigger manual renewal (see step 4 above)

Required Permissions:

  • kubectl read access to Certificate CRs in all namespaces

  • cert-manager RBAC (cluster-scoped list on certificates.cert-manager.io)

See also:

  • test_tls_secret_expiration.py — storage-layer expiry check (RE-3163); cross-check not_after here against the actual x509 to catch OT-3440-class CA/cert mismatches

  • test_cert_manager_issuers (below) — Issuer/ClusterIssuer readiness; check this first when certificates are stuck mid-renewal

  • test_cert_manager.py — cert-manager controller deployment count and CRD health

  • test_domino_url_cert.py — live TLS handshake against the Domino frontend

  • info/test_certificate_inventory.py — full chain validation with x509 parsing

  • Confluence: cert-manager runbook, page 2716729345

  • Confluence: Domsed webhook runbook, page 1368555604 (covers operator-webhook-certificate)

  • Incidents: OT-3440, DOM-71255, cert-manager#5864 (Ready=True / secret mismatch)

domino_admin_toolkit.checks.test_cert_manager_resources.test_cert_manager_issuers(cert_manager_issuers_data, runner)
Description:

Checks the readiness of all Issuer and ClusterIssuer custom resources managed by cert-manager. An unready Issuer silently breaks future certificate renewals — existing certs look fine today but will not renew when they expire. This check surfaces Issuer problems before they become cert-expiry incidents.

Empty ClusterIssuer list is not an error — some clusters use only namespaced Issuers (validated on paulle122101: 15 Issuers, 0 ClusterIssuers).

Failure Conditions:

  • Issuer or ClusterIssuer is not Ready

Error Conditions:

  • Issuer has no status.conditions — controller has not reconciled

Troubleshooting Steps:

  1. Inspect the Issuer directly: kubectl describe issuer -n <ns> <name> kubectl describe clusterissuer <name>

  2. Check ready_reason and the cert-manager controller logs for details: kubectl logs -n cert-manager deploy/cert-manager –tail=100

  3. For ACME issuers: verify DNS/HTTP-01 challenge reachability and ACME account registration (check status.acme.uri)

  4. For CA issuers: verify the referenced CA secret exists and is valid

  5. If this check fails while test_cert_manager_certificates passes, existing certs are still valid but renewals are blocked — fix the Issuer before any cert expires

  6. Confluence cert-manager runbook: page 2716729345

Resolution Steps:

  1. Fix the Issuer config (ACME credentials, CA secret reference, Vault token, etc.)

  2. Delete and re-create the Issuer if it is stuck in an unrecoverable error state

  3. Check cert-manager controller health (test_cert_manager.py) if all Issuers are failing simultaneously — the controller may be down or have lost API connectivity

Required Permissions:

  • kubectl read access to Issuer CRs in all namespaces

  • kubectl read access to ClusterIssuer CRs (cluster-scoped)

  • cert-manager RBAC (list on issuers.cert-manager.io, clusterissuers.cert-manager.io)

See also:

  • test_cert_manager_certificates (above) — Certificate CR readiness; an unready Issuer causes new Certificate CRs to stay stuck in a non-Ready state

  • test_cert_manager.py — cert-manager controller deployment count and CRD health; check this if all Issuers are simultaneously unready

  • test_tls_secret_expiration.py — storage-layer cert expiry (RE-3163)

  • Confluence: cert-manager runbook, page 2716729345

  • Confluence: Domsed webhook runbook, page 1368555604