domino_admin_toolkit.checks.test_platform_pvcs module

class domino_admin_toolkit.checks.test_platform_pvcs.PlatformPvcRow

Bases: TypedDict

allow_volume_expansion: bool | None
disk_usage_pct: float | None
inode_usage_pct: float | None
persistentvolumeclaim: str
phase: str | None
requested_capacity: str | None
storage_class: str | None
pydantic model domino_admin_toolkit.checks.test_platform_pvcs.PlatformPvcUsageAnalyzer

Bases: AnalyzerBase[PlatformPvcRow]

Flags platform PVCs approaching capacity, aligned with the Grafana Platform PV Usage alert.

Fields:
field fail_threshold_pct: float = 85.0
field warn_threshold_pct: float = 80.0
analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'PlatformPvcUsageAnalyzer'
domino_admin_toolkit.checks.test_platform_pvcs.platform_pvc_usage_data(prometheus_client_v2, platform_namespace, k8s_client)

Collect platform-namespace PVC disk/inode usage from Prometheus, enriched with storage class / capacity / phase / allowVolumeExpansion from the Kubernetes API.

Return type:

DataFrame

domino_admin_toolkit.checks.test_platform_pvcs.test_platform_pvcs(platform_pvc_usage_data, runner)
Description: Checks disk and inode usage of platform-namespace PVCs against Grafana-aligned

thresholds.

Failure Conditions: Any platform PVC’s disk or inode usage is at or above 85%, matching the

Grafana “PV Usage Critical” alert. 80-84.9% is reported as a WARN and does not fail the check – it is a toolkit-only early-warning band, not a mirror of Grafana’s own “PV Usage Warning” alert (that alert is a 7-day linear-projection forecast, so a stable PVC parked at 80-84% may never trigger it, and a fast-filling PVC below 80% might).

Troubleshooting Steps:

  1. Identify the affected PVC from the table (sorted worst-first).

  2. In Grafana, open the “Kubernetes / Views / Persistent Volumes” dashboard for the same PVC to see the fill trend – a WARN with a flat trend is chronic pressure, not an active incident; a fast-rising trend is a bigger concern than the static % alone shows.

  3. Storage class, requested capacity, phase, and expandability are already in the table – no need to kubectl describe pvc for those. If they show blank, PVC/StorageClass metadata collection failed; fall back to kubectl -n <platform-namespace> get pvc <name> and kubectl get storageclass <name> -o yaml.

Resolution Steps:

  1. If the StorageClass allows expansion, resize the PVC per the platform storage runbook.

  2. If usage is from log/cache buildup rather than genuine data growth, clear the offending path from inside the pod.

  3. For domino-shared-store or other EFS/NFS-backed volumes, treat WARN as capacity-planning input – do not suppress or allowlist it away.

Required Permissions: Platform admin access, kubectl access to the platform namespace and to

cluster-scoped StorageClass objects

See also:

  • test_node_ephemeral_storage.py – node root disk usage, same node-exporter family, was the same 80% convention until this uplift

  • info/test_pvc_usage_rate.py – fill-rate trend for the same PVCs, never fails

  • Grafana “PV Usage Critical” alert and “Kubernetes / Views / Persistent Volumes” dashboard – the FAIL threshold here mirrors that alert; WARN has no Grafana equivalent