domino_admin_toolkit.checks.test_node_pool_oversubscription module

pydantic model domino_admin_toolkit.checks.test_node_pool_oversubscription.NodePoolOversubscriptionAnalyzer

Bases: AnalyzerBase[NodePoolOversubscriptionRow]

Evaluates per-pool resource request totals against allocatable capacity.

Fields:
field fail_threshold: float = 100

FAIL when requests_pct is at or above this value.

field warn_threshold: float = 80

WARN when requests_pct is at or above this value.

analyze(data)

Analyzes one row and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: One row dict (TRow). The Runner calls this once per DataFrame row.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'NodePoolOversubscriptionAnalyzer'
class domino_admin_toolkit.checks.test_node_pool_oversubscription.NodePoolOversubscriptionRow

Bases: TypedDict

allocatable: float
limits_pct: float
node_count: int
node_pool: str
requests_pct: float
resource: str
total_limits: float
total_requests: float
domino_admin_toolkit.checks.test_node_pool_oversubscription.node_pool_oversubscription_data(prometheus_client_v2)

Collect per-pool CPU and memory request/limit totals from Prometheus.

Return type:

DataFrame

domino_admin_toolkit.checks.test_node_pool_oversubscription.test_node_pool_oversubscription(node_pool_oversubscription_data, runner)

Description: Aggregates CPU and memory requests and limits per Domino node pool and compares request totals against pool allocatable capacity. Surfaces pools where the scheduler has over-committed resources — the primary signal for memory triage and pending-pod incidents.

Failure Conditions:

  • A node pool’s CPU or memory requests meet or exceed 100% of pool allocatable (oversubscribed)

  • Missing requests_pct data for a pool/resource row

Troubleshooting Steps:

  1. Identify the worst-offending pool and resource from the output table (sorted by requests_pct)

  2. Correlate with the k8s-views-node-pools Grafana dashboard (RE-3038)

  3. Drill down: kubectl describe nodes -l dominodatalab.com/node-pool=<pool>

  4. Check per-pod sizing: test_k8s_platform_sizing and test_pods_mem_cpu_sizing

  5. Look for recent OOM kills or container restarts on nodes in the affected pool

Resolution Steps:

  1. Reduce resource requests on over-sized workloads in the affected pool

  2. Move workloads to a less loaded node pool or scale out the pool (add nodes)

  3. Review platform component sizing if platform pool is oversubscribed

  4. Cordon and drain severely affected nodes if active evictions are occurring

Required Permissions: Platform admin access, Prometheus read access (in-cluster), kubectl node describe

See also:

  • test_k8s_platform_sizing.py — per-pod memory/CPU sizing on platform pool nodes

  • test_pods_mem_cpu_sizing.py — per-pod memory/CPU sizing across the platform namespace

  • info/test_node_roles.py::test_node_roles — node-level enumeration and Pressure conditions

  • test_kubernetes.py::test_container_restarts — restart cadence that often correlates with oversubscription