domino_admin_toolkit.checks.test_mongodb module

pydantic model domino_admin_toolkit.checks.test_mongodb.MongoCollectionSizeAnalyzer

Bases: DataFrameAnalyzerBase

Flags any single collection consuming an outsized share of total storage. Relative (percent-of-total) rather than an absolute byte count, since “large” is deployment-dependent and no existing alert or dashboard anchors an absolute number for this signal.

Deliberately does not flag “unexpectedly empty” collections: this ticket’s design considered an empty-collection allow-list, but building one responsibly requires knowing which of Domino’s own ~100 domino-database collections are legitimately unused on a given deployment (feature-gated, low-traffic, etc.) – not something derivable from a live-cluster snapshot. Emptiness is surfaced as an informational count only, never thresholded.

Fields:
field oversized_pct_of_total: float = 0.25

Warn when a single collection’s storage size exceeds this fraction of total DB storage

analyze(data)

Analyzes the full DataFrame and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: The full DataFrame. Called once per check_df() invocation.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'MongoCollectionSizeAnalyzer'
pydantic model domino_admin_toolkit.checks.test_mongodb.MongoOversizedSagasAnalyzer

Bases: DataFrameAnalyzerBase

Flags completed sagas whose BSON size exceeds SAGA_SIZE_THRESHOLD_BYTES. Only reached with a non-empty DataFrame – check_df’s on_empty handling (Status.PASS, see test_mongo_sagas) covers the healthy “no oversized sagas” case before this analyzer ever runs.

Fields:

analyze(data)

Analyzes the full DataFrame and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: The full DataFrame. Called once per check_df() invocation.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'MongoOversizedSagasAnalyzer'
pydantic model domino_admin_toolkit.checks.test_mongodb.MongoReplicationAnalyzer

Bases: DataFrameAnalyzerBase

Validates that every MongoDB replica-set member is in a healthy steady state with an optime that isn’t materially behind the primary, and that the replica set retains enough voting members to tolerate losing one more without losing majority. This catches a member stuck in RECOVERING with tooStale=true for months while nominally passing – the pre-fix version of this check only ever measured members already labeled SECONDARY, so a stuck member in any other state was invisible to it. Same rules as before, moved from a bare function into this analyzer so mongo_replication_data owns every external call and this class owns every threshold decision.

Fields:

analyze(data)

Analyzes the full DataFrame and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: The full DataFrame. Called once per check_df() invocation.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'MongoReplicationAnalyzer'
pydantic model domino_admin_toolkit.checks.test_mongodb.MongoSagaIndexAnalyzer

Bases: DataFrameAnalyzerBase

Validates that the sagaId index exists on domino.sagas.

Fields:

analyze(data)

Analyzes the full DataFrame and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: The full DataFrame. Called once per check_df() invocation.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'MongoSagaIndexAnalyzer'
pydantic model domino_admin_toolkit.checks.test_mongodb.MongoServerHealthAnalyzer

Bases: DataFrameAnalyzerBase

Validates storage engine choice, connection headroom, wiredTiger cache pressure, and assert counters from a single serverStatus fetch. Thresholds below are toolkit-chosen defaults — no existing Grafana alert or SLO backs them (verified against operations-grafana-service: the “Connections” and “Assert Events” panels on the MongoDB Overview dashboard have no accompanying alert rule) — override via Field if a deployment’s baseline differs.

Fields:
field cache_dirty_warn_pct: float = 0.2

Warn when dirty wiredTiger cache bytes exceed this fraction of the configured max

field connections_fail_pct: float = 0.95

Fail when used/total connections exceeds this

field connections_warn_pct: float = 0.8

Warn when used/total connections exceeds this

analyze(data)

Analyzes the full DataFrame and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: The full DataFrame. Called once per check_df() invocation.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'MongoServerHealthAnalyzer'
pydantic model domino_admin_toolkit.checks.test_mongodb.MongoStorageAnalyzer

Bases: DataFrameAnalyzerBase

Reports MongoDB storage totals and disk headroom from a single dbStats call. Thresholds are toolkit-chosen defaults – no fsUsedSize/fsTotalSize-based alert or dashboard panel exists anywhere in operations-grafana-service (verified this session), so this is presently the only point-in-time signal for Mongo’s disk headroom.

Fields:
field disk_fail_pct: float = 0.9

Fail when used/total disk exceeds this fraction

field disk_warn_pct: float = 0.8

Warn when used/total disk exceeds this fraction

analyze(data)

Analyzes the full DataFrame and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: The full DataFrame. Called once per check_df() invocation.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'MongoStorageAnalyzer'
pydantic model domino_admin_toolkit.checks.test_mongodb.MongoVersionAnalyzer

Bases: DataFrameAnalyzerBase

Validates that the server version and featureCompatibilityVersion major versions agree.

Fields:

analyze(data)

Analyzes the full DataFrame and returns a list of CheckResult instances.

Return type:

list[CheckResult]

Args:

data: The full DataFrame. Called once per check_df() invocation.

Returns:

List[CheckResult]: A list containing the results of the analysis.

Raises:

NotImplementedError: If this method is not implemented by subclasses.

name: ClassVar[str] = 'MongoVersionAnalyzer'
domino_admin_toolkit.checks.test_mongodb.mongo_collection_stats_data(mongo_client)

Collect per-collection $collStats for domino, excluding system.views and config.

Return type:

DataFrame

domino_admin_toolkit.checks.test_mongodb.mongo_dbstats_data(mongo_client)

Collect MongoDB dbStats (totals + disk headroom, one call regardless of collection count).

Return type:

DataFrame

domino_admin_toolkit.checks.test_mongodb.mongo_oversized_sagas_data(mongo_client)

Collect completed sagas exceeding the 16MB size threshold.

Return type:

DataFrame

domino_admin_toolkit.checks.test_mongodb.mongo_replication_data(mongo_client, k8s_client)

Collect per-member MongoDB replica-set status, config votes, and per-member oplog/tooStale detail.

Return type:

DataFrame

domino_admin_toolkit.checks.test_mongodb.mongo_saga_index_data(mongo_client)

Collect whether the sagaId index exists on domino.sagas.

Return type:

DataFrame

domino_admin_toolkit.checks.test_mongodb.mongo_server_status_data(mongo_client)

Collect MongoDB serverStatus + featureCompatibilityVersion (one fetch, shared by two checks).

Return type:

DataFrame

domino_admin_toolkit.checks.test_mongodb.test_mongo_replication_status(mongo_replication_data, runner)
Description:

Checks that every MongoDB replica-set member is in a healthy steady state (PRIMARY, SECONDARY, or ARBITER) with an optime that isn’t materially behind the primary, and that the replica set retains enough voting members to tolerate losing one more without losing majority.

Failure Conditions:

A member is DOWN, UNKNOWN, or REMOVED; any member (including a nominally SECONDARY one) has an optime more than 24h behind the primary or MongoDB itself reports tooStale=true for it; there isn’t exactly one PRIMARY; or the replica set has already lost majority (fault-tolerance margin below zero).

Troubleshooting Steps:

  1. From the Domino UI or kubectl get pods -n <platform-namespace> -l app.kubernetes.io/name=mongodb-replicaset, identify the member named in the failure output.

  2. Connect to the primary (kubectl exec -it <primary-pod> -n <platform-namespace> – mongosh) and run rs.status() to confirm the member’s current state, optime, and (if present) tooStale.

  3. Check the Grafana “MongoDB Replication” dashboard for the same member’s replication lag and oplog recovery window over time.

Resolution Steps:

  1. If the member is legitimately mid-resync (recently reseeded or restarted), monitor – no action needed unless it exceeds the grace window noted in the failure output.

  2. If tooStale=true, or the member has been stuck past the grace window, follow the MongoDB Runbook’s stale-member recovery procedure.

  3. If the fault-tolerance margin has reached zero, prioritize restoring the unhealthy member before any other maintenance that could take down another voting member.

Required Permissions: Platform admin access; read access to the MongoDB admin database. Public Facing KB:

See also:

  • test_mongo_version.py — MongoDB server version / FCV alignment

  • test_mongo_server_health.py — storage engine, connections, cache pressure, asserts

  • test_mongodb_sizes.py — collection size and storage breakdown

  • pre_upgrade/test_mongodb.py::test_mongodb_replicaset — the pre-upgrade-time counterpart, which also checks for split-brain across polled members

domino_admin_toolkit.checks.test_mongodb.test_mongo_sagas(mongo_saga_index_data, mongo_oversized_sagas_data, runner)
Description:

Checks that the sagaId index exists on domino.sagas, and flags completed saga records that have grown past 16MB – oversized sagas can cause Mongo to time out, and if Mongo heartbeats stop, Model API image builds may not succeed and deployments may never start.

Failure Conditions:

  • The sagaId index does not exist on domino.sagas.

  • Any completed saga’s BSON document exceeds 16MB.

Troubleshooting Steps:

  1. From the Domino UI or kubectl exec -it <mongodb-pod> -n <platform-namespace> – mongosh, run db.sagas.getIndexes() to confirm the missing index, or re-run the oversized-saga aggregation directly to inspect a flagged document’s _id/started fields.

Resolution Steps:

  1. Missing index: recreate via db.sagas.createIndex({sagaId: 1}).

  2. Oversized saga: investigate why the workflow it belongs to accumulated so much state (e.g. an unbounded retry loop) per the saga-cleanup runbook.

Required Permissions: Platform admin access; read access to the MongoDB admin database. See also:

  • test_mongo_version.py — MongoDB server version / FCV alignment

  • test_mongo_server_health.py — storage engine, connections, cache pressure, asserts

  • test_mongodb_sizes.py — collection size and storage breakdown

  • test_mongo_replication_status.py — replica-set member health

domino_admin_toolkit.checks.test_mongodb.test_mongo_server_health(mongo_server_status_data, runner)
Description:

Checks storage engine health, connection headroom, wiredTiger cache pressure, and assert counters – signal already present in serverStatus but historically discarded.

Failure Conditions:

  • storageEngine is MMAPv1 (deprecated, can lock up under big operations).

  • Connections used exceed 95% of capacity (WARN at 80%).

Troubleshooting Steps:

  1. From the Domino UI or kubectl exec -it <mongodb-pod> -n <platform-namespace> – mongosh, run db.serverStatus().connections / .wiredTiger.cache / .asserts to confirm the reported values.

  2. Check the Grafana “MongoDB Overview” dashboard’s Connections / Assert Events panels for the trend leading up to now – neither panel has an alert today, so this check is the only point-in-time signal.

Resolution Steps:

  1. For connection exhaustion: identify and fix a connection leak in a calling service, or raise net.maxIncomingConnections if the deployment is legitimately at higher scale.

  2. For MMAPv1: migrate to WiredTiger per the MongoDB storage-engine migration runbook.

Required Permissions: Platform admin access; read access to the MongoDB admin database. See also:

  • test_mongo_version.py — server version / FCV alignment (shares this check’s serverStatus fetch)

  • test_mongo_replication_status.py — replica-set member health

  • test_mongodb_sizes.py — collection size and storage breakdown

domino_admin_toolkit.checks.test_mongodb.test_mongo_version(mongo_server_status_data, runner)
Description:

Retrieves the version of Mongo and its FeatureCompatibilityVersion, and checks that the two major versions agree.

Failure Conditions:

  • The server version’s major version doesn’t match featureCompatibilityVersion’s major version (a sign FCV was never raised after an upgrade).

  • serverStatus or featureCompatibilityVersion couldn’t be retrieved.

Troubleshooting Steps:

  1. From the Domino UI or kubectl exec -it <mongodb-pod> -n <platform-namespace> – mongosh, run db.version() and db.adminCommand({getParameter: 1, featureCompatibilityVersion: 1}) to confirm the reported mismatch.

Resolution Steps:

  1. If FCV is behind the server’s major version after a completed upgrade, raise it via db.adminCommand({setFeatureCompatibilityVersion: “<version>”}) per the MongoDB upgrade runbook for the target version.

Required Permissions: Platform admin access; read access to the MongoDB admin database. Public Facing KB:

See also:

  • test_mongo_server_health.py — connection headroom, storage engine, cache pressure, assert counters (shares this check’s serverStatus fetch)

  • test_mongo_replication_status.py — replica-set member health

  • pre_upgrade/test_mongodb.py::test_mongo_version — the pre-upgrade-time re-export of this same check

domino_admin_toolkit.checks.test_mongodb.test_mongodb_sizes(mongo_dbstats_data, mongo_collection_stats_data, runner)
Description:

Retrieves MongoDB storage totals, disk headroom, and per-collection size/count breakdown for the domino database.

Failure Conditions:

  • Disk usage (fsUsedSize/fsTotalSize) exceeds 90% (WARN at 80%).

  • A single collection exceeds 25% of total DB storage (WARN only – no existing alert to anchor a FAIL to; this is new triage signal, not a known incident pattern).

Troubleshooting Steps:

  1. From the Domino UI or kubectl exec -it <mongodb-pod> -n <platform-namespace> – mongosh, run db.stats() to confirm the reported totals, or db.<collection>.aggregate([{$collStats: {storageStats: {}}}]) for a specific flagged collection.

Resolution Steps:

  1. For disk pressure: expand the underlying PVC/volume per the MongoDB storage-scaling runbook.

  2. For a single collection dominating storage: investigate why (e.g. a missing TTL index or runaway writes) per the MongoDB collection-growth runbook.

Required Permissions: Platform admin access; read access to the MongoDB admin database. See also:

  • test_mongo_version.py — MongoDB server version / FCV alignment

  • test_mongo_server_health.py — storage engine, connections, cache pressure, asserts

  • test_mongo_replication_status.py — replica-set member health

  • test_mongo_sagas.py — saga index + oversized saga documents

  • test_central_config.py — domino.config emptiness (excluded from this check’s collection set)