APIRoutes

The Admin Toolkit provides a REST API for interacting with the toolkit programmatically. All API routes are prefixed with /toolkit/.

Reports API

GET /toolkit/

Returns the main landing page with the list of reports.

Response: HTML page with reports list

POST /toolkit/run

Initiates a new toolkit report execution.

Query Parameters: - exclude (optional): Comma-separated list of check categories to exclude - log_level (optional): Logging level (default: “INFO”) - upload_report (optional): Whether to upload report to Domino (default: false)

Response: JSON object with report details (status code 201)

Example:

{
  "id": 1,
  "status": "In Progress",
  "started": "2023-09-19 08:11:06"
}

GET /toolkit/report/<id>

Retrieves a specific report by its ID.

Parameters: - id: Report ID

Response: HTML report file (status code 200) or 404 if not found

GET /toolkit/reports

Returns a JSON list of all reports.

Response: JSON array of report objects

Example:

[
  {
    "id": 1,
    "status": "Completed",
    "started": "2023-09-19 08:11:06",
    "local_url": "https://my_domino/toolkit/report/1"
  }
]

Documentation API

GET /toolkit/docs/

Returns the documentation index page.

Response: HTML documentation index

GET /toolkit/docs/<path>

Returns a specific documentation page.

Parameters: - path: Path to the documentation file (e.g., “Reports.html”, “Settings.html”)

Response: HTML documentation page (status code 200) or 404 if not found

Settings API

GET /toolkit/settings

Returns the settings page.

Response: HTML settings page

GET /toolkit/opt_in_domino

Opts in to sending reports to Domino.

Response: JSON response with status

GET /toolkit/opt_out_domino

Opts out of sending reports to Domino.

Response: JSON response with status

POST /toolkit/schedule

Schedules automated report generation.

Request Body: JSON object with cron schedule in standard cron format

Example:

{
  "crontab_entry": "0 0 * * *"
}

Response: JSON response with status

Status API

GET /toolkit/status

A synthetic health signal for Keycloak, Postgres, Vault, and the Domino File System (dominofs). Unauthenticated, and exposed on its own ingress to an allowlist of external monitoring IP ranges (see status.ingress in helm/values.yaml / helm/templates/status-ingress.yaml) — do not add fields that could leak sensitive cluster details to that audience.

Consumer: New Relic Synthetics, via the Domino Synthetics via Toolkit scripted API monitor defined in terramonitor (standard-deploy/modules/services/toolkit-synth-monitor). It parses the JSON body and fails the check when status.system.status is not green, listing each red service’s name and msg; it does not key on the HTTP status code. The allowlisted ranges in helm/values.yaml are its public minion ranges — see #1230, which pruned the deprecated ones.

HTTP status code: always 200, whether every service is green or any service is red — the per-service status field in the body is the source of truth, and the body is served as application/json. Treat both the status code and the body shape as a contract with that monitor: change either only after confirming what it parses, since a monitor keying off the HTTP status would read a degraded cluster as healthy, and one keying off the body would break on a renamed field.

Latency: the four probes run concurrently under a single shared deadline (PROBE_TIMEOUT_SECONDS in ui/utils/sla_aggregate.py, currently 10s) that bounds the whole request however many dependencies are hung, with each individual dependency call handed the strictly smaller PROBE_DEPENDENCY_TIMEOUT_SECONDS (currently 7s). A probe that misses the deadline is reported red with a timeout message instead of stalling the response, so pick a monitor polling interval comfortably above 10s.

Message redaction: because this endpoint is unauthenticated, the msg field is redacted before it is published — first line only, run through the shared deny-list redactor (lib/redaction.py, which catches credential-shaped text such as the password libpq echoes back from an unparsable connection string), capped at 200 characters. The unredacted text is written to the toolkit log. Curated messages such as Vault is sealed and not operational pass through unchanged.

Isolation: a single dependency being down (including an exception raised while constructing the client a probe needs) is reported as red for only that service; it never produces a 500 and never prevents the other services from being reported. If even the base Kubernetes client can’t be constructed, every service is reported red.

What each probe exercises: the Keycloak probe performs a live password grant on every poll, deliberately bypassing the admin-token cache that exists to cut HTTP calls during pytest runs. The Vault probe execs vault status in a Vault pod selected by label — a Ready one, else any Running one — rather than in a hardcoded vault-0, so a single missing replica no longer reports the whole service down. Probing Vault over HTTP (/v1/sys/health through the vault Service) would be the better shape, and was deliberately not taken here: whether Vault serves TLS, whether its Service publishes not-ready addresses (which would route to a sealed replica and invent a new false red), and whether a NetworkPolicy permits toolkit → vault:8200 all need verifying against a live cluster first. Revisit once those are known.

Resource footprint: the probes share one process-wide, bounded thread pool, each probe closes the thread-local Kubernetes API client it opened, and the dominofs probe closes the MongoClient that mongo_client.get_client() builds per call. To re-verify after changing any of that, count the waitress process’s threads either side of a batch of polls — they should match:

kubectl -n domino-platform exec <toolkit-pod> -- sh -c 'ls /proc/1/task | wc -l'
for i in $(seq 200); do curl -so /dev/null https://<host>/toolkit/status; done
kubectl -n domino-platform exec <toolkit-pod> -- sh -c 'ls /proc/1/task | wc -l'

Response:

{
  "status": {
    "services": {
      "auth": [
        {"name": "keycloak", "status": "green", "msg": "healthy", "uptime": "2:00:00"},
        {"name": "postgres", "status": "green", "msg": "healthy", "uptime": "2:00:00"},
        {"name": "vault", "status": "red", "msg": "Vault is sealed and not operational", "uptime": "down"}
      ],
      "dominofs": [
        {"name": "dominofs", "status": "green", "msg": "healthy", "uptime": "2:00:00"}
      ]
    },
    "system": {"status": "red", "msg": "vault is currently unavailable"}
  }
}

status per service: "green" only when the underlying check is PASS; every other outcome (FAIL, WARN, ERROR, or an unhandled exception) is "red" — a fail-safe default so an unrecognized future status never reads as healthy. Because of that mapping, a check must only report non-PASS for evidence of a problem, never for the absence of evidence: dominofs reports PASS when its project sample gave it nothing to read (and says so in msg), since a deployment whose sampled projects carry no README is routine, not an outage.
uptime: time since the service’s last recorded failure, or "down" while it is currently red. Backed by best-effort history (SQLite ServiceCheck rows) — a failure to write that history is logged and never fails the response, so uptime can read "unknown" if the row can’t be read back.

DaemonSet API

The DaemonSet API provides endpoints for checking node-level information when the toolkit is running in DaemonSet mode.

GET /toolkit/docker_version

Returns Docker version information for the node.

Response: JSON object with Docker version

GET /toolkit/mongo_ping

Checks MongoDB connectivity from the node.

Response: JSON object with ping status

GET /toolkit/network_stats

Returns network statistics for the node.

Response: JSON object with network statistics