domino_admin_toolkit.checks.test_grafana_alerting module

pydantic model domino_admin_toolkit.checks.test_grafana_alerting.GrafanaAlertingAnalyzer

Bases: AnalyzerBase

Analyzes Grafana alerting configuration to detect unconfigured or default-only setups.

Evaluates:

  • Presence of alert rules

  • Availability of non-default contact points

  • Root notification policy configuration

  • Active channels in referenced contact points

Fields:

analyze(data)

Analyze Grafana alerting configuration.

Return type:

list[CheckResult]

Args:
data: Dictionary containing:

  • ‘_full_dataframe’: Full DataFrame with all contact points

  • ‘_meta’: Dictionary with metadata (total_rules, root_policy_exists)

Returns:

List of CheckResult objects

name: ClassVar[str] = 'GrafanaAlertingAnalyzer'
domino_admin_toolkit.checks.test_grafana_alerting.grafana_alerting_data(k8s_client)

Collect Grafana alerting configuration data

domino_admin_toolkit.checks.test_grafana_alerting.test_grafana_alerting(skip_domino_cloud, grafana_alerting_data)

Detects whether Grafana alerting is effectively unconfigured (default-only or no routing).

This check validates that Grafana alerting is properly configured to deliver alerts to real destinations. It ensures that:

  • If alert rules exist, there are custom contact points configured (not just default)

  • Contact points have active notification channels (Slack, Email, PagerDuty, etc.)

  • The root notification policy is present and routes to valid destinations

Failure Conditions:

  • Alert rules exist but NO contact points are configured

  • Alert rules exist but only default/built-in contact points are present

  • Default contact points contain placeholder email addresses (e.g., example@email.com)

  • Contact points with default configurations (empty UID, placeholder emails)

  • Root notification policy is missing

  • Root notification policy references contact points with no active channels

  • All referenced contact points have no notification channels configured

Troubleshooting Steps:

  1. Access Grafana via Domino UI or port-forward: kubectl port-forward -n domino-platform svc/grafana 3000:80

  2. Navigate to Alerting → Contact Points in Grafana UI

  3. Verify that custom contact points exist (beyond default)

  4. Check that contact points have valid notification channels configured

  5. Navigate to Alerting → Notification Policies

  6. Verify root policy routes to active contact points

Resolution Steps:

  1. Create at least one custom contact point: - In Grafana: Alerting → Contact Points → New contact point - Configure destination (Slack, Email, PagerDuty, etc.) - Test the contact point to ensure connectivity

  2. Update the root notification policy: - Navigate to Alerting → Notification Policies - Edit the default/root policy - Set it to route to your custom contact point(s)

  3. Verify configuration: - Send a test notification to confirm alerts reach destinations - Check that alert rules are assigned to appropriate policies

Required Permissions:

  • Read access to Grafana secret in platform namespace (for authentication)

  • Grafana API access for: - /api/ruler/grafana/api/v1/rules (alert rules) - /api/v1/provisioning/contact-points (contact points) - /api/v1/provisioning/policies (notification routing)