test_rabbitmq module

test_rabbitmq.test_get_rabbit_pod_logs(test_rabbit_api, retrieve_rabbit_pods)

Description:: Info check to spit the recent 100 lines from rabbit logs
Result:: String of last 100 log lines from each rabbit pod in the cluster
Public Facing KB:: https://tickets.dominodatalab.com/hc/en-us/articles/13022565244436-test-rabbitmq-py-Runbook

test_rabbitmq.test_rabbit_api()

Description:: Test the Rabbit API is returned successfully
Result:: Fails if the API does not exist this will prevent other services accessing RabbitMQ
Public Facing KB:: https://tickets.dominodatalab.com/hc/en-us/articles/13022565244436#h_01GRPYGWEYQAWV0J8P91RE9BSR

test_rabbitmq.test_rabbit_container_queues(test_rabbit_api)

Description:: Looks at rabbit queues w/ ‘container’ in the name and adds the number of consumers together. This test is applicable only when forge is enabled, and not for hephaestus (Domino 5.2) Lists each rabbit pod w/ corresponding total number of consumers More details here: https://www.rabbitmq.com/monitoring.html
Results:: Validates Rabbit has container queues.
Public Facing KB:: https://tickets.dominodatalab.com/hc/en-us/articles/13022565244436#h_01GRPYHCKN4ZCSMN834RW90762

test_rabbitmq.test_rabbit_data_plane_queues_consumers(test_rabbit_api)

Description:: Ensure Rabbit data-plane entity queue has 3 or more consumers(Dispatcher, 2 FEs) More details here: https://www.rabbitmq.com/monitoring.html
Result:: Fail if less than 3 consumers for data-plane queue found
Public Facing KB:: https://tickets.dominodatalab.com/hc/en-us/articles/13022565244436#h_01GRPYHHKQ6DS6V0FQ1F719EGF

test_rabbitmq.test_rabbit_dead_queues(test_rabbit_api)

Description:: Ensure Rabbit has no dead queues https://www.rabbitmq.com/monitoring.html
Result:: Fail if any queues not in state running returned
Public Facing KB:: https://tickets.dominodatalab.com/hc/en-us/articles/13022565244436#h_01GRPYHQSMSZJPK21XKE9N3391

test_rabbitmq.test_rabbit_get_creds()

Description:: Uses rabbitmq_client rabbit_credentials() function to fetch rabbit credentials from right secret.
Result:: Validates credentials exist as required

test_rabbitmq.test_rabbit_list_connections(test_rabbit_api)

Description:: Gets a list of all of the cluster’s current connections
Result:: Lists user, host, port, and state of each connection.
Public Facing KB:: https://tickets.dominodatalab.com/hc/en-us/articles/13022565244436#h_01GRPYJGXNW45KFDK1NDYJRGKV

test_rabbitmq.test_rabbit_local_alarm_diagnostics(test_rabbit_api)

Description:: For each rabbitmq-ha pod, runs ‘rabbitmq-diagnostics check_local_alarms’-like command
Result:: Health check that fails (returns with a non-zero code) if there are alarms in effect on the target node. Gathers information that is useful for debugging
Public Facing KB:: https://tickets.dominodatalab.com/hc/en-us/articles/13022565244436#h_01GRSA7FJG6ZVXFDFJP1BZ6V8J

test_rabbitmq.test_rabbit_node_status(test_rabbit_api)

Description:: Test the health of each rabbit node and list the status of node as a table.
Result:: Checks none status, mem, disk, file descriptor, socket and procs usage to determine health of each node in the cluster
Public Facing KB:: https://tickets.dominodatalab.com/hc/en-us/articles/13022565244436#h_01GRPYJ481SDPF74Y6Y8KM6BBT

test_rabbitmq.test_rabbit_pod_schedule(test_rabbit_api, retrieve_rabbit_pods)

Description:: Gathers info on nodes where rabbit pods are scheduled and tabulates the result. Asser when more than one rabbit pod got scheduled on the same k8s node.
Result:: Fails when more than one rabbit pods on rabbimq-ha cluster scheduled on the same worker node.
Public Facing KB:: https://tickets.dominodatalab.com/hc/en-us/articles/13022565244436-test-rabbitmq-py-Runbook

test_rabbitmq.test_rabbit_port_check_diagnostics(test_rabbit_api)

Description:: For each rabbitmq-ha pod, runs rabbitmq-diagnostics ‘check_port_connectivity’-like command
Result:: Health check that fails if any listener ports on the target node cannot accept a new TCP connection Gathers information that is useful for debugging
Public Facing KB:: https://tickets.dominodatalab.com/hc/en-us/articles/13022565244436#h_01GRSA7KTVKF2VW8WAD3TBR796

test_rabbitmq.test_rabbit_queue_depth(test_rabbit_api)

Description:: Ensure RabbitMQ queues are healthy, according to these criteria: - The depth of all queues is under 1000 messages - The number of unacknowledged messages in all queues is under 10 - There is at least one consumer for each queue More details here: https://www.rabbitmq.com/monitoring.html
Result:: Fail if any queue fails one of the criteria
Public Facing KB:: https://tickets.dominodatalab.com/hc/en-us/articles/13022565244436-test-rabbitmq-py-Runbook#h_01GRPYH6WH4BWCCZ7ZER1Z1P3C

test_rabbitmq.test_rabbit_service_stats(test_rabbit_api)

Description:: Ensure Rabbit has stats returning zero indicates an issue See more here: https://www.rabbitmq.com/monitoring.html
Result:: Fail if no stats returned
Public Facing KB:: https://tickets.dominodatalab.com/hc/en-us/articles/13022565244436#h_01GRPYH12M17T3346JNV0KPMB5

test_rabbitmq.test_rabbit_split_brain(test_rabbit_api, retrieve_rabbit_pods)

Description:: Checks to see whether any of the rabbitmq-ha pods has a different cluster name.
Results:: Fails if rabbitmq is in split brain
Public Facing KB:: https://tickets.dominodatalab.com/hc/en-us/articles/13022565244436#h_01GRPYJBMZBAR2FDP4ATZPMV88

test_rabbitmq.test_rabbitmq_feature_flags()

Description:: This function gets the Rabbit Feature flags info from the rabbitmq_admin api and list the details.
what is rabbit featureflag?: Rabbit feature flags serve as a mechanism for controlling which features are enabled or available across all nodes in the cluster.When a feature flag is enabled, the associated feature or behavior is also enabled. On the other hand, if a feature flag is not enabled, the feature or behavior will be disabled across all nodes in the cluster.
why this check required ?: This check can help identify the list of feature flags and their status in case the rabbit enters a problematic state due to disabled feature flags.
Result:: Rabbit Cluster Feature flags details in table format.
Public Facing KB:: https://www.rabbitmq.com/feature-flags.html