Test Alerting Flow with the API¶
The alerting integration needs to be properly configured. The default installation provides the required ConfigMaps, and in case of errors please refer to the production installation guide
Firing a test alert¶
The Seldon Deploy alerting integration is built with a flexible architecture and allows users to monitor and define SLAs, SLOs, and SLIs for their models and the operation of the platform. Prometheus metrics exposed by Deploy and the models form the basis of SLIs and defined alerts form the SLOs.
In this example, we’ll demonstrate how to push a test alert manually through the API to provide intuition on what would happen if SLO is breached.
Using the API to fire a test alert
You can use the API to fire a test alert. If you have configured Alertmanager correctly this will then show up in the Deploy frontend.
You can make an authorized curl request as below, getting a token using the API auth guide here.
curl http://<DEPLOY_IP>/seldon-deploy/api/v1alpha1/alerting/test -X POST -H "Authorization: Bearer $TOKEN"
Alert shows in notifications drawer
The test alert will send a notification to the Deploy frontend within seconds, along with informing any other receivers (Pagerduty/Opsgenie/Slack/Email) that you may have configured.
Alert shows on alerts page
View All Firing Alertsin the alerts tray and, you will see the test alert along with any other currently firing alerts. This allows you to diagnose and fix any issues you may have missed when away from the Deploy UI.
Alert removed from alerts page
The test alert will resolve after 1 minute and will no longer be visible on the alerts page once refreshed.
Once the alert is resolved, after some time the frontend will be notified about the resolution. If you use the default configuration this will be after 5 minutes, but otherwise depends on Alertmanager’s
This test alert only relies on Alertmanager, but real alerts will send resolution notifications as soon as Prometheus reports the alert as resolved.