Slack is an application to send messages to your team. A Slack channel is one of the notification channels available in Grafana. You can use it to send an alert to a Slack channel when a metrics match with the alert rule. It is very beneficial to have alerts sent to slack, because you can spot problems immediately.
Grafana send the message to slack using a Slack incoming webhook URL. To create a grafana alert to a Slack channel you need to create a Slack incoming webhook URL, create a notification channel, then create alerting rule.
How To Create Slack Incoming Webhook URL
First, we need to create a Slack Incoming Webhook URL for grafana to send the alert.
Go to https://api.slack.com/apps/new to create a slack app
Input App Name and select Workspace, then click
Create App
- Click Incoming Webhooks
- Switch the radio button to
On, then clickAdd New Webhook to Workspace
- Select the channel to send alerts, then click
Allow
- You will get a Webhook URL. Copy it.
How To Create Grafana Notification Channel
After you created a webhook URL, you need to create a notification channel in Grafana
- Go to your Grafana dashboard
- Click
Alertingicon, then clickNotification channels
- Click
Add Channelto create a new channel - Input the name of your channel
- Change Type to
Slackand input a Webhook URL that you have created
- Click
Send Testto test the channel. You will get message like this in Slack:
- If all good, click
Save
How To Create Alert Rule
Now that we have a notification channel, we can create an alerting rule to notify us when something wrong happen. In Grafana, an alert is attached to a dashboard panel. So in order to create an alert rule, you must already have a dashboard panel.
For this example, I am using a Request per second graph that I have created in the previous post.
There is metrics A that calculate the rate of http_request_get_books_count. We will create an alert for this metrics if the value dropped below a threshold. To create the alert, click Alert icon, then click Create Alert.
We will create an alert with a condition of the average value of metrics A for the last 5 minutes is below 7. The rule for our condition will look like this:
Also from the image you can see that the rule will Evaluate every "1m" For "1m". It means that every minute grafana will evaluate the rule. If the metrics violate the rule, grafana will wait for 1 minute. If after 1 minute the metrics is not recovered, grafana will trigger an alert.
Go to notification section at the bottom. Select the notification channel and write the alert message.
Save the dashboard. If the alert is triggered, you will see this message on your Slack channel.
Conclusion
Creating an alert to a messaging application like Slack is very beneficial. This ensures you and your teammates get notifications immediately if something wrong happens. You can mention your team slack group or use @here or @channel to make sure your team gets the message.