Today, the Prometheus Team released Alertmanager v0.25.0 with a large set of changes, new features, and bug fixes. As one of the new highlights, Alertmanager can now send alert notifications to Discord! This feature was added by Matthias Loibl of Polar Signals.

While the new feature still needs to be properly documented at the time of writing, we'll already show you how you can send alerts to Discord in a short from-scratch walkthrough here.

Setting up Prometheus

First, let's get a minimal Prometheus server running to generate some test alerts.

Download and unpack Prometheus (adjust the operating system and platform, if necessary):

wget https://github.com/prometheus/prometheus/releases/download/v2.41.0/prometheus-2.41.0.linux-amd64.tar.gz
tar xvfz prometheus-2.41.0.linux-amd64.tar.gz
cd prometheus-2.41.0.linux-amd64

Create an alerting rule file demo-alerts.yml with the following error rate test alert:

groups:
- name: demo-service-alerts
  rules:
  - alert: DemoServiceHighErrorRate
    expr: |
      (
        sum without(status, instance) (
          rate(demo_api_request_duration_seconds_count{status=~"5..",job="demo"}[1m])
        )
      /
        sum without(status, instance) (
          rate(demo_api_request_duration_seconds_count{job="demo"}[1m])
        ) * 100 > 0.5
      )
    for: 1m
    labels:
      severity: critical
    annotations:
      title: 'High 5xx rate for {{ $labels.method }} on {{ $labels.path }}'
      description: 'The 5xx error rate for path {{$labels.path}} with method {{ $labels.method }} in {{ $labels.job }} is {{ printf "%.2f" $value }}%.'

Create a prometheus.yml file to scrape three demo targets, load the alerting rule file, and send alerts to the Alertmanager we're going to run:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - demo-alerts.yml

alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']

scrape_configs:
  - job_name: 'demo'
    static_configs:
      - targets:
        - 'demo.promlabs.com:10000'
        - 'demo.promlabs.com:10001'
        - 'demo.promlabs.com:10002'

Start Prometheus as usual with the configs you just created:

./prometheus

Setting up a webhook on Discord

We assume you're already using Discord and have a channel that you want to send alerts to (in this example, we're using #alerts).

Edit the channel settings by clicking the "Edit Channel" cog button:

Next, head to the "Integrations" menu item:

Click on "Create Webhook":

Click on the newly added hook:

Adjust the name, copy the webhook URL, and save the hook:

Configuring Alertmanager

Download Alertmanager v0.25.0 (in a separate directory):

wget https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz
tar xvfz alertmanager-0.25.0.linux-amd64.tar.gz
cd alertmanager-0.25.0.linux-amd64

Set the alertmanager.yml config file to:

route:
  group_by: ['alertname', 'job']

  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h

  receiver: discord

receivers:
- name: discord
  discord_configs:
  - webhook_url: <DISCORD_WEBHOOK_URL>

ATTENTION: You'll need to replace the <DISCORD_WEBHOOK_URL> placeholder with the webhook URL you just copied from Discord. It should look something like this: https://discord.com/api/webhooks/XXX/YYY.

The config above just sends all alerts (grouped by job and alertname) to a single Discord receiver.

Start Alertmanager with the above config:

./alertmanager

Receiving and viewing alerts in Discord

The Prometheus server's /alerts page should eventually start showing some firing (red) alerts:

Shortly afterwards, you should also see alert notifications arriving on your Discord channel:

When all alerts in a group resolve (you can simulate this by stopping the Prometheus server and waiting 5 minutes for the alerts to expire in Alertmanager), you get a resolve notification as well:

If you would not like to send notifications for resolved alert groups, you can set the send_resolved: false option in your Discord config in the alertmanager.yml.

The above notifications are based on the default notification templates compiled into the Alertmanager (showing the alert name and label values in the title as well as full label and annotation details in the message), but it's possible to override the contents and style of the Discord notification, if you need to. For example, the Discord config in the Alertmanager can also take templateable title and message fields that allow you to adjust the message contents. There are also already users asking to add support for setting custom avatars and usernames, so Discord alerts will likely become even more flexible in the future.

Conclusion

This was just a quick walkthrough to show you how you can send Prometheus alerts to Discord. This new feature should also be documented in the Alertmanager configuration documentation soon, but in the meantime, you know how to get started!

And by the way: If you would like to learn Prometheus in a structured way and from the ground up, be sure to also check out our self-paced training courses that cover the basics of Prometheus, alerting and Alertmanager, as well as advanced use cases and integrations.