Subscribe to this blog via RSS at /blog/rss.xml
Using Metrics-Based Custom Thresholds in Prometheus Alerting Rules
April 04, 2024 by Julius Volz
Often you will want to have different alerting thresholds for different label combinations. For example, you may want to have individual thresholds for the error rates of each HTTP path and method combination, rather than a single hardcoded threshold…
Tags: alerting, thresholds, promql
Monitoring TLS Endpoint Certificate Expiration with Prometheus
February 06, 2024 by Julius Volz
If you have HTTPS endpoints in your infrastructure, you will want to make sure that none of their TLS certificates are about to expire without you noticing. Let's have a look at how you can use Prometheus' Blackbox Exporter to monitor for TLS…
Tags: alerting, tls, certificates, expiration, blackbox exporter
Errors, Successes, Totals: Which Metrics Should I Expose to Prometheus?
September 19, 2023 by Julius Volz
If your service is handling operations that can either succeed or fail, you may be wondering how to best expose them as Prometheus metrics. Should you expose errors and successes or errors and totals? Should you have a single metric split up by a…
Tags: prometheus, metrics, failures, errors, successes, totals, promql
Dealing with Missing Time Series in Prometheus
September 13, 2023 by Julius Volz
Due to Prometheus' dimensional data model and the way it tracks and collects metrics, some of the metrics referenced in your PromQL expressions may sometimes be missing. Let's have a look what causes missing series, how they can cause trouble for…
Tags: prometheus, time series, promql, instrumentation
High Availability for Prometheus and Alertmanager: An Overview
August 31, 2023 by Julius Volz
If you use Prometheus for mission-critical monitoring and alerting, you will want to make sure that your Prometheus stack is just as reliable as the systems and services it is monitoring. Let's take a look at the main strategies for achieving high…
Tags: prometheus, alertmanager, high availability, meta-monitoring
Sending Prometheus Alerts to Discord with Alertmanager v0.25.0
December 23, 2022 by Julius Volz
Today, the Prometheus Team released Alertmanager v0.25.0 with a large set of changes, new features, and bug fixes. As one of the new highlights, Alertmanager can now send alert notifications to Discord! This feature was added by Matthias Loibl of…
Tags: discord, alerting, alertmanager
Understanding Duplicate Samples and Out-of-order Timestamp Errors in Prometheus
December 15, 2022 by Julius Volz
Have you ever encountered any of the following errors in your Prometheus server logs? "Error on ingesting out-of-order samples" "Error on ingesting samples with different value but same timestamp" "duplicate sample for timestamp" Then you likely have…
Tags: prometheus, tsdb, out-of-order, duplicate, timestamps, collisions
Avoid These 6 Mistakes When Getting Started With Prometheus
December 11, 2022 by Julius Volz
Let's talk about 6 common mistakes to avoid when getting started with Prometheus-based monitoring: Mistake 1: Cardinality bombs This is the classic pitfall that everyone hits at least once when they're starting out with Prometheus. Once you discover…
Tags: prometheus, promql, monitoring, best practices, mistakes
Prometheus Turns 10
November 24, 2022 by Julius Volz
Today, we want to congratulate Prometheus on its tenth birthday! From Google to SoundCloud Ten years ago today, the first public commit was pushed to the Prometheus repo. Back then, Matt T. Proud and myself had just left Google to join SoundCloud. At…
Tags: prometheus, birthday
PromLabs and Chronosphere Open-Source the PromLens Query Builder
October 25, 2022 by Julius Volz
At PromLabs we created PromLens, the ultimate power tool for building, understanding, and visualizing PromQL queries, with the goal of helping users get the most out of PromQL-compatible monitoring solutions. A year ago, we partnered with…
Tags: promlens, promql, chronosphere, prometheus, open-source
What's New in Prometheus 2.39?
October 05, 2022 by Julius Volz
Today, the Prometheus Team released Prometheus 2.39. The main highlights are a new experimental TSDB feature that allows appending old or out-of-order data, as well as major improvements in resource usage. Besides that, there are a number of small…
Tags: tsdb, memory-usage
Testing the PromQL Compatibility of VMware Tanzu Observability by Wavefront
May 19, 2022 by Julius Volz
In the past, we have run and reported on multiple rounds of tests to verify the level of PromQL compatibility of various vendors and projects: Our initial comparison run from August 2020, Our update a few months later from November 2020, Our third…
Tags: promql, compliance, testing, vendors, wavefront, vmware, tanzu
What's New in Prometheus 2.33?
January 30, 2022 by Julius Volz
Today, the Prometheus Team released Prometheus 2.33. While there are only a few completely new features, this release promotes multiple important experimental features to a stable state. It also improves on existing functionality in Prometheus. Let's…
Tags: promql, remote write
Bringing the PromLens Fire to Chronosphere Users
November 04, 2021 by Julius Volz
At PromLabs we created PromLens, the ultimate power tool for building, understanding, and visualizing PromQL queries, with the goal of helping users get the most out of PromQL-compatible monitoring solutions. Today we are happy to announce that we…
Tags: promlens, promql, chronosphere
PromQL Vendor Compatibility, Round Three
October 14, 2021 by Julius Volz
Time for round three of our PromQL compatibility tests! In two previous blog posts, we compared PromQL correctness across vendors and then provided an update a few months later. Using an open-source compliance checker tool, we evaluated each…
Tags: promql, compliance, testing, vendors, cortex, thanos, victoriametrics, promscale, timescaledb, sysdig, metricfire, newrelic, m3, grafanacloud, chronosphere, amazon, amp, google, gmp, vmware, wavefront
What's New in Prometheus 2.30?
September 14, 2021 by Julius Volz
Today, the Prometheus Team released Prometheus 2.30. While this release is relatively light on major new features, it brings a number of enhancements to configurability and resource usage efficiency (in addition to some bug fixes). Let's take a look…
Tags: tsdb, wal, snapshots, scrape settings, timestamps, compression
What's New in Prometheus 2.28?
June 21, 2021 by Julius Volz
Today I had the pleasure of releasing Prometheus 2.28 in my function as a member of the Prometheus Team. While there are many changes in this release, let's take a look at some of the most relevant new features for users: Displaying trace exemplars…
Tags: exemplars, promql, http-sd, service-discovery, linode
Showcasing PromLens 1.0.0
May 26, 2021 by Julius Volz
Today we are happy to release version 1.0.0 of PromLens! PromLens is the ultimate power tool for PromQL that helps you build, analyze, explain, and visualize your queries. In case you are unfamiliar with it, try a quick demo here. Here is a summary…
Tags: promlens, promql, exploration
What's New in Prometheus 2.27?
May 16, 2021 by Julius Volz
Last Wednesday, the Prometheus Team released Prometheus 2.27. While there are many changes in this release, let's take a look at some of the most relevant new features for users: Dark mode Do you prefer darker user interfaces, especially when using…
Tags: exemplars, rules, backfilling, service-discovery
What's New in Prometheus 2.26?
April 01, 2021 by Julius Volz
Yesterday, the Prometheus Team released Prometheus 2.26. While there are many changes in this release, let's take a look at some of the most relevant new features for users: New PromQL editor Since the creation of Prometheus in 2012, the built-in…
How Exactly Does PromQL Calculate Rates?
January 29, 2021 by Julius Volz
Counters are a Prometheus metric type whose value only goes up, and which represent cumulative total counts like "How many requests have we handled in total?" or "How many seconds have we spent handling requests?". Since the value of a counter…
PromQL Queries for Exploring Your Metrics
December 17, 2020 by Julius Volz
When building PromQL queries, a frequent need for users is to get an idea of the data they are working with, and a way to explore the space of all available metrics and their labels. While the UI tooling for this is still evolving (PromLens and…
Tags: promql, exploration, metrics, promlens
Showcasing PromLens 0.11.0
December 15, 2020 by Julius Volz
Today we are happy to announce version 0.11.0 of PromLens. PromLens is the ultimate power tool for PromQL that helps you build, analyze, explain, and visualize your queries. In case you are unfamiliar with it, try a demo here. Here is a summary of…
An Update on PromQL Compatibility Across Vendors
December 01, 2020 by Julius Volz
Prometheus's query language PromQL is one of the cornerstones of the Prometheus ecosystem. Back in our August blog post, Comparing PromQL Correctness Across Vendors, we looked at several external projects and monitoring vendors that claimed to offer…
Tags: promql, compliance, testing, vendors, cortex, thanos, victoriametrics, promscale, timescaledb, metricfire, newrelic, m3, grafanacloud, chronosphere
Showcasing PromLens 0.10.0
October 19, 2020 by Julius Volz
Today we are happy to announce version 0.10.0 of PromLens. PromLens is the ultimate power tool for PromQL that helps you build, analyze, explain, and visualize your queries. In case you are unfamiliar with it, this is what it looks like: PromLens…
The Meaning of "Prometheus" - A Tale of Implementations and Interfaces
October 13, 2020 by Julius Volz
What people mean when they say that they "use Prometheus" or "support Prometheus" is evolving over time. Initially, the Prometheus landscape was very focused on a specific set of component implementations, whereas nowadays the focus is shifting more…
Tags: prometheus, promql
Metric Types in Prometheus and PromQL
September 25, 2020 by Julius Volz
Prometheus has the concept of different metric types: counters, gauges, histograms, and summaries. If you've ever wondered what these terms were about, this blog post is for you! We'll look at the meaning of each metric type, how to use it when…
Tags: promql, data model, metric types, instrumentation
Comparing PromQL Correctness Across Vendors
August 06, 2020 by Julius Volz
UPDATE: See our follow-up blog post from December 1 for the latest round of PromQL vendor test results. Prometheus' query language PromQL is one of the most important interfaces in the Prometheus ecosystem. Organizations rely on it to build their…
Tags: promql, compliance, testing, vendors, cortex, thanos, victoriametrics, timescaledb
Selecting Data in PromQL
July 02, 2020 by Julius Volz
In our last post, we looked at the overall anatomy of a PromQL query. Today, let's look a bit closer at the two ways of selecting data in PromQL: instant vector selectors and range vector selectors. We will examine their use cases, the reasoning…
The Anatomy of a PromQL Query
June 18, 2020 by Julius Volz
Let's have a look at the anatomy of a PromQL query. PromQL has operators, functions, selectors, and more, but let's not get bogged down by those details in this post. Let's instead look at the overall nature of a query: How are PromQL queries…