Last Wednesday, the Prometheus Team released Prometheus 2.27. While there are many changes in this release, let's take a look at some of the most relevant new features for users:

Dark mode

Do you prefer darker user interfaces, especially when using Prometheus at night? In pull request 8604, Łukasz Mierzwa now added the ability to switch the web UI from its default light version to a dark mode:

It's also possible to tell Prometheus to use the browser-preferred theme (using the right-most black/white button in the video).

The dark UI mode is still early in its development and some of its color choices will still be refined over time.

Retroactive backfilling of rule results

Normally, recording rules in Prometheus only start recording results starting from the time at which they are added to Prometheus. There has been a long-standing issue to allow evaluating rules retroactively on old data, so that Prometheus users can access rule results directly for those older time ranges. As a first step to address this, Jessica Greben added a new subcommand to the promtool CLI tool in pull request 7675 to support manual backfilling of rule results.

To backfill rule results, you can create files with your desired rules and then point promtool at the new rule files, your Prometheus server, and a TSDB output directory in which to create new blocks with the rule result time series:

promtool tsdb create-blocks-from rules \
  --start=<start-timestamp> \
  --end=<end-timestamp> \
  --output-dir <output-tsdb-dir> \
  --url <prometheus-server-url> \
  rules1.yml,rules2.yml

You can then move the newly generated TSDB blocks into the data directory of a Prometheus server to start accessing them.

Environment variable expansion for external labels

To achieve high availability for monitoring, it's common to run multiple Prometheus servers with identical configuration. However, as an exception to their identical configuration, you often want to be able to assign different external_labels to each Prometheus server, to differentiate replicas (e.g. replica: prometheus-a and replica: prometheus-b). To make this use case easier and not require two different configuration files on disk just to have different external labels, Prometheus 2.27 now allows you to expand environment variables in the external_labels configuration section.

To enable this experimental feature, you will have to set the flag --enable-feature=expand-external-labels. You can then use $envvar or ${envvar} in that configuration section, e.g.:

global:
  external_labels:
    replica: prometheus-${PROMETHEUS_REPLICA}

# [... rest of the configuration file ...]

Julien Pivotto added this feature in pull request 8649.

Better support for small use cases

Prometheus normally requires pre-allocating 512MiB of disk space for new TSDB chunk segment files. This can make it hard to use Prometheus in small environments, where the actual data to be stored in each chunk segment is only a few MiB and the available disk space is low. To make Prometheus work better for those use cases, Nathan Berkley added a flag --storage.tsdb.max-block-chunk-segment-size=<size> in pull request 8478 to allow configuring the maximum chunk segment file size.

For example, you could start Prometheus with a maximum chunk segment size of 64MiB like this:

./prometheus --storage.tsdb.max-block-chunk-segment-size=64MB

New service discoveries

In an ongoing effort to make Prometheus work better in more environments, the new release furthermore adds two new service discovery mechanisms:

Robert Jacob added support for discovering Docker engine containers as scrape targets in pull request 8629. You can check out the Prometheus documentation for all configuration details.

A contributor going by the name of N888 added support for discovering AWS Lightsail compute instances in pull request 8693. Again, you can find exact configuration details for this in the Prometheus documentation.

OAuth 2 support everywhere

When talking to other systems (scrape targets, Alertmanager, remote-write endpoints, and so on), Prometheus already supported multiple authentication schemes: bearer tokens, basic authentication, TLS client certificates, and more (depending on the situation). In addition to those, in pull request 8761 Levi Harrison now added support for configuring OAuth 2 in all relevant places. This just makes it easier to use Prometheus in environments that already use OAuth 2 for authentication purposes.

You can find details on configuring OAuth 2 in the Prometheus configuration documentation.

Exemplars support in remote_write

In our blog post about Prometheus 2.26, we already mentioned the importance of trace exemplars and how Prometheus 2.26 added support for storing them in memory and retrieving them via an HTTP API. This required setting the --enable-feature=exemplar-storage flag.

In Prometheus 2.27, Callum Styan now went one step further and added support for sending exemplars over Prometheus's remote_write protocol in pull request 8296. A TimeSeries protobuf message may now contain Exemplar fields to convey exemplar information. To enable this feature, you need to set the send_exemplars: true option in your remote_write configuration section. Remote storage implementors can now start making use of this to allow for better integration of metrics and trace data in their services.

Conclusion

A lot has happened in Prometheus 2.27 to make it work better for more use cases and to integrate it more with other systems. Go try it out now!