Today I had the pleasure of releasing Prometheus 2.28 in my function as a member of the Prometheus Team. While there are many changes in this release, let's take a look at some of the most relevant new features for users:

Displaying trace exemplars in the graphing interface

In our blog posts about Prometheus 2.26 and 2.27, we already talked about the new support for storing and retrieving trace exemplars in Prometheus, as well as for sending exemplar data over Prometheus's remote_write interface.

To take this integration of exemplars with metrics one step further, Levi Harrison now added support for displaying trace exemplars in Prometheus's graphing interface. When a PromQL query accesses series that have exemplar data attached to them, you can now show those exemplars along with the returned PromQL query data by enabling a per-graph "Show Exemplars" setting:

For example, in a latency-related graph, this allows you to easily find trace exemplars for requests with high latency, so that you can further investigate those requests in a tracing system. Since you may want to copy part of the exemplar metadata (such as the trace ID) to correlate it with another system, you can click on any exemplar to show its metadata in a more persistent way under the graph.

As described in the pull request that added the exemplar display support, an easy way to get a demo running where you can try out the exemplar display is by running Grafana's "The New Stack" demo app:

# Clone the "The New Stack" demo app.
git clone git@github.com:grafana/tns

# Check out a predictable version of the repo.
git checkout ec5673d

# Use Prometheus 2.28 instead of 2.26.
cd production/docker-compose
sed -i 's/prometheus:v2.26/prometheus:v2.28/' docker-compose.yml

# Run components necessary to expose and store metrics and trace exemplars.
docker-compose up tempo db app loadgen prometheus

Then head to http://localhost:9090/ and run the query:

histogram_quantile(0.99, sum by(le, method) (rate(tns_request_duration_seconds_bucket[1m])))

Click the "Show Exemplars" button to show trace exemplars.

Generic HTTP-based service discovery

Prometheus has supported generic service discovery integrations for a long time via its file_sd discovery mechanism, which watches and reads a set of target files from disk. The downside of the file_sd approach is that it requires sharing a filesystem between Prometheus and a sidecar process that generates the target files. This can be problematic in some environments, and users have asked for a network-based alternative for a long time.

Thus, Julien Pivotto introduced a new HTTP-based generic discovery mechanism in Prometheus 2.28. This mechanism works almost like file_sd, but loads target information from a specified URL rather than from a local file:

- job_name: http-sd
  http_sd_configs:
    - url: 'https://infra-db/prometheus-sd'

It also only supports JSON as the targets format, whereas file_sd supports both JSON and YAML.

We hope that this new HTTP-based discovery mechanism not only encourages people to build remote sidecar processes that can pull target information from various sources of truth, but that developers will add native Prometheus HTTP discovery support directly into their infrastructure and service databases, getting rid of the need to run a separate process.

You can find more information on the Prometheus website about writing your own HTTP-based service discovery integration.

Defaulting to the new expression editor

Prometheus 2.26 introduced an shiny new PromQL editor with advanced autocompletion, inline linting, and syntax highlighting, that you had to explicitly enable as an experimental feature. Since this new editor is so much more usable than the previous bare text input field, and since we have not received any major complaints about it, Prometheus 2.28 now defaults to this new editor:

Note that this default is stored in the browser's local storage and is only set to the new default value if no previously stored value is present for it yet. Thus if you used Prometheus 2.26 or 2.27 in the meantime and now access 2.28 using the same URL, you may still have to explicitly enable the new editor.

In case you encounter any issues with this new editor, please file an issue. For the time being, you can still revert to the old editor by disabling the "Use experimental editor" checkbox.

Other features and enhancements

There are several other features, enhancements, and bugfixes that made it into Prometheus 2.28. For example, there is now support for Linode service discovery, discovering Kubernetes API servers using a kubeconfig file, and a new startup screen in the web UI while the TSDB is starting up. You can find the full list of changes in the changelog for Prometheus 2.28.

Conclusion

A lot has happened in Prometheus 2.28 to integrate it with other systems and make it more usable. Go try it out now!