Today, the Prometheus Team released Prometheus 2.30. While this release is relatively light on major new features, it brings a number of enhancements to configurability and resource usage efficiency (in addition to some bug fixes). Let's take a look at some of the most relevant ones:

Faster server restart times via snapshotting

For large Prometheus servers that track many millions of concurrent time series, a server restart can take up to multiple (or even many) minutes. This is because the Prometheus server has to rebuild its latest in-memory state for recent time series data from the write-ahead-log (WAL) on disk. This process can be quite slow because Prometheus needs to effectively replay the entire ingestion process for every sample in the WAL to reconstruct the final time series chunks in memory.

Prometheus 2.30 introduces an experimental snapshotting feature, implemented by Ganesh Vernekar and enabled using the flag --enable-feature=memory-snapshot-on-shutdown. With this feature enabled, Prometheus writes out a more raw snapshot of its current in-memory state upon shutdown, which can then be re-read into memory more efficiently when the server restarts. In first experiments with this code, it reduced restart times by 50-80% in one example.

Note that Prometheus only writes out this snapshot when it completes an orderly shutdown, and not on a periodic basis during normal operation (to reduce overall write load while Prometheus is running). So this new feature only helps speed up restarts after clean shutdowns. In case of crashes or unclean shutdowns, the most recent data is still just replayed more slowly from the WAL as before.

Controlling scrape intervals and timeouts via relabeling

Special target labels, along with target relabeling, already allowed Prometheus users to control certain scrape behaviors, such as the address, the HTTP path, or the HTTP parameters sent along with the scrape request. In Prometheus 2.30, Levi Harrison now extended this configurability to two more parameters:

• Scrape intervals: How frequently a target should be scraped, controlled by a new __scrape_interval__ meta label, e.g. __scrape_interval__="15s".
• Scrape timeouts: How long a target scrape may take before failing, controlled by a new __scrape_timeout__ meta label, e.g. __scrape_timeout__="15s".

Importantly, this now allows setting per-target scrape intervals within the same scrape configuration section, and since the behavior is controlled via target labels, service discovery mechanisms can now control this behavior on a more dynamic basis as well, such as temporarily telling Prometheus to scrape certain targets more frequently (to get higher-resolution data in certain scenarios) or less frequently (to cause less load).

Improving storage efficiency by tuning timestamp tolerances

Within each time series, Prometheus uses a double delta compression algorithm for storing sample timestamps. This compression algorithm performs better if the intervals between subsequent timestamps are completely regular. Although Prometheus already tries to scrape targets on a regular interval, actual scrape timestamps can deviate slightly (by a few milliseconds) from the intended schedule. This was exacerbated by a Go regression that caused more timer jitter. In older versions, Prometheus already allowed setting the --scrape.adjust-timestamps boolean flag to adjust scrape timestamps by up to 2ms to align them to the intended scrape schedule and thus achieve better timestamp compression.

In Prometheus 2.30, Julien Pivotto now added support for tuning this scrape timestamp tolerance duration using the experimental flag --scrape.timestamp-tolerance=<duration>. Setting this flag to 10ms (vs. its default of 2ms), Julien observed a 30% chunk storage size reduction in one example use case.

For more background, there is also a proposal around timestamp alignment with more context, and whether such alignment flags should remaing temporary workarounds or not.

Other enhancements

There are a lot of other smaller improvements in Prometheus 2.30, such as improving WAL load memory usage by 24% and CPU usage by 19%, adding two more optional per-scrape metrics (scrape_timeout_seconds and scrape_sample_limit) in addition to up and friends, and more. To see the full list of changes, check out the Prometheus 2.30.0 release changelog.

Conclusion

While Prometheus 2.30 is relatively light on new features, it adds a few great improvements around configurability and efficiency. Go try it out now!