In our last post, we looked at the overall anatomy of a PromQL query. Today, let's look a bit closer at the two ways of selecting data in PromQL: instant vector selectors and range vector selectors. We will examine their use cases, the reasoning behind them, and some implementation details you should be aware of. Selecting data from Prometheus's TSDB forms the basis of almost any useful PromQL query before transforming, aggregating, or filtering that data further, so it's good to understand this fundamental step first.

Refresher: Prometheus data model

Before we talk about selecting data, let's refresh our knowledge of the shape of the data that Prometheus stores. Prometheus's data model is simple: it stores numeric time series. Each time series has an identity (its metric name and a unique set of key="value" label pairs that create dimensionality within that metric name), as well as a set of samples. Each sample has an int64 Unix timestamp in milliseconds and a float64 sample value (even if it just contains an integer value). That is it! Here is a visual representation of one series:

In queries, we either want to select the latest sample of a set of series at a given query step, or a historical range of samples to aggregate over in some way. Let's look at how to do that.

Instant vector selectors

An instant vector selector allows you to select the latest sample value at every evaluation resolution step of a range query, or in an instant query, at a single evaluation timestamp. It's called a vector selector because it returns a whole set (vector) of series, with one sample for each series (thus an instant vector).

Matcher syntax

To select the latest value for any series with the metric name api_http_requests_total, you simply write the metric name:

api_http_requests_total

To only select series with the method POST and the handler /api/widgets, you can add some label matchers in curly braces that restrict the output:

api_http_requests_total{method="POST", handler="/api/widgets"}

Besides equality matching, you can also use != for inequality, =~ for full-string regex matches, or !~ for negative regex matches against label values. So the following would select requests with the method POST on any handler that starts with /api/:

api_http_requests_total{method="POST", handler=~"/api/.*"}

Under the hood the metric name (api_http_requests_total) is just stored as another label with the special (reserved) label name __name__. So the query api_http_requests_total would be 100% equivalent to the following query:

{__name__="api_http_requests_total"}

This can be useful if you ever want to do regex matching against the metric name itself (e.g. for debugging purposes). For example, to select any series with a metric name starting with api_, just write:

{__name__=~"api_.*"}

Lookback delta

An instant vector selector serves the use case of "give me the latest sample value at a given time for a set of series". It's useful for graphing the development of series over time, looking at their latest sample values in a table, or basing transformations (such as aggregations) on a single sample value at each time slice.

However, the trickiness comes in when having to define "latest". Prometheus does not have a concept of a fixed interval between samples, so it's hard to say absolutely whether there is a "current" latest sample for a given series. If the latest sample for a series is a week old, you probably wouldn't want to include it in the result of an instant vector selector at the current timestamp (this would lead to many outdated series "flatlining" in your graphs). On the other hand, you can't expect samples to exactly match the evaluation timestamp either, since samples in the Prometheus TSDB are not aligned on some time grid and can have arbitrary timestamps.

So some kind of middle ground is needed. To select latest samples that are neither too outdated, nor require super fast scrape intervals or even grid-aligned sample timestamps, instant vector selectors look back a maximum of 5 minutes relative to the evaluation timestamp. Samples with a timestamp older than 5 minutes are dropped from the result.

Staleness

The 5-minute rule above is a good compromise between excluding old data and including recent-enough samples. However, there are situations where it is annoying to have series stick around in instant vector selector results longer than strictly needed. For example, if you are running a service on Kubernetes and track the pod name in a label value, then all series identities (label sets) change when you re-deploy that service (because the pod names change). You will have old series with one set of pod names and new series with a new set. However, for a period of 5 minutes, you will see both the new and the old series in your graphs, which can be confusing. It's even more confusing if you base aggregations on this data. If you calculate the sum over the memory usage of all pods, then for a period of 5 minutes you will see double the total usage that you should be seeing!

To resolve this, Prometheus supports marking series as explicitly stale: When Prometheus detects that a series will go extinct (because its target has disappeared, or it's no longer returned in the scrape of its target, or because a recording rule stopped returning it), Prometheus writes out an explicit staleness marker for this series. Under the hood, staleness markers are just a normal sample for that series in the TSDB, but with a special kind of NaN sample value to signal staleness.

When instant vector selectors encounter a staleness marker as the last-seen sample value before their evaluation timestamp, the corresponding series will not be included in the result. This solves the "double count" and "flatlining" issues more quickly.

Range vector selectors

Range vector selectors allow you to select an entire range of samples over time, again at each evaluation step of a query. This allows you to run aggregations over the time window: rates of increase, averages, min/max, quantiles, and so on.

You write range vector selectors exactly like instant vector selectors, but with the desired range time window appended in square brackets. For example, this would return the last 5 minutes of samples of each series with the api_http_requests_total metric name:

api_http_requests_total[5m]

You always need to include a time unit (m for minute in this case). The available units are: s (seconds), m (minutes), h (hours), d (days), w (weeks), y (years).

Note that you cannot graph a range vector selector by itself, since it would mean somehow graphing multiple samples for each series at each graph resolution step. You always need to transform the returned range vector into an instant vector first (giving you a single output sample for each series), for example by computing its per-second rate of increase:

rate(api_http_requests_total[5m])

Conceptually, range vector selectors are actually much simpler than instant vector selectors: They have no notion of lookback deltas or staleness, as the selection time range is completely determined by the user-provided time window.

Offsets

Sometimes you want to make data from a while ago appear like current data in your queries. For example, you might want to compare the request rates of a week ago with the current request rates. For this, PromQL supports a <vector selector> offset <duration> modifier on individual vector selectors which allows time-shifting data from the past into the current time.

For example, to query for the API request rate from a week ago:

rate(api_http_requests_total[5m] offset 1w)

You could then calculate the difference between the current request rate and the rate from last week:

rate(api_http_requests_total[5m] offset 1w) - rate(api_http_requests_total[5m])

Note: Offsets can only look back, not forward. This is consistent with PromQL's general computational model, where data selectors can look backwards in time, but not into the future (where data may still be missing or changing). If you really want to look into the future, then run the overall query with a timestamp (or time range) in the future.

Confusion alert! Instant/range selectors vs. instant/range queries

Beware: The terms "instant" and "range" can be confusing in Prometheus, because they are used both for types of entire PromQL queries as well as for types of individual selectors contained in queries. For an explanation of range and instant queries, see the blog post about the anatomy of PromQL queries. That concept is about the overall query execution, and whether a query as a whole is evaluated at a single timestamp (instant), or at multiple subsequent time steps (range). Instant and range vector selectors, on the other hand, are about whether you select a single sample at each evaluation timestamp, or a whole time range.

Conclusion

In this post we looked at the basis for useful PromQL queries: selecting current samples values, as well as selecting ranges of data to aggregate over. We learned how to use label matchers, how instant vectors decide which samples to include, and how to time-shift old data into the present. This is a good foundation for then processing that data further in PromQL.

Related Content

We also made a video about data selection in PromQL: