Filling in Missing Series in Binary Operator Matching

A long-requested PromQL feature has just landed in Prometheus 3.10: the ability to fill in default values for missing series in binary operations. In this post, we'll explore the motivation behind this feature, how the new fill(), fill_left(), and fill_right() modifiers work, and what caveats to watch out for when using them.

I want to specifically thank Dash0, an OpenTelemetry-native observability platform, for sponsoring the development of this feature.

The challenge: missing series in binary operations

When performing binary operations between two vectors in PromQL, only series that find a matching counterpart on the other side are included in the result vector by default. While this behavior is often exactly what you want, there are some situations where you may want a different behavior.

For example, you may want to add up the rates of successful and failed requests:

rate(successful_requests_total[5m]) + rate(failed_requests_total[5m])

If a particular label combination (e.g., {method="GET", status="200"}) only exists on one side of the operation (perhaps because there haven't been any failures for that combination yet) it will be completely dropped from the result. This is sometimes not what you want: you might prefer to treat the missing side as 0 and still get a result, in this case the total rate of either successful or failed requests for any method and status code combination that is present on either side.

As another example, you might want to compare a metric against a custom threshold that doesn't exist for every label set:

api_error_rates > api_error_rate_thresholds

If some of the label sets in api_error_rates do not have a matching threshold value, the filter operator will always drop that combination from the result, even if the error rate is very high. Maybe you would like to provide a default value for the threshold instead.

The old workaround: using or

Until now, the standard workaround (besides trying to avoid missing series) was to use the or operator to fill in potentially missing series:

vector1 + vector2 or vector1 or vector2

For the example query above, this becomes:

rate(successful_requests_total[5m]) + rate(failed_requests_total[5m])
or rate(successful_requests_total[5m])
or rate(failed_requests_total[5m])

This works but quickly becomes unwieldy. For default values other than 0, it gets even messier:

# Use 23 as the default for the right side, 42 for the left side.
(vector1 + vector2) or (vector1 + 23) or (42 + vector2)

And if you're using on() or ignoring() clauses, you need to carefully replicate that label matching logic with aggregations:

  vector1
+ on(label1, label2)
  vector2
or
  sum by(label1, label2) (vector1) + 23
or
  42 + sum by(label1, label2) (vector2)

This approach is error-prone, verbose, and inefficient. It also inadvertently reintroduces the metric name on resulting series in some cases, which can lead to unexpected behavior.

The new solution: Fill modifiers for providing default values

Prometheus 3.10 introduces three new experimental modifiers that let you specify default values for missing series in a binary operation:

• fill(<value>): Fill in missing series on both sides of the operation with the specified value.
• fill_left(<value>): Fill in missing series on the left side only.
• fill_right(<value>): Fill in missing series on the right side only.

Note: These modifiers are still experimental for now, so you need to enable them explicitly using the --enable-feature=promql-binop-fill-modifiers flag when starting Prometheus.

Basic usage

The simplest form is to use fill() to provide a default value for missing series on either side:

rate(successful_requests_total[5m]) + fill(0) rate(failed_requests_total[5m])

Now, if a series exists on only one side, a 0-valued sample is filled in on the other side, and you get a result for every label combination that exists on either side.

Filling only one side

Often, you'll want to fill in only one of the two sides. For example, when filtering error rates by custom thresholds that might not exist for every label set:

api_error_rates > fill_right(42) api_error_rate_thresholds

Here, if a series from api_error_rates doesn't have a matching threshold in api_error_rate_thresholds, the default threshold of 42 is used instead. The left side (api_error_rates) is not filled in, so only series that actually exist on the left are considered.

Using different values for each side

You can combine fill_left() and fill_right() to specify different default values for each side:

vector1 + fill_left(10) fill_right(20) vector2

This fills in missing left-side series with 10 and missing right-side series with 20.

Visualization

The following diagram shows how fill() works. We're adding two vectors, while using 0 as the default for any missing series:

Using fill_right(0) instead would only fill in the right side, omitting the last row from the result:

Working with group_left and group_right

The fill modifiers work together with many-to-one matching via group_left() and group_right(), but there are some important caveats to understand.

Filling the "one" side

When filling the "one" side (the side opposite to the grouping direction), things work as expected. For example, with group_left(cluster), you might want to fill in missing info metrics on the right side:

some_metric + on(status) group_left(cluster) fill_right(0) info_metric

If no matching info_metric exists for a given status, a 0-valued sample is filled in. However, note that the cluster label from the group_left(cluster) clause cannot be filled in – there's simply no source for that value, and we can't just invent one. So the result will have an empty cluster label for those filled-in series:

Filling the "many" side

When filling in the "many" side of a match, we face an ambiguity: if multiple series would have existed on that side (if their data hadn't gone missing for some reason), we don't know what their differentiating labels would have been. Therefore, only a single series can be filled in for each missing match group, using only the matching labels as its identity.

For example:

method_status_metric + on(status) group_left fill_left(0) status_only_metric

If status_only_metric has a {status="404"} series but method_status_metric has no series with that status, only a single {status="404"} series is filled in (with value 0), not multiple separate series for each possible method value:

Supported operators

The fill modifiers work with:

• Arithmetic operators: +, -, *, /, %, ^
• Comparison operators: ==, !=, >, <, >=, <=
• Trigonometric operators: atan2

They are not supported for set operators (and, or, unless), since the whole point of those operators is to filter series based on their presence or absence.

Caveats and pitfalls

Please be careful and don't just add fill operators to all of your binary operations without careful thought. While they can make it much easier to handle missing series in some situations, they also come with dangers and caveats that you really should be aware of before deciding to use them:

Caveat 1: Danger of masking incorrect selectors or broken data

Since the fill modifiers operate silently to fill in missing series and don't tell you what they did, they can mask real mistakes in your queries or your monitoring setup. This can happen due to a number of reasons:

• Mismatched metric names, label names, or label values between your PromQL selectors and your data (causing empty selectors).
• Incorrect matching modifiers (on, ignoring, group_left, group_right) on a binary operator.
• Missing data that you expect to always be present, but which is absent for some reason.

With the old default behavior, you would get no results for the affected label combinations, making it clear that something is wrong and needs to be investigated. With fill(), you will get plausible-looking outputs with incorrect values, and you might never know that something is broken. So be extra careful when using the new fill modifiers, especially in alerting rules.

Caveat 2: Changing output series labels over time

When a series needs to be filled in, its labels are derived from the match group's matching labels only. If the original series would have had additional labels (like a metric name), those won't be present on the filled-in result. This can lead to results with inconsistent label sets across different time steps if a series intermittently appears and disappears.

For example, in a range query, at timestamps where a series exists, you might see an output like failed_requests_total{method="GET"}, but at timestamps where it had to be filled in, you'll just see {method="GET"}. This is expected behavior: PromQL evaluates each timestamp independently and cannot know what the "real" labels would have been, had the series been present.

Caveat 3: No native histogram support (yet)

Currently, fill values must be numeric literals. Filling in native histograms with a default value is not supported yet, but could be added in the future if there's demand.

Caveat 4: You can't create series from nothing

Perhaps this is stating the obvious, but it's important to note that the fill modifiers can only fill in missing series when a matching series exists on at least one of the two sides. They do not create series out of thin air when both sides are missing. So you still need at least one series on one side of the operation for any output to be produced.

Conclusion

The new fill(), fill_left(), and fill_right() modifiers provide a clean and efficient way to provide default values for missing series in PromQL binary operations. They address a long-standing PromQL pain point and make some previously awkward queries much simpler to write.

This feature is available starting in Prometheus 3.10. I encourage you to try it out and provide feedback on the Prometheus GitHub repository or one of our community channels. If you run into any edge cases or have suggestions for improvements, I'd love to hear about them!

And if you want to learn more about PromQL and Prometheus, check out our PromQL training courses, especially Understanding PromQL for a deep dive into the query language.

Finally, many thanks go to Dash0 for inspiring this feature and sponsoring its development!