In the past, we have run and reported on multiple rounds of tests to verify the level of PromQL compatibility of various vendors and projects:

• Our initial comparison run from August 2020,
• Our update a few months later from November 2020,
• Our third and latest survey of the field from October 2021.

In our last post, we hadn't been able to fully test the VMware Tanzu Observability service by Wavefront yet, since the only Wavefront test account we had access to was being fed test data by an overly outdated reference Prometheus server. Since then, we have worked with engineers at VMware to establish an updated testing setup, with VMware making multiple engineering iterations on their service to improve their PromQL compatibility level along the way. In this post, we'll present the latest PromQL testing results for VMware's service.

Note: The testing and reporting of the results in this post are being sponsored by VMware. Despite such support, we aim to provide compatibility testing reporting that is as neutral and fact-based as possible.

General notes on test scores

The general notes on interpreting test scores from the last testing round still apply here. Please note especially that the numeric test scores (while good to have) are not alone a great indicator of the quality of test failures. Manual interpretation is needed to understand the actual impact, although full Prometheus compatibility requires a 100% test score without any cross-cutting issues in any case.

Updates to the test query set

Since the last testing round, we started allowing queries with negative offsets (e.g. some_metric offset -10m), as Prometheus 2.33 from January 29, 2022, started allowing these too. Other than this, there have been no further changes to the test query set. Generally, the test query set still needs to be updated at some point to test the new @ modifier for vector selectors, which was also promoted to a stable level in Prometheus 2.33.

Testing Tanzu Observability by Wavefront

Tanzu Observability advertises support for PromQL: "Wavefront supports both PromQL and WQL (Wavefront Query Language) queries". This PromQL support is being built on top of Wavefront's existing query language using a transpilation approach (which is similar to New Relic). To be able to test Wavefront's compatibility level, engineers at VMware provided us access to a reference Prometheus server (version 2.34) and a Wavefront test environment into which this Prometheus server remote-writes the usual testing reference data. We were then able to run the PromQL compliance tester tool against this setup.

Since Wavefront is sometimes off by 1ms when parsing floating-point start/end timestamps of queries, we added a query tweak in the tester configuration to only send full-second-aligned queries. With this tweak applied, we still noticed a number of behavioral deviations from standard PromQL:

• Hexadecimal scalar literals like 0x3d cannot be parsed.
• Special floating point values like Inf, -Inf, and NaN cannot be parsed.
• Prometheus returns an error if a vector selector does not contain at least one matcher that does not also match an empty string (for safety reasons). For example, Prometheus would reject the query {__name__=~".*"}, but allow the more explicit query {__name__=~".+"}. Wavefront executes both of these selectors successfully, causing a test comparison mismatch. To be fair, this could be interpreted as Wavefront supporting a superset of PromQL.
• Wavefront does not yet support negative offsets in selectors (e.g. my_metric offset -10m). Prometheus only started officially supporting negative offsets in version 2.33, so perhaps the behavior in Wavefront has not caught up yet with the latest PromQL changes.
• In several examples, the topk() / bottomk() functions return significantly different results, both in sample values and timestamps.
• Series staleness handling is not supported yet, so a purposefully intermittently present metric is returned throughout the query range without any gaps in Wavefront.
• Quantile calculations for the nonsensical target quantiles <0 or >1 return -Inf and +Inf in Prometheus, but return an error in Wavefront.
• The binary operators + and * applied between two instant vectors yield sample values that are totally different from the ones in Prometheus.
• The group_left() many-to-one matching modifier on a binary operator allows including a label from the "one" side in the result in Prometheus, but Wavefront's results don't include that label.
• The == / <= / >= binary operators between two vectors do not always work as expected and return empty results in examples where Prometheus produces an output.
• On a / binary operator query example that causes a clean one-to-one match and returns results in Prometheus, Wavefront returns an error reporting that cross-joins (which should not occur in the example) are not supported.
• On another / operator query example that causes a many-to-one match and returns valid results in Prometheus, Wavefront returns an error reporting that the output series would not be unique after dropping dimensions.
• Applying the - unary operator to an instant vector should drop the metric name, but doesn't in Wavefront.
• The query -1 ^ 2 returns -1 in Prometheus, but 1 in Wavefront, likely due to differences in operator precedence.
• When using the time() function in a binary filtering operation together with an instant vector (e.g. time() > some_timestamp_metric), the output sample timestamps are off, and not all original output timesteps are returned.
• Range vector selectors for ranges too small to select any samples (e.g. some_metric[1s]) produce empty results in Prometheus, whereas they still produce a non-empty output in Wavefront.
• The last_over_time() function is not yet supported in Wavefront.
• Many of the test queries produce special float values like Inf or NaN as output sample values, but samples with these values are filtered out in Wavefront, yielding empty result sets.
• The delta(), rate(), and increase() functions return very similar, but not always identical results as in Prometheus.
• label_replace() function calls with invalid target label names or invalid regular expressions fail in Prometheus, but are executed successfully in Wavefront.
• The clamp() function is not yet supported in Wavefront, although clamp_min() and clamp_max() are.
• The changes() function sometimes reports a value that is too large by 1.
• The histogram_quantile() function returns similar, but not identical values to Prometheus, and it does not remove the input metric name when it should.
• The count_values() function is not yet supported in Wavefront.
• A test subquery example with an offset applied returns similar, but different results in Wavefront.

Overall, the service passed 78.10% of tests after applying the described input timestamp query tweak. The exact test score varies by a few percent on each run, depending on the exact data covered.

The only other transpilation-based approach to creating a PromQL-like interface is New Relic, which received a 27.18% score in our last test run from October 2021. This currently places Wavefront (78.10%) first in the transpilation-based implementation category.

Conclusion

In this post, we looked at the PromQL compatibility of the Tanzu Observability by Wavefront service. While Wavefront already passes the majority of tests, the service still has a number of significant behavioral differences to Prometheus. We are looking forward to VMware continually improving their service's compatibility over time.

To view all test results, head to our PromQL Compliance Tests page, also available from our "Resources" menu item at the top of the page.