Clarification about histogram metric and buckets

[tbolon]

Hello,

I have a question about current histogram metric and buckets. I was trying to exploit histogram metrics in a rendering solution (grafana or influxdb) and have hard time trying to obtain percentiles based on current histogram bucket format.

It seems the payload is quite different from openmetrics and prometheus:

In OpenMetrics, histogram buckets counts are cumulative: buckets contains count value from interval AND previous buckets.

[...] Buckets MUST be cumulative. As an example for a metric representing request latency in seconds its values for buckets with thresholds 1, 2, 3, and +Inf MUST follow value_1 <= value_2 <= value_3 <= value_+Inf. If ten requests took 1 second each, the values of the 1, 2, 3, and +Inf buckets MUST equal 10.

It's the same in prometheus:

A histogram with a base metric name of <basename> exposes multiple time series during a scrape:

• cumulative counters for the observation buckets, exposed as <basename>_bucket{le="<upper inclusive bound"}
• the total sum of all observed values, exposed as <basename>_sum
• the count of events that have been observed, exposed as <basename>_count (identical to <basename>_bucket{le="+Inf"} above)

But in OpenTelemetry, buckets only contains count inside the bucket interval (see also the spec):

  // bucket_counts is an optional field contains the count values of histogram
  // for each bucket.
  //
  // The sum of the bucket_counts must equal the value in the count field.
  //
  // The number of elements in bucket_counts array must be by one greater than
  // the number of elements in explicit_bounds array.
  repeated fixed64 bucket_counts = 6;

  // explicit_bounds specifies buckets with explicitly defined bounds for values.
  //
  // The boundaries for bucket at index i are:
  //
  // (-infinity, explicit_bounds[i]] for i == 0
  // (explicit_bounds[i-1], explicit_bounds[i]] for 0 < i < size(explicit_bounds)
  // (explicit_bounds[i-1], +infinity) for i == size(explicit_bounds)
  //
  // The values in the explicit_bounds array must be strictly increasing.
  //
  // Histogram buckets are inclusive of their upper boundary, except the last
  // bucket where the boundary is at infinity. This format is intentionally
  // compatible with the OpenMetrics histogram definition.
  repeated double explicit_bounds = 7;

The important part is: The sum of the bucket_counts must equal the value in the count field.

In prometheus there is already some function to handle histogram buckets, where the bucket bound is defined as a tag, and extract percentile information expecting cumulative buckets.

But with this difference, I don't see an easy way to do something useful with common metrics rendering systems, as the bucket system seems to be totally different from common system.

Do you have any comment or explanation about this? Is there any reason for having a different format?

Thanks,

[tbolon]

Sorry to bump this question, but I have seen other discussions about the spec of metrics, but nothing regarding histograms.

Is there anyone here able to anwser my question? Perhaps I was not clear enough? Or I missed something obvious?

For me, it seems there is no clear end-to-end method to read and analyze histogram metrics emitted by opentelemetry with, for example, grafana.

[jkowall]

Otel generates the metrics, and the backend provides the query/analysis required for analysis. Grafana is a frontend for the analysis. In Prometheus implements functions and query language for hisograms which is where this happens.

We are generating histograms with the tracemetrics processor for example which are then exported via metrics. these are easily analyzed by Prometheus which enables the analysis of histograms.

[tbolon]

After reading that, I have checked the obvious: the prometheus exporter converts the buckets by summing them.

It allows you to use prometheus histogram_quantile function to generate analysis.

I have missed this conversion on the exporter, I was also focused on InfluxDb metrics export, which does not convert buckets (and was my initial pain point).

Thank you for giving me an hint on where to search. And sorry for this obvious question.

[jkowall]

Not that obvious at all, it's a good question for sure.

[tbolon]

Based on your feedback, I have created an issue on the influxdb repo used by the influxdb collector. I hope it will be fixed. Thanks again. 🙏