How to create slo for kafka latency?

[ashishvaishno]

Hello,

I wanted to create an slo for kafka for the below metrics kafka_network_requestmetrics_totaltimesms_produce_99th (https://docs.confluent.io/platform/current/kafka/monitoring.html#totaltimems)

I am using promethues with thanos where enabling experimental histogram is not an option as thanos compactor is yet to release downsampling of histogram thanos-io/thanos#6350
So usage of latencyNative in my situation is out of question. What would be a better way to handle this use case?

Regards
Ashish

[metalmatze]

Can you send more information on the kafka_network_requestmetrics_totaltimesms_produce_99th metric? Then we might be able to help.
You don't need native histograms for Pyrra and latency SLOs. The classic histograms are totally fine too.

What would be the time window and the latency objectives?

[ashishvaishno]

@metalmatze Sure :)
The metrics is mentioned here -> https://docs.confluent.io/platform/current/kafka/monitoring.html#totaltimems
I wanted to create the slo for all 3 type of request i.e Produce|FetchConsumer|FetchFollower
I would like to use it to see if the 99th Percentile is above 300ms for the latency SLO. The time window will be for a month

I started of with the below SLO

apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: kafka-network-request-metrics-producer-99-latency
  namespace: monitoring
  labels:
    release: prometheus-stack
spec:
  target: '95'
  window: 30d
  alerting:
    disabled: true
  indicator:
    ratio:
      errors:
        metric: kafka_network_requestmetrics_totaltimesms_produce_99th > 300
      total:
        metric: kafka_network_requestmetrics_totaltimesms_produce_99th

But that didnt work. Error : failed to get objective: ratio total metric is not a VectorSelector

[metalmatze]

I don't think the given metric will work. kafka_network_requestmetrics_totaltimesms_produce_99th.

You need the raw histogram metrics for what you want. Then you could do something along those lines:

pyrra/examples/caddy-response-latency.yaml

Lines 9 to 13 in 74a3b81

	latency:
	success:
	metric: caddy_http_response_duration_seconds_bucket{job="caddy",handler="subroute",code!~"5..",le="0.05"}
	total:
	metric: caddy_http_response_duration_seconds_count{job="caddy",handler="subroute",code!~"5.."}

The metric should look something like this:
kafka_network_request_duration_seconds_bucket{le="0.3"}
le="0.3" means Less Equal than 0.3s.

If something like that doens't exist you'll have to add it to the producer of these Kafka metrics. Nothing Pyrra can do.

[ashishvaishno]

@metalmatze do you need any other information?

[metalmatze]

I need time and energy between SRECon, KubeCon, and having a cold. A proper reply might need some time still. Sorry.

[ashishvaishno]

@metalmatze No worries :) All the best for the conferences and wish you a healthy recovery 💪

[ashishvaishno]

@metalmatze Happy New year 🎉 Were you able to look into this topic?

[metalmatze]

Hey, I responsed above.