Thanos Receive Http Request Latency High
critical

Description Thanos Receive {{$labels.job}} has a 99th percentile latency of {{ $value }} seconds for requests.
Query for 10m
>>>
	
				
					(
				
			
				
					
					
						histogram_quantile
					
				
			
				
					(0.99, sum by (job, le) (
				
			
				
					
					
						rate
					
				
			
				
					(
				
			
				
					
				
			
				
					{job=~".*thanos-receive.*", handler="receive"}[5m]))) > 10 and sum by (job) (
				
			
				
					
					
						rate
					
				
			
				
					(
				
			
				
					
				
			
				
					{job=~".*thanos-receive.*", handler="receive"}[5m])) > 0)
				
			
    
Query Explanation

The rule computes the 99th‑percentile latency of Thanos‑receive HTTP requests (handler="receive") over the last 5 minutes using histogram_quantile on the per‑bucket rates, and fires when that latency exceeds 10 seconds and there is at least one request observed in the same window (rate(..._count) > 0). In short, the alert triggers if Thanos‑receive is handling requests and the tail latency (p99) is higher than 10 s.

Get Alert
Download
Copy to Clipboard