OpenTelemetry & Tempo 2,5392,539
Description:
This dashboard provides comprehensive observability into the performance and health of key components within an OpenTelemetry and Tempo tracing pipeline.
For OpenTelemetry, it covers essential telemetry pipeline stages, including:
Receivers: Track ingestion performance and data throughput.
Processors: Monitor the efficiency and latency of processors like batch and memory limiter.
Exporters: Visualize export success rate and performance to backend systems.
Tail Sampling Processor: Gain insights into the sampling decision performance and dropped spans.
For Tempo, the dashboard includes detailed metrics on:
Querier: Monitor query latency, throughput, and cache hit/miss rates.
Writer: Track write operations, latency, and error rates.
Cache: Observe cache utilization and efficiency.
Blocks: Visualize block compaction, size, and retention metrics.
Metrics Generator: Display metrics related to trace-to-metrics generation.
This dashboard is designed to help users quickly identify bottlenecks, optimize pipeline performance, and ensure the reliability of their distributed tracing infrastructure.
Used Metrics 6363
tempo_metrics_generator_active_processors
tempo_distributor_queue_pushes_total
tempo_distributor_queue_pushes_failures_total
tempo_distributor_metrics_generator_pushes_total
tempo_distributor_metrics_generator_pushes_failures_total
tempo_metrics_generator_spans_discarded_total
prometheus_remote_storage_samples_failed_total
prometheus_remote_storage_samples_dropped_total
prometheus_remote_storage_exemplars_failed_total
prometheus_remote_storage_exemplars_dropped_total
tempo_metrics_generator_registry_series_limited_total
traces_service_graph_request_failed_total
traces_spanmetrics_calls_total
traces_service_graph_request_total
tempo_ingester_blocks_cleared_total
tempo_ingester_blocks_flushed_total
tempo_ingester_failed_flushes_total
tempodb_compaction_blocks_total
tempodb_compaction_bytes_written_total
tempodb_compaction_objects_combined_total
tempodb_compaction_objects_written_total
tempodb_blocklist_length
tempo_ring_members
memcached_current_connections
memcached_max_connections
memcached_current_bytes
memcached_limit_bytes
memcached_current_items
memcached_commands_total
memcached_read_bytes_total
memcached_written_bytes_total
tempo_memcache_request_duration_seconds_bucket
tempo_memcache_request_duration_seconds_sum
tempo_memcache_request_duration_seconds_count
tempo_discarded_spans_total
tempo_distributor_spans_received_total
tempo_ingester_traces_created_total
tempo_distributor_bytes_received_total
tempo_warnings_total
tempo_querier_worker_request_executed_total
tempo_ingester_live_traces
otelcol_receiver_accepted_spans
otelcol_receiver_refused_spans
otelcol_process_runtime_total_sys_memory_bytes
otelcol_process_memory_rss
otelcol_processor_refused_spans
otelcol_processor_batch_batch_send_size_bucket
otelcol_exporter_send_failed_spans
otelcol_exporter_sent_spans
otelcol_exporter_queue_size
otelcol_exporter_queue_capacity
otelcol_processor_tail_sampling_new_trace_id_received
otelcol_processor_tail_sampling_count_traces_sampled
otelcol_processor_tail_sampling_sampling_trace_removal_age_sum
otelcol_processor_tail_sampling_sampling_trace_removal_age_count
otelcol_processor_tail_sampling_sampling_trace_dropped_too_early
otelcol_processor_tail_sampling_sampling_trace_removal_age_bucket
otelcol_processor_tail_sampling_sampling_traces_on_memory
otelcol_processor_tail_sampling_sampling_decision_latency_bucket
otelcol_loadbalancer_num_backends
otelcol_loadbalancer_num_backend_updates
otelcol_loadbalancer_backend_outcome
otelcol_loadbalancer_backend_latency_bucket