Comprehensive Guide to Kafka Monitoring: Metrics, Problems, and Solutions

Pavithra Parthiban -

Apache Kafka has become the backbone of modern data pipelines, enabling real-time data streaming and processing for a wide range of applications. However, maintaining a Kafka cluster's reliability, performance, and scalability requires continuous monitoring of its critical metrics.

This blog provides a comprehensive guide to Kafka monitoring, including key metrics, their units, potential issues, and actionable solutions. We will conclude with how Atatus Kafka Monitoring simplifies the process of keeping your Kafka cluster running smoothly.

Table of Contents:

Key metrics to monitor in Kafka

A well-maintained Kafka cluster can handle a large amount of data. It is important to monitor the health of your Kafka setup to keep the applications that depend on it running smoothly. Kafka metrics are usually grouped into six categories:

Kafka Metrics

Here is a breakdown of the metrics:

1. Broker-level metrics

(i). Request latency

Request latency measures the time taken by Kafka brokers to process produce, fetch, or admin requests, with high latency often indicating bottlenecks in the broker or network.

Unit: Milliseconds (ms)

Problematic Scenario: If request latency spikes, consumers might face delays in receiving messages. This could stem from broker overload, slow disk I/O, or network issues.

Solution:

  1. Check broker CPU and disk utilization.
  2. Scale brokers horizontally or upgrade their hardware.
  3. Optimize network configuration and review the replication factor.

(ii). Under-replicated partitions

Under-replicated partitions indicates the number of partitions whose replicas are not in sync with the leader partition.

Unit: Count

Problematic Scenario: High values may occur due to slow network, broker failures, or replication throttling.

Solution:

  1. Inspect broker logs for replication delays.
  2. Ensure adequate bandwidth for replication.
  3. Increase num.replica.fetchers for faster recovery.

(iii). Disk Usage

Disk usage monitors how much disk space brokers consume.

Unit: Bytes (B)

Problematic Scenario: Disk usage nearing capacity can result in write failures and data loss.

Solution:

  1. Set up disk alerts for early warnings.
  2. Use Kafka log retention policies to delete older logs.
  3. Add more storage or brokers to the cluster.

2. Topic-level metrics

(i). Partition size

Partition size tracks the size of data stored in each partition.

Unit: Bytes (B)

Problematic Scenario: Uneven partition sizes might indicate skewed producer key distribution, causing load imbalance.

Solution:

  1. Review producer keying logic.
  2. Use Kafka’s partition reassignment tool to rebalance partitions.

(ii). Log Flush Latency

Log flush latency measures the time taken to write data from memory to disk.

Unit: Milliseconds (ms)

Problematic Scenario: High flush latency can result in message loss during broker crashes.

Solution:

  1. Tune the log.flush.interval.messages and log.flush.interval.ms settings.
  2. Optimize disk I/O or upgrade storage hardware.

3. Producer metrics

(i). Record send rate

Record send rate tracks the number of records the producer sends per second.

Unit: Records per second

Problematic Scenario: A sudden drop in send rate might indicate producer application issues or network congestion.

Solution:

  1. Check producer logs for errors (e.g., TimeoutException).
  2. Validate broker availability and network health.

(ii). Record error rate

Record error rate measures the rate of failed produce requests.

Unit: Errors per second

Problematic Scenario: An increase in errors could stem from broker unavailability or misconfigured producer settings.

Solution:

  1. Verify broker connectivity and configuration.
  2. Adjust retries and linger.ms producer properties.

4. Consumer metrics

(i). Consumer lag

Consumer lag represents the difference between the last committed offset and the latest offset available in a partition.

Unit: Offset count

Problematic Scenario: High lag may indicate slow consumers or an overloaded broker.

Solution:

  1. Scale consumer groups horizontally.
  2. Check consumer application for bottlenecks.
  3. Monitor broker health and resource utilization.

(ii). Fetch latency

Fetch latency measures the time taken for consumers to fetch records from brokers.

Unit: Milliseconds (ms)

Problematic Scenario: Increased fetch latency can result in delayed data processing.

Solution:

  1. Optimize consumer configuration (e.g., fetch.min.bytes, fetch.max.wait.ms).
  2. Ensure sufficient broker capacity to handle requests.

5. Cluster-level metrics

(i). Controller election rate

Tracks the rate at which controller elections occur. Frequent elections can disrupt cluster stability.

Unit: Elections per second

Problematic Scenario: High election rates often indicate broker failures or network partitions.

Solution:

  1. Investigate broker logs for errors.
  2. Improve broker availability with better hardware or networking.
  3. Ensure Zookeeper is stable and responsive.

(ii). Active controller count

Active controller count should always be 1 in a healthy cluster.

Unit: Count

Problematic Scenario: If more than one active controller is detected, it suggests cluster misconfiguration.

Solution:

  1. Restart affected brokers.
  2. Validate and fix Zookeeper configuration.

6. ZooKeeper metrics

(i). Connection count

Connection count monitors the number of active connections to ZooKeeper.

Unit: Count

Problematic Scenario: If connections approach the maxClientCnxns limit, brokers may fail to connect.

Solution:

  1. Increase maxClientCnxns in the ZooKeeper configuration.
  2. Optimize ZooKeeper client connections.

(ii). Request latency

Request latency tracks the time ZooKeeper takes to process requests.

Unit: Milliseconds (ms)

Problematic Scenario: High latency can affect Kafka’s metadata operations.

Solution:

  1. Scale ZooKeeper nodes.
  2. Optimize ZooKeeper hardware or network setup.

Best practices for kafka monitoring

  1. Centralized Monitoring: Use tools like Prometheus, Grafana, or Confluent Control Center to visualize metrics.
  2. Set Alerts: Define thresholds and alerts for critical metrics to respond proactively.
  3. Capacity Planning: Regularly review cluster utilization and scale as needed.
  4. Benchmark Regularly: Simulate peak loads to test cluster stability.

By monitoring these metrics and addressing issues promptly, you can ensure your Kafka setup remains reliable and performant.

Kafka monitoring with Atatus

Manually monitoring Kafka metrics can be challenging, especially for large-scale deployments. Atatus Kafka Monitoring offers a unified platform to simplify this process. Here is how Atatus can help:

  1. Real-Time Metrics: Track key Kafka metrics such as request latency, consumer lag, and disk usage in real-time with intuitive dashboards.
  2. Customizable Alerts: Receive instant alerts for critical events like under-replicated partitions or high controller election rates.
  3. Anomaly Detection: Identify unusual patterns in Kafka performance with AI-driven anomaly detection.
  4. End-to-End Visibility: Correlate Kafka metrics with application performance to gain insights into end-to-end system health.
  5. Historical Analysis: Access historical data for capacity planning and trend analysis.

Why choose Atatus?

With Atatus, you can reduce the complexity of Kafka monitoring and focus on optimizing your data pipelines. Whether it’s troubleshooting latency issues or ensuring even partition distribution, Atatus offers the tools you need to maintain a robust Kafka environment.

Conclusion

Monitoring Kafka is an ongoing process that requires attention to detail, proactive planning, and the right tools. By keeping track of critical metrics and leveraging platforms like Atatus, you can ensure your Kafka cluster performs optimally, even under heavy workloads.