Cloudera Docs

Performance considerations

A collection of basic recommendations for Kafka clusters.

The simplest recommendation for running Kafka with maximum performance is to have dedicated hosts for the Kafka brokers and a dedicated cluster for the Kafka metadata service. Depending on your deployment mode, this means either a dedicated ZooKeeper cluster (for ZooKeeper-based deployments) or dedicated KRaft controllers (for KRaft mode). If that is not an option, consider these additional guidelines for resource sharing with the Kafka cluster.

Running in VMs
It is common practice in modern data centers to run processes in virtual machines or containers. Kafka is sufficiently sensitive to I/O throughput and network performance that virtualized environments can impact broker operation. When running Kafka in a virtual environment, ensure adequate I/O performance and resource allocation. You may need to work with your infrastructure team or cloud provider to optimize Kafka performance in virtualized deployments.
Do not run other processes with brokers, ZooKeeper, or KRaft controllers
Due to I/O contention with other processes, it is generally recommended to avoid running other processes on the same hosts as Kafka brokers. Similarly, for optimal performance, ZooKeeper nodes and KRaft controllers should not be colocated with brokers or other services.
Keep the metadata service connection stable
Kafka relies heavily on having a stable connection to its metadata service. Putting an unreliable network between Kafka brokers and the metadata service (ZooKeeper or KRaft controllers) will cause connectivity issues and cluster instability. Examples of configurations to avoid:

  • Do not put Kafka brokers and metadata service nodes on separated networks

  • Do not put Kafka brokers and metadata service nodes on the same network with other high network loads

KRaft controller deployment considerations
When running Kafka in KRaft mode, follow these additional recommendations:

  • For redundancy, a Kafka cluster should use 3 or more controllers. The number of controllers depends on factors like cost and the number of concurrent failures your system should withstand without availability impact. To withstand N concurrent failures, the KRaft controller cluster must include 2N + 1 controllers.

  • KRaft controllers store all cluster metadata in memory and on disk. For a typical Kafka cluster, 5GB of main memory and 5GB of disk space for the metadata log directory is sufficient.