Cloudera Docs

What is Cloudera Surveyor for Apache Kafka?

Cloudera Surveyor is a standalone application that allows you to efficiently monitor and manage your Kafka clusters through a graphical user interface. As a proprietary Cloudera application, Cloudera Surveyor provides an optimized and fully supported solution for managing Kafka within your Cloudera ecosystem, simplifying operations and enhancing visibility beyond standard open-source tools.

Main benefits

  • Enhanced Kafka management – Cloudera Surveyor provides a user-friendly interface for managing Kafka clusters, topics, and consumer groups. This reduces the need for command-line tools and streamlines administrative tasks.

  • Scalability – Cloudera Surveyor is designed to efficiently monitor and manage a large number of Kafka clusters and topics, adapting to growing data streams and infrastructure demands.

  • Flexible deployment and configuration – Packaged as a Helm chart, Cloudera Surveyor leverages Helm’s capabilities to simplify application installation and deployment, configuration, as well as management in Kubernetes environments.

  • High availability – Cloudera Surveyor supports active-active high availability configurations. This ensures minimal downtime and enables rolling restarts for updates and upgrades.

Key features

Cloudera Surveyor provides a comprehensive set of features for Kafka cluster management and monitoring.

User interface (UI)
Cloudera Surveyor offers a centralized and intuitive interface for managing and monitoring Kafka clusters. The UI eliminates the need for command-line tools and provides a single pane of glass for interacting with your Kafka deployments.
Multi-Cluster support and tagging
You can manage multiple Kafka clusters from a single interface. Adding or removing clusters is as simple as specifying basic Kafka cluster connection properties in Cloudera Surveyor's configuration. Once registered, clusters are available for management and monitoring through the UI.

Administrators can assign tags to clusters for organizational purposes. For example, tags can be used to group clusters by department, geographical location, or environment (such as production, development, or QA). Tags can also indicate cluster versions, making it easier to filter and organize clusters in the UI.

Periodic Kafka snapshots
Cloudera Surveyor periodically fetches real-time operational metrics and metadata from configured Kafka clusters, providing up-to-date insights into cluster activity and performance trends.

  • Kafka entities such as brokers, topics (including internal topics), and consumer groups.

  • Metadata quorum, log directories, and configurations for topics and brokers.

  • Offset information for partitions.

Kafka entity alerts
Cloudera Surveyor includes robust alerting capabilities for various Kafka entities. Each entity instance has an associated status and alerts, helping you quickly identify and address issues. Alerts from sub-entities are rolled up into their parent entities for a comprehensive view. For example, cluster-level details include alerts from brokers, log directories, topics, partitions, and replicas.
You can configure trigger conditions for certain alerts on a per-cluster basis. Alerts are available for the following entities.

  • Clusters

  • Brokers

  • Log directories

  • Topics

  • Partitions

  • Replicas

  • Consumer groups

Broad Kafka compatibility
Cloudera Surveyor is compatible with any Kafka distribution that provides an API compatible with Apache Kafka 2.4.1 or higher. This includes Kafka clusters deployed with the Strimzi Cluster Operator in Cloudera Streams Messaging - Kubernetes Operator, Kafka clusters running in Cloudera on premises and Cloudera on cloud, as well as third-party Kafka distributions.
Management capabilities

Cloudera Surveyor offers a range of features allowing you to manage your Kafka environment.

  • Cluster management – View all configured Kafka clusters and their details, including status, alert summaries, and information on brokers, topics, partitions, replicas, consumer groups, and log directories.

  • Consumer group management – Manage consumer groups, including viewing details and assignments, deleting consumer groups, and resetting consumer group offsets.

  • Topic management – Manage topics within a cluster, including creating, deleting, increasing partition counts, as well as viewing and modifying topic configurations.

Security and authorization

Cloudera Surveyor prioritizes security with a secure-first deployment approach. It supports LDAP-based authentication, allowing you to verify user identities and manage access to the application. TLS encryption ensures that all communication channels are secure, protecting sensitive data in transit. Additionally, Kafka ACLs provide fine-grained authorization, enabling you to define and enforce access control policies for your Kafka environment.