Cloudera Docs

Installing Streams Messaging Manager on an existing Cloudera Private Cloud Base cluster

Learn how to install Streams Messaging Manager on an already running Cloudera Private Cloud Base cluster.

Overview

The following sections guide you through the process of installing Streams Messaging Manager on an already running cluster. Installing Streams Messaging Manager involves adding the service to the cluster and enabling optional integration between Streams Messaging Manager and other services like Schema Registry, Streams Replication Manager and Cruise Control, or service roles like Kafka Connect.

Integration with other services

Streams Messaging Manager is capable of integrating with other services. Enabling integration enables additional features and functions on the Streams Messaging Manager UI. For example, enabling integration with Streams Replication Manager enables the Cluster Replications section in Streams Messaging Manager, which you can use to monitor Streams Replication Manager replications.

If other services that Streams Messaging Manager integrates with are located on the same cluster as Streams Messaging Manager, enabling the integration happens when you add Streams Messaging Manager to your cluster. However, Streams Messaging Manager can integrate with some services, like Streams Replication Manager, even if the service is located on a remote cluster. In this case, the integration between Streams Messaging Manager and the remote service is configured after you add Streams Messaging Manager to your cluster. Additionally, the integration with other services can be enabled, disabled, or further configured separately following installation. As a result, the service integration steps are also documented as separate tasks.

Deployment recommendations

The Streams Messaging Manager service is comprised of two service roles, which are as follows:

  • Streams Messaging Manager Rest Admin Server: This role is responsible for processing Rest API requests from the Streams Messaging Manager UI and collecting metrics information from Cloudera Manager and Kafka.
  • Streams Messaging Manager UI Server: This role provides the Streams Messaging Manager UI. You use the UI to manage and monitor Kafka and related services.

Cloudera recommends that you add the Streams Messaging Manager service to a management host on your cluster. Cloudera does not recommend adding the Streams Messaging Manager service to hosts where Kafka broker roles are running. Each cluster can only have a single instance of the Streams Messaging Manager service deployed on it. In addition, the two Streams Messaging Manager roles must be installed on the same host.

Cloudera Manager and Cloudera Service Monitor requirements

Streams Messaging Manager requires high levels of Cloudera Manager and Service Monitor Service (SMON) memory usage. Ensure that both are tuned according to Cloudera recommendations.

Prometheus

By default Streams Messaging Manager uses Cloudera Manager as its metrics store. This means that the metrics that are available for monitoring on the Streams Messaging Manager UI are sourced from Cloudera Manager. If required, you can create a deployment where instead of Cloudera Manager, Streams Messaging Manager uses Prometheus as its metrics store. Prometheus is an open-source systems monitoring and alerting toolkit that stores its metrics as time series data.

Prometheus supports a significantly larger number of time series entities compared to Cloudera Manager. As a result, Cloudera recommends that you use Prometheus instead of Cloudera Manager if you need to monitor a large number of Kafka entities (topics, partitions and so on). Additionally, using Prometheus enables you to configure the roll up policy, delete specific time series entities, and configure scrape intervals and metrics retention periods. Prometheus setup for Streams Messaging Manager is done after adding Streams Messaging Manager service to your cluster.

Adding the Streams Messaging Manager service to an existing cluster

Learn how to add the Streams Messaging Manager service to an already running cluster.

The following list of steps walk you through how you can add Streams Messaging Manager on an already existing Cloudera Private Cloud Base Cluster. The following steps are aimed to provide you with the basic process of installation and do not go into detail how you can set up or configure security for the service.

  • Ensure that a Kafka service is installed on your cluster. Additionally, ensure that the Enable Producer Metrics Kafka service property is selected. This property is enabled by default.
  • Streams Messaging Manager requires a backend database to function. The database is used to store Kafka metadata, metrics, and alert definitions. As a result, a database and database user must be set up for Streams Messaging Manager before adding the service. The following SQL snippet gives an example of the setup that you are required to do.
    create database streamsmsgmgr;
CREATE USER 'streamsmsgmgr'@'localhost' IDENTIFIED BY 'streamsmsgmgr';
GRANT ALL PRIVILEGES ON streamsmsgmgr.* TO 'streamsmsgmgr'@'localhost' WITH GRANT OPTION;
CREATE USER 'streamsmsgmgr'@'%' IDENTIFIED BY 'streamsmsgmgr';
GRANT ALL PRIVILEGES ON streamsmsgmgr.* TO 'streamsmsgmgr'@'%' WITH GRANT OPTION;
    For more information on database setup as well as supported databases. Review the following resources:
  • Streams Messaging Manager is capable of establishing a secure (encrypted) connection with its database if the database has TLS 1.2/TCPS enabled. This is done by specifying the database connection using a JDBC URL. If you plan on using an encrypted connection, ensure that you have a valid JDBC URL on hand and have completed additional prerequisites. For more information and example JDBC URLs, see Database setup details for Streams Messaging Manager for TLS 1.2/TCPS-enabled databases.
  1. In Cloudera Manager, select the cluster that you want to install Streams Messaging Manager on.
  2. Click Actions > Add Service.
  3. Select Streams Messaging Manager from the list of services and click Continue.
  4. Select service dependencies.
    The options on the Select Dependencies page differ depending on what other services are available on your cluster. Review the following and select the dependencies based on your requirements.
    Schema Registry
    If selected, integration between Streams Messaging Manager and Schema Registry will be enabled. Schema Registry related features become available on the Streams Messaging Manager UI. This is a recommended optional dependency.
    Streams Replication Manager
    If selected, integration between Streams Messaging Manager and Streams Replication Manager will be enabled. Streams Replication Manager integration enables the Cluster Replications section on the UI, which you use to monitor replications. This is a recommended optional dependency.
    Kafka
    Streams Messaging Manager requires a Kafka service as a dependency. Kafka services will only be visible on this page if there are multiple Kafka services available on the cluster. Otherwise, if only a single Kafka is available, Streams Messaging Manager automatically selects it as a dependency as Kafka is a mandatory dependency.
  5. Click Continue.
  6. Assign role instances to hosts.
    On the Assign Roles page you select which host you want to install Streams Messaging Manager roles on. Because both Streams Messaging Manager roles must be installed on the same host, you can only select a host for the Rest Admin Server role. The UI Server role is installed on the same host.

    A host is selected by default. If required you can change the host by doing the following:

    1. Click the field below Streams Messaging Manager Rest Admin Server to display a dialog containing a list of hosts.
    2. Select a single host and click Ok.
  7. Click Continue.
  8. Set up the database connection.
    The database that you specify here is the one that you already set up for Streams Messaging Manager prior to installation.
    1. Select the database type and enter the database name, username, and password.
    2. Click Test Connection.
      If the connection test fails, review your configuration, fix any errors, and rerun the connection test. You can only continue if the validation check is successful.
  9. Click Continue.
  10. Review changes.
    The Review Changes pages lists default and suggested settings for several configuration properties. Review and if required configure these properties.
  11. Click Finish.

The Streams Messaging Manager service is added to your cluster, however, it is not yet started.

Depending on your requirements, you have multiple options on how to continue. Choose one of the following.

  • Start the Streams Messaging Manager service. To do this, click Actions > Start on the homepage of the Streams Messaging Manager service.
  • Learn more about or enable integration with other services and service roles. Continue with service specific integration steps.
  • Set up Prometheus as the metrics store. Continue with Setting up Prometheus for Streams Messaging Manager.

Integrating Schema Registry with Streams Messaging Manager

Learn how to configure Streams Messaging Manager to connect and integrate with Schema Registry. Enabling integration between the two services enables Schema Registry related functionality within the Streams Messaging Manager UI.

Streams Messaging Manager is capable of connecting to and integrating with a Schema Registry service. Enabling integration makes it possible for the Streams Messaging Manager UI Data Explorer to use schemas stored in Schema Registry to deserialize Kafka messages. Additionally, the Data Explorer will include a quick access link to the schema associated with the topic you are viewing.

  • In most cases Schema Registry integration is enabled when you add the Streams Messaging Manager service to the cluster. As a result, it is highly likely that integration is already enabled. If integration is enabled, you do not need to complete the following steps.
  • Streams Messaging Manager can only integrate with a Schema Registry service if the service is co-located with Streams Messaging Manager. Specifying remote Schema Registry services is not possible.
  1. In Cloudera Manager, select the Streams Messaging Manager service.
  2. Go to Configuration.
  3. Find and select the Schema Registry Service property.
  4. Click Save Changes.
  5. Restart Streams Messaging Manager.
Integration between Streams Messaging Manager and Schema Registry is enabled. Schema Registry related functionality is now available in the Data Explorer of the Streams Messaging Manager UI.

Integrating Streams Replication Manager with Streams Messaging Manager

Learn how to configure Streams Messaging Manager to connect and integrate with Streams Replication Manager. Enabling integration between the two services enables Streams Replication Manager related functionality within the Streams Messaging Manager UI.

Streams Messaging Manager is capable of integrating with Streams Replication Manager. Enabling integration between the two services enables the Cluster Replications section within the Streams Messaging Manager UI. You can use this section of the UI to monitor the replication flows you create with Streams Replication Manager.

Service integration is enabled in either of two ways. If Streams Replication Manager is located on the same cluster as Streams Messaging Manager, you integrate the two services by enabling a service dependency. If the two services are located remote to each other (on different clusters), you need to manually specify the host, port, and protocol of one or multiple Streams Replication Manager Service role instances that Streams Messaging Manager should interface with.

  • Ensure that a co-located or remote Streams Replication Manager service is available.
  • If Streams Replication Manager is co-located with Streams Messaging Manager, it is likely that integration between the two services was enabled when you added Streams Messaging Manager to your cluster. Check if the Cluster Replications section is available in the Streams Messaging Manager UI. If it is, you do not need to complete the following steps.
  1. In Cloudera Manager, select the Streams Messaging Manager service.
  2. Go to Configuration.
  3. Enable integration between the two services.
    How you complete this step depends on whether Streams Replication Manager is colocated with Streams Messaging Manager. Use a service dependency if the two services are colocated. Otherwise, specify the connection using dedicated properties.
    1. Find and select the STREAMS_REPLICATION_MANAGER Service property.
    1. Find and select the Configure Streams Replication Manager Manually property.
    2. Find and configure the Streams Replication Manager Rest Protocol property.

      Select http or https depending on configuration of Streams Replication Manager. If Enable TLS/SSL for Streams Replication Manager Service is selected, select https, otherwise, select http.

    3. Find and configure the Streams Replication Manager Rest HostPorts property.

      Enter the host and port pairs of one or multiple the Streams Replication Manager Service roles. You can find the hostname by going to Streams Replication Manager > Instances. For the port, use the value set in the Streams Replication Manager Service Https Port or Streams Replication Manager Service Port properties. If the Streams Replication Manager Service roles are TLS/SSL enabled (Enable TLS/SSL for Streams Replication Manager Service is selected), use the port in Streams Replication Manager Service Https Port. Otherwise, use the port in Streams Replication Manager Service Port.

  4. Click Save Changes.
  5. Restart Streams Messaging Manager.
Integration between Streams Messaging Manager and Streams Replication Manager is enabled. The Cluster Replications tab is now available on the Streams Messaging Manager UI.
Go to the Streams Messaging Manager UI and access the Cluster Replications section from the navigation sidebar.

Integrating Kafka Connect with Streams Messaging Manager

Learn how to set up Streams Messaging Manager for monitoring and managing Kafka Connect.

Cloudera recommends that you use Streams Messaging Manager to manage and monitor Kafka Connect. In order for Streams Messaging Manager to be able to interact with Kafka Connect, it must be configured and provided with the host, port, and protocol of the Kafka Connect role instances. This is done by configuring the Kafka Connect Rest HostPort and Kafka Connect Protocol Streams Messaging Manager configuration properties. Configuring these properties enables the Connect section in Streams Messaging Manager, which allows you to interact with Kafka Connect through a UI interface. Additionally, configuring these properties also enables you to use the Streams Messaging Manager REST API to interact with Kafka Connect.

If the Kafka Connect Rest HostPort property is left empty (default), Streams Messaging Manager is automatically configured with the host, port, and protocol of all Kafka Connect role instances belonging to the Kafka service selected with the Kafka Service Streams Messaging Manager property. This means that if you want Streams Messaging Manager to monitor and manage the Kafka Connect instance that belongs to the Kafka service that Streams Messaging Manager depends on, you do not need to complete the following steps.

  1. Select the Streams Messaging Manager service.
  2. Go to Configuration.
  3. Find and configure the following properties.
    • Kafka Connect Rest HostPort

      Enter the host and port pairs of the Kafka Connect roles. If you have multiple instances of the Kafka Connect role, Cloudera recommends that you add all of them for high availability. If there are multiple host and port pairs specified, Streams Messaging Manager will automatically failover between the specified endpoints. For the port, use the value set in the Kafka Connect Rest Port or Secure Kafka Connect Rest Port Kafka Connect properties. If the Kafka Connect role is not TLS/SSL enabled, use the port specified in Kafka Connect Rest Port. If TLS/SSL is enabled, use the port specified in Secure Kafka Connect Rest Port.

    • Kafka Connect Protocol

      Select http if SSL/TLS is not enabled for the Kafka Connect roles. Select https if SSL/TLS is enabled for the Kafka Connect roles.

  4. Click Save.
  5. Restart the service.
Streams Messaging Manager is configured and is able to manage and interact with Kafka Connect.
Go to the Streams Messaging Manager UI and access the Connect section from the left column. Alternatively, use the Streams Messaging Manager REST API to interact with Kafka Connect.

Authorizing users to access Cruise Control in Streams Messaging Manager

Learn how to authorize users to access the Cruise Control UI in Streams Messaging Manager. This is required to enable users access the UI or manage rebalancing. You can set access by assigning users the required authorization levels.

The integration between Streams Messaging Manager and Cruise Control is automatic. The Cruise Control User Interface (UI) can be accessed through an Streams Messaging Manager instance running on the same cluster as Cruise Control. In environments where security is disabled, anyone will be able to access Cruise Control in Streams Messaging Manager. In secured environments, users need to be added to the authorization level groups before they can see the Cruise Control UI. The group a user belongs to determines what they are allowed to do in the UI.

  1. Select the Cruise Control service in Cloudera Manager.
  2. Click Configuration.
  3. Search for auth_admins, auth_users or auth_viewers based on which authorization level is needed.
  4. Assign users to the required authorization level.
    Based on the authorization level, users have the following access to Cruise Control in Streams Messaging Manager:
    • Viewer level users: can only view the metrics
    • User level users: includes Viewer level permissions and manage goals, estimations
    • Admin level users: includes Viewer and User level permissions, starting and stopping rebalance
  5. Click Save changes.
Cruise Control is visible in Streams Messaging Manager. Users can access the Cruise Control UI using the on the navigation sidebar.