Cloudera Docs

Streams Replication Manager security example

Streams Replication Manager is capable of replicating data between Cloudera and non-Cloudera clusters that are secured. A setup like this involves configuring security for the Streams Replication Manager service (Driver and Service roles) and the srm-control command line tool.

Consider the following replication environment.

There are two clusters, US-West and US-East. US-West is a Cloudera cluster that has both Kafka and Streams Replication Manager deployed on it. The Kafka service on this cluster is both Kerberos and TLS/SSL enabled. That is, clients connecting to this cluster (including Streams Replication Manager) use the SASL_SSL protocol.

US-East has Kafka deployed on it but not Streams Replication Manager. Kafka on this cluster has TLS/SSL encryption enabled. More importantly, the platform that this cluster is running on is not defined. This is because this example is meant to demonstrate that Streams Replication Manager can connect to and replicate data to or from non-Cloudera clusters. As long as that cluster is running Kafka, Streams Replication Manager will be able to connect to it.

Data replication is unidirectional. The Streams Replication Manager service deployed in US-West is replicating Kafka data from US-East to US-West. From the perspective of the Streams Replication Manager service, US-West is its co-located Kafka cluster, while US-East is an external Kafka cluster.

The following example walks you through the steps required to set up both the Streams Replication Manager service and the srm-control tool for this replication environment.

The following steps assume that the Streams Replication Manager service is already installed and available on US-West. Additionally, it is also assumed that key and truststore files as well as other credentials required to establish a connection with each cluster are known and are available. The instructions do not go into detail on how you can generate or acquire these credentials.

  1. Define the external Kafka cluster (US-East):
    1. Log into the Cloudera Manager instance managing US-West.
    2. Go to Administration > External Accounts.
    3. Go to the Kafka Credentials tab.
    4. Click Add Kafka credentials.
    5. Create a Kafka credential for US-East:
      The security configuration of the external cluster determines which of the available properties you must set. Because in this example US-East has TLS/SSL encryption enabled, the following properties must be set:
      • Name
      • Bootstrap servers
      • Security protocol
      • Truststore Password
      • Truststore Path
      • Truststore Type
    6. Click Add.
      If credential creation is successful, a new entry corresponding to the Kafka credential you specified appears on the page.
  2. Define the co-located Kafka cluster (US-West):
    Co-located Kafka clusters can be defined in two separate ways. Either with a service dependency or with Kafka credentials. Because in this example US-West is Kerberos and TLS/SSL enabled, the service dependency method is used. This is in-line with Cloudera best practices.
    1. In Cloudera Manager, go to Clusters and select the Streams Replication Manager service.
    2. Go to Configuration.
    3. Find and enable the Kafka Service property.
    4. Find and configure the Streams Replication Manager Co-located Kafka Cluster Alias property.
      The alias you configure represents the co-located cluster. Enter an alias that is unique and easily identifiable. For example:
      uswest
    5. Enable relevant security feature toggles
      Because US-West is both TLS/SSL and Kerberos enabled, you must enable all feature toggles for both the Driver and Service roles. The feature toggles are the following:
      • Enable TLS/SSL for SRM Driver
      • Enable TLS/SSL for SRM Service
      • Enable Kerberos Authentication
  3. Add both clusters to the configuration of Streams Replication Manager:
    1. Find and configure the External Kafka Accounts property.
      Add the name of the Kafka credential you created for US-East to this property. This can be done by clicking the add button to add a new line to the property and then entering the name of the Kafka credential. For example:
      useast
    2. Find and configure the Streams Replication Manager Cluster alias property.
      Add all cluster aliases to this property. This includes the aliases present in both the External Kafka Accounts and Streams Replication Manager Co-located Kafka Cluster Alias properties. Delimit the aliases with commas. For example:
      useast, uswest
  4. Configure replications:
    In this example, data is replicated unidirectionally from US-West to US-East. As a result, only a single replication must be configured.
    1. Find the Streams Replication Manager's Replication Configs property.
    2. Click the add button and add new lines for each unique replication you want to add and enable.
    3. Add and enable your replications.
      For example:
      useast->uswest.enabled=true
  5. Configure Driver and Service role targets:
    1. Find and configure the Streams Replication Manager Service Target Cluster property.
      Add the co-located cluster’s alias to the property. In the case of this example:
      useast
    2. Find and configure the Streams Replication Manager Driver Target Cluster property.
      This property should contain all cluster aliases. In the case of this example, you can either add the aliases of both US-West and US-East or leave the property empty. Leaving the property empty is the same as adding all aliases.
  6. Click Save Changes.
  7. Restart the Streams Replication Manager service.
  8. Configure and run the srm-control tool:
    1. Click Gateway in the Filters pane.
    2. Find and configure the following properties:
      SRM Client's Secure Storage Password
      The password used to access the secure storage. Take note of the password you configure. You need to provide it in your CLI session before running the tool.
      Environment Variable Holding SRM Client's Secure Storage Password
      The name of the environment variable that stores the secure storage password. Take note of the name that you configure. You need to set it in your CLI session before running the tool.
      Gateway TLS/SSL Trust Store File
      The path to the TLS/SSL truststore file containing the server (co-located cluster’s) certificate and public key. Ensure that this file is available on all Streams Replication Manager hosts.
      Gateway TLS/SSL Truststore Password
      The password used to access the truststore file specified in the Gateway TLS/SSL Trust Store File property.
      SRM Client's Kerberos Principal Name
      The kerberos principal that the tool uses for authentication when connecting to the co-located Kafka cluster.
      SRM Client's Kerberos Keytab Location
      The path to the Kerberos keytab file that the tool uses for authentication when connecting to the co-located Kafka cluster.
    3. Click Save Changes.
    4. Re-deploy client configuration.
    5. SSH into one of the Streams Replication Manager hosts in your cluster.
    6. Set the secure storage password as an environment variable.
      export [***SECURE STORAGE ENV VAR***]=”[***SECURE STORAGE PASSWORD***]”
      Replace [***SECURE STORAGE ENV VAR***] with the name of the environment variable you specified in Environment Variable Holding SRM Client's Secure Storage Password. Replace [***SECURE STORAGE PASSWORD***] with the password you specified in SRM Client's Secure Storage Password. For example: 
      export SECURESTOREPASS=”mypassword
    7. Run the tool.
      For example:
      srm-control topics --source useast --target uswest --add topic1,topic2