Cloudera Docs

Migrate Kafka Using the DefaultReplicationPolicy

How to migrate Kafka with Streams Replication Manager using the DefaultReplicationPolicy.

  • Be aware that remote topics on the source cluster are renamed during the migration.

    As a result, you must make significant changes to all Kafka producers and consumers. Otherwise, they will not be able to connect to the correct topics in the Cloudera Private Cloud Base cluster. If this behavior is not suitable, review Migrate Kafka Using Streams Replication Manager and use the IdentityReplicationPolicy or the MigratingReplicationPolicy instead.

  • Setup and Configure Streams Replication Manager in the Cloudera Private Cloud Base cluster.

    For more information, see Add and Configure Streams Replication Manager and the Streams Replication Manager Configuration Examples. Setup instructions might differ in different versions, ensure that you view the version of the documentation that matches your Runtime version.

  1. Use the srm-control tool to include every topic and every consumer group.

    Including consumer groups is required for checkpointing.

    srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add ".*"

    srm-control groups --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add ".*"
  2. Validate that data is being migrated.

    Use the Cluster Replications page on the Streams Messaging Manager UI to monitor and validate the status of the migration. Alternatively, you can use the various endpoints available on the Streams Replication Manager Service REST API.

  3. Wait until replication is caught up.
  4. Stop producers.
  5. Stop consumers.
  6. Reconfigure all consumers to read from Cloudera Private Cloud Base Kafka and apply offset translation using Streams Replication Manager.
  7. Start consumers.
  8. Reconfigure all producers to write to Cloudera Private Cloud Base Kafka.

    The HDF instances of Kafka and Streams Messaging Manager are no longer required.

  9. Start producers.

Kafka is migrated. Kafka clients produce and consume from the Cloudera Private Cloud Base cluster. Migration is complete.