Replication Manager on Cloudera Base on premises

Cloudera Base on premises Replication Manager can replicate HDFS directories, Hive external tables, Impala data, Hive ACID tables, Iceberg tables, Ranger policies and roles for HDFS, Hive, and HBase services, and data in Ozone buckets.

See the following sections for the supported cluster and runtime versions:

Replicate from CDH and Cloudera Base on-premises source clusters section lists the cluster and runtime versions to:
- replicate data from CDH source clusters
- replicate data between Cloudera Base on premises clusters using same storage
- replicate data between Cloudera Base on premises clusters using different storage
Replicate HDFS and Hive data to cloud storage
Replicate from HDP 2 and HDP 3 source clusters

Replication policies support the following scenarios:

Kerberos

Replication Manager supports the following replication scenarios when Kerberos authentication is used on a cluster:

Secure source to a secure destination.
Insecure source to an insecure destination.
Insecure source to a secure destination. The following requirements must be met for this scenario:
- When a destination cluster has multiple source clusters, all the source clusters must either be secure or insecure. Replication Manager does not support a mix of secure and insecure source clusters.
- The destination cluster must run Cloudera Base on premises 7.x or higher.
- The source cluster must run a compatible Cloudera Base on premises version.
- This replication scenario requires additional configuration. For more information, see Replicating from unsecure to secure clusters .

Transport Layer Security (TLS)

You can use TLS with Replication Manager. Additionally, Replication Manager supports replication scenarios where TLS is enabled for non-Hadoop services (Hive/Impala) and TLS is disabled Hadoop services (such as HDFS,YARN, and MapReduce).

Apache Knox

When Cloudera Base on premises is configured with Knox and the source and target clusters are Knox-SSO enabled, you must ensure that you use the Cloudera Base on premises port in the peer URL when you add the source and target clusters as peers.

Replicate from CDH and Cloudera Base on premises source clusters

The following tables list the source and destination clusters, lowest supported versions of Cloudera Manager, and the services that are available for each supported cloud provider for CDH and Cloudera Base on premises source clusters; ensure that the target database name is the same as the source database name, otherwise issues appear during or after data replication:

Table 1. Replicate data from CDH source clusters
Source cluster	Lowest supported source Cloudera Manager version	Lowest supported source Cloudera Runtime version	Lowest supported destination cluster version	Supported services on Replication Manager
CDH 5 CDH 6	6.3.0	5.10	Cloudera Base on premises 7.0.3	HDFS, Sentry to Ranger*, Hive external tables
*To perform Sentry to Ranger replication using HDFS and Hive external table replication policies, you must have installed Cloudera Manager version 6.3.1 and higher on the source cluster and Cloudera Manager version 7.1.1 and higher on the target cluster.

Table 2. Replicate data between Cloudera Base on premises clusters using same storage
Source cluster	Lowest supported source Cloudera Manager version	Lowest supported source Cloudera Runtime version	Destination cluster	Supported services on Replication Manager
Cloudera Base on premises	7.1.1	7.1.1	Cloudera Base on premises	HDFS Hive external tables
Cloudera Base on premises	7.7.1	7.1.8	Cloudera Base on premises	Hive ACID tables* Use Cloudera Manager APIs to replicate Ozone buckets.
Cloudera Base on premises	7.7.1 CHF4	7.1.8	Cloudera Base on premises	Ozone buckets
Cloudera Base on premises	7.11.3	7.1.9	Cloudera Base on premises	Iceberg tables Ranger policies and roles, and Ranger audit logs in HDFS**
Cloudera Base on premises	7.11.3 CHF7	7.1.9 SP1	Cloudera Base on premises	Atlas replication policies***
You can use REPL commands or Replication Manager to replicate Hive ACID tables between Cloudera Base on premises 7.1.8 or higher versions using Cloudera Manager versions 7.7.1 or higher. You can also create Ranger replication policies on Kerberos-enabled Cloudera Base on premises 7.1.8 or higher clusters using Cloudera Manager 7.7.1 CHF6 and higher, if the Ranger replication feature flag is enabled. **Replicating Atlas metadata using Hive external table replication policies and Iceberg replication policies, and replicating the metadata and data lineage of all the Hive external tables, Iceberg tables, and any other Atlas supported entities in the source cluster to the target cluster using Atlas replication policies is a technical preview feature. It is `not` recommended for production deployments. Cloudera recommends that you try this feature in development or test environments. To enable this feature, contact your Cloudera account team.

Table 3. Replicate data between Cloudera Base on premises clusters using different storage
Source cluster	Lowest supported source Cloudera Manager version	Lowest supported source Cloudera Runtime version	Destination cluster	Supported services on Replication Manager
Cloudera Base on premises	7.11.3 CHF1	7.1.9	Cloudera Base on premises	Replicate the data and metadata for Hive external tables from: source cluster using HDFS to a target cluster using Dell EMC Isilon storage. source cluster using Dell EMC Isilon storage to a target cluster using HDFS.
Cloudera Base on premises	7.11.3 CHF2	7.1.9	Cloudera Base on premises	Replicate Hive ACID tables and Iceberg tables from: source cluster using HDFS to a target cluster using Dell EMC Isilon storage. source cluster using Dell EMC Isilon storage to a target cluster using HDFS.
Cloudera Base on premises	7.11.3 CHF7	7.1.9 SP1	Cloudera Base on premises	Replicate metadata-only for Ozone storage-backed Hive external tables using Hive external table replication policies. You must replicate the data using Ozone replication policies.

important

Hive external table replication policies do not support managed to managed table replication. When you replicate from a CDH cluster to a Cloudera Base on premises cluster, Replication Manager converts managed tables to external tables.

Therefore, to replicate managed tables (ACID) and external tables in a database successfully, you must perform the following steps in the order shown below:

Create Hive ACID table replication policy for the database to replicate the managed data.
After the replication completes, create the Hive external table replication policy to replicate the external tables in the database.

Ensure that the target cluster version is Cloudera Base on premises 7.1.8 or higher.

Replicate HDFS and Hive data to cloud storage

Cloudera Base on premises Replication Manager supports the following replication scenarios:

Replicate to and from Amazon S3 from CDH 5.14+ and Cloudera Manager version 5.13+.
Replication Manager does not support S3 as a source or destination when S3 is configured to use SSE-KMS.
Replicate to and from Microsoft ADLS Gen1 from CDH 5.13+ and Cloudera Manager 5.15, 5.16, 6.1+.
Replicate to Microsoft ADLS Gen2 (ABFS) from CDH 5.13+ and Cloudera Manager 6.1+.
Supports snapshots from CDH 5.15+ and Cloudera Manager 5.15+.
Replicate HDFS and Hive external tables from Cloudera Base on premises 7.1.9 CHF3 and higher clusters using Dell EMC Isilon storage to Cloudera on cloud clusters on AWS, Azure, and GCP.
Replicate HDFS and Hive external tables from Cloudera Base on premises 7.1.9 SP1 and higher to Cloudera on cloud clusters on GCP.

Starting in Cloudera Manager 6.1.0, Replication Manager ignores Hive tables backed by Kudu during replication. The change does not affect functionality since Replication Manager does not support tables backed by Kudu. This change was made to guard against data loss due to how the Hive Metastore, Impala, and Kudu interact.

Replicate from HDP 2 and HDP 3 source clusters

Replicating to and from HDP to Cloudera Manager 7.x is not supported by Replication Manager. However, you can replicate data using other methods. The following table lists the methods and the supported data replications to Cloudera Base on premises clusters that are supported:

Table 4. Replicate data from HDP 2 and HDP 3 source clusters
Lowest supported source version	Services that require alternate replication methods
HDP 2.6.5	HDFS. Use DistCp to replicate data.
HDP 3.1.1	HDFS. Use DistCp to replicate data.
HDP 3.1.1	HBase. Use HBase replication to replicate HBase data. Hive external tables. For information to replicate data, contact Cloudera Support.
HDP 3.1.5	Hive ACID tables to Cloudera 7.1.6 and higher clusters. Use REPL commands to replicate data. note Requires HDP 3.1.5 hotfixes.