How Hive ACID replication policy works

Hive ACID replication is a two-step process which consists of a DUMP operation on the source cluster and a LOAD operation on the target cluster. The results of the DUMP operation are stored in a staging directory on HDFS, and then it is loaded onto the target cluster.

The following sections explain how Hive ACID replication policies work, how to track scheduled replication policies, how synchronization works between the source and target clusters, monitoring, metrics, and other related information:

Phases of Hive ACID replication

Hive ACID replication operates in the following two phases:

Bootstrap phase

This phase captures a consistent snapshot of the source database, and then dumps the snapshot metadata into the staging location. Before the bootstrap process starts, the process waits for the hive.repl.bootstrap.dump.open.txn.timeout parameter to ensure all the active transactions are complete.

Incremental phase

The incremental replication process relies on the notification_log table as the source of truth. The table records event entries for every transaction performed on the source cluster.

note

During each incremental replication cycle, only the pending events are retrieved from the source cluster, dumped into the staging location (also known as the dump directory), and then replayed on the target cluster.
Incremental replication processes only the new events, therefore this phase is generally faster than the initial bootstrap phase. However, if there is a large event backlog, the time required for replication increases accordingly.
The retention period for these entries are controlled by metastore.repl.event.db.listener.timetolive and hive.repl.event.db.listener.timetolive parameters:
After the retention time expires, the ‘cleaner thread’ removes the older entries from the table.

Replication process in Cloudera Manager

Replication Manager performs the following steps after you create a Hive ACID replication policy in Cloudera Manager > Replication Manager:

Creates a REPL DUMP scheduled command on the source cluster. The REPL DUMP operation copies only the metadata from the source cluster and stores it in the staging location on HDFS on the target side.
Creates a REPL LOAD scheduled command on the target cluster. The REPL LOAD operation retrieves the metadata from the staging location and applies it to the target cluster. Additionally, REPL LOAD performs direct data copy from the source to the target cluster.

Tracking scheduled policies

The Hive ACID replication schedules are stored in the SCHEDULED_QUERIES table within the Hive database. Cloudera Manager does not have direct access to the Hive’s internal scheduled commands, therefore Cloudera Manager uses the Hive On Tez Replication Metrics Getter Command to continuously poll the details from the Hive database to the Cloudera Manager database.

This Hive On Tez Replication Metrics Getter Command retrieves the replication history from the following Hive tables:

SCHEDULED_EXECUTIONS table which tracks the scheduled commands.
REPLICATION_METRICS table which stores the replication statistics.
To monitor replication history and for error handling, click Actions > Show History for the required replication policy on the Cloudera Manager > Replication > Replication Policies page. The replication details that appear are retrieved from the REPLICATION_METRICS table.

Synchronization between the source and target clusters

The source and target clusters do not directly communicate with each other. Instead, the staging directory on HDFS acts as the common access point. You can schedule a replication policy to run at fixed intervals. For example, every N hours.

During each replication policy job run:

The DUMP and LOAD operations start simultaneously.
After the DUMP operation is complete, the operation creates a _finished_dump acknowledgment file.
After the LOAD operation is complete, the operation creates a _finished_load acknowledgment file.

Because of the lack of visibility between the source and target, one skip might occur during every replication cycle.

Understanding replication metrics: Table Count vs. Event Count

The bootstrap replication process operates on the specified tables, and the replication history shows the number of successfully replicated ACID-managed tables for the specific replication job run.

The incremental replication process relies on the notification_log events to track and replicate the changes. Therefore:

the replication history shows the Event Count which represents the number of events replicated during the incremental cycle.
the replication history shows the Table Count along with the event count for that cycle only if a new table is added on the source cluster which gets replicated during the incremental replication job run.

Errors, debugging failures, and misconfiguration issues

Types of errors

Hive ACID replication errors are categorized as:

Table 1.
Error Type	Error Code	Resolution
Recoverable errors	lower than 40000	Retry replication
Non-recoverable errors	greater than or equal to 40000	Requires manual intervention (re-bootstrap needed) note Handling non-recoverable errors If a non-recoverable error occurs, the replication process creates a _non_recoverable marker file in the staging directory containing the error details. The replication cannot proceed without the administrator intervention.

Debugging failures

Perform the following steps to debug a failure:

If an issue occurs, check the Show History page for more details about the failure.
If additional details are required, inspect the REPLICATION_METRICS and SCHEDULED_EXECUTIONS tables.
For deeper investigation, review the HS2 and HMS logs:
- DUMP failures → Check logs on the source cluster.
- LOAD failures → Check logs on the target cluster.

Misconfiguration

Some common issues that might appear because of misconfiguration include:

HS2 and HMS configuration mismatch: HiveServer2 (HS2) and the Hive Metastore (HMS) operate independently, therefore any configuration changes made in HS2 must also be applied to HMS.
Visibility differences: Configuration settings applied only to HS2 are reflected in the Beeline queries, but if the configuration setting pertains to HMS, then the value is read from HMS instead.
Best practice: To avoid inconsistencies and potential issues, always configure the settings in both HS2 and HMS as a rule of thumb. This ensures seamless replication and avoids unexpected behavior.

Re-bootstrap

To perform a re-bootstrap on a database, you must perform the following steps:

Drop the database on the target cluster - This step ensures a clean slate for replication.
Delete the existing replication policy – This step ensures there are no conflicts.
Re-create the replication policy and run it – This step triggers a new bootstrap process from the source.

These steps are necessary because of the complex dependencies that exist between the internal Hive tables that manage transactions and for data visibility. Failure to follow these steps might result in data visibility issues or replication incompatibilities.

For more information, see https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development