Known Issues

In the diagnostic bundle, open the service-diagnostics/[***CLUSTER NAME***]/[***KNOX SERVICE NAME***]/ folder. The Knox Gateway role diagnostics archive there includes the gateway logs (for example gateway.log and gateway-audit.log).

If a deployment fails due to a stale alternatives state, manually clear the temporary files on the affected host and retry the deployment from the Cloudera Manager UI.

1. Verify no active alternatives processes: SSH into the affected host and ensure no other instances of the alternatives tool are currently running:

   pgrep -af 'update-alternatives|alternatives'

   If the command returns no active processes, proceed to Step 2.

2. Identify the stale temporary files:List the temporary files to confirm which entries are blocked:

   ls -l /var/lib/alternatives/*.new

   To check a specific service (for example, HDFS or Hive), use:

   ls -l /var/lib/alternatives/<alt_name>*

3. Back up and remove the stale .new files: Create a backup of the stale file in a temporary directory before deleting it:

   sudo cp -a /var/lib/alternatives/<alt_name>.new /var/tmp/<alt_name>.new.$(date +%s).bak
   sudo rm -f /var/lib/alternatives/<alt_name>.new

4. Verify the alternatives state: Confirm the current status of the link:

   /usr/sbin/update-alternatives --display <alt_name>

5. Retry the operation: Return to the Cloudera Manager UI and re-run Deploy Client Configuration for the affected service.

If services fail to start after an upgrade due to missing alternatives or symbolic links (symlinks), manually complete the interrupted activation steps on the affected host or use the Cloudera Manager UI to reconcile the state.

1. Verify the missing symlink: SSH into the affected host machine and check the status of the failing JAR or symlink. For example:

   /usr/sbin/update-alternatives --display ozone-filesystem-hadoop3.jar

   If the output shows the link is missing or broken, proceed to Step 2.

2. Manually run the interrupted command:

   Search the Agent log at /var/log/cloudera-scm-agent/cloudera-scm-agent.log for Exit Code: -15 to locate the failed update-alternatives command immediately preceding that error. Copy the full path to the parcel library from that log entry.

   For the ozone-filesystem-hadoop3.jar failure, run the following installation command manually as the root user.:

   sudo /usr/sbin/update-alternatives --install /var/lib/hadoop-hdfs/ozone-filesystem-hadoop3.jar ozone-filesystem-hadoop3.jar /opt/cloudera/parcels/<CDH-VERSION-PATH>/lib/hadoop-ozone/ozone-filesystem-hadoop3.jar 5

3. Verify the fix: Run the display command again to ensure the link now correctly points to the new parcel directory:

   /usr/sbin/update-alternatives --display ozone-filesystem-hadoop3.jar

4. Restart Services: After you verify the fix, restart the failing service (such as Solr) through the Cloudera Manager UI.

Option 2: Cloudera Manager UI (Automatic Fix)

Instead of manual CLI intervention, you can force Cloudera Manager to recreate all symbolic links:

1. Navigate to the Hosts > Parcels page in the Cloudera Manager UI.

2. Select the affected parcel and click Activate again. This process declaratively identifies and recreates any missing symbolic links across all hosts the cluster.

If you encounter this error, do not restart the finalization command. Instead, manually verify the progress directly on the SCM by following these steps:

1. Check the Status: Run the following command from the command line to monitor the actual SCM finalization state:

   ozone admin scm finalizationstatus

2. Wait for Completion: Monitor the output until it indicates that finalization is complete.

3. Verify in Cloudera Manager: Once the CLI command confirms a successful finalization, you can safely ignore the previous failure message in Cloudera Manager and proceed with your post-upgrade tasks.

Use MySQL or PostgreSQL for the Knox gateway database when you run Knox in HA with JWT token features.

If you encounter an error stating that a controller is "missing" or "not enabled at root," follow these steps to restore and persist the controllers.

f you encounter service startup failures on large clusters, manually increase the fetch timeout through the Cloudera Manager Admin Console:

1. Log in to the Cloudera Manager Admin Console.

2. Navigate to Administration > Settings.

3. Search for the parameter: Cloudera Manager Descriptor Fetch Timeout.

4. Increase the value from the default 10 seconds to 60 seconds.

5. Click Save Changes.

6. Restart the Cloudera Management Service.

API Endpoint: POST /api/v57/hosts/reallocateMemory

Include the host name (the host where HiveServer and management roles run) in the API request body. This ensures that memory assignments are recomputed correctly, taking all roles on the host into account.

<property>
<name>ozone.snapshot.deep.cleaning.enabled</name>
<value>false</value>
</property>

<property>
<name>ozone.snapshot.ordered.deletion.enabled</name>
<value>true</value>
</property>

ln -s /opt/cloudera/parcels/CDH/lib/bigtop-utils/bigtop-detect-javahome /opt/cloudera/parcels/CDH/bin/bigtop-detect-javahome

To mitigate this issue and prevent startup failures after a Datahub restart, you can perform either of the following actions, or both:

• Reduce the Java heap size for affected services to prevent nodes from exceeding the available memory.
• Increase physical memory for on cloud or on-premises instances running the affected services.

1. ssh to Knox host.
2. export KNOX_GATEWAY_DATA_DIR="/var/lib/knox/idbroker/data"; export KNOX_GATEWAY_CONF_DIR="/var/lib/knox/idbroker/conf"
3. /opt/cloudera/parcels/CDH/lib/knox/bin/knoxcli.sh create-alias <ALIAS_NAME> --cluster <CLUSTER_NAME> --value <ALIAS_VALUE>
4. Verify the addition using /opt/cloudera/parcels/CDH/lib/knox/bin/knoxcli.sh list-alias --cluster <CLUSTER_NAME>

For HA deployments, users must do it on every Knox hosts (whereas the Save Alias command applies the change to all hosts automatically).

1. You must open Cloudera Manager on the second browser and restart the Ozone service separately.
2. After the Ozone service restarts, you can resume the cluster restart from the first browser.

The assumption is that you can access the external custom repository (such as Nexus or JFrog, or others) using LDAP credentials. In case an applicative user is used to fetch the external content (as done in Data Services with the docker imager repository), the customer should ensure that this applicative user is located under the user's base search path where the real users are being retrieved during LDAP authentication check (so the external repository will find it and will allow it to gain access for fetching the files).

Once done, you can use the current custom URL fields in the Cloudera Manager Wizard and enter the URL for the RPMs or parcels/other files in the format of "https://USERNAME:PASSWORD@server.example.com/XX".

While using the password, you are advised to use only the printable ASCII character range (excluding space), whereas in case of a special character (not letter/number) it can be replaced with HEX value (For example, you can replace Aa1234$ with Aa1234%24 as '%24' is translated into $ sign).

1. Shutdown the Passive Cloudera Manager Server.
2. Add and manage the parcel as usual, as described in Install Parcels.
3. Restart the Passive Cloudera Manager server after parcel operations are complete.

In case your OS is either RHEL 9 or SLES 15 or Ubuntu 2004 or Ubuntu 2204 and if you install the Cloudera Manager 7.13.1.x version, then create a symbolic link using root user privileges on the node that host the Service Monitor service (cloudera-scm-firehose) at /opt/cloudera/cm/lib/cdh71/cdh71-hive-client-7.13.1-shaded.jar, pointing to /opt/cloudera/cm/lib/cdh7/cdh7-hive-client-7.13.1-shaded.jar.

Restart the Service Monitor service post the change. This will allow the Service Monitor to perform Canary testing correctly on the HMS (Hive Metastore) service.

1. In Cloudera Manager, go to Tez > Configuration
2. Append the following values to both tez.am.launch.cmd-opts and tez.task.launch.cmd-opts:

   
   --add-opens=java.base/java.net=ALL-UNNAMED
   --add-opens=java.base/java.util=ALL-UNNAMED
   --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
   --add-opens=java.base/java.util.regex=ALL-UNNAMED
   --add-opens=java.base/java.lang=ALL-UNNAMED
   --add-opens=java.base/java.time=ALL-UNNAMED
   --add-opens=java.base/java.io=ALL-UNNAMED
   --add-opens=java.base/java.nio=ALL-UNNAMED

3. Save and restart

# === Custom Appender with Filters ===
log4j.appender.filteredlog=org.apache.log4j.ConsoleAppender
log4j.appender.filteredlog.layout=org.apache.log4j.PatternLayout
log4j.appender.filteredlog.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
# === Filter #1: Drop warning ===
log4j.appender.filteredlog.filter.1=org.apache.log4j.varia.StringMatchFilter
log4j.appender.filteredlog.filter.1.StringToMatch=Received Process Heartbeat for unknown (or duplicate) process.
log4j.appender.filteredlog.filter.1.AcceptOnMatch=false
# === Filter #2: Drop telemetry config warning ===
log4j.appender.filteredlog.filter.2=org.apache.log4j.varia.StringMatchFilter
log4j.appender.filteredlog.filter.2.StringToMatch=TELEMETRY_ALTUS_ACCOUNT is not configured for Otelcol
log4j.appender.filteredlog.filter.2.AcceptOnMatch=false
# === Accept all other messages ===
log4j.appender.filteredlog.filter.3=org.apache.log4j.varia.AcceptAllFilter
# === Specific logger for AgentProtocolImpl ===
log4j.logger.com.cloudera.server.cmf.AgentProtocolImpl=WARN, filteredlog
log4j.additivity.com.cloudera.server.cmf.AgentProtocolImpl=false
# === Specific logger for BaseMonitorConfigsEvaluator === 
log4j.logger.com.cloudera.cmf.service.config.BaseMonitorConfigsEvaluator=WARN, filteredlog
log4j.additivity.com.cloudera.cmf.service.config.BaseMonitorConfigsEvaluator=false

Once done, restart the Cloudera Manager server(s) for the updated configuration to be picked.

To resolve this issue temporarily, you must perform the following steps:

• Upgrade the target Cloudera Manager before you upgrade the source Cloudera Manager for 7.11.3 CHF12 version only.
• Pause all replication policies, upgrade source Cloudera Manager, upgrade destination Cloudera Manager, and unpause the replication policies.
• Upgrade source Cloudera Manager, upgrade target Cloudera Manager, and rerun the failed Ozone replication policies between the source and target clusters.

If you add Zeppelin or Knox services later to the existing cluster, you must manually add the respective service user to the livy_admin_users configuration in the Livy configuration page.

To resolve the issue temporarily, you can either comment out or disable the line require_secure_transport in the configuration file located at /etc/my.cnf.

1. Go to the Cloudera Manager > Replication > Replication > Replication Policies page.
2. Click Actions > Edit for the required Iceberg replication policy.
3. Go to the Advanced tab on the Edit Iceberg Replication Policy modal window.
4. Enter the mapreduce.job.hdfs-servers.token-renewal.exclude = ns1, ns2 key value pair for Advanced Configuration Snippet (Safety Valve) for source hdfs-site.xml and Advanced Configuration Snippet (Safety Valve) for destination hdfs-site.xml fields.
5. Save the changes.
6. Click Actions > Run Now to run the replication policy.

• Remove some or all the properties in ozone_replication_core_site_safety_valve, and move them to ozone-conf/ozone-site.xml_service_safety_valve.
• Add a dummy property with no value in ozone_replication_core_site_safety_valve. For example, add <property><name>dummy_property</name><value></value></property>, save the changes, and run the Ozone replication policy.

1. kinit with the scm custom kerberos principal
2. ozone admin scm finalizeupgrade
3. ozone admin scm finalizationstatus

1. In the Cloudera Manager, go to Hive > Configuration
2. Add the value for hive.compactor.initiator.on to true in the "Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml"
3. Save the changes and Restart.

• Log in to the Ranger Admin host(s) on the source cluster.
• Identify the Cloudera Manager agent PEM file using the # cat /etc/cloudera-scm-agent/config.ini | grep -i client_cert_file command. For example, the file might reside in client_cert_file=/myTLSpath/cm_server-cert.pem location.
• Create the path for the new PEM file using the # mkdir -p /var/lib/cloudera-scm-agent/agent-cert/ command.
• Copy the client_cert_file from config.ini as cm-auto-global_cacerts.pem file using the # cp /myTLSpath/cm_server-cert.pem /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem command.
• Change the ownership to 644 using the # chmod 644 /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem command.

• Resume the Ranger replication policy in Replication Manager.

Additionally, you must ensure that you do not update the property values in Ozone Replication Advanced Configuration Snippet (Safety Valve) for core-site.xml for Ozone replication jobs. This is because the values in this advanced configuration snippet overrides the property values in core-site.xml and not the ozone-site.xml file.

Different property values in Ozone Service Advanced Configuration Snippet (Safety Valve) for ozone-conf/ozone-site.xml and Ozone Replication Advanced Configuration Snippet (Safety Valve) for core-site.xml result in a nondeterministic behavior where the replication job picks up either value during the job run which leads to incorrect results or replication job failure.

1. SSH to the host on which the Ranger Admin role is running.
2. Find the ranger-replication.sh file.
3. Create a backup copy of the file.
4. Locate substituteEnv SOURCE_DESTINATION_RANGER_SERVICE_NAME_MAPPING ${RANGER_REPL_SERVICE_NAME_MAPPING} in the file.
5. Modify it to substituteEnv SOURCE_DESTINATION_RANGER_SERVICE_NAME_MAPPING "'${RANGER_REPL_SERVICE_NAME_MAPPING//\"}'"
6. Save the file.
7. Rerun the Ranger replication policy.

• Use different bucket names on the source and target clusters.
• Set the fs.s3a.delegation.token.binding property to an empty value in ozone_replication_core_site_safety_valve to disable the delegation tokens for Ozone replication policy jobs.

1. In Cloudera Manager, go to SMM > Configuration > Streams Messaging Manager Rest Admin Server Advanced Configuration Snippet (Safety Valve)
2. Override bootstrap server URL (hostname:port as set in the listeners for broker) for streams-messaging-manager.yaml.
3. Save your changes.
4. Restart SMM.

1. Access the Ranger Console (Ranger Admin web UI).
2. Click the cm_atlas resource-based service.
3. Add the schemaregistry user to the all - * policies.
4. Click Manage Service > Edit Service.
5. Add the schemaregistry user to the default.policy.users property.

1. When using table filters, split the policy into multiple chunks so that the Hive Data Replication command created by the chunked policies can be persisted.
2. Increase the arguments column size of the COMMANDS table in the target Cloudera Manager database using the ALTER TABLE COMMANDS ALTER COLUMN arguments TYPE character varying(10485760); command. The maximum varchar column size is 10,485,760.

• For existing databases or tables that you created without keys, enter the location mapping of the source and target om service IDs in the Location Mapping field in the Iceberg replication policy. For example, ofs://srcomid, ofs://tgtomid.
• For new databases and tables, ensure that you provide a key when you create the database. For example, CREATE DATABASE db1 LOCATION ‘ofs://omid/volume1/bucket1/db1.db'; and CREATE EXTERNAL TABLE tb1 (id int) STORED BY ICEBERG LOCATION 'ofs://omid/volume1/bucket1/tb1’.