Fixed issues in 7.1.9 SP1 CHF 6

Know more about the cumulative hotfix 6 for 7.1.9 SP1.

Following are the list of fixes that were shipped for CDP Private Cloud Base version 7.1.9-1.cdh7.1.9.p1034.63559640.

CDPD-76868: Upgrade and unify commons-lang3 to version 3.13

Upgraded the commons-lang3 dependency to version 3.13.0 throughout the stack.

CDPD-75742: Upgrade commons-lang3 to 3.13.0

The commons-text version 1.11.0 in Spark was calling Range.of(..) method of commons-lang3 that was not available in version 3.12.0 resulting in a Oozie-Spark job failure. This issue is now resolved and commons-lang3 is now upgraded to version 3.13.0 which contains the required method.

CDPD-77188: Atlas can consume Iceberg CREATE TABLE statements from Cloudera Data Warehouse

Incompatibilities between Cloudera Data Warehouse version 2024.0.18.2-4 with Cloudera Data Hub version 7.2.17 and Apache Atlas in Cloudera Runtime 7.1.9 CHF6 are resolved. Add the following property to the Atlas configuration to resolve the issue: atlas.notification.consumer.preprocess.iceberg.entities=true

CDPD-77856: Issue in APPEND mode while writing Ranger audits to HDFS

Ranger audits to HDFS currently use APPEND mode in case of errors or exceptions encountered in writing audits to HDFS destination (to prevent large number of audit files), and fallbacks to WRITE mode if unable to APPEND.

The issue is fixed now. A configuration parameter to enable APPEND mode while writing Ranger audits to HDFS has been added.

CDPD-77736: Livy for Spark 3 Web UI was missing in the Knox gateway homepage in 7.1.8

This issue is resolved by adding a metadata section to Livy for Spark 3 service.xml file. Knox can now display the appropriate section in its web UI.

CDPD-78066: Iceberg_tables when created, does not show under hive_db entity

After the update, the UI will show Iceberg tables if they have been created under the same hive_db. Iceberg table can been seen in the Tables tab of a Hive_db in basic search. Both hive_table and iceberg_table typenames will be visible.

CDPD-78193: CSV injection vulnerability during CSV and Excel file export

When policies were created with the special characters mentioned in a document, there were some vulnerabilities.

The issue is fixed now. Checks are now added to ensure whenever such characters are present, a space is added after it.

CDPD-77092: Hbase scan operation returns denied columns in result

In some cases, Ranger authorization returned access results of some HBase data even when the user was not entitled to do so.

This issue is fixed now.

CDPD-79591: Impala base folder identification aligns with Hive

Impala used a different approach to identify valid base folders for reading Hive ACID tables, leading to inconsistencies. Hive allowed reading a base folder even if an open writeId existed before a newer base writeId, but Impala did not. This discrepancy caused read issues, particularly when using INSERT OVERWRITE or TRUNCATE operations.

Impala’s logic now treats base directories created by INSERT OVERWRITE and TRUNCATE the same way as Hive, ensuring consistent read behavior.

Apache Jira: IMPALA-13759

CDPD-79456: RMS full-sync breaks due to unsupported schema

RMS supports HDFS and Ozone file-system in private cloud and S3 in public cloud. The supported file schema types are hdfs, s3a, o3fs, and ofs. If Hive table location was stored at other file-system which was not supported by the RMS, then full-sync threw exception, due to unsupported schema. The full-sync was never completed.

This fix skips the unsupported file schema types while processing table and database metadata during full-sync and delta-sync in RMS. Therefore, table and database locations stored at other file-system, which is not supported by the RMS, are not mapped and appropriate messages are logged in RMS server log file.

CDPD-78850: Impala event processor now ignores CREATE_TABLE events

Previously, if a database was invisible to the Impala cluster due to authorization restrictions, CREATE_TABLE events in such databases caused the event processor to enter an ERROR state.

Impala now ignores CREATE_TABLE events when the database is not found, preventing errors in the event processor.

Apache Jira: IMPALA-11735

CDPD-78836: vulnerable version of jquery.datatables is in use

Upgrades to the latest supported version of jquery.datatables to address security issues.

CDPD-79088: Impala now reloads metadata when table properties change in ALTER_TABLE events

After upgrading to Cloudera Data Platform 7.1.9 SP1 CHF 1, Impala automatically detected newly added partitions but did not refresh appended data in existing partitions unless a manual REFRESH or INVALIDATE METADATA command was run.

Impala now reloads metadata for ALTER_TABLE events when table properties change, ensuring appended data is detected automatically.

Apache Jira: IMPALA-13403

CDPD-79108: KerberosBasicAuthenticationHandler cannot handle colons in passwords

KerberosBasicAuthenticationHandler can now handle colons in passwords. This handler is used by Schema Registry and Streams Messaging Manager.

CDPD-72782: Ozone write does not work when http proxy is set for the JVM

GRPC uses HTTP internally for its connections and due to this, if HTTP proxy is configured for any Ozone process using GRPC , it directs each call through the proxy even for GRPC which is not desirable for performance. Hence this fix disables proxy for GRPC connections that Ozone uses.

Apache Jira: HDDS-11257

CDPD-65714: Allow FS client to specify EC as default filesystem replication

This fix allows you to specify EC as the default replication type for a file uploaded through the Hadoop FileSystem API to Ozone through a client side configuration option.

Apache Jira: HDDS-10336

CDPD-78069: Oozie action configuration's Java options are not applied due to CDPD-60551

Java options configured in workflow xml's action configurations such as yarn.app.mapreduce.am.command-opts and mapreduce.map.java.opts are not applied to the Oozie Launcher AM's JVM. This issue is fixed now.

COMPX-18589: YARN ResourceManager raised an exception during comparison of queues

YARN ResourceManager raised an exception, java.lang.IllegalArgumentException: Comparison method violates its general contract!. The RCA was with the AND condition that caused the exception of TimSort algorithm during comparison of queues. This issue is now resolved.

Apache Jira: YARN-11745

CDPD-65645: Cache sort results in ContainerBalancerSelectionCriteria

The sorting of all the containers was time consuming. The sort result is now cached and the performance of Container Balancer in large clusters is now improved.

Apache Jira:HDDS-10160

CDPD-77399: HBase fails to register the servlet metrics and throws ClassNotFoundException: org.apache.hadoop.metrics.MetricsServlet

This issue is fixed now. HBase does not warn about the Hadoop 2-based metric servlet class on a Hadoop 3 deployment.

Apache Jira:: HBASE-28315

CDPD-75089: Restrict trusted packages in ReflectData and SpecificData

Schema parsing in the Java SDK of Apache Avro had an issue that could allow malicious actors to execute arbitrary code when reading Avro data. This issue is now resolved by restricting trusted packages in ReflectData and SpecificData.

Apache Jira:AVRO-3985