Fixed issues in 7.1.9 SP1 CHF 5

CDPD-76634: RMS server throws ConcurrentModificationException

The original ConcurrentModificationException is likely thrown when the resource-mappings are modified in response to changes in the Hive metadata while they are being serialized for downloading to the NameNode (or secondary-namenode).

The fix is to create a shallow copy of resource-mappings before applying deltas which ensures that resource-mappings are not modified while they are being serialized for downloading to the NameNode.

CDPD-77877: Exclude guava from arrow-vector

Fixed an issue where a transitive dependency would force custom applications to be built with a particular version of guava.

CDPD-59896: Hue fails to clean up documents if Oozie is on the blocked list of services

Earlier, when you ran the desktop_document_cleanup script to clean old documents, and if you added Oozie to the blocked list of apps in the Hue Advanced Configuration Snippet, the script failed. To run the script successfully, you had to remove the Oozie service from the app_blacklist property manually. This issue has been resolved. You no longer need to manually remove Oozie from the blocked list of apps.

CDPD-62397: Improved default editor selection in Hue configuration

Previously, when you tried to log in and access Hue, the first available editor was used by default if multiple editors were configured. This issue has now been resolved, allowing you to specify a default editor in the Hue configuration file for all users.

CDPD-75286: Spark History UI - StreamConstraintsException: String length exceeds the maximum length

Fixed an issue with Jackson to allow unlimited json string length in Spark event logs.

Zeppelin Livy interpreter paragraph outputs are being misplaced

Fixed an issue where Zeppelin paragraph outputs were being misplaced. Users are now able to use pyspark to run os.system commands without issues.

CDPD-78260: Partition value mismatch during INSERT event processing

Impala failed to process some INSERT events due to partition value mismatches. Hive encodes partition strings in file paths using URL encoding, but partition strings in HMS events are not encoded. Impala mismatched partitions during HMS events processing by decoding them incorrectly, causing event processing failures.

The fix includes decoding partition strings only when derived from file paths and not from HMS events to ensure accurate partition mapping.

Apache Jira: IMPALA-13691

CDPD-77922: Prometheus metric name incompatibility with JDK11 and JDK17

Impala metrics derived from JVM MemoryPoolMXBean names contain apostrophe characters in JDK11 and JDK17. Since Prometheus does not support apostrophes in metric names, these metrics cannot be consumed by Prometheus.

To address the issue, apostrophes in Prometheus metric names are now automatically translated to underscores, ensuring compatibility with Prometheus requirements.

Apache Jira: IMPALA-13638

CDPD-73427: Upgrade Spring Framework to resolve vulnerabilities

CVE-2024-38808: Vulnerable to resource exhaustion through user-supplied SpEL expressions.

CVE-2024-38809: DoS attack possible due to improper ETag validation in HTTP requests.

Upgraded Spring Framework to versions 6.1.12, 6.0.23, or 5.3.39 to mitigate these issues.

CDPD-78221: Hive query execution failures

HiveServer2 experienced intermittent query failures during DAG submission. The failures occurred due to a null value for the dagId variable, which resulted during DAG submission. This caused disrupted connectivity and failed Hive operations.

The fix includes optimizing Tez execution paths to prevent compilation errors and ensure seamless query processing.

CDPD-75665: Importing a table generates a DDL with an incorrect location

When creating a table using the IMPORT command, an external imported table can have this partitions located under the managed warehouse directory, which violates the metastore.warehouse.external.dir and metastore.warehouse.dir that intend to host different type of tables.

The fix ensures that the table path is fetched from HMS before importing partitions, ensuring that all data is copied to the correct location in the newly imported table.

Apache Jira: HIVE-28580

CDPD-78077: Kafka Connect connectors might fail during deployment or while being moved to another node

Kafka Connect connectors no longer fail due to incompletely unpacked dependencies while being deployed or moved to another node.

CDPD-76135: Schema check database connection throws PSQLException

The schema check now completes successfully during Oozie startup for secure database connections or when custom connection properties are used, preventing warnings from appearing in the logs.

CDPD-76604: CDEP fails to install CDP Private Cloud Base as Ozone does not start up. The OZONE-1 has failed in startup so CDEP hasn't executed next set of commands. error is displayed.

This issue is resolved by fixing the race condition in creation and deletion of rocksdbjni native library.

CDPD-77188: Atlas can consume Iceberg CREATE TABLE statements from Cloudera Data Warehouse

Incompatibilities between Cloudera Data Warehouse version 2024.0.18.2-4 with Cloudera Data Hub version 7.2.17 and Apache Atlas in Cloudera Runtime 7.1.9 CHF5 are resolved.

COMPX-18909: NodeManager can be marked unhealthy if an application is killed

A logic was introduced to mark the NodeManager unhealthy if the container-executor encountered a unrecoverable/configuration error, that is, when the container-executor script was missing. If the application was killed just before when one of the containers tried to access the localizer syslog file an IOException occured due to the deleted app log directory, and then the NodeManager was marked unhealthy incorrectly.

This issue is now resolved and the error checking is more specific and the false positive NodeManager markings are now removed.

Apache Jira: YARN-11753

CDPD-49702: NodeManager must be shut down when the program /var/lib/yarn-ce/bin/container-executor cannot be run

Previoulsy, a job failed when NodeManager encountered the No such file or directory error when running the /var/lib/yarn-ce/bin/container-executor program. This issue is now resolved and NodeManager is marked as unhealthy and shut down when it cannot run the program.

Apache Jira: YARN-11709