Fixed issues in 7.1.9 SP1 CHF 4

Know more about the cumulative hotfix 4 for 7.1.9 SP1. This cumulative hotfix was released on 10 Jan, 2025.

Following are the list of fixes that were shipped for CDP Private Cloud Base version 7.1.9-1.cdh7.1.9.p1023.60818430.

COMPX-18033: Fixed the order of updating CPU controls with cgroup v1

This fix updates cpu.cfs_period_us before cpu.cfs_quota_us, to keep the ratio between the two values and not to overcome the limit defined at parent level.

Apache Jira: YARN-11733

COMPX-17463: Setting maximum-application-lifetime using AQCv2 templates did apply to the first submitted application

Previously, when the maximum-application-lifetime property was set using the AQC v2 templates (yarn.scheduler.capacity.root.test.auto-queue-creation-v2.template.maximum-application-lifetime), the property did not apply to the first submitted application, and was only applied to the subsequent ones. This issue is now resolved.

Apache Jira: YARN-11708

CDPD-76503: Query with UNION operators fails due to OutOfMemoryError

Queries containing multiple UNION operators failed with a Java heap space OutOfMemoryError when executing or generating the explain plan.

The issue was addressed by optimizing how metadata is managed for UNION operators, preventing excessive memory usage during query execution, or explaining plan generation. This ensures queries with UNION operators run without causing memory errors.

Apache Jira: HIVE-25220

CDPD-76378: fs.s3a.connection.expect.continue controls 100 CONTINUE behavior

Added fs.s3a.connection.expect.continue option to control whether or not a PUT request to the S3 store sets the "Expect: 100-continue" header, and waits for the 100 CONTINUE response before uploading any data. The configuration parameter is set to true by default.

Apache Jira: HADOOP-19317

CDPD-71872: Failed table loads not retried after metastore recovery

Tables failed to load when the metastore was down, and queries continued to fail even after the metastore was back up, requiring manual invalidation.

The system now automatically retries loading failed tables when a query is executed, ensuring successful loads once the metastore is up, without manual intervention.

Apache Jira: IMPALA-13120

CDPD-75891: Missing partitions in COPY TESTCASE with LocalCatalog mode

In LocalCatalog mode, COPY TESTCASE did not include partition and file metadata, only exporting table schema and column stats. Importing the test case caused partition ID conflicts, leading to incomplete or incorrect results.

• Partition and file metadata are now correctly included during export.
• Unique partition IDs are generated during import to avoid conflicts.
• Logic for handling partitions and unpartitioned tables during import has been fixed.

These updates ensure that COPY TESTCASE works reliably in LocalCatalog mode.

Apache Jira: IMPALA-13340

CDPD-76186: Ozone does not support snapshots on linked buckets. This can impact Replication Manager jobs running on linked W3 buckets.

This issue is resolved by supporting Ozone snapshots on linked buckets. However, if you have created any snapshots on the linked buckets from the versions lower than 7.1.9 SP1 CHF4, you must delete the snapshots on the linked buckets by running the ozone sh snapshot delete <vol>/<linked bucket name> <snapshot name> command before upgrading to 7.1.9 SP1 CHF4.

Apache Jira: HDDS-11705

CDPD-75721: LOAD DATA INPATH fails with ICEBERG table in Impala

If you are using Impala to run a LOAD DATA INPATH statement on an Iceberg table, the statement fails if the target table is fully qualified.

This issue is now resolved.

CDPD-76136: Schema resolution does not work for migrated partitioned Iceberg tables that have complex types

Schema resolution does not work correctly for migrated partitioned Iceberg tables that have complex data types. This fix addresses the field ID generation by taking the number of partitions into account. If none of the partition columns are included in the data file (common scenario), file-level field IDs are adjusted accordingly. You could also come across a scenario where all the partition columns are included in the data files.

However, if some partition columns are included in the data file while other partition columns are not, an error is generated.

Apache JIRA: IMPALA-13364

CDPD-76137: Impala should ignore character case of Iceberg schema elements

Impala cannot read Iceberg tables written by Apache Spark that contain schema elements in uppercase or lowercase letters.

Scema is case insensitive in Impala, however, Spark allows creation of schema elements with uppercase or lowercase letters and stores them in the metadata JSON files of Iceberg.

With this fix, Impala invokes Scan.caseSensitive(boolean caseSensitive) on the TableScan object to set case insensivity.

Apache JIRA: IMPALA-13463

CDPD-76091: Duplicate entries in TXN_WRITE_NOTIFICATION_LOG causing replication failures

Oracle's treatment of empty strings as NULL led to duplicate entries in TXN_WRITE_NOTIFICATION_LOG during Hive ACID transactions with MERGE statements. This caused mismatches in _files and _dumpmetadata files, resulting in incremental LOAD failures and blocking replication

The query logic was updated to exclude partition predicates for non-partitioned tables, ensuring consistent dump files and resolving replication issues.

CDPD-75941: Spark Iceberg vectorized Parquet read of decimal column is incorrect

Fixed an issue with Spark reading Iceberg Parquet data files with a decimal column incorrectly when vectorized read was enabled.

CDPD-75925: Append realm of the Kerberos ticket to the navencrypt-gen-keytab command

It is now necessary to add the realm of the kerberos ticket to the "navencrypt-gen-keytab" command when creating the ticket. For example, navencrypt-gen-keytab https://abc-1.cloudera.com:7183 admin ABC.CLOUDERA.COM.

CDPD-75488: Fails to update to Phoenix 5.1.3 with PhoenixNonRetryableRuntimeException

While updating to the Phoenix 5.1.3 version, the PhoenixNonRetryableRuntimeException is thrown which states that the org.apache.phoenix.query.DefaultGuidePostsCacheFactory class cannot be loaded or instantiated from SquirrelSQL

Apache Jira: PHOENIX-7290

CDPD-75383: Ranger Audit Filter for the HBase service is not working as expected

From the UI perspective, while creating services configuring the Audit Filter resources, incorrect values were set for the 'Include' and 'Exclude' flag setup. This issue has now been resolved and is tracked in this Jira ticket

Apache Jira: RANGER-4983

CDPD-75008: Performance fixes for Ozone plugin

Fixes the performance issues observed while evaluating policies for multi-level resources. RANGER-4893 : Improves policy evaluation for multi level resource hierarchies. RANGER-4922 : Reduces time to find tags associated with multi-level resources.

CDPD-74821: Incorrect query results when running concurrent Hive queries

Running multiple concurrent Hive queries through Beeline could produce incorrect results. This issue occurred when the hive.query.result.cached.enabled setting was set to false. The queries fetched results from other parallel queries instead of their own, leading to inconsistencies in workflows.

The issue has been resolved by ensuring that files are set in a thread-safe manner within HiveSequenceFileInputFormat.

Apache Jira: HIVE-28530

CDPD-74730: Backport SPARK-46239: Hide the Jetty server's version

[SPARK-46239] Hide the Jetty server's version.

Apache Jira: SPARK-46239

CDPD-74483: Ozone replication from source to destination fails when ozone.client.bytes.per.checksum is set to 16 KB.

When ozone.client.bytes.per.checksum is set to a lower value, say 16 KB, the parity checksum calculation during the validation phase is wrong and displays the IllegalArgumentException error on the client side.

This issue is now resolved by setting ozone.client.bytes.per.checksum to 1 MB.

CDPD-72174: During the FileSizeCountTask, the ClassCastException error is displayed in the Recon server where RepeatedOmKeyInfo is incorrectly cast to OmKeyInfo causing task processing to fail.

This issue is resolved by implementing a safeguard that checks for value mismatches and ignores such events.

Apache Jira: HDDS-11187

CDPD-71931: Ranger - Upgrade commons-compress to 1.26.0 due to CVE-2024-25710 and CVE-2024-26308

Upgraded commons-compress to 1.26.0 due to CVE-2024-25710 and CVE-2024-26308

Apache Jira: RANGER-4729

CDPD-71411: Atlas Hbase import fails with jackson-databind expection

The Issue was reproducible for JDK 11. While running the hbase-import script, the jackson databind (from version 2.15.0) was looking for the enum READ_UNKNOWN_ENUM_VALUES_USING_DEFAULT_VALUE which was not found because there was an older version (2.12.7) of jackson-annotations jar on the class path. As a result, a java.lang.NoSuchFieldError occurred and eventually the import script failed.

CDPD-68425: Oozie should validate the backend database identifiers in case-insensitive way

Resolved the issue where Oozie incorrectly reported missing tables due to issues with table name casing.

CDPD-62726: Atlas 'updateTime' parameter is not updated when business metadata, labels is edited.

The change, fixed the issue when the updateTime system attribute was not updated for business metadata and labels when added and removed from an entity. This caused the customer significant issues with audit and accountability of user actions in this governance tool, and severely limits its use within the organisation, as the business metadata functionality is used for all data.

Also, this helped the incremental import export where updated actions for business metadata and labels were performed on the entity.

Apache Jira: ATLAS-4802

CDPD-44060: Stale locks preventing materialized view rebuilds

The ALTER MATERIALIZED VIEW <view_name> REBUILD command failed with the following error:

Error: Error while compiling statement: FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: Another process is rebuilding the materialized view <view_name>

This occurred because stale lock entries in the backend materialization_rebuild_locks table were not removed when the rebuild operation failed or was interrupted. These stale locks prevented subsequent rebuild attempts for the same materialized view.

The issue was addressed by ensuring that stale lock entries are removed from the materialization_rebuild_locks table even when a rebuild transaction is aborted. This ensures that subsequent rebuild commands for the same materialized view can proceed without errors.

Apache Jira: HIVE-28416