Known Issues in Spark

Learn about the known issues in Spark, the impact or changes to the functionality, and the workaround.

CDPD-60862: Rolling restart fails during ZDU when DDL operations are in progress

During a Zero Downtime Upgrade (ZDU), the rolling restart of services that support Data Definition Language (DDL) statements might fail if DDL operations are in progress during the upgrade. As a result, ensure that you do not run DDL statements during ZDU.

The following services support DDL statements:

• Impala
• Hive – using HiveQL
• Spark – using SparkSQL
• HBase
• Phoenix
• Kafka

Data Manipulation Lanaguage (DML) statements are not impacted and can be used during ZDU. Following the successful upgrade, you can resume running DDL statements.

None. Cloudera recommends modifying applications to not use DDL statements for the duration of the upgrade. If the upgrade is already in progress, and you have experienced a service failure, you can remove the DDLs in-flight and resume the upgrade from the point of failure.

INSIGHT-10232: Spark 3.5 jobs fail with Can't get Kerberos realm on JDK 17 Kerberos clusters

On CDP Private Cloud Base 7.1.9 with CDS 3.5 Powered by Apache Spark and JDK 17, Spark 3 jobs can fail immediately at submission on Kerberos-secured clusters, including after upgrading from RHEL 7 to RHEL 8 and from Java 8 to JDK 17. Jobs running on YARN in client mode terminate with an error similar to the following:

Exception in thread "main" java.lang.IllegalArgumentException: Can't get Kerberos realm
Caused by: java.lang.IllegalAccessException: ... org.apache.hadoop.security.authentication.util.KerberosUtil cannot access class sun.security.krb5.Config

The root cause is a JDK 17 module restriction: the java.security.jgss module does not export sun.security.krb5 to the unnamed module, so Hadoop cannot initialize Kerberos. Apache Spark addressed this in SPARK-37345 (Spark 3.3 and later), but the Cloudera CDS 3.5 parcel build on 7.1.9 does not include that fix.

Add JVM module-access flags for the Spark driver and executors using the following steps:

1. Go to the Cloudera Manager.
2. Go to the Spark 3 service.
3. Click the Configuration tab.
4. Select Scope > Gateway.
5. Select Category > Advanced.
6. Locate the Spark 3 Client Advanced Configuration Snippet (Safety Valve) for spark3-conf/spark-defaults.conf_client_config_safety_valve property.
7. Add the following entries:

   spark.driver.extraJavaOptions=--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED
   spark.executor.extraJavaOptions=--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED

8. Enter a Reason for change, and then click Save Changes to commit the changes.
9. Deploy the client configuration.

If you submit Spark jobs through Livy for Spark 3, add the same spark.driver.extraJavaOptions and spark.executor.extraJavaOptions entries in the Livy Server for Spark 3 Advanced Configuration Snippet (Safety Valve) for livy3-conf/livy.conf property.

INSIGHT-8106: Spark 3 returns zero rows when reading some Hive external Parquet tables

On CDP Private Cloud Base 7.1.9 with CDS Powered by Apache Spark 3.x, queries against Hive external Parquet tables through spark3-shell can return zero rows even though the same query in spark-shell (Spark 2) returns the expected results. For example, spark.sql("SELECT COUNT(1) FROM db.table_name").show() can report a count of 0 in Spark 3 while Spark 2 reports the correct row count.

This behavior often affects tables whose data files are stored in nested subdirectories under the table location rather than at the table root. A common cause is a Hive CTAS (create table as select) statement where one or more source tables used UNION ALL. Hive stores the output in subdirectories such as HIVE_UNION_SUBDIR_1, HIVE_UNION_SUBDIR_2, and so on. Spark 3 does not recursively list Parquet files under the table location by default, so it finds no data files and treats the table as empty.

Use one of the following workarounds:

• Recreate the table so that Parquet data files are stored in a flat layout at the table location (for example, create a new external Parquet table and load the data with a CTAS that does not produce union subdirectories).
• Read the table location with the DataFrame API and enable recursive file discovery, then register a temporary view for SQL queries. The recursiveFileLookup option applies to DataFrame reads, not to direct spark.sql scans of metastore-registered tables.

  val df = spark.read
    .option("recursiveFileLookup", "true")
    .parquet("/warehouse/tablespace/external/hive/db_name/table_name")
  df.createOrReplaceTempView("table_temp")
  spark.sql("SELECT COUNT(1) FROM table_temp").show()

INSIGHT-8865: Spark 3 cluster-mode submit fails after JDK 17 upgrade due to Java module access restrictions

After upgrading to JDK 17, Spark 3 jobs submitted in cluster mode with spark3-submit can fail with java.lang.IllegalAccessError. Stricter Java Platform Module System (JPMS) rules block Hadoop libraries from accessing internal JNDI/LDAP packages (for example com.sun.jndi.ldap) unless you explicitly export or open the required modules.

Set JDK_JAVA_OPTIONS on the Spark client through a spark-env.sh safety valve in Cloudera Manager. Use the steps for your Runtime version, then add the same JVM options in both cases.

Cloudera Runtime 7.1.9 (CDS Spark 3)

1. Go to the Cloudera Manager.
2. Go to the Spark 3 service.
3. Click the Configuration tab.
4. Select Scope > Gateway.
5. Select Category > Advanced.
6. Locate the Spark 3 Client Advanced Configuration Snippet (Safety Valve) for spark3-conf/spark-env.sh property.
7. Add the following entry:

   export JDK_JAVA_OPTIONS="--add-exports=java.base/sun.security.x509=ALL-UNNAMED \
   --add-exports=java.base/sun.security.pkcs=ALL-UNNAMED \
   --add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED \
   --add-opens=java.base/sun.security.util=ALL-UNNAMED"

8. Enter a Reason for change, and then click Save Changes.
9. Deploy the client configuration and restart stale services as prompted.

Cloudera Runtime 7.3.1 (Spark 3 in Runtime)

1. Go to the Cloudera Manager.
2. Go to the Spark service.
3. Click the Configuration tab.
4. Select Scope > Gateway.
5. Select Category > Advanced.
6. Locate the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh property.
7. Add the same JDK_JAVA_OPTIONS entry as shown above for Runtime 7.1.9.
8. Enter a Reason for change, and then click Save Changes.
9. Deploy the client configuration and restart stale services as prompted.

If you see Can't get Kerberos realm errors on JDK 17 instead, configure spark-defaults.conf as described in INSIGHT-10232.

INSIGHT-8092: Spark 3 submit fails with Could not create delegation token for Atlas client when Kerberos and Spark Lineage are enabled

On CDP Private Cloud Base 7.1.9 with Kerberos and Atlas configured, spark3-submit can fail during YARN application submission when Spark Lineage (spark.lineage.enabled) is enabled, which is the default after installing the CDS Spark 3 parcel. The client exits with an error similar to the following:

Exception in thread "main" java.lang.RuntimeException: Could not create delegation token for Atlas client (see Caused by)
...
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user.

The Spark Atlas Connector obtains Atlas delegation tokens during client submission. When Spark Lineage is enabled on a Kerberos-secured cluster, that step requires Kerberos credentials on the submit path. A prior kinit in the shell might not be sufficient for this client-side token acquisition.

Use one of the following workarounds:

Option 1: Disable Spark Lineage if Atlas metadata from Spark is not required

1. Go to the Cloudera Manager.
2. Go to the Spark 3 service.
3. Click the Configuration tab.
4. Search for and clear Spark Lineage (spark.lineage.enabled).
5. If you do not need Atlas integration from Spark, also clear Atlas Service (atlas_service).
6. Enter a Reason for change, and then click Save Changes.
7. Deploy the client configuration and retry spark3-submit.

Option 2: Pass Kerberos credentials on spark3-submit if Spark Lineage is required

spark3-submit \
  --conf spark.kerberos.keytab=/path/to/user.keytab \
  --conf spark.kerberos.principal=user@REALM \
  ...

Equivalent forms include legacy spark.yarn.keytab and spark.yarn.principal, or --keytab and --principal.

For cluster-mode submission when Kafka brokers are unavailable and you need fault-tolerant lineage behavior, see Spark Connector configuration in Apache Atlas.

CDPD-94393: RuntimeWarning Failed to add file" message appears even when Spark successfully loads files

In both Spark 2 and 3, due to an exception when attempting to add files to the Python path, the RuntimeWarning: Failed to add file message appears even when the Python JAR file is successfully loaded.

None. You can safely ignore the message as the file is loaded successfully and the message does not affect job completion.

CDPD-67517: Spark3 tests are failing if /tmp is mounted as noexec.

Map the tmpdir to a writable path in spark3-conf/spark-defaults.conf using the following steps:

1. Go to the Cloudera Manager.
2. Go to the Spark service.
3. Click the Configuration tab.
4. Select Scope > Gateway.
5. Select Category > Advanced.
6. Locate the Spark Client Advanced Configuration Snippet (Safety Valve) for spark3-conf/spark-defaults.conf_client_config_safety_valve property.
7. Map the tmpdir to a writable path:

   spark.driver.extraJavaOptions=-Djava.io.tmpdir=/var/tmp
                   spark.executor.extraJavaOptions=-Djava.io.tmpdir=/var/tmp

8. Enter a Reason for change, and then click Save Changes to commit the changes.
9. Deploy the client configuration.

CDPD-23817: In the upgraded Cluster, the permission of /tmp/spark is restricted due to the HDP configuration hive.exec.scratchdir=/tmp/spark.

If you are using the /tmp/spark directory in the CDP cluster, you must provide the required additional Policies/ACL permissions.

CDPD-22670 and CDPD-23103: There are two configurations in Spark, "Atlas dependency" and "spark_lineage_enabled", which are conflicted. The issue is when Atlas dependency is turned off but spark_lineage_enabled is turned on.

Run Spark application, Spark will log some error message and cannot continue. That can be restored by correcting the configurations and restarting Spark component with distributing client configurations.

CDPD-23007: Mismatch in the Spark Default DB Location. In HDP 3.1.5, hive_db entities have one attribute - 'location' which is configured to the '/managed' path. In fresh install of CDP 7.1.7, hive_db entities now have 2 attributes 'location' configured to '/external' path and 'managedLocation' configured to '/managed' path. In. the AM2CM migration (HDP 3.1.5 -> CDP 7.1.7), the 'location' attribute from hive_db entities in HDP 3.1.5 comes unaltered to CDP 7.1.7 and hence maps to '/managed' path.

This issue arises only if you are upgrading from HDP 3.1.5 to CDP 7.1.7. If you are performing a fresh install of CDP 7.1.7, you can ignore this issue.

None

CDPD-217: The Apache Spark connector is not supported

The old Apache Spark - Apache HBase Connector (shc) is not supported in CDP releases.

Use the new HBase-Spark connector shipped in CDP release.

CDPD-3038: Launching pyspark displays several HiveConf warning messages

When pyspark starts, several Hive configuration warning messages are displayed, similar to the following:

23/08/02 08:37:26 WARN conf.HiveConf: HiveConf of name hive.metastore.runworker.in does not exist
            23/08/02 08:37:26 WARN conf.HiveConf: HiveConf of name hive.masking.algo does not exist
            23/08/02 08:37:34 WARN conf.HiveConf: HiveConf of name hive.metastore.runworker.in does not exist
            23/08/02 08:37:34 WARN conf.HiveConf: HiveConf of name hive.masking.algo does not exist

These errors can be safely ignored.