Pre-upgrade - Upgrading Cloudera Data Engineering service

You can seamlessly upgrade a previous Cloudera Data Engineering service version to a new version.

important

Upgrading Cloudera Data Engineering service supports endpoint stability only when you are upgrading from 1.5.4 SP2 or earlier versions to 1.5.5 or 1.5.5 SP1. Endpoint stability enables you to access the Cloudera Data Engineering service of the new version with the original endpoint. Thus, you can use the existing endpoints without changing configurations at the application level. The Cloudera Data Engineering service endpoint migration process lets you migrate your resources, jobs, job run history, Spark jobs logs, and event logs from your old cluster to the new cluster.
If you are upgrading Cloudera Data Engineering to 1.5.5 SP2 or higher versions, endpoint stability is not supported. This means that the links to your Cloudera Data Engineering Service and Virtual Cluster will change after the upgrade.
After upgrading Cloudera Data Engineering from 1.5.4 SP2 or earlier versions to 1.5.5 or higher versions, the Cloudera Data Engineering Services and Virtual Clusters that were created in the earlier versions does not work in Cloudera Data Engineering 1.5.5 or higher versions. Cloudera does not recommend using the old Cloudera Data Engineering Services and Virtual Clusters that were created before the upgrade. Instead, create new Services and Virtual Clusters in Cloudera Data Engineering 1.5.5 or higher version that you upgraded to and use them.
After upgrading Cloudera Data Engineering, the upgraded Virtual Cluster retains the same base OS images used in the source Virtual Cluster. This ensures maximum compatibility, particularly for jobs that depend on specific Python and Scala versions, such as Spark jobs. For example, if the source Virtual Cluster uses a standard Red Hat image, the upgraded Virtual Cluster retains that image type. The same applies to security-hardened images. No automation path is supported from a standard Red Hat image to a security-hardened image or the other way around.
Upgrading to Cloudera Data Services on premises 1.5.5 SP2 CHF1 triggers an unsupported, automatic upgrade of Cloudera Data Engineering Virtual Cluster (VC) Spark versions from 3.2.x or 3.3.x to 3.5. Furthermore, backup and restore operations in a Cloudera Data Services on premises environment fail during VC creation if the system attempts to restore an older, incompatible Spark version (such as 3.2.4 or 3.3.2) to the target runtime. To prevent upgrade and restoration failures, especially during Data Lake upgrades from 7.1.9 to 7.3.1.x or higher, do the following:
1. Upgrade your Data Lake to at least 7.1.9 SP1.
2. Create a new VC using Spark 3.5 for each existing 3.2.x or 3.3.x VCs.
3. Migrate the workloads from older VCs to the the new Spark 3.5 VC.
4. Validate the workloads on the new Spark 3.5 VC.
5. Delete the old VC(s) that are still on older Spark versions (for example, Spark 3.3).
6. Proceed with the Data Lake upgrade to 7.3.1 or higher.

Once you upgrade to Cloudera Data Engineering version 1.5.5 SP2 or higher, the endpoints that you were using in the previous version are not supported.