Cloudera Docs

Prerequisites for upgrading CDE service with endpoint stability

You must first download the docker image and create the cde-upgrade-util.properties file to back up the Cloudera Data Engineering (CDE) service.

  1. Login into the ECS Server host using SSH and create an external kubeconfig file. The following command assumes that your home directory, that is, ~/ is the working directory. 
    sed -e 's/certificate-authority-data/#&/' -e "s/server: .*/server: https:\/\/`hostname`:6443/" -e '/server/a \ \ \ \ insecure-skip-tls-verify: true' /etc/rancher/rke2/rke2.yaml > ~/kubeconfig && cat ~/kubeconfig

    This command creates a file named kubeconfig in the working directory which is the external kubeconfig file.

  2. Copy the CDP Credentials file named credentials of the DEAdmin user into the ECS Server host’s working directory as follows:
    1. In the Cloudera Data Platform (CDP) console, click the Management Console tile.
    2. Click User Management and select the user.
    3. Click Generate Access Key > Download credentials file.
    4. Copy the CDP Credentials file into the ECS Server host with the name credentials.
    5. Verify if the credentials are present in the ECS Server host:
      ls -l credentials
  3. Set the environment variables in the ECS Server host by running the following command:
 
    export PATH=$PATH:/opt/cloudera/parcels/ECS/installer/install/bin/linux/:/opt/cloudera/parcels/ECS/docker export KUBECONFIG=~/kubeconfig
  4. Download the dex-upgrade-utils docker image tarball. The file naming convention is dex-upgrade-utils-[***VERSION-NUMBER***]-[***BUILD-NUMBER***].tar.gz.
  5. Load the downloaded docker image into the host machine docker runtime: 
    docker load < dex-upgrade-utils-[***VERSION-NUMBER***]-[***BUILD-NUMBER***].tar.gz

    Example:

    docker load < dex-upgrade-utils-1.20.1-b48.tar.gz

    Sample output:

    368243204766.dkr.ecr.us-west-2.amazonaws.com/cdp-private/cloudera/dex/dex-upgrade-utils:1.20.1-b48
  6. Create the required folders on the ECS Server host and copy the credentials and kubeconfig secret files. 
    mkdir /opt/backup-restore
export BASE_WORK_DIR=/opt/backup-restore

cd $BASE_WORK_DIR
mkdir backup secrets
chmod 775 backup/

  7. Place the CDP credentials file of the DEAdmin user and administrator kubeconfig file in the $BASE_WORK_DIR/secrets directory. 
    cp ~/credentials secrets/
cp ~/kubeconfig secrets/
  8. Create the cde-upgrade-util.properties file as follows:
    1. Create the cde-upgrade-util.properties file and save it in the $BASE_WORK_DIR directory.
    2. Update the following information in the cde-upgrade-util.properties file: 
      cdp_k8s_namespace:<CDP control plane k8s namespace>
cdp_endpoint:<CDP control plane endpoint>
credential_file_path:<Path to the DEAdmin user CDP credentials file>
de_admin_user:<DEAdmin user-id>
de_admin_password:<DEAdmin user's password must be in base64 encoded format. Use the "echo -n [***PASSWORD***] | base64" command to encode the password. >
tls_insecure:<Keep it true if you are using a self-signed certificate>
auto_unpause_jobs: <Specify it as “true” if you want to automatically resume the jobs that were paused during the backup phase. The jobs will be resumed after you restore the CDE service.>
platform_type:ECS
use_stored_user:<(optional) Boolean property which can be true or false. Use this property in conjunction with do-as described below.>
do_as:<(optional) if the value of use_stored_user is set to true, this value is used as a fallback when the stored user is not valid. Otherwise, this is directly used as job owner. If the use_stored_user parameter is set to false and no value is supplied in the do_as parameter, then no validation will be performed for the job's username and it will be restored as it is.>

      For example: The following options are the minimum recommended options that you must include in the cde-upgrade-util.properties file:

      cdp_k8s_namespace=cdp
cdp_endpoint=https://console-cdp.apps.host-1.ecs-pvc1.kcloud.cloudera.com
credential_file_path=/home/dex/.cdp/credentials
de_admin_user=cdpuser1
de_admin_password=VGVzdDEyMw==
tls_insecure=true
auto_unpause_jobs=true
platform_type=ECS
user_stored_user=false
  9. Make a note of the details of the CDE service that is being migrated. This information is required if you are using a CDP database that is external and is not accessible from the container which is running the cde-upgrade endpoint stability commands. Identify the cluster endpoint:
    1. In the Cloudera Data Platform (CDP) console, click the Data Engineering tile. The CDE Home page displays.
    2. Click Administration in the left navigation menu. The Administration page displays.
    3. In the Services column on the left, click the Cluster Details icon corresponding to the CDE service whose endpoint you want to migrate.
    4. Make a note of the CDE cluster ID.