Recovering a corrupted Cloudera Data Engineering Embedded database

In case you did not stop the jobs and scale down Cloudera Data Engineering embedded databases but completed the upgrade of OpenShift Container Platform (OCP), there is a chance of the Cloudera Data Engineering embedded database getting corrupted which causes the virtual clusters to become inaccessible. Follow the below steps to recover the Cloudera Data Engineering embedded database.

Identifying the CDE Namespace
1. Navigate to the Cloudera Data Engineering Overview page by clicking the Cloudera Data Engineering tile in the Cloudera Management Console .
2. In the CDE Services column, click Service Details for the Cloudera Data Engineering service.
3. Note the Cluster ID shown in the page. For example, if the Cluster ID is cluster-abcd1234, then the CDE Namespace is dex-base-abcd1234.
4. Use this CDE Namespace (in the above example, it is dex-base-abcd1234) in the following instructions to run kubernetes commands.

Edit the dex-base-db-server-config configuration map and add the innodb_force_recovery=4 configuration in the [mysqld] section.

OpenShift CLI

oc scale statefulset/cdp-cde-embedded-db --namespace <CDE Namespace> --replicas 1

Kubernetes CLI

kubectl scale statefulset/cdp-cde-embedded-db --namespace <CDE Namespace> --replicas 1

Example snippet:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file 
# will be reopened with the relevant failures.
#
apiVersion: v1
data:
  my.cnf: |-
    [mysqld]
    port=3306
    default_authentication_plugin = mysql_native_password
    bind-address = 0.0.0.0
    innodb_force_recovery=4

Scale down and then back up the Cloudera Data Engineering embedded database statefulset to restart it.

OpenShift CLI

oc scale statefulset/cdp-cde-embedded-db --namespace <CDE Namespace> --replicas 0

oc scale statefulset/cdp-cde-embedded-db --namespace <CDE Namespace> --replicas 1

Kubernetes CLI

kubectl scale statefulset/cdp-cde-embedded-db --namespace <CDE Namespace> --replicas 0

kubectl scale statefulset/cdp-cde-embedded-db --namespace <CDE Namespace> --replicas 1

Wait for 10 minutes for the Cloudera Data Engineering embedded database to complete the recovery.

Edit the dex-base-db-server-config configuration map again by removing the previously added innodb_force_recovery=4 configuration under the [mysqld] section.
OpenShift CLI
```
oc edit configmap/dex-base-db-server-config --namespace <CDE Namespace>
```
Kubernetes CLI
```
kubectl edit configmap/dex-base-db-server-config --namespace <CDE Namespace> 
```

Scale down and then back up the Cloudera Data Engineering embedded database statefulset to restart it again.

OpenShift CLI

oc scale statefulset/cdp-cde-embedded-db --namespace <CDE Namespace> --replicas 0
                        
oc scale statefulset/cdp-cde-embedded-db --namespace <CDE Namespace> --replicas 1

Kubernetes CLI

kubectl scale statefulset/cdp-cde-embedded-db --namespace <CDE Namespace> --replicas 0
                        
kubectl scale statefulset/cdp-cde-embedded-db --namespace <CDE Namespace> --replicas 1

Wait for all the Cloudera Data Engineering Virtual Clusters to be accessible. This usually takes about 10 minutes.