Manually revoking certificates from Venafi TPP

The certrevoke operator is responsible for revoking certificates in Venafi TPP when they are deleted from Kubernetes. When the operator is offline (either scaled down or offline due to some issue) and the certificates are deleted, the corresponding CertMeta custom resources may become orphaned without being properly marked for revocation. These orphaned CertMeta resources do not have the certificate.cdp.cloudera.com/deleted-during-downtime annotation set, making them invisible to the normal recovery process when the operator starts up again.

Impact

Certificates that were deleted while the certrevoke operator was offline remain valid in Venafi TPP, creating potential security risks if those certificates are compromised. The operator cannot automatically detect or revoke these certificates without manual intervention.

Detection

Use the following command to identify orphaned CertMeta resources that do not have corresponding Certificate resources anywhere in the cluster:


# Get all certificate names from all namespaces
CERT_NAMES=$(kubectl get certificate --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')

# Check each CertMeta against the list of all certificate names
kubectl get certmeta -n cert-manager -o custom-columns="NAME:.metadata.name,ISSUER:.metadata.annotations.certificate\.cdp\.cloudera\.com/issuer-name,PICKUP_ID:.spec.pickupId" | grep -v '^NAME' | while read -r name issuer pickup_id; do
  if ! echo "$CERT_NAMES" | grep -q "^$name$"; then
    echo "Orphaned CertMeta: $name, Issuer: $issuer, Pickup ID: $pickup_id"
  fi
done

Manual Revocation

For each orphaned CertMeta identified, manually revoke the certificates in Venafi TPP:

Use the information from the detection command to identify the certificates.
Revoke them manually through the Venafi TPP iterface.
Delete the CertMeta resources using the following command:
```
kubectl delete certmeta <certmeta-name> -n cert-manager
```