Service account for the provisioning credential

The provisioning credential for Google Cloud relies on a service account that can be assumed by Cloudera.

The following flow describes how the Google Cloud provisioning credential works:

  1. Your GCP account administrator creates a service account and assigns the minimum permissions allowing Cloudera to create and manage resources in your Google Cloud account. Next, the administrator generates a service account access key pair for the service account.
  2. The service account is registered as a credential in Cloudera and its access key is uploaded to Cloudera.
  3. The credential is then used for registering your Google Cloud environment in Cloudera.
  4. Once this is done, Cloudera uses the credential for provisioning environment-related resources, workload clusters, and resources for other Cloudera services that you run in Cloudera.

Review the following to learn about the permissions required for the credential and how to create the service account.

Permissions for the provisioning credential's service account

To allow Cloudera to access and provision resources in your Google Cloud project, you should create a service account in your Google Cloud project, assign the following roles or granular permissions. Next, you generate a JSON access key that can later be provided to Cloudera. Cloudera will assume this service account via the service account access key provided during credential creation for provisioning resources for your environment.

The service account must fulfill one of the following requirements (choose one of the options):

  • Option 1: Assign the following IAM roles at the project level. This is a simpler option.
  • Option 2: Alternatively, you can create custom IAM roles with the following granular IAM permissions assigned and then assign the role to the service account at the project level. This allows you to minimize the number of permissions granted to Cloudera.

Option 1: IAM roles

IAM role Scope Description
iam.serviceAccounts.list IAM permission Project This is required in order for Cloudera to be able to list service account names that you created in your GCP project.

You need to create a custom role in order to assign this permission.

Compute Instance Admin (v1)

roles/compute.instanceAdmin.v1

Project This is required for provisioning of Compute Engine instances, disks, and images in your VPC.
Storage Admin

roles/storage.admin

Project This is required for the creation of a storage bucket to store the Cloudbreak image objects. Delete permissions are not required.
Compute Network Viewer

roles/compute.networkViewer

Project This is required for read-only access to all networking resources.
Compute Load Balancer Admin

roles/compute.loadBalancerAdmin

Project This role is required for load balancing between HA components of the Data Lake.
Cloud SQL Admin

roles/cloudsql.admin

Project This is required in order for Cloudera to have the permission for creating and deleting a Data Lake and and heavy duty flow management Cloudera Data Hub clusters cleanly.
Compute Network User

roles/compute.networkUser

Project Required for shared VPC only

If you would like to use a shared VPC, you need this additional role in the scope of the host project of the VPC.

Compute Public IP Admin

roles/compute.publicIpAdmin

Project Required only when not using Cluster Connectivity Manager

This additional role is only required if you are planning to disable Cluster Connectivity Manager for your environment.

Option 2: Granular permissions

You should create a custom IAM role to assign these permissions.
Granular IAM permissions Scope Description

Required for data encryption and decryption.

cloudkms.cryptoKeys.list Project List all the keys inside any key ring.
cloudkms.keyRings.list Project List all the available key rings in the project.
cloudkms.cryptoKeyVersions.useToEncrypt Project Use a key to encrypt data.
cloudkms.cryptoKeyVersions.useToDecrypt Project Use a key to decrypt data.

Required to create, stop, start, and delete an external database from the Data Lake and Data Hub clusters.

cloudsql.instances.create Project Create a new Cloud SQL instance.
cloudsql.instances.delete Project Delete a Cloud SQL instance.
cloudsql.instances.get Project View details of a Cloud SQL instance.
cloudsql.instances.list Project List all the Cloud SQL instances.
cloudsql.instances.update Project Update a Cloud SQL instance.
cloudsql.users.create Project Create a new user on a Cloud SQL instance.
cloudsql.instances.startReplica Project Enables the creation of a read replica of an existing Cloud SQL instance.
cloudsql.instances.stopReplica Project Stop the replication process.
cloudsql.instances.restart Project Enable the restart process of a Cloud SQL instance.

Required to create VMs from images in your VPC.

compute.addresses.create Project Create external IP addresses that can be assigned to Google Cloud resources like virtual machine (VM) instances, load balancers, etc.
compute.addresses.get Project Display both internal and external IP addresses.
compute.addresses.use Project Use both internal and external IP addresses.
compute.disks.create Project Create disks for the VM instances.
compute.disks.delete Project Clean up the disks.
compute.disks.setLabels Project Set or modify labels on disks (you or a service account).
compute.disks.use Project Use the disk.
compute.firewalls.list Project List the firewall rules associated with a project (you or a service account).
compute.forwardingRules.create Project Create forwarding rules.
compute.forwardingRules.delete Project Delete forwarding rules.
compute.forwardingRules.list Project List all forwarding rules.
compute.globalOperations.get Project View the status of global operations in Google Cloud, such as creating or deleting global resources.
compute.images.get Project View details of a specific image (you or a service account).
compute.images.useReadOnly Project Use an image, but only in read-only mode.
compute.instanceGroups.create Project Create instance groups.
compute.instanceGroups.delete Project Delete instance groups.
compute.instanceGroups.get Project Get information about a particular instance group.
compute.instanceGroups.update Project Update an instance group.
compute.instanceGroups.use Project Use an instance group.
compute.instances.create Project Create VM instances.
compute.instances.delete Project Delete VM instances,
compute.instances.get Project Get information about a particular instance.
compute.instances.setLabels Project Set or modify labels on a VM instance.
compute.instances.setMetadata Project Set or update the metadata of a VM instance.
compute.instances.setServiceAccount Project Set or update the service account associated with a VM.
compute.instances.setTags Project Set or modify tags on a VM instance.
compute.instances.start Project Start the instances.
compute.instances.stop Project Stop the instances.
compute.instances.update Project Modify the configuration of a specific instance. Useful for vertical scaling.
compute.instances.use Project Use a VM instance.
compute.machineTypes.list Project List the virtual hardware configuration for a VM.
compute.networks.list Project List all the available networks.
compute.regionBackendServices.create Project Create a regional backend service. A regional backend service is part of the infrastructure that routes traffic to backend instances or groups within a specific region. This is useful for load balancers.
compute.regionBackendServices.delete Project Delete a regional backend service.
compute.regionBackendServices.use Project Use a regional backend service.
compute.regionHealthChecks.create Project Create health checks that monitor the health of the backend services, an important feature for load balancers.
compute.regionHealthChecks.delete Project Delete health checks.
compute.regionHealthChecks.useReadOnly Project Use health checks restricted to read-only mode.
compute.regionOperations.get Project View the status of region operations in Google Cloud. This is required regardless of globalOperations get permissions because both have different scopes.
compute.regions.get Project Get information about a specific region.
compute.regions.list Project List all the regions in the project.
compute.subnetworks.list Project List all the subnets in the project.
compute.subnetworks.use Project Use subnets.
compute.subnetworks.useExternalIp Project Assign external IP addresses to VM instances. If this is enabled, attempting to create a VM instance with an external IP address will fail.
compute.zoneOperations.get Project View the status of zone operations in Google Cloud. This is required for both global and regional level permissions.
compute.images.create Project Create a new image. This is required if you want to use a custom image.
compute.addresses.delete Project Delete static IP addresses that are allocated within a specific region or globally in Google Cloud (you or a service account).
compute.forwardingRules.setLabels Project Set or update forwarding rules labels.
compute.forwardingRules.use Project Use forwarding rules, most often used by load balancers to balance the traffic to backend services.
compute.regionHealthChecks.update Project Update the health checks that monitor the backend services.
compute.addresses.createInternal Project Create internal IP addresses within a Virtual Private Cloud (VPC). These internal IP addresses are used for private communication within your network, not exposed to the Internet.
compute.firewalls.create Project Create firewall rules (you or a service account).
compute.firewalls.delete Project Delete the firewall rules (you or a service account).
compute.subnetworks.get Project Get details about a specific subnet.
compute.networks.get Project Get details about a specific network.

These are required for Cloudera to access the service accounts that you created.

iam.serviceAccounts.actAs Project Allows you or a service account to impersonate a service account.
iam.serviceAccounts.list Project List all the service accounts within a specific project.

(Optional) By default, Cloudera creates this bucket, but you can pre-create it. This is not required if you are planning to pre-create the GCS bucket for storing OS images for VMs. See Storage bucket for OS images.

storage.buckets.get Project Get all the storage buckets from a specific project.
storage.objects.create Project Upload objects to the storage accounts.
storage.objects.delete Project Delete objects from the storage accounts.
storage.objects.get Project Retrieve objects from the storage account. This does not allow the listing of objects in the storage bucket.
For instructions on how to create the service account, refer to the following documentation:

Create provisioning credential's service account and generate access key

Create a service account and generate a JSON access key.

Before you begin

Review the above permissions to learn what IAM permissions and IAM roles you need to assign to the service account that you will create.

Steps

  1. Log in to your Google Cloud account.

  2. Navigate to the project used for Cloudera.

  3. Navigate to the IAM & Admin.
  4. To create a custom role:
    1. Navigate to the Roles page.
    2. Click +Create Role.
    3. Specify a Title.
    4. Specify an ID.
    5. Click +Add Permissions.
    6. Add the required granular permission(s).
    7. Use the same steps to add all the required permissions.
    8. Click Create.
  5. To create a service account:
    1. Navigate to the Service accounts page.
    2. Click Create service account.
    3. Enter a service account name.
    4. Click Create.
    5. Under Grant this service account access to project, choose the IAM roles to grant to the service account on the project. You need to assign all of the roles listed in the table.
    6. When you are done adding all the required roles, click Done to finish creating the service account.
  6. To generate an access key:
    1. Once your account has been created, find the row of the service account that you want to create a key for. In that row, click the (context menu) button, and then click Create key.
    2. Under Key type, select JSON and click Create.
    3. Clicking Create downloads the service account key file. You will use the JSON access key to register the service account as a credential in Cloudera.
  7. Additionally, once you create the Logger and IDBroker service accounts, you need to update each of these two service accounts to grant the provisioning service account the Service Account User (iam.serviceAccountUser) role. The instructions are provided as part of Minimum setup for cloud storage.

What to do next

Once you have this setup ready, you can Register a GCP credential in Cloudera.