Cloudera Docs

Cross-account access IAM role

To allow Cloudera to create resources in your AWS account, you create a cross-account access IAM role in your AWS account and grant Cloudera access to the role as a trusted principal by specifying a specific AWS account and an external ID.

The policy for the cross-account access IAM role must have the permissions enumerated in the documentation linked below. In addition, the IAM role must reference the specific AWS account ID and external ID provided in the Cloudera Management Console.

IAM policy definitions

When creating an IAM policy for the provisioning credential, use the following IAM policy definitions.

You have two options:

Option Description Policies
Default policy This policy is appropriate for registering a Cloudera environment and provisioning all Cloudera services. This is the same IAM policy as the Default policy provided in the Cloudera Management Console web interface.
Reduced access policies Instead of the extensive default policy, you can use minimal reduced access policies. Note that the first policy is required for provisioning a Cloudera environment. The remaining policies are optional and are only needed if you would like to use the specific Cloudera services.

Create a cross-account IAM role

In order to use role-based authentication, you must create an IAM role on AWS.

Prior to creating a cross-account IAM role on AWS, log in to the Cloudera web interface and obtain the parameters that need to provide for the IAM role:
  1. In the Cloudera Management Console, navigate to Shared Resources > Credentials > Create Credential:
  2. Note or copy the Cross-account Access Policy, Account ID, and External ID listed in the UI:

    You will need them to complete the following steps.

  1. Log in to the AWS Management Console.
  2. Navigate to the IAM console > Policies and click Create policy:
    1. Select the JSON tab and paste the Cross-account Access Policy to the Policy editor.
    2. Click Next.
    3. Enter a name for your policy in the Name field of the Review policy page:
    4. Click Create policy when finished.
  3. Navigate to the IAM console > Roles and click Create role:
  4. Select AWS account as Trusted Entity type.
  5. Check Another AWS account and provide the following information:
    • Copy and paste your Account ID provided in Cloudera to the Accound ID field.
    • Check Require external ID under Options.
    • Copy and paste the External ID from Cloudera to the External ID field.
  6. Click Next.
  7. Search for the previously created policy, and add the permission to the role by checking the box.
  8. Click Next.
  9. Enter a role name to the Role name field:
  10. Click Create role to finish the role creation process.
  11. Search for the created role, and obtain the IAM Role ARN. You will need it to create a role-based credential:

Once you are done creating the IAM role on AWS, create a role-based credential in Cloudera.

Reduced access policy definition

The following reduced policy definition can optionally be used to register a Cloudera environment and create Cloudera Data Hub clusters and Cloudera Operational Database databases in it.

The IAM role used for the provisioning credential can use the reduced access policy instead of the default policy.

You should:

  1. Copy this policy.

  2. Update it as described below.

  3. Once done, you can create the cross-account role using this policy.

Customize the scope of iam:PassRole

Cloudera utilizes the iam:PassRole as part of the cloud identity federation setup process via IDBroker on AWS. The default policy uses “*” as the resource scope for this action: 
{
     "Effect": "Allow
      "Action": [
        "iam:PassRole"
      ],
      "Resource": "*"
}

However the scope can be reduced to just a few roles. To do this, you need to edit the following block and replace the “*” with the ARNs of the following roles used for the environment:

  • IDBROKER_ROLE
  • LOG_ROLE
  • RANGER_AUDIT_ROLE
  • DATALAKE_ADMIN_ROLE

You choose the name for these roles when you set up Cloudera as part of the Minimal Setup for Cloud Storage.

After updating according to the minimal setup for cloud storage, the iam:PassRole policy excerpt should like similar to the following: 

{
      "Effect": "Allow
      "Action": [
        "iam:PassRole"
      ],
      "Resource": [
               "arn:aws:iam::$ACCOUNTID:role/IDBROKER_ROLE",
               "arn:aws:iam::$ACCOUNTID:role/LOG_ROLE",
               “arn:aws:iam::$ACCOUNTID:role/RANGER_AUDIT_ROLE",
               "arn:aws:iam::$ACCOUNTID:role/DATALAKE_ADMIN_ROLE"
           ]
}

If you have already created these roles and you don’t know the name of these roles, you can obtain them using the following steps:

IDBROKER_ROLE

  1. Log in to Cloudera web interface and navigate to the Cloudera Management Console service > Data Lakes.
  2. Scroll down to where you see Event History and select the Hardware tab.
  3. Locate the EC2 instance for the idbroker node (if you are running Medium Duty Data Lake, you will see two IDBroker nodes, you can use either):
  4. Click on the instance id. You will be redirected to the EC2 console.
  5. In the Instance Details section for this node, you can see the name of the IAM role being used by this instance. Click on this role to get to the details.
  6. Copy the ARN of the instance profile. It follows the following naming convention: arn:aws:iam::<12-digit-AWS-account-id>:instance-profile/<name-of-idbroker-role>

LOG_ROLE

  1. Navigate to Environments .
  2. Click on the existing environment
  3. Click on the Summary tab.
  4. Scroll to Logs Storage and Audits and you will see the Instance Profile entry.
  5. Copy the ARN of the Instance Profile:

RANGER_AUDIT_ROLE and DATALAKE_ADMIN_ROLE

  1. Navigate to Environments.
  2. Click on the existing environment.
  3. From the Actions menu, select Manage Access.
  4. Click on the IDBroker Mappings tab.
  5. Under Current Mappings you will see the mappings for the Data Access Role and Ranger Audit Role.
  6. Copy the ARNs of the Data Access Role and Ranger Audit Role:

Further permissions reduction (Optional)

You can reduce these policies even further by pre-creating some resources:

  • If you use your own security groups, then the following actions can be removed from the policy:
    ec2:CreateSecurityGroup
ec2:AuthorizeSecurityGroupIngress
ec2:RevokeSecurityGroupEgress
ec2:AuthorizeSecurityGroupEgress
  • If you use private IPs or set up Cluster Connectivity Manager to communicate with the Cloudera Control Plane, then the following actions can be removed from the policy:
    ec2:CreateSecurityGroup
ec2:AuthorizeSecurityGroupIngress
ec2:RevokeSecurityGroupEgress
ec2:AuthorizeSecurityGroupEgress
  • If you use private IPs or set up Cluster Connectivity Manager to communicate with the Cloudera Control Plane, then the following actions can be removed from the policy:
    ec2:AllocateAddress
ec2:ReleaseAddress

AWS IAM permission rds:DescribeOrderableDBInstanceOptions for fallback database instance types

When provisioning fails because a cloud provider experiences capacity issues for a specific database instance type in a specific region, the fallback feature lets a request specify a primary instance type and one or more fallback instance types to keep provisioning resilient.

Overview

Summary

Cloudera now lets a database creation request carry an ordered list of fallback instance types in addition to the primary instance type. If provisioning causes a capacity problem for the primary instance type, it automatically retries with consecutive fallbacks until one succeeds. Cloudera uses the AWS API DescribeOrderableDBInstanceOptions before provisioning to validate that the primary and fallback types can be ordered in the customer’s region. This requires adding the IAM action rds:DescribeOrderableDBInstanceOptions to the cross-account IAM role used for the AWS Environment (the provisioning credential's minimal IAM policy).

The DescribeOrderableDBInstanceOptions read-only AWS RDS API lists the RDS instance classes that can currently be ordered for a given database engine and version in a region (including their availability zones). Cloudera queries it for the PostgreSQL engine at the supported minimum engine version.

Prerequisites

  • Customers using the reduced access policy (the minimal/restricted cross-account policy) must add rds:DescribeOrderableDBInstanceOptions to the cross-account IAM role's attached policy. Without rds:DescribeOrderableDBInstanceOptions, the fallback-type validation cannot run and is silently skipped.
  • Customers using the default (broad rds:*) policy do not require any changes; the wildcard already covers it.
  • The feature must be enabled for the account (entitlement CDP_FALLBACK_DATABASE_INSTANCETYPE).
  • The feature is gated by the CDP_FALLBACK_DATABASE_INSTANCETYPE account entitlement. The fallback list is only attached to a request when the account has this entitlement. Without the entitlement, behavior is unchanged (single instance type, no fallback). This is separate from the CDP_FALLBACK_INSTANCETYPE compute-instance fallback entitlement.

Operation

Once prerequisites are met, the fallback feature works as follows.

Validation

The service validates the list before provisioning.
  • The primary instance type must be present.
  • The fallback instance list cannot include any blanks or duplicates.
  • The primary instance cannot be repeated in the fallback instance list.
  • Every instance type (primary and fallback) must actually be orderable in the target region. Invalid requests are rejected immediately with a clear error.

Provisioning

Provisioning attempts the primary instance type first. During a capacity failure, consecutive fallback instances are retried in order until one succeeds or the list is exhausted.

Use case

During validation, Cloudera fetches the set of orderable instance types for the customer's region and checks that the requested primary and fallback instance types are all included in the set. If an instance type is not orderable in the region, the request is rejected with the list of unavailable instance types and the list of available ones. The customer receives an actionable error instead of a failed provisioning attempt later.