Deploying a flow using the wizard

Deploy a flow definition to run an Apache NiFi flow in Cloudera Data Flow. You do this by selecting a flow definition and launching the deployment wizard, where you select an environment, provide values for required parameters, configure sizing, inbound connections, custom resources, and KPIs.

You have an enabled and healthy Cloudera Data Flow environment.
The flow definition you want to deploy has been added to the Catalog by someone with DFCatalogAdmin role.
You must have at least DFCatalogViewer role granting you access to the Catalog.
You must have at least DFCollectionViewer role granting you access to the collection where the flow definition is located.
You must have DFFlowAdmin role for the environment where you want to deploy the flow definition.
You must have at least DFProjectMember role for the project where you want to deploy the flow definition.
If you deploy custom processors or controller services, you may need to meet additional prerequisites.

For information on roles and authorization, see Cloudera Data Flow authorization.

Select the flow definition version you want to deploy from the catalog and start the deployment wizard

Select from flow definitions available to you in the Cloudera Data Flow Catalog and choose if you want to add the flow to an existing deployment or you want to create it in a new deployment.

In Cloudera Data Flow, select Catalog from the left navigation pane.
Flow definitions available for you to deploy are displayed, one definition per row.
Select a flow definition to view details and available versions.
The flow details pane opens on the right.
Select a flow definition version then click Deploy to launch the deployment wizard.
- Select Create New Deployment to deploy the flow on a newly created NiFi cluster and continue with .
- Select Add to Existing Deployment to deploy the flow on an existing NiFi cluster and proceed with
  note
  If your Cloudera Data Flow environment and/or deployment version is 2.11.0 or lower, you cannot add multiple flows to an existing deployment, therefore this option is not available to you. Select an environment and proceed with naming your flow and assigning it to a project.

Launch the deployment wizard

Select a flow definition version, decide whether you want to deploy it on an existing NiFi cluster or you want to create a new one along the way. Follow the instructions in the deployment wizard.

Select a flow definition version then click Deploy

to launch the deployment wizard.

Select Create New Deployment to deploy the flow on a newly created NiFi cluster and continue with .
Select Add to Existing Deployment to deploy the flow on an existing NiFi cluster and proceed with
note
If your Cloudera Data Flow environment and/or deployment version is 2.11.0 or lower, you cannot add multiple flows to an existing deployment, therefore this option is not available to you. Select an environment and proceed with naming your flow and assigning it to a project.

Name your flow

When you add a flow to an existing deployment you must provide a flow name that is unique within that deployment. You can import an existing flow configuration to speed up flow deployment.

warning

When you add a flow version that requires custom NARs or custom Python processors to an existing deployment which does not already have those resources in place, Cloudera Data Flow gives you a warning instructing you to add the missing resource before proceeding.

Failing to add a custom NAR to your deployment before deploying the flow version results in erroneous operation until you add the NAR and Cloudera Data Flow recognizes the change. This may not be an issue in testbed use, but you should avoid it in production use.

Adding a custom Python processor to a deployment after you deployed a flow version has absolutely no effect on the operation of the already deployed flow. You can only fix that issue by opening the flow description in Flow Designer for editing, removing and then re-adding the custom Python processor, publishing your changes as a new flow version, and performing a flow version change on your flow. This sequence of actions changes the ID of the component and causes Cloudera Data Flow to pick up the custom Python processor that you have added post-deployment.

Provide a unique Flow Name.
This name helps you distinguish between individual instances of a flow within a deployment.
note

Flow name must be unique within a deployment. The deployment wizard marks a name valid by displaying a green checkmark below the Flow Name text box.
Optional: If you have previously exported a configuration that closely aligns with the one you are deploying now, you can import it under Import Configuration to auto-fill as much of the wizard as possible.
You can overwrite imported configuration values in the deployment wizard.
Optional: If you do not want the flow to start running after deployment, unselect the Automatically start flow upon successful deployment checkbox.
If the flow version you are deploying requires an inbound connection and the target deployment does not have one, you see a warning. You cannot proceed until you either add an inbound conection configuration to the deployment in the Deployment Manager, or you select the I understand and choose to proceed with the configuration as is. option.

If you click To set a Custom Inbound Connections Configuration open the Deployment Manager option, Deployment Manager opens in a new window where you can configure an inbound connection before proceeding with adding the flow.

If you select the I understand and choose to proceed with the configuration as is. option, the Next button becomes active and you can proceed with adding the flow without configuring the inbound connection at this time.

tip
If you do not configure an inbound connection now, unselect the Automatically start flow upon successful deployment option and configure the inbound connection before starting the flow in Deployment Manager.
Click Next .

Configure NiFi

After selecting the target environment, project, and naming your flow, you need to set Apache NiFi version, possible inbound connections, and custom processors. Depending on the flow definition, you may also need to provide values for a number of configuration parameters. Finally, you need to set the capacity of the NiFi cluster servicing your deployment.

Pick a NiFi Runtime Version for your flow deployment.
Select if you want to use Apache NiFi 1.x or 2.x with your deployment.

Cloudera recommends that you always use the latest available version within the 1.x and 2.x lines, if possible.
Specify whether you want to use inbound connections that allow your flow to receive data from an external data source.

If yes, specify the endpoint host name and listening port(s) where your flow listens to incoming data.

See Creating an inbound connection endpoint for complete information on endpoint configuration options.
Specify whether you want to use NiFi Archives (NARs) to deploy custom NiFi processors or controller services.

If yes, specify the Cloudera Workload Username, password, and cloud storage location where the NARs are stored.

tip
If you want to provide a machine user as Cloudera Workload Username during flow deployment, make sure to note the full workload user name including the srv_ prefix.

Make sure that you click the Save button specific to Custom NAR Configuration before proceeding.
If you selected to run your flow with NiFi 2.x, specify whether you want to use custom Python processors with your flow.
If yes, specify the Cloudera Workload Username, password, and cloud storage location where the processors are stored.
tip
Create a dedicated directory in your object store where you keep all Python processors. Create one Python script per processor and store it in this directory.
tip
If you want to provide a machine user as Cloudera Workload Username during flow deployment, make sure to note the full workload user name including the srv_ prefix.

Make sure that you click the Save button specific to Custom Python Processors before proceeding.
Click Next .

Provide parameter values

Depending on the flow you deploy, you may need to specify parameter values like connection strings or credentials, and upload files like truststores or JARs.

Provide values to parameters required for your flow deployment.
You have to provide values for all parameters. Filter for empty fields by selecting the No value checkbox.

tip
If you are deploying a ReadyFlow, you can learn about required parameters and instructions on how to obtain parameter values by checking Prerequisites and Required parameters in the documentation of the respective ReadyFlow.
When you finish setting configuration parameters, click Next .
You may edit parameters individually by clicking the edit icon next to them or you may edit entire parameter groups by selecting the Edit Group option.

In either case, you have the option to define a custom parameter value that is specific to this deployment or, if available, select the value from a shared parameter group.
You may import a set of shared parameters from a parameter group to fill in values and facilitate flow deployment. To do so, select the Import Shared Parameters option.
- Parameters in the shared group that match a deployment parameter are preselected, unless you have already modified that parameter in the deployment wizard. In that case that particular parameter is left unselected to protect any changes you have already made and you need to manually select that parameter.

Configure sizing and scaling

Set the size and number of Apache NiFi nodes, auto-scaling, and the type of storage to be used.

Specify NiFi node size by selecting one of the options:
- Extra Small: 2 vCores per Node, 4 GB RAM per Node
- Small: 3 vCores per Node, 6 GB RAM per Node
- Medium: 6 vCores per Node, 12 GB RAM per Node
- Large: 12 vCores per Node, 24 GB RAM per Node
- Custom: If you select this option, you can set the number of vCores between 1-4 and the amount of RAM per node between 2-8 GB. The memory allocated here is the total memory for the NiFi container, of which 50% is allocated to the NiFi JVM heap.
Set the number of NiFi nodes and auto-scaling.
- Select the Auto Scaling checkbox, if you want your cluster to automatically scale based on CPU utilization. When you enable auto-scaling, the minimum number of NiFi nodes are used for initial size and the workload scales up or down depending on resource demands.
- Set the number of nodes between 1 and 32.
- If you enabled Auto Scaling, you may enable your flow to scale based on flow performance metrics. To do so, select the Flow Metrics Scaling checkbox.

Select storage type.

Select whether you want your deployment to use storage optimized for cost or for performance. You also have the option to set a custom storage size. Only use the custom option if you know how changing individual values affects your deployment.

Standard: 512 GB Content Repo Size, 512 GB Provenance Repo Size, 256 GB Flow File Repo Size, 2300 IOPS, 150 MB/s Max Throughput
Performance: 1024 GB Content Repo Size, 1024 GB Provenance Repo Size, 256 GB Flow File Repo Size, 5000 IOPS, 200 MB/s Max Throughput

Custom:

Depending on your deployment scenario, you have the following options:

On AWS, you can set the repo Size between 50 - 1000 GB, IOPS between 3000 - 7000 in increments of 1000, and Max Throughput between 125 MB/s - 500 MB/s in increments of 100 for both the Content Repo, Provenance Repo, and Flow File Repo. You can set the identical IOPS and Maximum throughput values to all repositories by setting the values for one repository and then selecting the Apply IOPS and Throughput values to all repositories. checkbox.

On Azure, you can select one of four predefined custom storage configurations for Content Repo, Provenance Repo, and Flow File Repo. Table Available Azure storage configurations describes the predefined custom storage values.

Table 1. Available Azure storage configurations
Storage configuration	Disk size (GB)	IOPS	Throughput (MB/s)
P6	64	240	50
P10	128	500	100
P15	256	1100	125
P30	1024	5000	200

If applicable, configure additional storage for Python processors.
The wizard automatically detects if the flow definition being deployed contains any Python processors and suggests the amount of extra disk space to add because of them.
note

If the flow definition does not contain Python processors and you do not plan to introduce Python processors to this flow in the future, you need to take no action. Skip this step.

If you plan to introduce Python processors to this flow in the future, you must add the extra storage now as you cannot change storage size after deployment.
Click Next .

Set Key performance indicators

Optionally add key performance indicators to help you track the performance of your flow deployment then review your settings and launch the deployment process.

From KPIs, you may choose to identify key performance indicators (KPIs), the metrics to track those KPIs, and when and how to receive alerts about the KPI metrics tracking.

tip

You can reorder the KPIs by dragging them up or down the list. The order you configure here will be reflected in the detailed monitoring view of the resulting deployment.

See Working with KPIs for complete information about the KPIs available to you and how to monitor them.

note
If you deploy a flow to an exsting deployment, you can only define KPIs for the new flow, you cannot add or change deployment-related KPIs.
Click Next .

Verify your settings and initiate deployment

Review deployment settings, make any necessary changes, and start deployment.

Review a summary of the information provided and make any necessary edits by clicking Previous.
When you are finished, complete your flow deployment by clicking Deploy.

tip

Click View CLI Command to see the equivalent Cloudera CLI syntax in a help pane.

After you click Deploy, you are redirected to the Alerts tab in the Flow Details where you can track how the deployment progresses.