Manual Installation - Hortonworks Data Platform

Installing HDP Manually

Also available as:

PDF

Contents

1. Getting Ready to Install
- Meet Minimum System Requirements
- Virtualization and Cloud Platforms
- Configure the Remote Repositories
- Decide on Deployment Type
- Collect Information
- Prepare the Environment
- Download Companion Files
- Define Environment Parameters
- [Optional] Create System Users and Groups
- Determine HDP Memory Configuration Settings
  - Running the HDP Utility Script
  - Manually Calculating YARN and MapReduce Memory Configuration Settings
- Configuring NameNode Heap Size
- Allocate Adequate Log Space for HDP
2. Installing HDFS and YARN
- Set Default File and Directory Permissions
- Install the Hadoop Packages
- Install Compression Libraries
  - Install Snappy
  - Install LZO
- Create Directories
3. Installing Apache ZooKeeper
- Install the ZooKeeper Package
- Securing ZooKeeper with Kerberos (optional)
- Set Directories and Permissions
- Set Up the Configuration Files
- Start ZooKeeper
4. Setting Up the Hadoop Configuration
5. Validating the Core Hadoop Installation
- Format and Start HDFS
- Smoke Test HDFS
- Configure YARN and MapReduce
- Start YARN
- Start MapReduce JobHistory Server
- Smoke Test MapReduce
6. Installing Apache HBase
- Install the HBase RPMs
- Set Directories and Permissions
- Set Up the Configuration Files
- Validate the Installation
- Starting the HBase Thrift and REST Servers
7. Installing Apache Phoenix
- Configuring HBase for Phoenix
- Configuring Phoenix to Run in a Secure Cluster
- Smoke Testing Phoenix
- Troubleshooting Phoenix
8. Installing and Configuring Apache Tez
- Prerequisites
- Install the Tez RPM
- Configure Tez
- Validate the Tez Installation
- Troubleshooting
9. Installing Apache Hive and Apache HCatalog
- Installing the Hive-HCatalog RPM
- Setting Directories and Permissions
- Setting Up the Hive/HCatalog Configuration Files
  - HDP-Utility script
  - Configure Hive and HiveServer2 for Tez
- Setting Up RDBMS for Use with the Hive Metastore
- Creating Directories on HDFS
- Validating the Installation
- Enabling Tez for Hive Queries
- Disabling Tez for Hive Queries
- Configuring Tez with the Capacity Scheduler
- Validating Hive-on-Tez Installation
10. Installing Apache Pig
- Install the Pig RPMs
- Set Up Configuration Files
- Validate the Installation
11. Installing Apache WebHCat
- Install the WebHCat RPMs
- Upload the Pig, Hive and Sqoop tarballs to HDFS
- Set Directories and Permissions
- Modify WebHCat Configuration Files
- Set Up HDFS User and Prepare WebHCat Directories
- Validate the Installation
12. Installing Apache Oozie
- Install the Oozie RPMs
- Set Directories and Permissions
- Set Up the Oozie Configuration Files
- Configure Your Database for Oozie
- Validate the Installation
13. Installing Apache Ranger
- Installation Prerequisites
- Manual Installation
- Installing Policy Manager
- Installing UserSync
- Installing Ranger Plug-ins
- Verifying the Installation
14. Installing Hue
- Prerequisites
- Configure HDP
- Install Hue
- Configure Hue
- Start Hue
- Configuring Hue for an External Database
- Using Hue with Oracle
- Using Hue with MySQL
- Using Hue with PostgreSQL
15. Installing Apache Sqoop
- Install the Sqoop RPMs
- Set Up the Sqoop Configuration
- Validate the Installation
16. Installing Apache Mahout
17. Installing and Configuring Apache Flume
- Understanding Flume
- Installing Flume
- Configuring Flume
- Starting Flume
- HDP and Flume
- A Simple Example
18. Installing and Configuring Apache Storm
- Install the Storm RPMs
- Configure Storm
- Configure a Process Controller
- (Optional) Configure Kerberos Authentication for Storm
- (Optional) Configuring Authorization for Storm
- Validate the Installation
19. Installing and Configuring Apache Spark
- Spark Prerequisites
- Installing Spark
- Configuring Spark
- Validating Spark
20. Installing and Configuring Apache Kafka
- Install Kafka
- Configure Kafka
- Validate Kafka
21. Installing Apache Accumulo
- Install the Accumulo RPM
- Configure Accumulo
- Validate Accumulo
22. Installing Apache Falcon
- Install the Falcon RPM
- Configuring Proxy Settings
- Configuring Falcon Entities
- Configuring Oozie for Falcon
- Configuring Hive for Falcon
- Configuring for Secure Clusters
- Validate Falcon
23. Installing Apache Knox
- Install the Knox RPMs on the Knox server
- Set up and Validate the Knox Gateway Installation
24. Installing Ganglia (Deprecated)
- Install the Ganglia RPMs
- Install the Configuration Files
- Extract the Ganglia Configuration Files
- Copy the Configuration Files
- Set Up Ganglia Hosts
- Set Up Configurations
- Set Up Hadoop Metrics
- Validate the Installation
25. Installing Nagios (Deprecated)
- Install the Nagios RPMs
- Install the Configuration Files
- Extract the Nagios Configuration Files
- Create the Nagios Directories
- Copy the Configuration Files
- Set the Nagios Admin Password
- Set the Nagios Admin Email Contact Address
- Register the Hadoop Configuration Files
- Set Hosts
- Set Host Groups
- Set Services
- Set Status
- Add Templeton Status and Check TCP Wrapper Commands
- Validate the Installation
26. Installing Apache Slider
27. Setting Up Security for Manual Installs
- Preparing Kerberos
- Configuring HDP
- Configuring Hue
- Setting up One-Way Trust with Active Directory
  - Configure Kerberos Hadoop Realm on the AD DC
  - Configure the AD Domain on the KDC and Hadoop Cluster Hosts
28. Uninstalling HDP

Manual Installation

This section describes how to:

Installing Policy Manager
Installing UserSync
Manually install Ranger Plug-ins for HDFS, HBase, Hive, Knox or Storm
Verifying the Installation