CDH6.3.1部署

基础环境准备

1.1基础环境准备

1.1.1基础环境

名称	ip	操作系统
node1	192.168.75.10	centos7.9
node2	192.168.75.11	centos7.9
node3	192.168.75.12	centos7.9

1.1.2镜像包准备

mysql-connector-java-8.0.26.jar

CM6.3.1
cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm
oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
allkeys.asc

CDH6.3.1
CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel
CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1
manifest.json

文件介绍：

1）所有rpm文件为CM二进制安装包，后续需要放到本地yum路径下面，供安装使用

2）CDH开头的两文件一个为基础组件安装包，sha文件为类似MD5值校验的文件，需要放到/opt/cloudera/parcel-repo/，进行组件安装

3）allkeys.asc 为密文检验，跟yum源放一起

1.2 SSH免密登录

修改主机名
hostnamectl set-hostname node1

主机名映射
vim /etc/hosts
192.168.75.10 node1
192.168.75.11 ndoe2
192.168.75.12 node3
scp /etc/hosts root@node2:/etc/hosts
scp /etc/hosts root@node3:/etc/hosts

ssh免密登录
ssh-keygen
cat .ssh/*.pub > .ssh/authorized_keys
scp -r .ssh root@node2:/root/
scp -r .ssh root@node3:/root/

1.3 关闭禁止防火墙

systemctl stop firewalld 
systemctl disable firewalld

vim /etc/sysconfig/selinux
SELINUX=enforcing 改为 SELINUX=disabled
setenforce 0

1.4开启时钟同步

yum install -y ntp
systemctl enable ntpd
systemctl start ntpd

验证
ntpstat

1.5 httpd服务安装配置,主节点

yum install -y httpd
systemctl enable httpd
systemctl start httpd

1.6安装mysql，主节点

卸载mariadb
rpm -qa | grep mariadb
rpm -e --nodeps mariadb

安装mysql8.0
具体安装过程略过

1.7 创建CDH源数据库、用户

创建scm用户
CREATE USER 'scm'@'%' IDENTIFIED BY 'mysql@Edmp.com';
grant all privileges on *.* to root@'%';
grant all privileges on *.* to 'scm'@'%' with grant option;
flush privileges;

创建数据库
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
 
grant all privileges on *.* to 'root'@'%' with grant option;
grant all privileges on *.* to 'scm'@'%' with grant option;
flush privileges; 

1.8部署JDBC

mkdir /usr/share/java
cd /usr/share/java
cp /opt/CDH/mysql-connector-java-8.0.26.jar ./
mv mysql-connector-java-8.0.26.jar mysql-connector-java.jar

分发给各个节点
scp mysql-connector-java.jar root@node2:/usr/share/java
scp mysql-connector-java.jar root@node3:/usr/share/java

1.9 JDK环境部署，每个节点

rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

配置环境变量
vim /etc/profile

echo 'export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera' >> /etc/profile
echo 'export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar' >> /etc/profile
echo 'PATH=$PATH:$HOME/bin:$JAVA_HOME/bin' >> /etc/profile
source /etc/profile
java -version

CDH部署

2.1 httpd部署离线parcel源

2.1.1安装包迁移

mkdir -p /var/www/html/cdh6_parcel/
cp manifest.json /var/www/html/cdh6_parcel/
cp allkeys.asc /var/www/html/cdh6_parcel/
cp *.rpm /var/www/html/cdh6_parcel/
cp CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel* /var/www/html/cdh6_parcel/
 
systemctl restart httpd

2.1.2编写/etc/yum.repos.d/cloduera-manger.repo

[cloduera-manger]
name=Cloduera Manger 6.3.1
baseurl=http://node1/cdh6_parcel/
enable=1
gpgcheck=0
autorefresh=0
type=rpm-md

2.1.3分发该文件到其他节点

scp /etc/yum.repos.d/cloduera-manger.repo root@node2:/etc/yum.repos.d/
scp /etc/yum.repos.d/cloduera-manger.repo root@node3:/etc/yum.repos.d/

2.1.4创建repodata，本地yum源

cd /var/www/html/cdh6_parcel/  #本地yum配置路径
createrepo .

yum生效
systemctl restart httpd
yum clean all
yum makecache

2.2 CM部署

安装依赖包
yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi portmap httpd mod_ssl openssl-devel python-psycopg2 MySQL-python /lib/lsb/init-functions

node1
rpm -ivh cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
rpm -ivh cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
rpm -ivh cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
rpm -ivh cloudera-manager-server-db-2-6.0.1-610811.el7.x86_64.rpm

node2 3
rpm -ivh cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
rpm -ivh cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm



禁用透明大页面压缩 node1 2 3都执行
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
上面两条写入开机自启动
vim /etc/rc.local

优化交换分区
vim /etc/sysctl.conf
	vm.swappiness = 10
sysctl -p /etc/sysctl.conf 

设置cloudera manager数据库
mysql在本地时执行
rm -rf /etc/cloudera-scm-server/db.mgmt.properties
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm mysql@Edmp.com


# 在cm02.spark.com的服务器执行命令
/usr/share/cmf/schema/scm_prepare_database.sh mysql -h cm01.spark.com -uroot -p123123 --scm-host cm02.spark.com scm scm scm
为Cloudera Manager指定数据库为Mysql
通过用户名为root，密码为123123，连接cm01.spark.com上的Mysql数据库。
为在cm02.spark.com的cloudera-scm-server服务，创建一个名叫scm的数据库(第一个scm)，并为这个数据库创建一个用户，用户名为scm(第二个scm)，密码也为scm(第三个scm)


将CHD6相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中，如果没有此目录，可以自己创建。（如果是在线安装，那么此步骤可以忽略。）

注意：
最后将CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256，重命名为CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha，这点必须注意，否则系统会重新下载CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel



启动Cloudera Manager Server：
systemctl start cloudera-scm-server

可以查看启动日志
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

看到此条信息，说明启动完成
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server. 

2.3 登录CM

http://node1:7180/cmf/login

默认凭证
Username: admin
Password: admin