Hadoop 1.2.1 安装笔记01 : Linux 与免密码

浏览数：212 / 时间：2015年06月20日

目标：配置一个hadoop 1.2.1 测试环境

用的JDK是: jdk-7u65-linux-x64.gz

选的hadoop 是： hadoop-1.2.1.tar.gz

均来源Apache和oracle网站

主机规划：

Linux版本： Centos 6.5 x64位

/boot：用来存放与Linux系统启动有关的程序，比如启动引导装载程序等，建议大小为100MB。

/usr：用来存放Linux系统中的应用程序，其相关数据较多，建议大于3GB以上。

/var：用来存放Linux系统中经常变化的数据以及日志文件，建议大于1GB以上。

/home：存放普通用户的数据，是普通用户的宿主目录，建议大小为剩下的空间。

/：Linux系统的根目录，所有的目录都挂在这个目录下面，建议大小为5GB以上。

/tmp：将临时盘在独立的分区，可避免在文件系统被塞满时影响到系统的稳定性。建议大小为500MB以上。

swap：实现虚拟内存，建议大小是物理内存的1~2倍。

Boot Loader密码是 D*****2014

安装选择 Basic Server 不然所有的包得一个个打，确实很麻烦.

配置sudo ，赋予hadoop用户权限，并配置sudo免密

useradd hadoop

passwd hadoop

vi /etc/sudoers

root ALL=(ALL) ALL

hadoop ALL=(ALL) ALL 添加一行

hadoop ALL=(ALL) NOPASSWD: ALL 添加一行免密码

配置本机的解析

[hadoop@master /]$ cat /etc/hosts
10.15.5.200 master.hadoop
10.15.5.201 slave01.hadoop
10.15.5.202 slave02.hadoop

配置主机名

vi etc/sysconfig/network

HOSTNAME=master.hadoop

配置IP 地址

[hadoop@master /]$ cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
UUID=721f9261-45d5-4335-9b47-64459173b2a9
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
HWADDR=00:50:56:82:00:0F
IPADDR=10.15.5.200
PREFIX=24
GATEWAY=10.15.5.1
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"

挂载安装盘，以便所需要的安装包。打上ftp包方便安装软件

[root@master home]# mount -t auto /dev/cdrom /home/cdrom
mount: block device /dev/sr0 is write-protected, mounting read-only

[root@master Packages]# rpm -ivh ftp-0.17-54.el6.x86_64.rpm

warning: ftp-0.17-54.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
1:ftp ########################################### [100%]

目标： master和slaves hadoop账户无密码互访

原理：

可以粗糙的理解为，我制造一把钥匙，把样子发给你，你在门禁系统里记下我钥匙的样子，我拿这把钥匙去开门时候你的门禁系统直接放行。

Master上，登录hadoop用户，并在/home/hadoop目录下操作

[hadoop@master ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory ‘/home/hadoop/.ssh‘.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
8d:81:09:c8:45:3f:c0:fb:3b:a0:cf:95:b6:dd:e9:b1 [email protected]
The key‘s randomart image is:
+--[ RSA 2048]----+
| . == |
| o .+ o |
| .= . |
| . . + |
| . S . |
| . .. |
| . .+. . |
| .. ooo . + |
| .o ....E |
+-----------------+

该隐藏文件夹下，生成了一个 id_rsa.pub文件

[hadoop@master .ssh]$ ls -al /home/hadoop/.ssh
total 16
drwx------. 2 hadoop hadoop 4096 Jul 30 22:09 .
drwx------. 3 hadoop hadoop 4096 Jul 30 22:09 ..
-rw-------. 1 hadoop hadoop 1671 Jul 30 22:09 id_rsa
-rw-r--r--. 1 hadoop hadoop 402 Jul 30 22:09 id_rsa.pub

[hadoop@master .ssh]$ cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAugxDdYElWX8URmERVTPsNtSW2mZoEMZyzmIKEe3UQjMbgYOCx0jV15/vtnOXJf4K+S6hccajih2oemcC4BmMI99nFWcyd9zcrvfJvn/DZHnw0Yog6MyMd9Qw2BQWUL265dhW2FNCaECuyG2U1CxR0W9wZlVz54JlToCRX6yUvWZzzGqUw/oR3ZWE7pUpQiOHV0ZNypUtbWEw/zj7N01LbVskNLqyY164APIVfZBONPMXjs/H6B8/VcxsA0lDwAqndnMxJ1iQHFkMntNgqcLkhs8oAmvl+/A6NH2I0ZBW+VOCUIjbNKNRO9BKwGvUQUZGcKthEPU0jK5ErsS6RPBrmQ== [email protected]

把这个文件的内容，追加到 authorized_keys 的文件里

[hadoop@master .ssh]$ cat id_rsa.pub >> authorized_keys
[hadoop@master .ssh]$ ls
authorized_keys id_rsa id_rsa.pub
[hadoop@master .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAugxDdYElWX8URmERVTPsNtSW2mZoEMZyzmIKEe3UQjMbgYOCx0jV15/vtnOXJf4K+S6hccajih2oemcC4BmMI99nFWcyd9zcrvfJvn/DZHnw0Yog6MyMd9Qw2BQWUL265dhW2FNCaECuyG2U1CxR0W9wZlVz54JlToCRX6yUvWZzzGqUw/oR3ZWE7pUpQiOHV0ZNypUtbWEw/zj7N01LbVskNLqyY164APIVfZBONPMXjs/H6B8/VcxsA0lDwAqndnMxJ1iQHFkMntNgqcLkhs8oAmvl+/A6NH2I0ZBW+VOCUIjbNKNRO9BKwGvUQUZGcKthEPU0jK5ErsS6RPBrmQ== [email protected]

此时需要修改authorized_keys的文件权限，不安全的设置安全设置，会让你不能使用RSA功能

[hadoop@master .ssh]$ ls -al
total 20
drwx------. 2 hadoop hadoop 4096 Jul 30 22:20 .
drwx------. 3 hadoop hadoop 4096 Jul 30 22:09 ..
-rw-rw-r--. 1 hadoop hadoop 402 Jul 30 22:20 authorized_keys
-rw-------. 1 hadoop hadoop 1671 Jul 30 22:09 id_rsa
-rw-r--r--. 1 hadoop hadoop 402 Jul 30 22:09 id_rsa.pub
[hadoop@master .ssh]$ sudo chmod 600 authorized_keys
[hadoop@master .ssh]$ ls -al
total 20
drwx------. 2 hadoop hadoop 4096 Jul 30 22:20 .
drwx------. 3 hadoop hadoop 4096 Jul 30 22:09 ..
-rw-------. 1 hadoop hadoop 402 Jul 30 22:20 authorized_keys
-rw-------. 1 hadoop hadoop 1671 Jul 30 22:09 id_rsa
-rw-r--r--. 1 hadoop hadoop 402 Jul 30 22:09 id_rsa.pub

修改 /etc/ssh/sshd_config 并且确认其中三个选项然后sudo service sshd restart 服务

[hadoop@master .ssh]$ sudo cat /etc/ssh/sshd_config
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

在所有Slave上建立 .ssh文件夹,改变权限

[hadoop@slave02 ~]$ mkdir .ssh
[hadoop@slave02 ~]$ ls -al
total 32
drwx------. 3 hadoop hadoop 4096 Jul 30 22:47 .
drwxr-xr-x. 4 root root 4096 Jul 30 20:10 ..
-rw-------. 1 hadoop hadoop 401 Jul 30 20:58 .bash_history
-rw-r--r--. 1 hadoop hadoop 18 Jul 18 2013 .bash_logout
-rw-r--r--. 1 hadoop hadoop 176 Jul 18 2013 .bash_profile
-rw-r--r--. 1 hadoop hadoop 124 Jul 18 2013 .bashrc
drwxrwxr-x. 2 hadoop hadoop 4096 Jul 30 22:47 .ssh
-rw-------. 1 hadoop hadoop 557 Jul 30 20:28 .viminfo

[hadoop@slave02 ~]$ sudo chmod 700 .ssh
[hadoop@slave02 ~]$ ls -al
total 32
drwx------. 3 hadoop hadoop 4096 Jul 30 22:47 .
drwxr-xr-x. 4 root root 4096 Jul 30 20:10 ..
-rw-------. 1 hadoop hadoop 401 Jul 30 20:58 .bash_history
-rw-r--r--. 1 hadoop hadoop 18 Jul 18 2013 .bash_logout
-rw-r--r--. 1 hadoop hadoop 176 Jul 18 2013 .bash_profile
-rw-r--r--. 1 hadoop hadoop 124 Jul 18 2013 .bashrc
drwx------. 2 hadoop hadoop 4096 Jul 30 22:47 .ssh
-rw-------. 1 hadoop hadoop 557 Jul 30 20:28 .viminfo
[hadoop@slave02 ~]$

把id_rsa.pub发送给slaves

[hadoop@master .ssh]$ scp id_rsa.pub [email protected]:/home/hadoop/.ssh/id_rsa_frommaster.pub
The authenticity of host ‘slave01.hadoop (10.15.5.201)‘ can‘t be established.
RSA key fingerprint is 76:14:2f:f9:d9:03:07:17:7c:d1:ad:1e:af:55:45:00.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘slave01.hadoop,10.15.5.201‘ (RSA) to the list of known hosts.
[email protected]‘s password:
id_rsa.pub 100% 402 0.4KB/s 00:00
[hadoop@master .ssh]$ scp id_rsa.pub [email protected]:/home/hadoop/.ssh/id_rsa_frommaster.pub
The authenticity of host ‘slave02.hadoop (10.15.5.202)‘ can‘t be established.
RSA key fingerprint is 76:14:2f:f9:d9:03:07:17:7c:d1:ad:1e:af:55:45:00.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘slave02.hadoop,10.15.5.202‘ (RSA) to the list of known hosts.
[email protected]‘s password:
id_rsa.pub 100% 402 0.4KB/s 00:00

把这个值追加到slaves上的authorized_keys里面,并修改安全权限

[hadoop@slave02 .ssh]$ cat id_rsa_frommaster.pub >> authorized_keys
[hadoop@slave02 .ssh]$ sudo chmod 600 authorized_keys

修改slaves主机上的 /etc/ssh/sshd_config内容，并重启sshd服务

[hadoop@slave01 .ssh]$ sudo cat /etc/ssh/sshd_config
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

[hadoop@slave01 .ssh]$ sudo service sshd restart
Stopping sshd: [ OK ]
Starting sshd: [ OK ]

最终验证方法

[hadoop@master .ssh]$ ssh slave01.hadoop

Last login: Wed Jul 30 23:00:25 2014 from master.hadoop

[hadoop@slave01 ~]$ logout

Connection to slave01.hadoop closed.

[hadoop@master .ssh]$ ssh slave02.hadoop

[hadoop@slave02 ~]$ logout

Connection to slave02.hadoop closed.

同理：从slaves 到master

slaves 生成公钥
把slaves把Slave的公钥追加到Master的".ssh"文件夹下的"authorized_keys"中

[hadoop@slave01 .ssh]$ cat id_rsa.pub

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqkRUrgiyD3W36WPRpL9ifJjgnVe+1R4x4mRhimKXOfmW+D3dcbi9FwE2j0/H+NaMdLWjQYw685ITdfHni0X5La7YXy6EIe0FQB/NxDKVslc44RuzJQJKQkosqSY/HGVfefF7OzIFCEcqVQDqN+OpWlrbXNtmO1UUhG2TFvj3MsbgwWHf7fc+uSN7Y7bMzLJpKhAVEjljjyTaHekJ8WMNzgT160vgr0mIZawDlRxKRlS2HtWQnndf74ZJDCeQKgJA6RUKuqBLop5x/0lvHBbUkn3MADnJUlIzd5pJW/aFEVySypfkGZTspj3+m8gNfQkqYJiB6sBfFgUYt1iPL/GNRw== [email protected]

[hadoop@slave02 ~]$ cat .ssh/id_rsa.pub

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3mHGmqOGjIkC/Elc/4CopgvKCQ7AdtqV87dhQPWnDdlljePVY/GbCWYqp4h8CmtmsMNreTva8RLBPSIP3jGMrbvNDCCuMo+mZGa034yR6JwekI9zVikzYscCTqWCk6w5Hs3u/pnb1Ym6a46ho+DNEm42qIaYzrhRn9Fc1F9Hd3/dXRq0KZlO/5xMKuhFt1GKy+gs+l7MiK6y7pTISx+ox/mDjDQfzxfpguRO68XX54+dD0gONSb/MaVVMdqiWK3Fh88ouN23sKi/cnzD1vLSm55kpT6ZRo792qcFmMe7CIYc6dwdkKZBgkdheayWy5PPfsZrGfDd/erVCWrmofDhiw== [email protected]

可以用cat >>的方式追加，或者vi直接写入。钥匙么，让master知道就行了.