使用shell脚本实现自动化部署hadoop集群

    有四台Linux服务器，其中一台为主，其它为从。服务器使用的是CentOS6.5，jdk选择1.6，hadoop选择1.0.4版本。要到实际环境中应用还要根据实际情况做修改。

    如果是新装好的系统，要先配置好ip，shell脚本如下：

1. #!bin/bash

2. read "input ip:" ip

3. echo ‘the default hostname is master‘

4. sed -i ‘$aIPADDR=‘$ip /etc/sysconfig/network-scripts/ifcfg-eth0 

5. sed -i ‘/BOOTPROTO/cBOOTPROTO="no"‘ /etc/sysconfig/network-scripts/ifcfg-eth0 

6. sed -i ‘/IPV6INIT/cIPV6INIT="no"‘ /etc/sysconfig/network-scripts/ifcfg-eth0 

7. sed -i ‘/NM_CONTROLLED/cNM_CONTROLLED="no"‘ /etc/sysconfig/network-scripts/ifcfg-eth0 

8. service network restart

9. chkconfig network on

10. service iptables stop

11. chkconfig iptables off

12. setenforce 0

13. hostname master

14. sed -i ‘/HOSTNAME/cHOSTNAME=master‘ /etc/sysconfig/network

    集群的ip配置如下，在实际环境中可以自己作调整。首先检查通信，通信成功才能下一步进行，否则安装终止。如果通信成功，才会执行接下来的步骤，则修改/etc/hosts文件和ssh_conf，然后重启ssh服务。密码要根据实际环境修改。

1. masterip=192.168.2.254

2. slave1ip=192.168.2.11

3. slave2ip=192.168.2.2

4. slave3ip=192.168.2.3

5. service iptables stop

6. chkconfig iptables off

7. if [ -e ip.txt ];then

8. rm -rf ip.txt

9. fi

10. touch ip.txt

11. echo $masterip >>ip.txt

12. echo $slave1ip >>ip.txt

13. echo $slave2ip >>ip.txt

14. echo $slave3ip >>ip.txt

15. NEWPASS=123456

16. NETWORK=TRUE

17. echo "before you install,please make sure network is ok!!!"

18. echo "now  test the network"

19. for ip in $(cat ip.txt)

20. do

21. ping $ip -c 5 &>/dev/null 

22. if [ $? == 0 ] ;then 

23. echo "${ip} is ok"

24. else 

25. echo "${ip} conn‘t connected" 

26. NETWORK=FALSE

27. fi

28. done

29. echo $NETWORK

30. if [ $NETWORK != FALSE   ];then

31. .........

32. fi

为了方便说明，笔者在此处做了步骤拆分.

    使用root用户实现master的root用户能够免秘钥ssh登陆其他主机。这样root用户就可以很方便的管理其他主机了。

1. PASS=123456

2. yum -y install tcl --nogpgcheck

3. yum -y install expect --nogpgcheck

4. expect <<EOF 

5.         spawn ssh-keygen

6.         expect "Enter file in which to save the key (/root/.ssh/id_rsa):" 

7.         send "\r"

8. expect "Enter passphrase (empty for no passphrase):"

9. send "\r"

10. expect "Enter same passphrase again:"

11. send "\r"

12. expect eof

13. EOF

14. for ip in $(cat ip.txt)

15. do

16. expect <<EOF

17.         spawn ssh-copy-id root@${ip} 

18. expect "(yes/no)?"

19.         send "yes\r"

20.         expect "password:"

21.         send "${PASS}\r"

22. expect eof

23. EOF

24. done

25. expect <<EOF

26.         spawn ssh-copy-id hadoop@master

27. expect "(yes/no)?"

28.         send "yes\r"

29.         expect "password:"

30.         send "${PASS}\r"

31. expect eof

32. EOF

   完成root无秘钥登陆其他主机后，各个主机添加hadoop用户，并修改/etc/hosts文件，主机上修改sshd的配置文件并重新启动服务。 

1. for ip in $(cat ip.txt)

2. do

3. ssh root@$ip "useradd hadoop"

4. ssh root@$ip "echo ‘123456‘ | passwd --stdin hadoop"

5. ssh root@$ip "echo $masterip‘  master‘ >>/etc/hosts"

6. ssh root@$ip "echo $slave1ip‘   slave1‘ >>/etc/hosts"

7. ssh root@$ip "echo $slave2ip‘   slave1‘ >>/etc/hosts"

8. ssh root@$ip "echo $slave3ip‘   slave1‘ >>/etc/hosts"

9. done

10. cp ip.txt /home/hadoop

11. cp hadoopsshconf.sh /home/hadoop

12. chown hadoop:hadoop /home/hadoop/ip.txt

13. chown hadoop:hadoop /home/hadoop/hadoopsshconf.sh

14. 
    

15. ssh hadoop@localhost "sh hadoopsshconf.sh"

16. 
    

17. sed -i ‘/#RSAAuthentication yes/cRSAAuthentication yes‘  /etc/ssh/sshd_config

18. sed -i ‘/#PubkeyAuthentication yes/cPubkeyAuthentication yes‘ /etc/ssh/sshd_config

19. sed -i ‘/#AuthorizedKeysFile     .ssh/authorized_keys/AuthorizedKeysFile     .ssh/authorized_keysc‘ /etc/ssh/sshd_config

20. service sshd restart

21. chkconfig sshd on

    只有root用户能够免秘钥登陆其他主机是不够的，还要本机的hadoop用户能够免密钥登陆其他hadoop用户，上面的脚本中已经将脚本hadoopsshconf.sh拷贝到hadoop用户下方，使用ssh远程命令以hadoop用户执行即可。hadoopsshconf.sh如下：

1. #!/bin/bash

2. PASS=123456

3. expect <<EOF 

4.         spawn ssh-keygen

5.         expect "Enter file in which to save the key (/root/.ssh/id_rsa):" 

6.         send "\r"

7. expect "Enter passphrase (empty for no passphrase):"

8. send "\r"

9. expect "Enter same passphrase again:"

10. send "\r"

11. expect eof

12. EOF

13. cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys

14. chmod 600  ~/.ssh/authorized_keys 

15. for ip in $(cat ip.txt)

16. do

17. expect <<EOF

18.         spawn ssh-copy-id hadoop@${ip} 

19. expect "(yes/no)?"

20.         send "yes\r"

21.         expect "password:"

22.         send "${PASS}\r"

23. expect eof

24. EOF

25. done

    安装jdk1.6脚本，首先清除掉系统上已经安装的jdk，避免版本冲突。

1. #!/bin/bash

2. rm -rf tmp.txt

3. rpm -qa | grep java* > tmp.txt

4. line=$(cat tmp.txt)

5. for i in $line

6. do

7. rpm -e $i --nodeps

8. done

9. mkdir -p /usr/java

10. cp ./jdk-6u32-linux-x64.bin /usr/java/

11. cd /usr/java/

12. ./jdk-6u32-linux-x64.bin 

13. rm -rf jdk-6u32-linux-x64.bin

14. echo ‘export JAVA_HOME=/usr/java/jdk1.6.0_32‘ >>/etc/profile

15. echo ‘export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib‘ >>/etc/profile

16. echo ‘export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin‘ >>/etc/profile

17. source /etc/profile

18. java -version

19. cd -

接下来安装hadoop1.4.0,安装在/usr下边。脚本内容包括/etc/profile的修改，hadoop配置文件，hadop-env.sh,conre-site.xml,hdfs-site.xml,mapred-site.xml，masters和slaves的配置文件修改,兵拷贝到其他机子上去。

1. mkdir -p /usr/hadoop

2. tar -zxf hadoop-1.0.4.tar.gz -C /usr/hadoop

3. mv /usr/hadoop/hadoop-1.0.4 /usr/hadoop/hadoop1

4. chown -R hadoop:hadoop /usr/hadoop

5. echo ‘export HADOOP_HOME=/usr/hadoop/hadoop1‘ >> /etc/profile

6. echo ‘export PATH=$PATH:$HADOOP_HOME/bin‘ >>/etc/profile

7. source /etc/profile

10. echo ‘export JAVA_HOME=/usr/java/jdk1.6.0_32‘ >> /usr/hadoop/hadoop1/conf/hadoop-env.sh

11. echo ‘export HADOOP_PID_DIR=/usr/hadoop/hadoop1/pids‘ >>/usr/hadoop/hadoop1/conf/hadoop-env.sh

12. 
    
    

13. sed -i ‘6a\\t</property>‘ /usr/hadoop/hadoop1/conf/core-site.xml 

14. sed -i ‘6a\\t<value>hdfs://master:9000</value>‘ /usr/hadoop/hadoop1/conf/core-site.xml 

15. sed -i ‘6a\\t<name>fs.default.name</name>‘ /usr/hadoop/hadoop1/conf/core-site.xml 

16. sed -i ‘6a\\t<property>‘ /usr/hadoop/hadoop1/conf/core-site.xml

17. 
    

18. 
    

19. sed -i ‘6a\\t</property>‘ /usr/hadoop/hadoop1/conf/hdfs-site.xml

20. sed -i ‘6a\\t<value>false</value>‘ /usr/hadoop/hadoop1/conf/hdfs-site.xml

21. sed -i ‘6a\\t<name>dfs.permissions</name>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

22. sed -i ‘6a\\t<property>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

23. sed -i ‘6a\\t</property>‘ /usr/hadoop/hadoop1/conf/hdfs-site.xml

24. sed -i ‘6a\\t<value>/usr/hadoop/hadoop1/tmp/</value>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

25. sed -i ‘6a\\t<name>hadoop.tmp.dir</name>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

26. sed -i ‘6a\\t<property>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

27. sed -i ‘6a\\t</property>‘ /usr/hadoop/hadoop1/conf/hdfs-site.xml

28. sed -i ‘6a\\t<value>/usr/hadoop/hadoop1/data/</value>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

29. sed -i ‘6a\\t<name>dfs.data.dir</name>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

30. sed -i ‘6a\\t<property>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

31. sed -i ‘6a\\t</property>‘ /usr/hadoop/hadoop1/conf/hdfs-site.xml

32. sed -i ‘6a\\t<value>/usr/hadoop/hadoop1/namenode/</value>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

33. sed -i ‘6a\\t<name>dfs.name.dir</name>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

34. sed -i ‘6a\\t<property>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

35. sed -i ‘6a\\t</property>‘ /usr/hadoop/hadoop1/conf/hdfs-site.xml

36. sed -i ‘6a\\t<value>3</value>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

37. sed -i ‘6a\\t<name>dfs.replication</name>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

38. sed -i ‘6a\\t<property>‘     /usr/hadoop/hadoop1/conf/hdfs-site.xml

39. 
    

40. 
    

41. sed -i ‘6a\\t</property>‘ /usr/hadoop/hadoop1/conf/mapred-site.xml

42. sed -i ‘6a\\t<value>2</value>‘     /usr/hadoop/hadoop1/conf/mapred-site.xml

43. sed -i ‘6a\\t<name>mapred.tasktracker.reduce.tasks.maximum</name>‘     /usr/hadoop/hadoop1/conf/mapred-site.xml

44. sed -i ‘6a\\t<property>‘     /usr/hadoop/hadoop1/conf/mapred-site.xml

45. sed -i ‘6a\\t</property>‘ /usr/hadoop/hadoop1/conf/mapred-site.xml

46. sed -i ‘6a\\t<value>2</value>‘     /usr/hadoop/hadoop1/conf/mapred-site.xml

47. sed -i ‘6a\\t<name>mapred.tasktracker.map.tasks.maximum</name>‘     /usr/hadoop/hadoop1/conf/mapred-site.xml

48. sed -i ‘6a\\t<property>‘     /usr/hadoop/hadoop1/conf/mapred-site.xml

49. sed -i ‘6a\\t</property>‘ /usr/hadoop/hadoop1/conf/mapred-site.xml

50. sed -i ‘6a\\t<value>master:9001</value>‘     /usr/hadoop/hadoop1/conf/mapred-site.xml

51. sed -i ‘6a\\t<name>mapred.job.tracker</name>‘     /usr/hadoop/hadoop1/conf/mapred-site.xml

52. sed -i ‘6a\\t<property>‘     /usr/hadoop/hadoop1/conf/mapred-site.xml

53. 
    

54. 
    

55. 
    

56. echo ‘master‘ >> /usr/hadoop/hadoop1/conf/masters

57. 
    

58. 
    

59. sed -i ‘1d‘ /usr/hadoop/hadoop1/conf/slaves

60. echo ‘slave1‘ >> /usr/hadoop/hadoop1/conf/slaves

61. echo ‘slave2‘ >> /usr/hadoop/hadoop1/conf/slaves

62. echo ‘slave3‘ >> /usr/hadoop/hadoop1/conf/slaves

63. 
    

64. 
    

65. 
    

66. mkdir -p  /usr/hadoop/hadoop1/data/

67. mkdir -p  /usr/hadoop/hadoop1/tmp/

68. chown -R hadoop:hadoop  /usr/hadoop/

69. chmod -R 755 /usr/hadoop/hadoop1/data/

70. chmod -R 755 /usr/hadoop/hadoop1/tmp/

71. for i in $(seq 3)

72. do

73. ssh slave$i "mkdir -p /usr/hadoop"

74. done

75. scp -r /usr/hadoop/hadoop1 root@slave1:/usr/hadoop

76. scp -r /usr/hadoop/hadoop1 root@slave2:/usr/hadoop

77. scp -r /usr/hadoop/hadoop1 root@slave3:/usr/hadoop

78. for i in $(seq 3)

79. do

80. ssh slave$i "chown -R hadoop:hadoop /usr/hadoop"

81. ssh slave$i "chmod -R 755 /usr/hadoop/hadoop1/data/"

82. ssh slave$i "chmod -R 755 /usr/hadoop/hadoop1/tmp/"

83. done

这样就完成了hadoop集群的自动化安装。然后就是格式化，启动并验证了