为Linux集群创建新账户，并配置hadoop集群

浏览数：40 / 时间：2015年06月20日

转自:http://blog.csdn.net/bluesky8640/article/details/6945776

之前装python、jdk、hadoop都是用的root账户，这是一个绝对的失策，linux对用户访问权限很严格，新创的hod账户根本无法启动hadoop，而root在hod模式下使用torque是不被建议的，所以只能用hod账户再重新装一遍.

创建用户、设置密码、修改用户、删除用户：

useradd testuser   创建用户testuser
passwd testuser   给已创建的用户testuser设置密码
说明：新创建的用户会在/home下创建一个用户目录testuser
usermod --help   修改用户这个命令的相关参数
userdel testuser  删除用户testuser
rm -rf testuser   删除用户testuser所在目录

上面的几个命令只有root账号才可以使用，如果你不知道自己的系统上面的命令在什么位置可以使用如下命令查找其路径：locate useradd

在root账户下为服务器集群(x101~x156)创建新账户hod.

addhoduser.exp

    #!/usr/bin/expect -f  
    set password 123456  
    for {set i 1} {$i<57} {incr i} {  
        if {$i<10} {  
        set ip x10$i  
        puts "it is the first if : $ip"  
        } else {  
        set ip x1$i  
        puts "it is the first else : $ip"  
        }  
      
        puts "$ip begins!"  
      
        spawn ssh $ip  
        expect "#"  
        send "useradd hod\r"  
        expect "#"  
        send "passwd hod\r"  
        expect "*New UNIX password*"  
        send "$password\r"  
        expect "*Retype new UNIX password*"  
        send "$password\r"  
        expect "#"  
        send "exit\r"  
        expect eof  
      
        puts "$ip end!"  
    }  
    close

在hod账户下进行如下的设置：

在每个服务器的hod目录下创建.ssh目录：（错！这部操作被证明是极大的错误，不用自己创建.ssh目录，使用“ssh-keygen -t rsa”会自动创建.ssh，自己创建.ssh目录会由于文件夹权限问题使得ssh无密码访问失效）

mkdirssh.exp(不需要使用)

    #!/usr/bin/expect -f  
    set password 123456  
    for {set i 1} {$i<57} {incr i} {  
        if {$i<10} {  
        set ip x10$i  
        puts "it is the first if : $ip"  
        } else {  
        set ip x1$i  
        puts "it is the first else : $ip"  
        }  
      
        spawn ssh $ip  
        expect "*password:*"  
        send "$password\r"  
        expect eof  
          
        expect "#"  
        send "mkdir /home/hod/.ssh\r"  
        expect "#"  
        send "exit\r"  
        expect eof  
    }  
    close

在x101服务器的/home/hod/.ssh目录下创建authorized_keys文件，并设置为644权限，这很关键.(这部操作也就不必了，后面ssh-copy-id -i ~/.ssh/id_rsa.pub hod@x101会自动创建authorized_keys文件，且mod为600，这个权限也是可以的)

SSH无密码登录steps:
为了让在A上可以无密码SSH到B上
首先在A上执行ssh-keygen -t rsa
一路回车
将公钥导入到authorized_keys文件中
cat ~/.ssh/rsa_id.pub >> authorized_keys
修改文件和文件夹权限
chmod 600 authorized_keys
chmod 700 ~/.ssh
将authorized_keys分发到远程主机
scp ~/.ssh/authorized_keys [email protected]:/home/meteor/.ssh

现在就可以无密码登录了。

ssh [email protected]

为x101~x156配置ssh无密码登录：

wumimadenglu.exp

    #!/usr/bin/expect -f  
    set password 123456  
    for {set i 1} {$i<57} {incr i} {  
        if {$i<10} {  
        set ip x10$i  
        puts "it is the first if : $ip"  
        } else {  
        set ip x1$i  
        puts "it is the first else : $ip"  
        }  
          
        spawn ssh $ip  
        expect "*password:*"  
        send "$password\r"  
        expect eof  
      
        expect "#"  
        send "ssh-keygen -t rsa\r"  
        expect "*save the key*"  
        send "\r"  
        expect "*Enter passphrase*"  
        send "\r"  
        expect "*Enter same passphrase again*"  
        send "\r"  
        expect eof  
      
        expect "#"   
        send "ssh-copy-id -i ~/.ssh/id_rsa.pub hod@x101\r"  
        expect "*password:*"  
        send "$password\r"  
        expect "#"  
        send "exit\r"  
        expect eof  
    }  
    close

最后只要将x101下的authorized_keys文件拷贝到x102~x156的.ssh目录即可.这方法会比之前为root账户配置ssh无密码登录快很多很多.

copyauthorized_keystoall.exp

#!/usr/bin/expect -f  
set password 123456  
for {set i 2} {$i<57} {incr i} {  
    if {$i<10} {  
    set ip x10$i  
    puts "it is the first if : $ip"  
    } else {  
    set ip x1$i  
    puts "it is the first else : $ip"  
    }  
     
    spawn scp /home/hod/.ssh/authorized_keys hod@$ip:/home/hod/.ssh/  
    expect "*password:*"  
    send "$password\r"  
    expect eof  
}  
close

这里需要说明一点，在进行scp的时候尽量使用/home/hod/.ssh/authorized_keys这样的绝对目录，使用~/.ssh/authorized_keys可能会由于找不到目录而出错

后续操作：

配置好ssh无密码登录之后，我在hod账户目录下安装了python和jdk，方法都是先拷贝安装包或压缩包再解压安装

copypythontoall.sh

    for((i=2;i<57;i++))  
    do  
       if [ $i -lt 10 ]  
       then  
          scp /home/hod/Python-2.5.1.tgz hod@x10$i:/home/hod/  
          echo "it is the first if: x10$i"  
       else  
          scp /home/hod/Python-2.5.1.tgz hod@x1$i:/home/hod/  
          echo "it is the first else: x1$i"  
       fi  
    done

installpythontoall.exp

#!/usr/bin/expect -f  
for {set i 2} {$i<57} {incr i} {  
    if {$i<10} {  
    set ip x10$i  
    puts "it is the first if : $ip"  
    } else {  
    set ip x1$i  
    puts "it is the first else : $ip"  
    }  
     
    spawn ssh $ip  
    expect "#"  
    send "tar zxvf /home/hod/Python-2.5.1.tgz\r"  
    expect "#"  
    send "cd /home/hod/Python-2.5.1\r"  
    expect "#"  
    send "./configure\r"  
    expect "#"  
    send "make\r"  
    expect "#"  
    send "sudo make install\r"  
    expect "*Password:*"  
    send "123456\r"  
    expect "#"  
    send "exit\r"  
    expect eof  
      
    puts "$ip completed"  
    sleep 3  
}  
close

copyjdktoall.sh

    for((i=2;i<57;i++))  
    do  
       if [ $i -lt 10 ]  
       then  
          scp /home/hod/jdk-7u1-linux-i586.tar.gz hod@x10$i:/home/hod/  
          echo "it is the first if: x10$i"  
       else  
          scp /home/hod/jdk-7u1-linux-i586.tar.gz hod@x1$i:/home/hod/  
          echo "it is the first else: x1$i"  
       fi  
    done

installjdktoall.exp

    #!/usr/bin/expect -f  
    for {set i 2} {$i<57} {incr i} {  
        if {$i<10} {  
        set ip x10$i  
        puts "it is the first if : $ip"  
        } else {  
        set ip x1$i  
        puts "it is the first else : $ip"  
        }  
         
        spawn ssh $ip  
        expect "#"  
        send "tar zxvf /home/hod/jdk-7u1-linux-i586.tar.gz\r"  
        expect "#"  
        send "exit\r"  
        expect eof  
      
        puts "$ip completed"  
        sleep 3  
    }  
    close

安装hadoop：

installhadooptoall.exp

    #!/usr/bin/expect -f  
    set password 123456  
    for {set i 2} {$i<57} {incr i} {  
        if {$i<10} {  
        set ip x10$i  
        puts "it is the first if : $ip"  
        } else {  
        set ip x1$i  
        puts "it is the first else : $ip"  
        }  
          
        spawn scp /home/hod/hadoop-0.20.2.tar.gz hod@$ip:/home/hod/  
          
        spawn ssh $ip  
        expect "#"  
        send "mkdir /home/hod/hadoop\r"  
        expect "#"  
        send "cp /home/hod/hadoop-0.20.2.tar.gz /home/hod/hadoop/\r"  
        expect "#"  
        send "tar zxvf /home/hod/hadoop/hadoop-0.20.2.tar.gz\r"  
        expect "#"  
        send "exit\r"  
        expect eof  
          
        puts "$ip completed"  
        sleep 3  
    }  
    close

tar zxvf /home/hod/hadoop/hadoop-0.20.2.tar.gz会有问题，该操作会把hadoop压缩包解压在/home/hod目录下，应该在这之前先执行cd /home/hod/hadoop操作

在x101按之前root的操作配置好参数：

copypeizhitoall.sh

    for((i=2;i<57;i++))  
    do  
       if [ $i -lt 10 ]  
       then  
          scp -r /home/hod/hadoop/hadoop-0.20.2/conf hod@x10$i:/home/hod/hadoop/hadoop-0.20.2/  
          scp /home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf/hodrc hod@x10$i:/home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf/  
          echo "it is the first if: x10$i"  
       else  
          scp -r /home/hod/hadoop/hadoop-0.20.2/conf hod@x1$i:/home/hod/hadoop/hadoop-0.20.2/  
          scp /home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf/hodrc hod@x1$i:/home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf/  
          echo "it is the first else: x1$i"  
       fi  
    done

最后用root用户在/etc/profile文件末尾加上下面语句，并把/etc/profile拷贝覆盖x102~x156

export JAVA_HOME=/home/hod/jdk1.7.0_01  
export PATH=$JAVA_HOME/bin:$PATH  
export CLASSPATH=.:$JAVA_HOME/lib:$CLASSPATH  
export HADOOP_HOME=/home/hod/hadoop/hadoop-0.20.2  
export PATH=$HADOOP_HOME/bin:$PATH  
export CLUSTER_NAME=clus  
export RM_QUEUE=batch  
export RM_HOME=/usr/local  
export PYTHON_HOME=/home/hod/Python-2.5.1  
export PATH=$PYTHON_HOME:$PATH  
export HOD_PYTHON_HOME=/home/hod/Python-2.5.1/python  
export HOD_CONF_DIR=/home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf  
export HOD_HOME=/home/hod/hadoop/hadoop-0.20.2/contrib/hod  
export PATH=$HOD_HOME/bin:$PATH