ubuntu安装hadoop1.x(伪分布式)

参考 kinglau - 二、Ubuntu14.04下安装Hadoop2.4.0 (伪分布模式)

接上篇 ubuntu安装hadoop(单机)

1. 配置core-site.xml

# hadoop1.x.x 的配置文件都在 $HADOOP_HOME/conf/下,hadoop2.x.x 的配置文件都在 $HADOOP_HOME/etc/hadoop/下
# 本文将都已hadoop1.x.x为例,hadoop2.x.x配置对应文件
$ sudo gedit /usr/local/hadoop/conf/core-site.xml # add   <configuration>   <properties>   <name>fs.default.name</name>   <value>hdfs://localhost:9000</value>   </properties>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop/tmp</value>
        <description>临时目录,其他临时目录的父目录</description>
        <final>true</final>
    </property>
  </configuration>

2. 配置 hdfs-site.xml

<configuration>
    <property>
        <name>dfs.data.dir</name>
        <value>/usr/local/hadoop/tmp/dfs/data</value>
        <description>data node的数据目录</description>
        <final>true</final>
    </property>
    <property>
        <name>dfs.name.dir</name>
        <value>/usr/local/hadoop/tmp/dfs/name</value>
        <description>name node的元数据</description>
        <final>true</final>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
        <description>文件复制的副本数</description>
    </property>
</configuration>

3. 格式化namenode

$ /usr/local/hadoop/bin/hadoop namenode -format
# when be asked "Re-format filesystem in /usr/local/hadoop/tmp/dfs/namenode ? (Y or N)" input Y
# when you see something like "Storage directory /usr/local/hadoop-1.2.1/tmp/dfs/name has been successfully formatted." ,you succeed.

4. 启动hadoop

$ /usr/local/hadoop/bin/start-dfs.sh
# starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-namenode-erbin-ubuntu.out
# localhost: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-datanode-erbin-ubuntu.out
# localhost: starting secondarynamenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-erbin-ubuntu.out
$ jps -m
# 14841 DataNode
# 14983 SecondaryNameNode
# 15073 Jps -m
# 14701 NameNode

5. WordCount测试

$ cd /usr/local/hadoop
## 在hdfs创建文件夹
$ bin/hadoop dfs -mkdir /user
$ bin/hadoop dfs -ls /
#drwxr-xr-x   - hadoop supergroup          0 2015-04-24 15:08 /user
$ bin/hadoop dfs -mkdir /user/hadoop
## hadoop会默认将/user/hadoop作为home
$ bin/hadoop dfs -mkdir input
$ bin/hadoop dfs -ls 
#Found 1 items
#drwxr-xr-x   - hadoop supergroup          0 2015-04-24 15:12 /user/hadoop/input
## 上传本地文件值hdfs
## hadoop dfs -put [local] [hdfs]
$ bin/hadoop dfs -put input input
$ bin/hadoop dfs -ls input
#Found 7 items
#-rw-r--r--   1 hadoop supergroup       7457 2015-04-24 15:12 /user/hadoop/input/capacity-scheduler.xml
#-rw-r--r--   1 hadoop supergroup        178 2015-04-24 15:12 /user/hadoop/input/core-site.xml
#-rw-r--r--   1 hadoop supergroup        327 2015-04-24 15:12 /user/hadoop/input/fair-scheduler.xml
#-rw-r--r--   1 hadoop supergroup       4644 2015-04-24 15:12 /user/hadoop/input/hadoop-policy.xml
#-rw-r--r--   1 hadoop supergroup        178 2015-04-24 15:12 /user/hadoop/input/hdfs-site.xml
#-rw-r--r--   1 hadoop supergroup       2033 2015-04-24 15:12 /user/hadoop/input/mapred-queue-acls.xml
#-rw-r--r--   1 hadoop supergroup        178 2015-04-24 15:12 /user/hadoop/input/mapred-site.xml
$ bin/hadoop jar hadoop-examples-1.x.x.jar wordcount input output 
$ bin/hadoop dfs -ls output
#Found 2 items
#-rw-r--r--   1 hadoop supergroup          0 2015-04-24 15:25 /user/hadoop/output/_SUCCESS
#-rw-r--r--   1 hadoop supergroup       6289 2015-04-24 15:25 /user/hadoop/output/part-r-00000
$ bin/hadoop dfs -cat output/*

6. 关闭hadoop

$ /usr/local/hadoop-1.2.1/bin/stop-dfs.sh 
# stopping namenode
# localhost: stopping datanode
# localhost: stopping secondarynamenode
$ jps -m
# 15317 Jps -m

7. 备注

有些文章,在配置core-site.xml,hdfs-site.xml时,里面的文件路径前总是会添加‘ file:/ ‘或‘ file:/// ‘.这种写法在2.x是可以的,但是在1.x是不可以的

测试过发现不加file前缀的路径写法,在1.x,2.x都是可以通过的。

如果dfs.name.dir和dfs.data.dir文件夹是实现创建的。dfs.data.dir的文件权限可能会报错,如果报错,就按错误提示更改一下文件夹的权限即可。

 

郑重声明:本站内容如果来自互联网及其他传播媒体,其版权均属原媒体及文章作者所有。转载目的在于传递更多信息及用于网络分享,并不代表本站赞同其观点和对其真实性负责,也不构成任何其他建议。