hadoop分布式配置

环境:ubuntu-12.04-server-amd64

3台 ubuntu-12.04-server-amd64 server

分别

192.168.122.101 server1      //主,运行NameNode+SecondaryNameNode+JobTracker

192.168.122.102 server2     //从1 运行DataNode+TaskTracker

192.168.122.103 server3   //从2 运行DataNode+TaskTracker

1、配置java环境(server1、server2、server3)

下载jdk-7u3-linux-x64.tar.gz

/opt/java/jdk/jdk1.7.0_03

配置一个软链接

ln -s /opt/java/jdk/jdk1.7.0_03 /opt/java/jdk/current

sudo vi /etc/profile

增加

export JAVA_HOME=/opt/java/jdk/current
export JRE_HOME=/opt/java/jdk/current/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

保存

source /etc/profile

sudo update-alternatives –install /usr/bin/java  java  /opt/java/jdk/current/bin/java 300;
sudo update-alternatives –install /usr/bin/javac javac /opt/java/jdk/current/bin/javac 300;

2、配置/etc/hosts(server1、server2、server3)

192.168.122.101 server1

192.168.122.102 server2

192.168.122.103 server3

3、安装ssh(server1、server2、server3)

sudo apt-get install ssh

4、ssh免登录(server1)

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

scp /home/ubuntu/.ssh/id_dsa.pub ubuntu@server2:/home/ubuntu

scp /home/ubuntu/.ssh/id_dsa.pub ubuntu@server3:/home/ubuntu

5、ssh免登录(server2、server3)

mkdir ~/.ssh

mv  ~/id_dsa.pub ~/.ssh

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

6、下载hadoop-1.0.1.tar.gz(server1、server2、server3)

解压缩到/opt/hadoop-1.0.1

vi /opt/hadoop-1.0.1/conf/hadoop-env.sh

打开注释并设置JAVA_HOME

export JAVA_HOME=/opt/java/jdk/current
 

vi /opt/hadoop-1.0.1/conf/core-site.xml

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://server1:9000</value>
</property>

<property>
    <name>hadoop.tmp.dir</name>
    <value>/opt/hadoop/tmp/</value>
</property>

</configuration>

vi /opt/hadoop-1.0.1/conf/hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>

<property>
    <name>dfs.name.dir</name>
    <value>/opt/hadoop/name/</value>
  </property>
  <property>
    <name>dfs.data.dir</name>
    <value>/opt/hadoop/data</value>
  </property>
</configuration>

vi /opt/hadoop-1.0.1/conf/mapred-site.xml:

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://server1:9001</value>
</property>
</configuration>

7、设置主从(server1)

vi /opt/hadoop-1.0.1/masters

server1

vi /opt/hadoop-1.0.1/slaves

server2

server3

8、启动hadoop

 

ubuntu@server1:~$ cd /opt/hadoop-1.0.1/bin/
ubuntu@server1:/opt/hadoop-1.0.1/bin$ ./hadoop namenode -format
 
 
 
 
ubuntu@server1:/opt/hadoop-1.0.1/bin$ ./start-all.sh 
 
starting namenode, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-namenode-server1.out
server2: starting datanode, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-datanode-server2.out
server3: starting datanode, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-datanode-server3.out
server1: starting secondarynamenode, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-secondarynamenode-server1.out
starting jobtracker, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-jobtracker-server1.out
server3: starting tasktracker, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-tasktracker-server3.out
server2: starting tasktracker, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-tasktracker-server2.out
 

9、查看启动情况(server1,server2,server3)

ubuntu@server1:/$ jps
1544 SecondaryNameNode
1749 Jps
1628 JobTracker
1273 NameNode
 

ubuntu@server2:~$ jps
1135 DataNode
1645 Jps
1327 TaskTracker
 

ubuntu@server3:~$ jps
1271 TaskTracker
1615 Jps
1084 DataNode
 

10、查看

查看任务调度

http://server1:50030/

查看name

http://server1:50070

原创文章,转载请注明: 转载自海波无痕

本文链接地址: hadoop分布式配置

文章的脚注信息由WordPress的wp-posturl插件自动生成

此条目发表在javaee, server分类目录,贴了, , , 标签。将固定链接加入收藏夹。

发表评论

电子邮件地址不会被公开。 必填项已用*标注

评论链接可以 移除 nofollow.