# HDFS

* Namenode（nn）
* Secondary Namenode（snn）
* Journalnode
* Datanode

# YARN

* Resourcemanager
* Nodemanager


# 计算模型

* MapReduce
* Spark


# 部署
## 说明
```bash
192.168.33.51   hdfs:NameNode,SecondaryNameNode     yarn:ResourceManager
192.168.33.52   hdfs:DataNode                       yarn:NodeManager
192.168.33.53   hdfs:DataNode                       yarn:NodeManager
192.168.33.54   hdfs:DataNode                       yarn:NodeManager

```
## Vagrantfile
```ruby
  [51,52,53,54].each do |i|
    config.vm.define "n#{i}" do |node|
      node.vm.network "private_network", ip: "192.168.33.#{i}"
      node.vm.synced_folder "/data/vagrant/shell", "/shell"
      node.vm.network :forwarded_port, guest: 22, host: "2#{i}22", host_ip: "0.0.0.0"
      node.vm.provider "virtualbox" do |vb|
        vb.memory = "2048"
        vb.cpus = 2
      end
      node.vm.provision "shell", inline: <<-SHELL
        echo "vagrant:vagrant" | sudo chpasswd
        mkdir -p /data
        chown -R vagrant:vagrant /data
        hostnamectl set-hostname n#{i}
      SHELL
    end
  end
```
## 安装
```bash
# 设置hosts
sudo vi /etc/hosts
192.168.33.51   n51
192.168.33.52   n52
192.168.33.53   n53
192.168.33.54   n54

# 设置系统时区
sudo timedatectl set-timezone Asia/Shanghai

# 下载解压缩
tar -xvf jdk-8u201-linux-x64.tar.gz
tar -xvf scala-2.12.8.tgz
tar -xvf apache-maven-3.6.0-bin.tar.gz
tar -xvf hadoop-3.2.0.tar.gz
tar -xvf spark-2.4.0-bin-hadoop2.7.tgz
tar -xvf apache-hive-3.1.1-bin.tar.gz
tar -xvf hbase-2.1.4-bin.tar.gz

# 设置环境变量
sudo touch /etc/profile.d/myenv.sh

# java
export JAVA_HOME=/data/java/jdk
export M2_HOME=/data/java/maven
export JRE_HOME=${JAVA_HOME}/jre
export SCALA_HOME=/data/java/scala
export CLASS_PATH=${JAVA_HOME}/lib:${JRE_HOME}/lib:${SCALA_HOME}/lib
export PATH=$PATH:${JAVA_HOME}/bin:${SCALA_HOME}/bin:$M2_HOME/bin
# hadoop
export HADOOP_HOME=/data/hadoop
#export HADOOP_MAPRED_HOME=$HADOOP_HOME
#export HADOOP_COMMON_HOME=$HADOOP_HOME 
#export HADOOP_HDFS_HOME=$HADOOP_HOME
#export YARN_HOME=$HADOOP_HOME
#export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
# spark
export SPARK_HOME=/data/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

# hive
export HIVE_HOME=/data/hive
export PATH=$PATH:$HIVE_HOME/bin

# hbase
export HBASE_HOME=/data/hbase
export PATH=$PATH:$HBASE_HOME/bin

# source
source /etc/profile

# 检查版本
[hduser@n51 ~]$ java -version
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
[hduser@n51 ~]$ scala -version
Scala code runner version 2.12.8 -- Copyright 2002-2018, LAMP/EPFL and Lightbend, Inc.

# 增加系统用户和组
sudo groupadd hadoop
sudo useradd -g hadoop hduser
sudo cat /etc/passwd |grep hduser
sudo cat /etc/group |grep hadoop
sudo vi /etc/sudoers
#hduser ALL=(ALL) ALL

#sudo usermod -aG hadoop vagrant
sudo passwd hduser
sudo su -l hduser

# 创建ssh密钥
mkdir ~/.ssh;
cd ~/.ssh/
# 都按回车
ssh-keygen -t rsa
# 加入授权
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
# 修改文件权限
chmod 600 ~/.ssh/authorized_keys

# 授权登录
ssh-copy-id -i ~/.ssh/id_rsa.pub hduser@n51
ssh-copy-id -i ~/.ssh/id_rsa.pub hduser@n52
ssh-copy-id -i ~/.ssh/id_rsa.pub hduser@n53
ssh-copy-id -i ~/.ssh/id_rsa.pub hduser@n54

#cat ~/.ssh/authorized_keys

# 创建haddop数据目录
sudo mkdir /data/hadoop_data
sudo chown -R hduser:hadoop /data/hadoop/
sudo chown -R hduser:hadoop /data/hadoop_data/
sudo chmod -R a+w /data/hadoop_data
su -l hduser
hdfs namenode -format

# 拷贝压缩包到各个节点
# cd /data 
# tar -czf ~/hadoop.tar.gz *;
# scp ~/hadoop.tar.gz hduser@n52:~/
# scp ~/hadoop.tar.gz hduser@n53:~/
# scp ~/hadoop.tar.gz hduser@n54:~/

# 拷贝配置文件到各节点
scp -r /data/hadoop/etc hduser@n52:/data/hadoop
scp -r /data/hadoop/etc hduser@n53:/data/hadoop
scp -r /data/hadoop/etc hduser@n54:/data/hadoop
```
vi etc/hadoop/core-site.xml
```xml
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/data/hadoop_data/</value>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://n51:9000</value>
    </property>

    <property>
        <name>hadoop.proxyuser.hduser.groups</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.hduser.hosts</name>
        <value>*</value>
    </property>

```
vi etc/hadoop/hdfs-site.xml
```xml
    <property>
        <name>dfs.namenode.http-address</name>
        <value>n51:50070</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>n51:50090</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>
```
vi etc/hadoop/mapred-site.xml
```xml
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
   <property>
    <name>mapreduce.jobhistory.address</name>
    <value>n51:10020</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>n51:19888</value>
  </property>
```
vi etc/hadoop/yarn-site.xml
```xml
 <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>n51</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>
```
vi etc/hadoop/workers
```toml
n52
n53
n54
```

## 验证
>
- HDFS页面：50070 
- YARN的管理界面：8088
- HistoryServer的管理界面：19888
- Zookeeper的服务端口号：2181
- Mysql的服务端口号：3306
- Hive.server1=10000
- Kafka的服务端口号：9092
- azkaban界面：8443
- Hbase界面：16010,60010
- Spark的界面：8080
- Spark的URL：7077
>

```bash
## HDFS页面：50070
http://n51:50070
## YARN界面：8088
http://n51:8088
## Spark的界面：8080
http://n51:8080
## Spark的URL：7077
```


# spark
## Hive & Beeline

```bash
vi $HIVE_HOME/conf/hive-site.xml
```
```xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:postgresql://192.168.33.26:30007/metastore</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>org.postgresql.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hiveuser</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>mypassword</value>
    </property>
    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://n51:9083</value>
    </property>
    <property>
        <name>hive.metastore.event.db.notification.api.auth</name>
        <value>false</value>
    </property>
    <!--
    <property> 
        <name>hive.server2.enable.doAs</name>  
        <value>false</value> 
    </property>
    <property>
        <name>hive.server2.webui.host</name>
        <value>n51</value>
    </property>
    <property>
        <name>hive.server2.webui.port</name>
        <value>10002</value>
    </property>
    <property>
        <name>hive.server2.thrift.port</name>
        <value>10000</value>
    </property>
    <property>
        <name>hive.server2.thrift.bind.host</name>
        <value>n51</value>
    </property>
    -->
</configuration>
```
```bash
#wget https://jdbc.postgresql.org/download/postgresql-42.2.5.jar
# 初始化metastore
../bin/schematool -dbType postgres -initSchema
# 启动metastore
hive --service metastore --hiveconf hive.root.logger=DEBUG,console > ~/metastore.log 2>&1 &
# 启动hiveserver2
hive --service hiveserver2 --hiveconf hive.root.logger=DEBUG,console > ~/hiveserver2.log 2>&1 &
# 没有10000端口,没有10002端口
# hive.metastore.event.db.notification.api.auth
# 启动hive cli
hive --service cli -hiveconf hive.root.logger=DEBUG,console
# Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
# hive on spark
hive --service cli -hiveconf hive.execution.engine=spark  -hiveconf hive.root.logger=DEBUG,console
# create table test_1(id int, name string, pid int, price int) ROW FORMAT delimited fields terminated by '\t' stored as textfile ;

# 启动 spark-sql
spark-sql --master yarn
vi $SPARK_HOME/conf/hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://n51:9083</value>
    </property>
</configuration>

## 错误1
hive> show databases;
FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
## 没有启动 metastore，先启动metastore
```

## spark on YARN
```bash
cp -r spark-2.4.0-bin-hadoop2.7 /data/
cd /data/
ln -s spark-2.4.0-bin-hadoop2.7 spark
sudo chown -R hduser:hadoop /data/spark/
```

```bash
# 提交应用程序时使用local-cluster[x,y,z]参数：x代表要生成的executor数，y和z分别代表每个executor所拥有的core和memory数。
spark-submit  或者 spark-submit --master local
spark-submit --master local-cluster[2, 3, 1024]
spark-submit --master spark://node1:7077  或者 spark-submit --master spark://node1:7077 --deploy-mode client
spark-submit --master spark://node1:6066 --deploy-mode cluster
spark-submit --master yarn 或者 spark-submit --master yarn --deploy-mode client
spark-submit --master yarn --deploy-mode cluster


# 测试
/data/spark/bin/spark-shell --class org.apache.spark.examples.JavaSparkPi --master yarn-client /data/spark/examples/jars/spark-examples*.jar 10

# 错误1
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
# 解决 
# vi /data/spark/conf/spark-env.sh
export SPARK_DIST_CLASSPATH=$(hadoop classpath)

# 错误2
Exception in thread "main" org.apache.spark.SparkException: When running with master 'yarn-client' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.

# 解决 
# vi /data/spark/conf/spark-env.sh
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop/

# 错误3
Warning: Master yarn-client is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.

> /data/spark/bin/spark-shell --class org.apache.spark.examples.JavaSparkPi --master yarn-client --jars /data/spark/examples/jars/spark-examples*.jar 10
< /data/spark/bin/spark-shell --class org.apache.spark.examples.JavaSparkPi --master yarn --deploy-mode client --jars/data/spark/examples/jars/spark-examples*.jar 10

# 错误4
WARN  NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
# export HADOOP_OPTS="${HADOOP_OPTS} -Djava.library.path=${HADOOP_HOME}/lib/native"
# 解决 
# vi /data/spark/conf/spark-env.sh
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/:$LD_LIBRARY_PATH


# 错误5
/data/spark/bin/spark-shell --class org.apache.spark.examples.JavaWordCount \
--master yarn \
--deploy-mode client \
--jars /data/spark/examples/jars/spark-examples_2.12-2.4.0.jar \
/data/spark/examples/src/main/resources/people.txt

Exception in thread "main" org.apache.spark.sql.AnalysisException: Path does not exist: hdfs://n51:9000/data/spark/examples/src/main/resources/people.txt;
	at org.apache.spark.sql.execution.datasources.DataSource.$anonfun$checkAndGlobPathIfNecessary$1(DataSource.scala:558)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)
	at scala.collection.immutable.List.foreach(List.scala:388)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)
	at scala.collection.immutable.List.flatMap(List.scala:351)
	at org.apache.spark.sql.execution.datasources.DataSource.checkAndGlobPathIfNecessary(DataSource.scala:545)
	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:359)
	at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
	at org.apache.spark.sql.DataFrameReader.text(DataFrameReader.scala:715)
	at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:757)
	at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:724)
	at org.apache.spark.examples.JavaWordCount.main(JavaWordCount.java:45)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

# 错误6
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: file://data/spark/examples/src/main/resources/people.txt, expected: file:///
	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:730)
	at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:87)
	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:661)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
	at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1683)
	at org.apache.spark.sql.execution.datasources.DataSource.$anonfun$checkAndGlobPathIfNecessary$1(DataSource.scala:557)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)
	at scala.collection.immutable.List.foreach(List.scala:388)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)
	at scala.collection.immutable.List.flatMap(List.scala:351)
	at org.apache.spark.sql.execution.datasources.DataSource.checkAndGlobPathIfNecessary(DataSource.scala:545)
	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:359)
	at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
	at org.apache.spark.sql.DataFrameReader.text(DataFrameReader.scala:715)
	at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:757)
	at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:724)
	at org.apache.spark.examples.JavaWordCount.main(JavaWordCount.java:45)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

# 错误7
2019-03-28 10:04:10 INFO  DAGScheduler:54 - Job 0 failed: collect at JavaWordCount.java:53, took 11.890651 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, n52, executor 2): java.io.FileNotFoundException: File file:/data/spark/examples/src/main/resources/people.txt does not exist
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:132)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:619)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:454)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:454)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:485)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:454)
	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:405)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1887)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1875)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1874)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1874)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:926)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:926)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2108)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2057)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2046)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
	at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:945)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:944)
	at org.apache.spark.api.java.JavaRDDLike.collect(JavaRDDLike.scala:361)
	at org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:360)
	at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
	at org.apache.spark.examples.JavaWordCount.main(JavaWordCount.java:53)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.FileNotFoundException: File file:/data/spark/examples/src/main/resources/people.txt does not exist
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:132)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:619)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:454)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:454)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:485)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:454)
	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:405)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


# 错误

2019-03-29 12:59:41 WARN  Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.

# 正确操作1
hadoop fs -put /data/spark/examples/src/main/resources/people.txt /user/hduser/
hadoop fs -ls -R /
drwxr-xr-x   - hduser supergroup          0 2019-03-28 09:12 /user
drwxr-xr-x   - hduser supergroup          0 2019-03-28 11:28 /user/hduser
drwxr-xr-x   - hduser supergroup          0 2019-03-28 10:35 /user/hduser/.sparkStaging
-rw-r--r--   3 hduser supergroup         32 2019-03-28 11:28 /user/hduser/people.txt

/data/spark/bin/spark-shell --class org.apache.spark.examples.JavaWordCount \
--master yarn --deploy-mode client \
--jars /data/spark/examples/jars/spark-examples*.jar \
/user/hduser/people.txt


# 正确操作2
/data/spark/bin/spark-shell --class org.apache.spark.examples.JavaSparkPi --master yarn --deploy-mode client --jars /data/spark/examples/jars/spark-examples*.jar 10

2019-03-28 09:12:21 INFO  SparkContext:54 - Running Spark version 2.4.0
2019-03-28 09:12:22 INFO  SparkContext:54 - Submitted application: Spark Pi
2019-03-28 09:12:22 INFO  SecurityManager:54 - Changing view acls to: hduser
2019-03-28 09:12:22 INFO  SecurityManager:54 - Changing modify acls to: hduser
2019-03-28 09:12:22 INFO  SecurityManager:54 - Changing view acls groups to: 
2019-03-28 09:12:22 INFO  SecurityManager:54 - Changing modify acls groups to: 
2019-03-28 09:12:22 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hduser); groups with view permissions: Set(); users  with modify permissions: Set(hduser); groups with modify permissions: Set()
2019-03-28 09:12:22 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 34268.
2019-03-28 09:12:22 INFO  SparkEnv:54 - Registering MapOutputTracker
2019-03-28 09:12:22 INFO  SparkEnv:54 - Registering BlockManagerMaster
2019-03-28 09:12:22 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2019-03-28 09:12:22 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2019-03-28 09:12:22 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockmgr-51798937-8c00-4270-a97e-f4b2e1c0ea1a
2019-03-28 09:12:22 INFO  MemoryStore:54 - MemoryStore started with capacity 413.9 MB
2019-03-28 09:12:22 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2019-03-28 09:12:22 INFO  log:192 - Logging initialized @3323ms
2019-03-28 09:12:22 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: 2018-06-06T01:11:56+08:00, git hash: 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
2019-03-28 09:12:22 INFO  Server:419 - Started @3450ms
2019-03-28 09:12:22 INFO  AbstractConnector:278 - Started ServerConnector@669253b7{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2019-03-28 09:12:22 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@547e29a4{/jobs,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5167268{/jobs/json,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1cfd1875{/jobs/job,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2c444798{/jobs/job/json,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1af7f54a{/stages,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6ebd78d1{/stages/json,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@436390f4{/stages/stage,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6d1310f6{/stages/stage/json,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3228d990{/stages/pool,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54e7391d{/stages/pool/json,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@50b8ae8d{/storage,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@255990cc{/storage/json,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@51c929ae{/storage/rdd,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3c8bdd5b{/storage/rdd/json,null,AVAILABLE,@Spark}
2019-03-28 09:12:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29d2d081{/environment,null,AVAILABLE,@Spark}
2019-03-28 09:12:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@40e4ea87{/environment/json,null,AVAILABLE,@Spark}
2019-03-28 09:12:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@58783f6c{/executors,null,AVAILABLE,@Spark}
2019-03-28 09:12:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a7b503d{/executors/json,null,AVAILABLE,@Spark}
2019-03-28 09:12:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@512d92b{/executors/threadDump,null,AVAILABLE,@Spark}
2019-03-28 09:12:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@62c5bbdc{/executors/threadDump/json,null,AVAILABLE,@Spark}
2019-03-28 09:12:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7bdf6bb7{/static,null,AVAILABLE,@Spark}
2019-03-28 09:12:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@e72dba7{/,null,AVAILABLE,@Spark}
2019-03-28 09:12:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@33c2bd{/api,null,AVAILABLE,@Spark}
2019-03-28 09:12:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3abd581e{/jobs/job/kill,null,AVAILABLE,@Spark}
2019-03-28 09:12:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4d4d8fcf{/stages/stage/kill,null,AVAILABLE,@Spark}
2019-03-28 09:12:23 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://n51:4040
2019-03-28 09:12:24 INFO  RMProxy:133 - Connecting to ResourceManager at n51/192.168.33.51:8032
2019-03-28 09:12:24 INFO  Client:54 - Requesting a new application from cluster with 3 NodeManagers
2019-03-28 09:12:24 INFO  Configuration:2752 - resource-types.xml not found
2019-03-28 09:12:24 INFO  ResourceUtils:419 - Unable to find 'resource-types.xml'.
2019-03-28 09:12:24 INFO  Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
2019-03-28 09:12:24 INFO  Client:54 - Will allocate AM container, with 896 MB memory including 384 MB overhead
2019-03-28 09:12:24 INFO  Client:54 - Setting up container launch context for our AM
2019-03-28 09:12:24 INFO  Client:54 - Setting up the launch environment for our AM container
2019-03-28 09:12:25 INFO  Client:54 - Preparing resources for our AM container
2019-03-28 09:12:25 WARN  Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2019-03-28 09:12:26 INFO  Client:54 - Uploading resource file:/tmp/spark-006831fe-7db9-4735-b02c-989d273409a7/__spark_libs__4818467303892521996.zip -> hdfs://n51:9000/user/hduser/.sparkStaging/application_1553701436298_0001/__spark_libs__4818467303892521996.zip
2019-03-28 09:12:30 INFO  Client:54 - Uploading resource file:/data/spark/examples/jars/spark-examples_2.12-2.4.0.jar -> hdfs://n51:9000/user/hduser/.sparkStaging/application_1553701436298_0001/spark-examples_2.12-2.4.0.jar
2019-03-28 09:12:31 INFO  Client:54 - Uploading resource file:/tmp/spark-006831fe-7db9-4735-b02c-989d273409a7/__spark_conf__1337752320075848084.zip -> hdfs://n51:9000/user/hduser/.sparkStaging/application_1553701436298_0001/__spark_conf__.zip
2019-03-28 09:12:31 INFO  SecurityManager:54 - Changing view acls to: hduser
2019-03-28 09:12:31 INFO  SecurityManager:54 - Changing modify acls to: hduser
2019-03-28 09:12:31 INFO  SecurityManager:54 - Changing view acls groups to: 
2019-03-28 09:12:31 INFO  SecurityManager:54 - Changing modify acls groups to: 
2019-03-28 09:12:31 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hduser); groups with view permissions: Set(); users  with modify permissions: Set(hduser); groups with modify permissions: Set()
2019-03-28 09:12:32 INFO  Client:54 - Submitting application application_1553701436298_0001 to ResourceManager
2019-03-28 09:12:33 INFO  YarnClientImpl:311 - Submitted application application_1553701436298_0001
2019-03-28 09:12:33 INFO  SchedulerExtensionServices:54 - Starting Yarn extension services with app application_1553701436298_0001 and attemptId None
2019-03-28 09:12:34 INFO  Client:54 - Application report for application_1553701436298_0001 (state: ACCEPTED)
2019-03-28 09:12:34 INFO  Client:54 - 
	 client token: N/A
	 diagnostics: [Thu Mar 28 09:12:34 +0800 2019] Scheduler has assigned a container for AM, waiting for AM container to be launched
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: default
	 start time: 1553735553132
	 final status: UNDEFINED
	 tracking URL: http://n51:8088/proxy/application_1553701436298_0001/
	 user: hduser
2019-03-28 09:12:35 INFO  Client:54 - Application report for application_1553701436298_0001 (state: ACCEPTED)
2019-03-28 09:12:36 INFO  Client:54 - Application report for application_1553701436298_0001 (state: ACCEPTED)
2019-03-28 09:12:37 INFO  Client:54 - Application report for application_1553701436298_0001 (state: ACCEPTED)
2019-03-28 09:12:38 INFO  Client:54 - Application report for application_1553701436298_0001 (state: ACCEPTED)
2019-03-28 09:12:39 INFO  Client:54 - Application report for application_1553701436298_0001 (state: ACCEPTED)
2019-03-28 09:12:40 INFO  Client:54 - Application report for application_1553701436298_0001 (state: ACCEPTED)
2019-03-28 09:12:41 INFO  Client:54 - Application report for application_1553701436298_0001 (state: ACCEPTED)
2019-03-28 09:12:42 INFO  Client:54 - Application report for application_1553701436298_0001 (state: ACCEPTED)
2019-03-28 09:12:43 INFO  YarnClientSchedulerBackend:54 - Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> n51, PROXY_URI_BASES -> http://n51:8088/proxy/application_1553701436298_0001), /proxy/application_1553701436298_0001
2019-03-28 09:12:43 INFO  JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /jobs, /jobs/json, /jobs/job, /jobs/job/json, /stages, /stages/json, /stages/stage, /stages/stage/json, /stages/pool, /stages/pool/json, /storage, /storage/json, /storage/rdd, /storage/rdd/json, /environment, /environment/json, /executors, /executors/json, /executors/threadDump, /executors/threadDump/json, /static, /, /api, /jobs/job/kill, /stages/stage/kill.
2019-03-28 09:12:43 INFO  YarnSchedulerBackend$YarnSchedulerEndpoint:54 - ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
2019-03-28 09:12:43 INFO  Client:54 - Application report for application_1553701436298_0001 (state: RUNNING)
2019-03-28 09:12:43 INFO  Client:54 - 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: 192.168.33.52
	 ApplicationMaster RPC port: -1
	 queue: default
	 start time: 1553735553132
	 final status: UNDEFINED
	 tracking URL: http://n51:8088/proxy/application_1553701436298_0001/
	 user: hduser
2019-03-28 09:12:43 INFO  YarnClientSchedulerBackend:54 - Application application_1553701436298_0001 has started running.
2019-03-28 09:12:43 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36330.
2019-03-28 09:12:43 INFO  NettyBlockTransferService:54 - Server created on n51:36330
2019-03-28 09:12:43 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2019-03-28 09:12:43 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, n51, 36330, None)
2019-03-28 09:12:43 INFO  BlockManagerMasterEndpoint:54 - Registering block manager n51:36330 with 413.9 MB RAM, BlockManagerId(driver, n51, 36330, None)
2019-03-28 09:12:43 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, n51, 36330, None)
2019-03-28 09:12:43 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, n51, 36330, None)
2019-03-28 09:12:43 INFO  JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /metrics/json.
2019-03-28 09:12:43 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@668cc9a2{/metrics/json,null,AVAILABLE,@Spark}
2019-03-28 09:12:50 INFO  YarnSchedulerBackend$YarnDriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.33.52:49272) with ID 1
2019-03-28 09:12:50 INFO  BlockManagerMasterEndpoint:54 - Registering block manager n52:44918 with 413.9 MB RAM, BlockManagerId(1, n52, 44918, None)
2019-03-28 09:12:51 INFO  YarnSchedulerBackend$YarnDriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.33.54:46766) with ID 2
2019-03-28 09:12:51 INFO  YarnClientSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
2019-03-28 09:12:51 INFO  BlockManagerMasterEndpoint:54 - Registering block manager n54:35494 with 413.9 MB RAM, BlockManagerId(2, n54, 35494, None)
2019-03-28 09:12:52 INFO  SparkContext:54 - Starting job: reduce at SparkPi.scala:38
2019-03-28 09:12:52 INFO  DAGScheduler:54 - Got job 0 (reduce at SparkPi.scala:38) with 10 output partitions
2019-03-28 09:12:52 INFO  DAGScheduler:54 - Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
2019-03-28 09:12:52 INFO  DAGScheduler:54 - Parents of final stage: List()
2019-03-28 09:12:52 INFO  DAGScheduler:54 - Missing parents: List()
2019-03-28 09:12:52 INFO  DAGScheduler:54 - Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
2019-03-28 09:12:52 INFO  MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 3.0 KB, free 413.9 MB)
2019-03-28 09:12:52 INFO  MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 1765.0 B, free 413.9 MB)
2019-03-28 09:12:52 INFO  BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on n51:36330 (size: 1765.0 B, free: 413.9 MB)
2019-03-28 09:12:52 INFO  SparkContext:54 - Created broadcast 0 from broadcast at DAGScheduler.scala:1161
2019-03-28 09:12:52 INFO  DAGScheduler:54 - Submitting 10 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
2019-03-28 09:12:52 INFO  YarnScheduler:54 - Adding task set 0.0 with 10 tasks
2019-03-28 09:12:52 INFO  TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, n54, executor 2, partition 0, PROCESS_LOCAL, 7402 bytes)
2019-03-28 09:12:52 INFO  TaskSetManager:54 - Starting task 1.0 in stage 0.0 (TID 1, n52, executor 1, partition 1, PROCESS_LOCAL, 7402 bytes)
2019-03-28 09:12:53 INFO  BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on n52:44918 (size: 1765.0 B, free: 413.9 MB)
2019-03-28 09:12:53 INFO  BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on n54:35494 (size: 1765.0 B, free: 413.9 MB)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Starting task 2.0 in stage 0.0 (TID 2, n54, executor 2, partition 2, PROCESS_LOCAL, 7402 bytes)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 1412 ms on n54 (executor 2) (1/10)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Starting task 3.0 in stage 0.0 (TID 3, n54, executor 2, partition 3, PROCESS_LOCAL, 7402 bytes)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Finished task 2.0 in stage 0.0 (TID 2) in 84 ms on n54 (executor 2) (2/10)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Starting task 4.0 in stage 0.0 (TID 4, n54, executor 2, partition 4, PROCESS_LOCAL, 7402 bytes)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Finished task 3.0 in stage 0.0 (TID 3) in 64 ms on n54 (executor 2) (3/10)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Starting task 5.0 in stage 0.0 (TID 5, n52, executor 1, partition 5, PROCESS_LOCAL, 7402 bytes)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Finished task 1.0 in stage 0.0 (TID 1) in 1520 ms on n52 (executor 1) (4/10)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Starting task 6.0 in stage 0.0 (TID 6, n54, executor 2, partition 6, PROCESS_LOCAL, 7402 bytes)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Finished task 4.0 in stage 0.0 (TID 4) in 58 ms on n54 (executor 2) (5/10)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Starting task 7.0 in stage 0.0 (TID 7, n52, executor 1, partition 7, PROCESS_LOCAL, 7402 bytes)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Finished task 5.0 in stage 0.0 (TID 5) in 55 ms on n52 (executor 1) (6/10)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Starting task 8.0 in stage 0.0 (TID 8, n54, executor 2, partition 8, PROCESS_LOCAL, 7402 bytes)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Finished task 6.0 in stage 0.0 (TID 6) in 73 ms on n54 (executor 2) (7/10)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Starting task 9.0 in stage 0.0 (TID 9, n52, executor 1, partition 9, PROCESS_LOCAL, 7402 bytes)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Finished task 7.0 in stage 0.0 (TID 7) in 93 ms on n52 (executor 1) (8/10)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Finished task 8.0 in stage 0.0 (TID 8) in 61 ms on n54 (executor 2) (9/10)
2019-03-28 09:12:54 INFO  TaskSetManager:54 - Finished task 9.0 in stage 0.0 (TID 9) in 60 ms on n52 (executor 1) (10/10)
2019-03-28 09:12:54 INFO  YarnScheduler:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool 
2019-03-28 09:12:54 INFO  DAGScheduler:54 - ResultStage 0 (reduce at SparkPi.scala:38) finished in 2.141 s
2019-03-28 09:12:54 INFO  DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 2.257604 s
Pi is roughly 3.1413311413311416
2019-03-28 09:12:54 INFO  AbstractConnector:318 - Stopped Spark@669253b7{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2019-03-28 09:12:54 INFO  SparkUI:54 - Stopped Spark web UI at http://n51:4040
2019-03-28 09:12:54 INFO  YarnClientSchedulerBackend:54 - Interrupting monitor thread
2019-03-28 09:12:54 INFO  YarnClientSchedulerBackend:54 - Shutting down all executors
2019-03-28 09:12:54 INFO  YarnSchedulerBackend$YarnDriverEndpoint:54 - Asking each executor to shut down
2019-03-28 09:12:54 INFO  SchedulerExtensionServices:54 - Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
2019-03-28 09:12:54 INFO  YarnClientSchedulerBackend:54 - Stopped
2019-03-28 09:12:54 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2019-03-28 09:12:54 INFO  MemoryStore:54 - MemoryStore cleared
2019-03-28 09:12:54 INFO  BlockManager:54 - BlockManager stopped
2019-03-28 09:12:54 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2019-03-28 09:12:54 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2019-03-28 09:12:54 INFO  SparkContext:54 - Successfully stopped SparkContext
2019-03-28 09:12:54 INFO  ShutdownHookManager:54 - Shutdown hook called
2019-03-28 09:12:54 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-ab81b4a5-dd8d-4f2e-bb9e-0aee50fc95fe
2019-03-28 09:12:54 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-006831fe-7db9-4735-b02c-989d273409a7

# 2019-04-02 08:25:27 WARN  Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.


# 正确操作3
/data/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi \
--master yarn --deploy-mode cluster \
--jars /data/spark/examples/jars/spark-examples_2.12-2.4.0.jar \
/data/spark/examples/jars/spark-examples_2.12-2.4.0.jar 10

## hdfs追加文件
hadoop fs -appendToFile ~/cp.sh /user/hduser/words.txt &&  hadoop fs -cat /user/hduser/words.txt

# Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
# 在hdfs上创建保存spark相关jars的目录：
hdfs dfs -mkdir -p  /jars/spark_jars
# 上传spark的jars
hdfs dfs -put /data/spark/jars/* /jars/spark_jars/
# 在spark的conf的spark-default.conf添加配置
spark.yarn.jars=hdfs://n51:9000/jars/spark_jars/*
# hive on spark
hive --service cli \
-hiveconf hive.root.logger=DEBUG,console \
-hiveconf hive.execution.engine=spark \
-hiveconf spark.kryo.referenceTracking=false \
-hiveconf spark.kryo.classesToRegister=org.apache.hadoop.hive.ql.io.HiveKey,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch
```

## sparkSQL
```bash
start-thriftserver.sh --hiveconf hive.server2.thrift.port=10011 --master yarn
#--driver-class-path /data/spark-2.2.0-bin-hadoop2.7/jars/mysql-connector-java-5.1.43-bin.jar --executor-memory 5g --total-executor-cores 5
spark-sql --master yarn
spark-shell --master yarn
```
```scala
val input = sc.textFile("/user/hduser/random_words.txt")
val words = input.flatMap(line => line.split(" "))
val counts = words.map(word => (word, 1)).reduceByKey(_ + _)
counts.saveAsTextFile("/user/hduser/random_words_out.txt")
//input.count()
//input.first()
```

# Hbase
## 说明
```bash
192.168.33.51   hdfs:NameNode,SecondaryNameNode     yarn:ResourceManager	zookpeer	hbase:HMaster
192.168.33.52   hdfs:DataNode                       yarn:NodeManager		zookpeer	hbase:HRegionServer
192.168.33.53   hdfs:DataNode                       yarn:NodeManager		zookpeer	hbase:HRegionServer
192.168.33.54   hdfs:DataNode                       yarn:NodeManager		        	hbase:HRegionServer
```
## 安装
vi /data/hbase/conf/hbase-site.xml
```xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>n51,n52,n53</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
  </property>
  <property>
    <name>hbase.tmp.dir</name>
    <value>/data/hbase_tmp</value>
  </property>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://n51:9000/hbase</value>
  </property>
  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.unsafe.stream.capability.enforce</name>
    <value>false</value>
  </property>
  <property>
    <name>hbase.wal.provider</name>
    <value>filesystem</value>
  </property> 
<!--
<property>
	<name>hbase.master.info.port</name>
	<value>16010</value>
</property>
-->
</configuration>
```
vi /data/hbase/conf/regionservers
```toml
n52
n53
n54
```
vi /data/hbase/conf/hbase-env.sh
```bash
export HBASE_MANAGES_ZK=false
```

## 启动
```bash
bin/start-hbase.sh
bin/stop-hbase.sh
bin/hbase-daemon.sh start master
bin/hbase-daemon.sh start regionserver
```

```bash
2019-04-11 14:49:56,938 ERROR [main] regionserver.HRegionServer: Failed construction RegionServer
java.lang.NoClassDefFoundError: org/apache/htrace/SamplerBuilder
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:644)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:628)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2701)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2683)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:372)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
	at org.apache.hadoop.hbase.util.CommonFSUtils.getRootDir(CommonFSUtils.java:362)
	at org.apache.hadoop.hbase.util.CommonFSUtils.isValidWALRootDir(CommonFSUtils.java:411)
	at org.apache.hadoop.hbase.util.CommonFSUtils.getWALRootDir(CommonFSUtils.java:387)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.initializeFileSystem(HRegionServer.java:704)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:613)
	at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:489)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3093)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
	at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3111)
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.SamplerBuilder
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 25 more
2019-04-11 14:49:56,959 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster.
	at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3100)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
	at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3111)
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/SamplerBuilder
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:644)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:628)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2701)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2683)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:372)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
	at org.apache.hadoop.hbase.util.CommonFSUtils.getRootDir(CommonFSUtils.java:362)
	at org.apache.hadoop.hbase.util.CommonFSUtils.isValidWALRootDir(CommonFSUtils.java:411)
	at org.apache.hadoop.hbase.util.CommonFSUtils.getWALRootDir(CommonFSUtils.java:387)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.initializeFileSystem(HRegionServer.java:704)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:613)
	at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:489)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3093)
	... 5 more
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.SamplerBuilder
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 25 more




2019-04-11 15:18:00,964 ERROR [master/n51:16000:becomeActiveMaster] master.HMaster: Failed to become active master
java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.
	at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1086)
	at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:423)
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:611)
	at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1458)
	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:890)
	at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2272)
	at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:581)
	at java.lang.Thread.run(Thread.java:748)
2019-04-11 15:18:00,965 ERROR [master/n51:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master n51,16000,1554967074912: Unhandled exception. Starting shutdown. *****
java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.
	at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1086)
	at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:423)
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:611)
	at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1458)
	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:890)
	at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2272)
	at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:581)
	at java.lang.Thread.run(Thread.java:748)


java.lang.ExceptionInInitializerError
	at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:79)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:362)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:339)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2633)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:964)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Failed to create local dir /data/hbase_tmp/local/jars, DynamicClassLoader failed to init
	at org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
	at org.apache.hadoop.hbase.util.DynamicClassLoader.<init>(DynamicClassLoader.java:98)
	at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException$ClassLoaderHolder.lambda$static$0(RemoteWithExtrasException.java:56)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException$ClassLoaderHolder.<clinit>(RemoteWithExtrasException.java:55)
	... 6 more


2019-04-11 17:04:26,903 INFO  [RpcServer.priority.FPBQ.Fifo.handler=18,queue=0,port=16020] regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
2019-04-11 17:04:26,925 INFO  [RS_OPEN_META-regionserver/n53:16020-0] wal.AbstractFSWAL: WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=n53%2C16020%2C1554973432966.meta, suffix=.meta, logDir=hdfs://n51:9000/hbase/WALs/n53,16020,1554973432966, archiveDir=hdfs://n51:9000/hbase/oldWALs
2019-04-11 17:04:26,936 ERROR [RS_OPEN_META-regionserver/n53:16020-0] handler.OpenRegionHandler: Failed open of region=hbase:meta,,1.1588230740
java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper
	at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
	at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:614)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:126)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:756)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:486)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.<init>(AsyncFSWAL.java:253)
	at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:73)
	at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:48)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:152)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:60)
	at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:284)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2126)
	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284)
	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-04-11 17:04:29,406 INFO  [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020] regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
2019-04-11 17:04:29,423 INFO  [RS_OPEN_META-regionserver/n53:16020-0] wal.AbstractFSWAL: WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=n53%2C16020%2C1554973432966.meta, suffix=.meta, logDir=hdfs://n51:9000/hbase/WALs/n53,16020,1554973432966, archiveDir=hdfs://n51:9000/hbase/oldWALs
2019-04-11 17:04:29,425 ERROR [RS_OPEN_META-regionserver/n53:16020-0] handler.OpenRegionHandler: Failed open of region=hbase:meta,,1.1588230740
java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper
	at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
	at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:614)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:126)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:756)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:486)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.<init>(AsyncFSWAL.java:253)
	at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:73)
	at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:48)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:152)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:60)
	at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:284)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2126)
	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284)
	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

# http://www.zhongruitech.com/4186901.html
```

```bash
bin/hbase shell
hbase(main)> create 'hbase_test',{NAME=>'cf1'},{NAME=>'cf2'}
# ERROR: Failed to create local dir /data/hbase_tmp/local/jars, DynamicClassLoader failed to init
# ERROR: Could not initialize class org.apache.hadoop.hbase.ipc.RemoteWithExtrasException$ClassLoaderHolder
# For usage try 'help "create"'
hbase(main)> put 'hbase_test', '001','cf1:name','Tom'
hbase(main)> put 'hbase_test', '001','cf1:age','18'
hbase(main)> put 'hbase_test', '001','cf2:phone’, ‘13309882999’
hbase(main)> put 'hbase_test', '001','cf2:address',’昆明’

# HBase表的创建
# 语法：create <table>, {NAME => <family>, VERSIONS => <VERSIONS>}
# 例如：创建表t1，有两个family name：f1，f2，且版本数前者为3，后者为1
hbase(main)> create 't1',{NAME => 'f1', VERSIONS => 3},{NAME => 'f2', VERSIONS => 1}
# disable表
hbase(main)> disable 't1'
# drop表
hbase(main)> drop 't1'
# HBase表的清空
hbase(main)> truncate 't1'

hbase(main)> balance_switch true
hbase(main)> create 'blog','article','author'
hbase(main)> create "test","col1","col2" 
hbase(main)> list
hbase(main)> list "test"
hbase(main)> status

ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2977)
	at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1973)
	at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:630)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)



# http://www.zhongruitech.com/4186901.html
# https://ken.io/note/hbase-cluster-deploy-guide
# http://www.cnblogs.com/jishilei/archive/2013/05/27/3101172.html

cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml $HBASE_HOME/conf/

```



# zookeeper集群

## 安装
vi conf/zoo.cfg
```toml
tickTime=2000
dataDir=/opt/zookeeper/data
clientPort=2181
initLimit=10
syncLimit=5
server.1=192.168.33.181:2888:3888
server.2=192.168.33.182:2888:3888
server.3=192.168.33.183:2888:3888
```
```bash
mkdir -p /opt/zookeeper/data/
mkdir -p /opt/zookeeper/logs/
echo "1" >> /opt/zookeeper/data/myid
./bin/zkServer.sh start
./bin/zkServer.sh stop
./bin/zkServer.sh status
```

## systemd
```toml
[Unit]
Description=zookeeper.service
After=network.target
[Service]
Type=forking
Environment=ZOO_LOG_DIR=/opt/zookeeper/logs/
Environment=ZOO_LOG4J_PROP=INFO,ROLLINGFILE
Environment=ZOOPIDFILE=/tmp/zookeeper_server.pid
Environment=PATH=/usr/local/jdk1.8.0_152/bin:/usr/local/jdk1.8.0_152/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
ExecStart=/opt/zookeeper-3.4.13/bin/zkServer.sh start
ExecStop=/opt/zookeeper-3.4.13/bin/zkServer.sh stop
ExecReload=/opt/zookeeper-3.4.13/bin/zkServer.sh restart
#PIDFile=/tmp/zookeeper_server.pid
#WorkingDirectory=
#User=www
[Install]
WantedBy=multi-user.target
```

# hadoop3端口

| PORT  | CONFIG NAME                                   | CONFIG VALUE                              |
| ----- | --------------------------------------------- | ----------------------------------------- |
| 0     | dfs.balancer.address                          | `0.0.0.0:0`                               |
| 9866  | dfs.datanode.address                          | `0.0.0.0:9866`                            |
| 9864  | dfs.datanode.http.address                     | `0.0.0.0:9864`                            |
| 9865  | dfs.datanode.https.address                    | `0.0.0.0:9865`                            |
| 9867  | dfs.datanode.ipc.address                      | `0.0.0.0:9867`                            |
| 8111  | dfs.federation.router.admin-address           | `0.0.0.0:8111`                            |
| 50071 | dfs.federation.router.http-address            | `0.0.0.0:50071`                           |
| 50072 | dfs.federation.router.https-address           | `0.0.0.0:50072`                           |
| 8888  | dfs.federation.router.rpc-address             | `0.0.0.0:8888`                            |
| 8480  | dfs.journalnode.http-address                  | `0.0.0.0:8480`                            |
| 8481  | dfs.journalnode.https-address                 | `0.0.0.0:8481`                            |
| 8485  | dfs.journalnode.rpc-address                   | `0.0.0.0:8485`                            |
| 0     | dfs.mover.address                             | `0.0.0.0:0`                               |
| 50100 | dfs.namenode.backup.address                   | `0.0.0.0:50100`                           |
| 50105 | dfs.namenode.backup.http-address              | `0.0.0.0:50105`                           |
| 9870  | dfs.namenode.http-address                     | `0.0.0.0:9870`                            |
| 9871  | dfs.namenode.https-address                    | `0.0.0.0:9871`                            |
| 9868  | dfs.namenode.secondary.http-address           | `0.0.0.0:9868`                            |
| 9869  | dfs.namenode.secondary.https-address          | `0.0.0.0:9869`                            |
| 50200 | dfs.provided.aliasmap.inmemory.dnrpc-address  | `0.0.0.0:50200`                           |
| 2181  | hadoop.registry.zk.quorum                     | `localhost:2181`                          |
| 10020 | mapreduce.jobhistory.address                  | `0.0.0.0:10020`                           |
| 10033 | mapreduce.jobhistory.admin.address            | `0.0.0.0:10033`                           |
| 19888 | mapreduce.jobhistory.webapp.address           | `0.0.0.0:19888`                           |
| 19890 | mapreduce.jobhistory.webapp.https.address     | `0.0.0.0:19890`                           |
| 0     | yarn.nodemanager.address                      | `${yarn.nodemanager.hostname}:0`          |
| 8049  | yarn.nodemanager.amrmproxy.address            | `0.0.0.0:8049`                            |
| 8048  | yarn.nodemanager.collector-service.address    | `${yarn.nodemanager.hostname}:8048`       |
| 8040  | yarn.nodemanager.localizer.address            | `${yarn.nodemanager.hostname}:8040`       |
| 8042  | yarn.nodemanager.webapp.address               | `${yarn.nodemanager.hostname}:8042`       |
| 8044  | yarn.nodemanager.webapp.https.address         | `0.0.0.0:8044`                            |
| 8032  | yarn.resourcemanager.address                  | `${yarn.resourcemanager.hostname}:8032`   |
| 8033  | yarn.resourcemanager.admin.address            | `${yarn.resourcemanager.hostname}:8033`   |
| 8031  | yarn.resourcemanager.resource-tracker.address | `${yarn.resourcemanager.hostname}:8031`   |
| 8030  | yarn.resourcemanager.scheduler.address        | `${yarn.resourcemanager.hostname}:8030`   |
| 8088  | yarn.resourcemanager.webapp.address           | `${yarn.resourcemanager.hostname}:8088`   |
| 8090  | yarn.resourcemanager.webapp.https.address     | `${yarn.resourcemanager.hostname}:8090`   |
| 8089  | yarn.router.webapp.address                    | `0.0.0.0:8089`                            |
| 8091  | yarn.router.webapp.https.address              | `0.0.0.0:8091`                            |
| 8047  | yarn.sharedcache.admin.address                | `0.0.0.0:8047`                            |
| 8045  | yarn.sharedcache.client-server.address        | `0.0.0.0:8045`                            |
| 8046  | yarn.sharedcache.uploader.server.address      | `0.0.0.0:8046`                            |
| 8788  | yarn.sharedcache.webapp.address               | `0.0.0.0:8788`                            |
| 10200 | yarn.timeline-service.address                 | `${yarn.timeline-service.hostname}:10200` |
| 8188  | yarn.timeline-service.webapp.address          | `${yarn.timeline-service.hostname}:8188`  |
| 8190  | yarn.timeline-service.webapp.https.address    | `${yarn.timeline-service.hostname}:8190`  |


- [core-default.xml](https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/core-default.xml)
- [hdfs-default.xml](https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml)
- [hdfs-rbf-default.xml](https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml)
- [yarn-default.xml](https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml)
- [mapred-default.xml](https://hadoop.apache.org/docs/r3.1.1/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml)


# 参考资料

* [zookeeper.service](https://gist.github.com/martinhynar/286dee76f4a5f3881f0ebcf711fea52f)
* [HowToSetupYourDevelopmentEnvironment](https://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment)
* [Submitting Applications](http://spark.apache.org/docs/latest/submitting-applications.html)
* [BigDataLearning](https://github.com/MOBIN-F/BigDataLearning)
* sqoop
* kudu vs hbase
* impala vs hive vs Spark SQL vs Presto
* [alluxio](https://github.com/Alluxio/alluxio) vs [Ignite](https://ignite.apache.org/)
* Apache Druid
* Apache Kylin
* Apache Storm
* Apache Samza
* Apache Flink
* Apache Kafka
* parquet
* Spark Thrift Server
* [scylla](https://github.com/scylladb/scylla)
* flume logstash
* ELK(ElasticSearch+Logstash+ Kibana)
* EFK(ElasticSearch+Filebeat+ Kibana)
* Fluent：fluentbit 和 fluentd.
* Kafka Streams
* Ganglia
* hue
* Apache Presto vs Apache Impala
* [Spark Configuration](https://spark.apache.org/docs/latest/configuration.html)
* [Hive on Spark: Getting Started](https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started)
* [Setting Up HiveServer2](https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2)