海波无痕

ubuntu安装hive

ziki — Wed, 09 May 2012 01:49:58 +0000

环境:ubuntu-12.04-server-amd64

JDK1.7

hidoop分布式请看http://blog.ziki.cn/984.html

如把hadoop当作数据库的存储部分的话，那么hive可以理解为一个sql语言解释器,或者可以叫执行类SQL语言的shell解释器。

使用hive中可以使用表的操作select/insert/create来映射到hadoop的hdfs系统中一系列文本操作，可以理解为sql对hdfs的行为映射.

更可以通过实现自定义的udf、udaf来实现复杂的筛选select,这些带有函数的select语句hive会自动翻译成hadoop的mapreduce来执行job最终返回结果

返回结果可以直接输出也可以直接insert到一张hive创建的表,这样的话就将算出的最终结果持久化回hadoop的hdfs中,可以通过haddop的命令行直接访问最终结果。

当然用hive更是可以访问。

1、获取hive-0.8.1.tar.gz

2、解压缩到/opt/hive-0.8.1

3、vi /etc/prfole

export HIVE_HOME=/opt/hive-0.8.1

export PATH=$HIVE_HOME/bin:$PATH

4、vi /opt/hive-0.8.1/conf/hive-env.sh

HADOOP_HOME=/opt/hadoop-1.0.1

5、准备一个mysql数据库并命名为hive

6、vi /opt/hive-0.8.1/conf/hive-site.xml

    hive.metastore.local
      true

    javax.jdo.option.ConnectionURL
      jdbc:mysql://192.168.122.104:3306/hive

    javax.jdo.option.ConnectionDriverName
      com.mysql.jdbc.Driver

      javax.jdo.option.ConnectionUserName
      hiveuser

      javax.jdo.option.ConnectionPassword
      hiveuser


      datanucleus.fixedDatastore
      false

7、打开hive

hive

当前hive的shell感觉跟mysql的客户端shell差不多,在下边就可以运行hivesql了。

建表

hive>create table access_log(url STRING,count_no STRING);

导入数据

hive>LOAD DATA LOCAL INPATH '/opt/access_log/access.log' OVERWRITE INTO TABLE access_log;

可以看到,这句sql产生了hadoop的hdfs文件

ubuntu@server1:/$ /opt/hadoop-1.0.1/bin/hadoop dfs -ls /user/hive/warehouse
Found 1 items
drwxr-xr-x - ubuntu supergroup 0 2012-05-07 19:03 /user/hive/warehouse/access_log

hive> show tables;

access_log
OK
Time taken: 4.003 seconds

hive>select count(*) from access_log;

可以看到select count(*) from access_log 语句的执行就产生了一个hadoop的mapreduce 来运行。这个过程被hive封装了

hive> select count(*) from access_log;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapred.reduce.tasks=
Starting Job = job_201205090916_0001, Tracking URL = http://server1:50030/jobdetails.jsp?jobid=job_201205090916_0001
Kill Command = /opt/hadoop-1.0.1/libexec/../bin/hadoop job -Dmapred.job.tracker=hdfs://server1:9001 -kill job_201205090916_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2012-05-09 09:46:07,510 Stage-1 map = 0%, reduce = 0%
2012-05-09 09:46:12,621 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.99 sec
2012-05-09 09:46:13,633 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.99 sec
2012-05-09 09:46:14,649 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.99 sec
2012-05-09 09:46:15,680 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.99 sec
2012-05-09 09:46:16,707 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.99 sec
2012-05-09 09:46:17,728 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.99 sec
2012-05-09 09:46:18,748 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.99 sec
2012-05-09 09:46:19,763 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.99 sec
2012-05-09 09:46:20,774 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.99 sec
2012-05-09 09:46:21,785 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 0.99 sec
2012-05-09 09:46:22,811 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 0.99 sec
2012-05-09 09:46:23,883 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 0.99 sec
2012-05-09 09:46:24,894 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.83 sec
2012-05-09 09:46:25,914 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.83 sec
2012-05-09 09:46:26,930 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.83 sec
2012-05-09 09:46:27,960 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.83 sec
2012-05-09 09:46:29,009 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.83 sec
2012-05-09 09:46:30,025 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.83 sec
2012-05-09 09:46:31,045 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.83 sec
MapReduce Total cumulative CPU time: 3 seconds 830 msec
Ended Job = job_201205090916_0001
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 Accumulative CPU: 3.83 sec HDFS Read: 4951 HDFS Write: 3 SUCESS
Total MapReduce CPU Time Spent: 3 seconds 830 msec
OK
21
Time taken: 35.107 seconds

原创文章，转载请注明： 转载自海波无痕

本文链接地址: ubuntu安装hive

文章的脚注信息由WordPress的wp-posturl插件自动生成

hadoop分布式配置

ziki — Sat, 05 May 2012 15:05:40 +0000

环境:ubuntu-12.04-server-amd64

3台 ubuntu-12.04-server-amd64 server

分别

192.168.122.101 server1 //主,运行NameNode+SecondaryNameNode+JobTracker

192.168.122.102 server2 //从1 运行DataNode+TaskTracker

192.168.122.103 server3 //从2 运行DataNode+TaskTracker

1、配置java环境(server1、server2、server3)

下载jdk-7u3-linux-x64.tar.gz

/opt/java/jdk/jdk1.7.0_03

配置一个软链接

ln -s /opt/java/jdk/jdk1.7.0_03 /opt/java/jdk/current

sudo vi /etc/profile

增加

export JAVA_HOME=/opt/java/jdk/current
export JRE_HOME=/opt/java/jdk/current/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

保存

source /etc/profile

sudo update-alternatives --install /usr/bin/java java /opt/java/jdk/current/bin/java 300;
sudo update-alternatives --install /usr/bin/javac javac /opt/java/jdk/current/bin/javac 300;

2、配置/etc/hosts(server1、server2、server3)

192.168.122.101 server1

192.168.122.102 server2

192.168.122.103 server3

3、安装ssh(server1、server2、server3)

sudo apt-get install ssh

4、ssh免登录(server1)

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

scp /home/ubuntu/.ssh/id_dsa.pub ubuntu@server2:/home/ubuntu

scp /home/ubuntu/.ssh/id_dsa.pub ubuntu@server3:/home/ubuntu

5、ssh免登录(server2、server3)

mkdir ~/.ssh

mv ~/id_dsa.pub ~/.ssh

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

6、下载hadoop-1.0.1.tar.gz(server1、server2、server3)

解压缩到/opt/hadoop-1.0.1

vi /opt/hadoop-1.0.1/conf/hadoop-env.sh

打开注释并设置JAVA_HOME

export JAVA_HOME=/opt/java/jdk/current

vi /opt/hadoop-1.0.1/conf/core-site.xml

fs.default.name
hdfs://server1:9000

hadoop.tmp.dir
/opt/hadoop/tmp/

vi /opt/hadoop-1.0.1/conf/hdfs-site.xml

dfs.replication
2

    dfs.name.dir
    /opt/hadoop/name/

    dfs.data.dir
    /opt/hadoop/data

vi /opt/hadoop-1.0.1/conf/mapred-site.xml:

mapred.job.tracker
hdfs://server1:9001

7、设置主从(server1)

vi /opt/hadoop-1.0.1/masters

server1

vi /opt/hadoop-1.0.1/slaves

server2

server3

8、启动hadoop

ubuntu@server1:~$ cd /opt/hadoop-1.0.1/bin/

ubuntu@server1:/opt/hadoop-1.0.1/bin$ ./hadoop namenode -format

ubuntu@server1:/opt/hadoop-1.0.1/bin$ ./start-all.sh

starting namenode, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-namenode-server1.out

server2: starting datanode, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-datanode-server2.out

server3: starting datanode, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-datanode-server3.out

server1: starting secondarynamenode, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-secondarynamenode-server1.out

starting jobtracker, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-jobtracker-server1.out

server3: starting tasktracker, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-tasktracker-server3.out

server2: starting tasktracker, logging to /opt/hadoop-1.0.1/libexec/../logs/hadoop-ubuntu-tasktracker-server2.out

9、查看启动情况(server1,server2,server3)

ubuntu@server1:/$ jps
1544 SecondaryNameNode
1749 Jps
1628 JobTracker
1273 NameNode

ubuntu@server2:~$ jps
1135 DataNode
1645 Jps
1327 TaskTracker

ubuntu@server3:~$ jps
1271 TaskTracker
1615 Jps
1084 DataNode

10、查看

查看任务调度

http://server1:50030/

查看name

http://server1:50070

原创文章，转载请注明： 转载自海波无痕

本文链接地址: hadoop分布式配置

文章的脚注信息由WordPress的wp-posturl插件自动生成

ubuntu系统连接思科vpn

ziki — Wed, 02 May 2012 14:13:46 +0000

1、sudo apt-get install network-manager-vpnc;

2、新建/data/remote.conf

IPSec gateway 11.11.11.11
IPSec ID cisco
IPSec secret 1234567890
Xauth username haha
Xauth password hehe

3、mkdir ~/.sudopass;

touch ~/.sudopass/sudopass.secret

gedit ~/.sudopass/sudopass.secret

写入root的密码,并保存。

4、输入命令

sudo -u root -S vpnc /data/remote --enable-1des --local-port 0 --dns-update no < ~/.sudopass/sudopass.secret

原创文章，转载请注明： 转载自海波无痕

本文链接地址: ubuntu系统连接思科vpn

文章的脚注信息由WordPress的wp-posturl插件自动生成

ubuntu为当前用户增加组

ziki — Mon, 30 Apr 2012 09:51:49 +0000

将wangliang 增加到vboxusers组中

sudo usermod -a -G vboxusers wangliang

http://www.cyberciti.biz/faq/howto-linux-add-user-to-group/

原创文章，转载请注明： 转载自海波无痕

本文链接地址: ubuntu为当前用户增加组

文章的脚注信息由WordPress的wp-posturl插件自动生成

ubuntu 12.04 LTS 版eclipse svn报Failed to load JavaHL Library

ziki — Mon, 30 Apr 2012 01:15:02 +0000

Failed to load JavaHL Library.
These are the errors that were encountered:
no libsvnjavahl-1 in java.library.path
no svnjavahl-1 in java.library.path
no svnjavahl in java.library.path
java.library.path = /usr/lib/jni

按照以前的装法

1、sudo apt-get install libsvn-java

2、eclipse.ini中增加参数

-vmargs
-Dosgi.requiredJavaVersion=1.5
-Djava.library.path=/usr/lib/jni

就可以了,但安装配置好后还是报错Failed to load JavaHL Library.重新安装libsvn-java还是不灵,去usr/lib/jni发现jni不是个目录。原来是安装目录发生了改变。将jni的path改为如下就正常了。

-Djava.library.path=/usr/lib/x86_64-linux-gnu/jni/

参考资料http://pkgs.org/ubuntu-12.04/ubuntu-universe-amd64/libsvn-java_1.6.17dfsg-3ubuntu3_amd64.deb.html

原创文章，转载请注明： 转载自海波无痕

本文链接地址: ubuntu 12.04 LTS 版eclipse svn报Failed to load JavaHL Library

文章的脚注信息由WordPress的wp-posturl插件自动生成

run multiple instances of sshd on ubuntu

ziki — Fri, 06 Apr 2012 08:51:32 +0000

sudo cp -r /etc/ssh /etc/ssh2
sudo cp /etc/init.d/ssh /etc/init.d/ssh2
sudo cp /etc/default/ssh /etc/default/ssh2
sudo ln /usr/sbin/sshd /usr/sbin/sshd2

sudo vi /etc/init.d/ssh2
SSHD_OPTS='-f /etc/ssh2/sshd_config'

sudo vi /etc/ssh2/sshd_config'

Port 28222
PidFile /var/run/sshd2.pid
HostKey /etc/ssh2/ssh_host_rsa_key
HostKey /etc/ssh2/ssh_host_dsa_key
HostKey /etc/ssh2/ssh_host_ecdsa_key

ssh-keygen -b 1024 -t rsa -f /etc/ssh2/ssh_host_key;
ssh-keygen -b 1024 -t rsa -f /etc/ssh2/ssh_host_rsa_key;
ssh-keygen -b 1024 -t dsa -f /etc/ssh2/ssh_host_dsa_key;

参考资料

http://ubuntuforums.org/showthread.php?t=1497376

原创文章，转载请注明： 转载自海波无痕

本文链接地址: run multiple instances of sshd on ubuntu

文章的脚注信息由WordPress的wp-posturl插件自动生成

避免javax.xml.bind.UnmarshalException兼容一个更新的webservice

ziki — Wed, 04 Apr 2012 13:54:28 +0000

CXF的jaxb客户端调用webservice的情况下，默认情况下服务端更新了wsdl如xml增加了一个属性,假如不再根据wsdl重新生成client就会报错 javax.xml.bind.UnmarshalException: unexpected element ，但新增的这个元素对原有的业务又是无用的多余的。

那对于客户端来说就可以不升级client，解决办法是将set-jaxb-validation-event-handler 设置为false

JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();

Class clazz = MyService.class;

List interceptors = new ArrayList();

factory.setAddress("http://localhost:8080/MyService?wsdl");

factory.setServiceClass(clazz);

if(factory.getProperties()==null){

Map properties = new HashMap();

factory.setProperties(properties);

}

factory.getProperties().put("set-jaxb-validation-event-handler", "false");

client = (MyService)factory.create();

参考资料

http://cxf.547215.n5.nabble.com/How-to-deal-with-an-updated-WSDL-td552046.html

http://stackoverflow.com/questions/2195034/server-side-xml-validation-with-cxf-webservice

原创文章，转载请注明： 转载自海波无痕

本文链接地址: 避免javax.xml.bind.UnmarshalException兼容一个更新的webservice

文章的脚注信息由WordPress的wp-posturl插件自动生成

cxf自动生成的wsdl被nginx代理后soap:address location

ziki — Sat, 31 Mar 2012 01:46:00 +0000

由于开发webservice使用的是cxf的java first、wsdl是自动生成的。其中生成的

binding="tns:MyServiceSoapBinding" name="MyServicePort">

location="http://www.domain.com/appcontext/Service/MyService"/>

location 是根据请求的request来自动生成的

如果用127.0.0.1来访问生成的wsdl就是

 binding="tns:MyServiceSoapBinding" name="MyServicePort">
 location="http://127.0.0.1/appcontext/Service/MyService"/>
如果用localhost来访问生成的wsdl就是
 binding="tns:MyServiceSoapBinding" name="MyServicePort">
 location="http://localhost/appcontext/Service/MyService"/>
这就会产生一个问题如果用nginx或者apache放在tomcat的前端的话,tomcat获取的request是内网的地址

比如访问nginx的

www.domain.com--->用nginxproxy到 nginxserver所在的内网地址是localhost:8080

那么动态生成的wsdl

就变成了

 binding="tns:MyServiceSoapBinding" name="MyServicePort">
 location="http://localhost:8080/appcontext/Service/MyService"/>

这将导致wsdl的不准确。

解决办法是用publishedEndpointUrl参数来明确这个location的地址

如下

原创文章，转载请注明： 转载自海波无痕

本文链接地址: cxf自动生成的wsdl被nginx代理后soap:address location

文章的脚注信息由WordPress的wp-posturl插件自动生成

tomcat在windows下安装后台服务

ziki — Thu, 29 Mar 2012 13:22:34 +0000

1、用命令行进入tomcat_home/bin/目录,运行service install tomcat_c4。进入服务列表设置为自动服务。

2、使用regedit进入注册表

找到

HKEY_LOCAL_MACHINE\SOFTWARE\Apache Software Foundation\Procrun 2.0\tomcat_c4\Parameters\Java

修改

JvmMs 选10进制设置为512

JvmMx 选10进制设置为512

Options 的值增加

-XX:NewRatio=3

-XX:PermSize=128m

-XX:MaxPermSize=256m

-XX:MaxDirectMemorySize=128m

-Duser.country=US

-Duser.language=en

-Duser.timezone=Asia/Shanghai

-Dfile.encoding=utf-8

-Dsun.jnu.encoding=utf-8

-Dcom.sun.management.jmxremote=true

server.xml配置

原创文章，转载请注明： 转载自海波无痕

本文链接地址: tomcat在windows下安装后台服务

文章的脚注信息由WordPress的wp-posturl插件自动生成

Another unnamed CacheManager already exists in the same VM.

ziki — Wed, 21 Mar 2012 15:02:47 +0000

ehcache版本2.5.1

nested exception is net.sf.ehcache.CacheException: Another unnamed CacheManager already exists in the same VM. Please provide unique names for each CacheManager in the config or do one of following:

1. Use one of the CacheManager.create() static factory methods to reuse same CacheManager with same name or create one if necessary

2. Shutdown the earlier cacheManager before creating new one with same name.

The source of the existing CacheManager is: DefaultConfigurationSource [ ehcache.xml or ehcache-failsafe.xml ]

at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:389)

at org.springframework.web.context.ContextLoader.loadParentContext(ContextLoader.java:358)

at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:193)

at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)

at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4779)

at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5273)

at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)

at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)

at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)

at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:649)

at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1581)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

at java.util.concurrent.FutureTask.run(FutureTask.java:138)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:662)

classpath*:com/xxx/*.hbm.xml

hibernate.bytecode.use_reflection_optimizer={hibernate.bytecode.use_reflection_optimizer}

hibernate.dialect=${hibernate.dialect}

hibernate.jdbc.fetch_size=${hibernate.jdbc.fetch_size}

hibernate.jdbc.batch_size=${hibernate.jdbc.batch_size}

hibernate.hbm2ddl.auto=${hibernate.hbm2ddl.auto}

hibernate.show_sql=${hibernate.show_sql}

hibernate.format_sql=${hibernate.format_sql}

hibernate.query.factory_class=${hibernate.query.factory_class}

hibernate.default_batch_fetch_size=${hibernate.default_batch_fetch_size}

hibernate.transaction.factory_class=${hibernate.transaction.factory_class}

hibernate.transaction.auto_close_session=${hibernate.transaction.auto_close_session}

hibernate.cache.use_query_cache=${hibernate.cache.use_query_cache}

hibernate.cache.use_second_level_cache=${hibernate.cache.use_second_level_cache}

hibernate.cache.region.factory_class=net.sf.ehcache.hibernate.SingletonEhCacheRegionFactory

net.sf.ehcache.configurationResourceName=/com/xxx/ehcache.xml

用个项目中多个sessionfactory使用hibernate的ehcache或者hibernate和业务共同使用ehcache使用SingletonEhCacheRegionFactory。

原创文章，转载请注明： 转载自海波无痕

本文链接地址: Another unnamed CacheManager already exists in the same VM.

文章的脚注信息由WordPress的wp-posturl插件自动生成