Hive on Coinidea的博客

[Java]Java通过JDBC连接运行Hive

Thu, 28 Apr 2016 07:11:02 +0000

Hadoop的版本繁多，一旦版本不对Hive的连接和操作可能会报各种各样的错误，这个时候需要用到上一篇博文提到的Maven来管理这些lib。

Hadoop 以及 Hive 的版本查看：

1
2


hadoop version
hive --version

pom 的配置如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>com.coinidea</groupId>
 <artifactId>hive</artifactId>
 <packaging>war</packaging>
 <version>1.0-SNAPSHOT</version>
 <name>hive.web Maven Webapp</name>
 <url>http://maven.apache.org</url>
 <properties>
 <hadoop.version>2.5.0-cdh5.3.2</hadoop.version>
 <hive.version>0.13.1-cdh5.3.2</hive.version>
 </properties>
 <dependencies>
 <!--hadoop和hive-->
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-jdbc</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-common</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-exec</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-metastore</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-service</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hadoop</groupId>
 <artifactId>hadoop-common</artifactId>
 <version>${hadoop.version}</version>
 <exclusions>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jetty</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jetty-util</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jsp-2.1</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jsp-api-2.1</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>servlet-api-2.1</artifactId>
 </exclusion>
 <exclusion>
 <groupId>javax.servlet</groupId>
 <artifactId>servlet-api</artifactId>
 </exclusion>
 <exclusion>
 <groupId>javax.servlet.jsp</groupId>
 <artifactId>jsp-api</artifactId>
 </exclusion>
 <!--tomcat7中自带jasper.jar-->
 <exclusion>
 <groupId>tomcat</groupId>
 <artifactId>jasper-compiler</artifactId>
 </exclusion>
 <exclusion>
 <groupId>tomcat</groupId>
 <artifactId>jasper-runtime</artifactId>
 </exclusion>
 </exclusions>
 </dependency>
 <dependency>
 <groupId>cglib</groupId>
 <artifactId>cglib-nodep</artifactId>
 <version>2.2.2</version>
 </dependency>
 </dependencies>
 <build>
 <finalName>hive</finalName>
 </build>
 <repositories>
 <repository>
 <id>cloudera</id>
 <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
 <releases>
 <enabled>true</enabled>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </repository>
 <repository>
 <id>people.apache.snapshots</id>
 <url>http://repository.apache.org/content/groups/snapshots-group/</url>
 <releases>
 <enabled>false</enabled>
 </releases>
 <snapshots>
 <enabled>true</enabled>
 </snapshots>
 </repository>
 </repositories>
</project>

新建 HiveTest.java

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


package com.coinidea;

import java.sql.*;

public class HiveTest {
 /* Start server: hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000 */
 private static String driverName = "org.apache.hive.jdbc.HiveDriver";

 /**
 * @param args
 * @throws SQLException
 */
 public static void main(String[] args) throws SQLException {
 try {
 Class.forName(driverName);
 } catch (ClassNotFoundException e) {
 e.printStackTrace();
 System.exit(1);
 }
 //replace "hive" here with the name of the user the queries should run as
 Connection con = DriverManager.getConnection("jdbc:hive2://192.168.*.*:10000/default", "hive", "hive");
 Statement stmt = con.createStatement();
 String table_user = "test.user";
 String user_sql = "select uri from " + table_user + " limit 10";
 System.out.println("Running: " + user_sql);
 ResultSet users = stmt.executeQuery(user_sql);
 while (users.next()) {
 System.out.println("USER:" + users.getString(1));
 }
 stmt.close();
 }
}

运行 HiveTest 之前，需要启动 HiveServer2

1

hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000

[Hive]Need to specify partition columns because the destination table is partition

Tue, 23 Feb 2016 13:36:11 +0000

当向已经分区的表插入数据时，有可能会报错：

1

Need to specify partition columns because the destination table is partition

原因是需要在插入的数据中指定分区字段的数值是多少。

比如：

建表语句：

1
2
3
4
5
6
7
8
9


CREATE TABLE test (
 starttime STRING,
 endtime STRING,
 title STRING
)
PARTITIONED BY (username STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
STORED AS TEXTFILE;

1
2
3
4
5
6
7
8
9


CREATE TABLE test2 (
 starttime STRING,
 endtime STRING,
 title STRING
)
PARTITIONED BY (username STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
STORED AS TEXTFILE;

插入语句：

1
2


INSERT INTO TABLE test PARTITION(username='admin')
SELECT starttime, endtime, title FROM test2 WHERE username = 'admin';

[Hive]Hive SQL 分区间统计问题

Wed, 18 Nov 2015 07:45:26 +0000

需求：

表table中存在一个text字段，需要统计出文本长度为[0,20]、[20,40]、[40,60]…按区间划分的数量。其中，需要自动化到包含最大值max的区间。

实现：

第一种方法：

1
2
3
4
5
6
7


select count(case when length(text) between 0 and 20 then 1 end) as text1,
count(case when length(text) between 21 and 40 then 1 end) as text2,
count(case when length(text) between 41 and 60 then 1 end) as text3,
count(case when length(text) between 61 and 80 then 1 end) as text4,
count(case when length(text) between 81 and 100 then 1 end) as text5,
count(case when length(text) > 100 then 1 end) as text6
from table;

这样写有一个问题，就是必须先知道max的值，然后随着max的增大，sql语句会越来越长，而且涉及到人工的干预。

第二种方法：

1

select floor(length(text)/20) * 20, count(*) from table group by floor(length(text)/20);

这种方法，先将每个text的文本长度除以20，向下取整，再乘以20得到一个分组依据，以这个依据进行group by就可以得到最终结果，在规避循环的情况下，用简单的语句，巧妙的解决了自动分区间统计的问题。