Big Data on Coinidea's Blog

[Java] Connecting to and Running Hive via JDBC

Thu, 28 Apr 2016 07:11:02 +0000

Hadoop has many versions, and if the version is wrong, connecting to and operating Hive may throw all sorts of errors. This is where Maven, mentioned in the previous blog post, comes in handy for managing these libraries.

Checking Hadoop and Hive versions:

1
2


hadoop version
hive --version

POM configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>com.coinidea</groupId>
 <artifactId>hive</artifactId>
 <packaging>war</packaging>
 <version>1.0-SNAPSHOT</version>
 <name>hive.web Maven Webapp</name>
 <url>http://maven.apache.org</url>
 <properties>
 <hadoop.version>2.5.0-cdh5.3.2</hadoop.version>
 <hive.version>0.13.1-cdh5.3.2</hive.version>
 </properties>
 <dependencies>
 <!--hadoop and hive-->
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-jdbc</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-common</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-exec</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-metastore</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-service</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hadoop</groupId>
 <artifactId>hadoop-common</artifactId>
 <version>${hadoop.version}</version>
 <exclusions>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jetty</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jetty-util</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jsp-2.1</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jsp-api-2.1</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>servlet-api-2.1</artifactId>
 </exclusion>
 <exclusion>
 <groupId>javax.servlet</groupId>
 <artifactId>servlet-api</artifactId>
 </exclusion>
 <exclusion>
 <groupId>javax.servlet.jsp</groupId>
 <artifactId>jsp-api</artifactId>
 </exclusion>
 <!--tomcat7 includes jasper.jar-->
 <exclusion>
 <groupId>tomcat</groupId>
 <artifactId>jasper-compiler</artifactId>
 </exclusion>
 <exclusion>
 <groupId>tomcat</groupId>
 <artifactId>jasper-runtime</artifactId>
 </exclusion>
 </exclusions>
 </dependency>
 <dependency>
 <groupId>cglib</groupId>
 <artifactId>cglib-nodep</artifactId>
 <version>2.2.2</version>
 </dependency>
 </dependencies>
 <build>
 <finalName>hive</finalName>
 </build>
 <repositories>
 <repository>
 <id>cloudera</id>
 <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
 <releases>
 <enabled>true</enabled>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </repository>
 <repository>
 <id>people.apache.snapshots</id>
 <url>http://repository.apache.org/content/groups/snapshots-group/</url>
 <releases>
 <enabled>false</enabled>
 </releases>
 <snapshots>
 <enabled>true</enabled>
 </snapshots>
 </repository>
 </repositories>
</project>

Create HiveTest.java

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


package com.coinidea;

import java.sql.*;

public class HiveTest {
 /* Start server: hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000 */
 private static String driverName = "org.apache.hive.jdbc.HiveDriver";

 /**
 * @param args
 * @throws SQLException
 */
 public static void main(String[] args) throws SQLException {
 try {
 Class.forName(driverName);
 } catch (ClassNotFoundException e) {
 e.printStackTrace();
 System.exit(1);
 }
 //replace "hive" here with the name of the user the queries should run as
 Connection con = DriverManager.getConnection("jdbc:hive2://192.168.*.*:10000/default", "hive", "hive");
 Statement stmt = con.createStatement();
 String table_user = "test.user";
 String user_sql = "select uri from " + table_user + " limit 10";
 System.out.println("Running: " + user_sql);
 ResultSet users = stmt.executeQuery(user_sql);
 while (users.next()) {
 System.out.println("USER:" + users.getString(1));
 }
 stmt.close();
 }
}

Before running HiveTest, you need to start HiveServer2

1

hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000

[Hive] Need to Specify Partition Columns Because the Destination Table Is Partitioned

Tue, 23 Feb 2016 13:36:11 +0000

When inserting data into an already partitioned table, you may encounter this error:

1

Need to specify partition columns because the destination table is partition

The reason is that you need to specify the partition field values in the insert statement.

For example:

Create Table Statements:

1
2
3
4
5
6
7
8
9


CREATE TABLE test (
 starttime STRING,
 endtime STRING,
 title STRING
)
PARTITIONED BY (username STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
STORED AS TEXTFILE;

1
2
3
4
5
6
7
8
9


CREATE TABLE test2 (
 starttime STRING,
 endtime STRING,
 title STRING
)
PARTITIONED BY (username STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
STORED AS TEXTFILE;

Insert Statement:

1
2


INSERT INTO TABLE test PARTITION(username='admin')
SELECT starttime, endtime, title FROM test2 WHERE username = 'admin';

[Hive] Hive SQL Range-Based Statistics

Wed, 18 Nov 2015 07:45:26 +0000

Requirement:

A table contains a text field. We need to count the number of records whose text length falls into ranges [0,20], [20,40], [40,60], etc. The ranges should automatically extend to cover the maximum value.

Implementation:

Method 1:

1
2
3
4
5
6
7


select count(case when length(text) between 0 and 20 then 1 end) as text1,
count(case when length(text) between 21 and 40 then 1 end) as text2,
count(case when length(text) between 41 and 60 then 1 end) as text3,
count(case when length(text) between 61 and 80 then 1 end) as text4,
count(case when length(text) between 81 and 100 then 1 end) as text5,
count(case when length(text) > 100 then 1 end) as text6
from table;

The problem with this approach is that you need to know the max value first. As the max increases, the SQL statement gets longer and longer, and it requires manual intervention.

Method 2:

1

select floor(length(text)/20) * 20, count(*) from table group by floor(length(text)/20);

This method divides each text’s length by 20, floors the result, then multiplies by 20 to get a grouping key. Using this key for GROUP BY gives us the final result. Without any loops, this simple statement elegantly solves the automatic range-based statistics problem.

[Storm] Installing Storm

Thu, 01 Oct 2015 05:12:26 +0000

A distributed real-time computation system for processing high-speed, large-volume data streams. It adds reliable real-time data processing capabilities to Hadoop.

Home link: http://storm.apache.org/

Download link: http://storm.apache.org/downloads.html

Setup

Java

CentOS comes with JDK 1.7, so this step can be skipped.

Zookeeper

Home link: http://zookeeper.apache.org/releases.html

Download the package, upload, and extract

http://mirror.bit.edu.cn/apache/zookeeper/stable/
1

tar -xf zookeeper-3.4.6.tar.gz

Configure Zookeeper

1
2
3
4
5
6
7
8


tickTime=2000
dataDir=/var/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
server.1=zookeeper1:2888:3888
server.2=zookeeper2:2888:3888
server.3=zookeeper3:2888:3888

Start Zookeeper
1

bin/zkServer.sh start
Test
1

bin/zkCli.sh -server 127.0.0.1:2181
Dependency installation
Java

Pre-installed.
Python

Pre-installed.
Storm
Download the package, upload, and extract

Extract command:
1

tar -xf apache-storm-0.9.5.tar.gz
Configure the Zookeeper address in storm.yaml
1 2

storm.zookeeper.servers: - "127.0.0.1"
Create and configure the workdir
1

storm.local.dir: "/home/admin/storm/workdir"
Note: storm.local.dir is the local disk directory used by Nimbus and Supervisor processes to store a small amount of state (such as jars, confs, etc.). This directory must be created in advance with sufficient access permissions.
1

nimbus.host: "127.0.0.1"
1 2 3 4 5

supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703

Start all Storm services

Nimbus

1

bin/storm nimbus >/dev/null 2>&1 &

Supervisor

1

bin/storm supervisor >/dev/null 2>&1 &

1

bin/storm ui >/dev/null 2>&1 &

Logview

1

bin/storm logviewer > /dev/null 2>&1

Access Storm UI

http://ip:8080

[Hadoop] Setting Up the CDH Environment

Mon, 28 Sep 2015 14:50:42 +0000

1. Connect server

Use CRT or Xshell to connect to the server remotely.

2. Network config

First, configure DNS for the server and check whether it can access the internet.

Enter the following command:

1

ping [www.baidu.com][1]

If you receive a reply from [www.baidu.com][1], it means the server can access the internet. If you get “unknown host,” the DNS configuration is incorrect and needs to be set up.

There are two ways to configure DNS:

Temporary solution: edit the resolv.conf file and add “nameserver 8.8.8.8”.
1 2

vi /etc/resolv.conf nameserver 8.8.8.8
Permanent solution: edit the ifcfg_eth0 file and add “DNS1=8.8.8.8”.
1 2

vi /etc/sysconfig/network-scripts/ifcfg-eth0 DNS1=8.8.8.8

After configuration, run the command “service network restart”.

The second approach is recommended, as the first one will be lost when the server restarts.

Disable the firewall

Enter the command:

1

service iptables stop

3. File transfer

During the installation process, you may need to upload local files to the server. Here is one approach.

Enter the command:

1

sudo yum install lrzsz

After installing the lrzsz package, enter:

1

rz

Then select the file you want to upload.

Yum (Yellow dog Updater Modified) is an RPM-based package manager that can automatically download and install RPM packages from specified servers, handling dependency relationships automatically and installing all dependent packages at once, eliminating the tedious process of downloading and installing them one by one.

4. Install CDH

HDFS: A distributed file storage system, the open-source Java implementation of GFS.

MapReduce: A parallel computing programming model for large-scale datasets.

HBase: NoSQL column database

Hive: Data warehouse

Zookeeper: Distributed lock service

Pig: Big data analysis platform interface

Home link: http://www.cloudera.com/content/cloudera/en/products-and-services/cdh.html

Setup:

1. Java

CentOS 6.7 comes with JDK 1.7, so this step can be skipped. Any JDK-related steps during installation can be ignored.

2. Cloudera Manager Server

Cloudera Manager makes it easy to manage Hadoop deployments of any scale in production. Quickly deploy, configure, and monitor your cluster through an intuitive UI – complete with rolling upgrades, backup and disaster recovery, and customizable alerting.

Download:

http://www.cloudera.com/content/cloudera/en/downloads/cloudera_manager/cm-5-4-7.html

Download the latest version 5.4.7.

After downloading, you will get a file of about 503KB named cloudera-manager-installer.bin. Upload this file (using “rz”) to a designated folder on the server.

Close selinux:

1
2


/etc/selinux/config
selinux=disabled

Hostname

You need to modify the hostname of the machine. To check the current hostname, simply type hostname.

Enter:
1

vi /etc/hosts
Edit the file and add the line “192.168._._ master.com master”.

Note: when adding datanode nodes, do not name them “master” – use names like “datanode1” instead.

Run the “reboot” command to restart the server.

3. CDH

Using cloudera manager

Navigate to the directory containing cloudera-manager-installer.bin and run:

1
2


chmod u+x cloudera-manager-installer.bin
./ cloudera-manager-installer.bin

Keep clicking next. The selected items will be highlighted in bold.

This may take a while, so please be patient…

Choose express free version.
Display the packages to be installed
Specify CDH cluster hosts
Install CDH
Enter username and password
Install the specified Parcel
Select services

4. Storm

A distributed real-time computation system for processing high-speed, large-volume data streams. It adds reliable real-time data processing capabilities to Hadoop.

5. Spark

Spark uses in-memory computing. Starting from multi-iteration batch processing, it allows data to be loaded into memory for repeated queries. It also integrates multiple computing paradigms such as data warehousing, stream processing, and graph computing. Spark is built on top of HDFS and integrates well with Hadoop.

6. Problems

Refreshing metadata packages

1
2


Another app is currently holding the yum lock; waiting for it to exit...
rm -f /var/run/yum.pid

Uninstall Cloudera Manager

1

$ sudo /usr/share/cmf/uninstall-cloudera-manager.sh

Installation failed: Unable to receive heartbeat from Agent

1
2
3
4
5


cat /etc/hosts
ip 域名 主机名
192.168.*.1 master.com master
cat /etc/sysconfig/network
HOSTNAME=master.com

Uninstall Cloudera Manager

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


sudo rm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera*
sudo /usr/share/cmf/uninstall-cloudera-manager.sh
sudo service cloudera-scm-server stop
sudo service cloudera-scm-server-db stop
cloudera-manager-server-db
sudo yum remove cloudera-manager-server
sudo yum remove cloudera-manager-server-db-2
sudo service cloudera-scm-agent hard_stop
sudo yum remove 'cloudera-manager-*'
sudo yum clean all
sudo rm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera* /var/log/cloudera* /var/run/cloudera*
sudo rm -Rf /var/cache/apt/archives/cloudera*
sudo rm /tmp/.scm_prepare_node.lock
sudo rm -Rf /var/lib/flume-ng /var/lib/hadoop* /var/lib/hue /var/lib/navigator /var/lib/oozie /var/lib/solr /var/lib/sqoop* /var/lib/zookeeper
sudo rm -Rf /dfs /mapred /yarn