Big Data on Coinidea's Blog

[vsftpd] vsftpd Passive Mode Configuration

Wed, 10 Apr 2019 02:56:05 +0000

vsftpd.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140


# Example config file /etc/vsftpd/vsftpd.conf
#
# The default compiled in settings are fairly paranoid. This sample file
# loosens things up a bit, to make the ftp daemon more usable.
# Please see vsftpd.conf.5 for all compiled in defaults.
#
# READ THIS: This example file is NOT an exhaustive list of vsftpd options.
# Please read the vsftpd.conf.5 manual page to get a full idea of vsftpd's
# capabilities.
#
# Allow anonymous FTP? (Beware - allowed by default if you comment this out).
anonymous_enable=NO
#
# Uncomment this to allow local users to log in.
# When SELinux is enforcing check for SE bool ftp_home_dir
local_enable=YES
#
# Uncomment this to enable any form of FTP write command.
write_enable=YES
#
# Default umask for local users is 077. You may wish to change this to 022,
# if your users expect that (022 is used by most other ftpd's)
local_umask=022
#
# Uncomment this to allow the anonymous FTP user to upload files. This only
# has an effect if the above global write enable is activated. Also, you will
# obviously need to create a directory writable by the FTP user.
# When SELinux is enforcing check for SE bool allow_ftpd_anon_write, allow_ftpd_full_access
#anon_upload_enable=YES
#
# Uncomment this if you want the anonymous FTP user to be able to create
# new directories.
#anon_mkdir_write_enable=YES
#
# Activate directory messages - messages given to remote users when they
# go into a certain directory.
dirmessage_enable=YES
#
# Activate logging of uploads/downloads.
xferlog_enable=YES
#
# Make sure PORT transfer connections originate from port 20 (ftp-data).
connect_from_port_20=YES
#
# If you want, you can arrange for uploaded anonymous files to be owned by
# a different user. Note! Using "root" for uploaded files is not
# recommended!
#chown_uploads=YES
#chown_username=whoever
#
# You may override where the log file goes if you like. The default is shown
# below.
#xferlog_file=/var/log/xferlog
#
# If you want, you can have your log file in standard ftpd xferlog format.
# Note that the default log file location is /var/log/xferlog in this case.
xferlog_std_format=YES
#
# You may change the default value for timing out an idle session.
#idle_session_timeout=600
#
# You may change the default value for timing out a data connection.
#data_connection_timeout=120
#
# It is recommended that you define on your system a unique user which the
# ftp server can use as a totally isolated and unprivileged user.
#nopriv_user=ftpsecure
#
# Enable this and the server will recognise asynchronous ABOR requests. Not
# recommended for security (the code is non-trivial). Not enabling it,
# however, may confuse older FTP clients.
#async_abor_enable=YES
#
# By default the server will pretend to allow ASCII mode but in fact ignore
# the request. Turn on the below options to have the server actually do ASCII
# mangling on files when in ASCII mode.
# Beware that on some FTP servers, ASCII support allows a denial of service
# attack (DoS) via the command "SIZE /big/file" in ASCII mode. vsftpd
# predicted this attack and has always been safe, reporting the size of the
# raw file.
# ASCII mangling is a horrible feature of the protocol.
#ascii_upload_enable=YES
#ascii_download_enable=YES
#
# You may fully customise the login banner string:
ftpd_banner=Welcome to blah FTP service.
#
# You may specify a file of disallowed anonymous e-mail addresses. Apparently
# useful for combatting certain DoS attacks.
#deny_email_enable=YES
# (default follows)
#banned_email_file=/etc/vsftpd/banned_emails
#
# You may specify an explicit list of local users to chroot() to their home
# directory. If chroot_local_user is YES, then this list becomes a list of
# users to NOT chroot().
# (Warning! chroot'ing can be very dangerous. If using chroot, make sure that
# the user does not have write access to the top level directory within the
# chroot)
chroot_local_user=YES
chroot_list_enable=NO
# (default follows)
chroot_list_file=/etc/vsftpd/chroot_list
#
# You may activate the "-R" option to the builtin ls. This is disabled by
# default to avoid remote users being able to cause excessive I/O on large
# sites. However, some broken FTP clients such as "ncftp" and "mirror" assume
# the presence of the "-R" option, so there is a strong case for enabling it.
#ls_recurse_enable=YES
#
# When "listen" directive is enabled, vsftpd runs in standalone mode and
# listens on IPv4 sockets. This directive cannot be used in conjunction
# with the listen_ipv6 directive.
listen=YES
#
# This directive enables listening on IPv6 sockets. By default, listening
# on the IPv6 "any" address (::) will accept connections from both IPv6
# and IPv4 clients. It is not necessary to listen on *both* IPv4 and IPv6
# sockets. If you want that (perhaps because you want to listen on specific
# addresses) then you must run two copies of vsftpd with two configuration
# files.
# Make sure, that one of the listen options is commented !!
listen_ipv6=NO

pam_service_name=vsftpd
userlist_enable=YES
tcp_wrappers=YES
local_root=/ftp
anon_root=/ftp
use_localtime=YES
user_config_dir=/etc/vsftpd/userconfig
pasv_enable=yes
pasv_min_port=41000
pasv_max_port=42000
allow_writeable_chroot=YES
listen_port=21
xferlog_file=/var/log/vsftpd.log
pasv_address=your_ip
pasv_addr_resolve=yes
pasv_promiscuous=YES

user_list

1

# Comment out everything in this file

chroot_list

1
2


ftpuser
ftpsubuser

/etc/vsftpd/userconfig/ftpsubuser

1

local_root=/ftp/ftpsubuser/

Commands to execute:

1
2
3
4
5


sudo useradd ftpuser -d /ftp -s /sbin/nologin -m
passwd ftpuser

sudo useradd ftpsubuser -d /ftp/ftpsubuser -s /sbin/nologin -m
passwd ftpsubuser

[Docker]Docker Common Commands Cheat Sheet

Wed, 10 Apr 2019 02:45:13 +0000

A recent project involved MongoDB clusters, Spark clusters, Hadoop clusters, Python 2.7, Python 3.5, various recommendation algorithms, Java Resin environment, LNMP environment, vsftpd, and more. Because it needed to be deployed across intranet, testing, and production environments, each deployment was painful. This is where Docker’s benefits really shone – packing Python environments, LNMP environments, vsftpd, and everything else into Docker containers was extremely satisfying. After 2 months of deployment and integration testing, I compiled some practical Docker commands. System environment: CentOS 7

Download, install Docker, and start the service
https://download.docker.com/linux/centos/7/x86_64/stable/Packages/
Download docker-ce-18.03.1.ce-1.el7.centos.x86_64.rpm
1 2

yum install docker-ce-18.03.1.ce-1.el7.centos.x86_64.rpm service docker start

Common Docker commands
Fix error: WARNING: IPv4 forwarding is disabled. Networking will not work.

Step 1: On the host machine, run echo "net.ipv4.ip_forward=1" >>/usr/lib/sysctl.d/00-system.conf

Step 2: Restart network and Docker services

1

[root@localhost /]# systemctl restart network && systemctl restart docker

Failed to get D-Bus connection: Operation not permitted

1

docker run -d --name centos7 --privileged=true --net=host -v /home/code:/home/code centos:mysql_python3_jdk1.8 /usr/sbin/init

Run a Docker container for a long time with host path mounting

1

docker run -dit -v /home/code:/home/code python35:latest /bin/bash

Enter a running Docker container

1

docker exec -it container_id /bin/bash

Start a container image

1

docker run -it python35:latest /bin/bash

Docker export:

1

docker export -o /home/python35.tar container_id

docker save *.tar

Docker import:

1

docker import /home/python35.tar

docker load < *.tar

Docker tagging

1

docker tag image_id REPOSITORY:TAG

Docker specify host network

1

docker run -dit --net=host python35:latest /bin/bash

Docker sync container and host time

1

docker run -dit -v /etc/localtime:/etc/localtime python35:latest /bin/bash

Install JDK 1.8

1

yum install java-1.8.0-openjdk* -y

Install MySQL 5.6

1
2
3
4
5
6
7
8


rpm -Uvh http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm
yum -y install mysql-community-server
systemctl enable mysqld
systemctl start mysqld
Initialize:
mysql_secure_installation
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'remote_login_password' WITH GRANT OPTION;
FLUSH PRIVILEGES;

Python 3 encoding

1

echo 'export LANG=zh_CN.UTF-8' >> ~/.bashrc

[ElasticSearch]PHP Operations with ElasticSearch

Thu, 04 Jan 2018 09:04:18 +0000

Official API:
You can download the official PHP ElasticSearch API and read the documentation for related operations.
Example code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


static public function initSearchClient() {
 if (null == BaseFrontController::$client) {
 $hosts['hosts'] = array(
 "host" => '127.0.0.1',
 "port" => '9200',
 'scheme' => 'http'
 );
 BaseFrontController::$client = \Elasticsearch\ClientBuilder::create()
 ->setSSLVerification(false)
 ->setHosts($hosts)
 ->build();
 }
}

// Elastic search
static public function actionSearch($area, $keyword, $start, $end, $size=100, $agg = null) {
 BaseFrontController::initSearchClient();
 $params['index'] = BaseFrontController::getSearchIndex();
 $params['type'] = 'raw';
 $params['body']["size"] = $size;
 // your aggs
 if (null != $agg) {
 $params['body']["aggs"] = $agg;
 }
 $params['body']["query"]["filtered"]['filter']["and"] = [
 ["term"=>['Your area' => $area]], // your area
 ["range"=>["Your time" => array("gte"=>$start, "lte"=>$end)]] // your time
 ];

 if ($keyword != '') {
 $params['body']["query"]["filtered"]['query']['bool']['should']['match_phrase'] = array();// your query word
 }

 $response = BaseFrontController::$client->search($params);
 return $response;
}

[ElasticSearch]Modifying ES Mapping

Sun, 19 Nov 2017 12:23:18 +0000

Having used MySQL-like databases for so long, one day I ran into a major pitfall with ES.
When posting the mapping, I didn’t properly distinguish between “analyzed” and “not_analyzed” for a field, which accidentally caused all data in that column to be tokenized. The dataset had approximately 150 million records.
I naively thought I could just modify the field attributes like in MySQL. ES is based on Lucene, and there’s no other way — in simple terms, you either delete the index and re-import, or reindex. Reindexing means creating a new index and copying the data from the old index over. There are many tutorials online for this, for example:
http://blog.csdn.net/loveyaqin1990/article/details/77684599
https://www.cnblogs.com/wmx3ng/p/4112993.html

As for actual implementation code, there’s very little available online. After a long search, I only found a Python implementation. This article provides a migration implementation based on the official ES PHP SDK and bulk API.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58


<?php
require 'vendor/autoload.php';
$hosts['hosts'] = array(
 "host" => '127.0.0.1',
 "port" => '9200',
 'scheme' => 'http'
);
$client = Elasticsearch\ClientBuilder::create()
 ->setSSLVerification(false)
 ->setHosts($hosts)
 ->build();
for ($i = 1; $i <= 10; $i++) {
 if ($i != 10) {
 $params['index'] = 'index-0'.$i;
 }
 else {
 $params['index'] = 'index-'.$i;
 }
 echo $params["index"]."\r\n";
 $params['type'] = 'raw';
 $params['scroll'] = '120s';
 $params["size"] = 50000;
 $params["body"] = array(
 "query" => array(
 "match_all" => array()
 )
 );
 $response = $client->search($params);
 $step = 1;
 while (isset($response['hits']['hits']) && count($response['hits']['hits']) > 0) {
 echo $step++."\t";
 $scroll_id = $response['_scroll_id'];
 unset($response);
 $response = $client->scroll(
 array(
 "scroll_id" => $scroll_id,
 "scroll" => "120s"
 )
 );
 if (count($response['hits']['hits']) > 0) {
 $bulk = array('index'=>$params['index']."-reindex",'type'=>$params['type']);
 foreach ($response["hits"]["hits"] as $key=>$val) {
 $bulk['body'][]=array(
 'index' => array(
 '_id'=>$val['_id']
 ),
 );
 $bulk['body'][] = $val['_source'];
 }
 // insert reindex
 $res = $client->bulk($bulk);
 unset($bulk);
 } else {
 break;
 }
 }
}
?>

[Java] Connecting to and Running Hive via JDBC

Thu, 28 Apr 2016 07:11:02 +0000

Hadoop has many versions, and if the version is wrong, connecting to and operating Hive may throw all sorts of errors. This is where Maven, mentioned in the previous blog post, comes in handy for managing these libraries.

Checking Hadoop and Hive versions:

1
2


hadoop version
hive --version

POM configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>com.coinidea</groupId>
 <artifactId>hive</artifactId>
 <packaging>war</packaging>
 <version>1.0-SNAPSHOT</version>
 <name>hive.web Maven Webapp</name>
 <url>http://maven.apache.org</url>
 <properties>
 <hadoop.version>2.5.0-cdh5.3.2</hadoop.version>
 <hive.version>0.13.1-cdh5.3.2</hive.version>
 </properties>
 <dependencies>
 <!--hadoop and hive-->
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-jdbc</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-common</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-exec</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-metastore</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-service</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hadoop</groupId>
 <artifactId>hadoop-common</artifactId>
 <version>${hadoop.version}</version>
 <exclusions>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jetty</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jetty-util</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jsp-2.1</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jsp-api-2.1</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>servlet-api-2.1</artifactId>
 </exclusion>
 <exclusion>
 <groupId>javax.servlet</groupId>
 <artifactId>servlet-api</artifactId>
 </exclusion>
 <exclusion>
 <groupId>javax.servlet.jsp</groupId>
 <artifactId>jsp-api</artifactId>
 </exclusion>
 <!--tomcat7 includes jasper.jar-->
 <exclusion>
 <groupId>tomcat</groupId>
 <artifactId>jasper-compiler</artifactId>
 </exclusion>
 <exclusion>
 <groupId>tomcat</groupId>
 <artifactId>jasper-runtime</artifactId>
 </exclusion>
 </exclusions>
 </dependency>
 <dependency>
 <groupId>cglib</groupId>
 <artifactId>cglib-nodep</artifactId>
 <version>2.2.2</version>
 </dependency>
 </dependencies>
 <build>
 <finalName>hive</finalName>
 </build>
 <repositories>
 <repository>
 <id>cloudera</id>
 <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
 <releases>
 <enabled>true</enabled>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </repository>
 <repository>
 <id>people.apache.snapshots</id>
 <url>http://repository.apache.org/content/groups/snapshots-group/</url>
 <releases>
 <enabled>false</enabled>
 </releases>
 <snapshots>
 <enabled>true</enabled>
 </snapshots>
 </repository>
 </repositories>
</project>

Create HiveTest.java

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


package com.coinidea;

import java.sql.*;

public class HiveTest {
 /* Start server: hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000 */
 private static String driverName = "org.apache.hive.jdbc.HiveDriver";

 /**
 * @param args
 * @throws SQLException
 */
 public static void main(String[] args) throws SQLException {
 try {
 Class.forName(driverName);
 } catch (ClassNotFoundException e) {
 e.printStackTrace();
 System.exit(1);
 }
 //replace "hive" here with the name of the user the queries should run as
 Connection con = DriverManager.getConnection("jdbc:hive2://192.168.*.*:10000/default", "hive", "hive");
 Statement stmt = con.createStatement();
 String table_user = "test.user";
 String user_sql = "select uri from " + table_user + " limit 10";
 System.out.println("Running: " + user_sql);
 ResultSet users = stmt.executeQuery(user_sql);
 while (users.next()) {
 System.out.println("USER:" + users.getString(1));
 }
 stmt.close();
 }
}

Before running HiveTest, you need to start HiveServer2

1

hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000

[Hive] Need to Specify Partition Columns Because the Destination Table Is Partitioned

Tue, 23 Feb 2016 13:36:11 +0000

When inserting data into an already partitioned table, you may encounter this error:

1

Need to specify partition columns because the destination table is partition

The reason is that you need to specify the partition field values in the insert statement.

For example:

Create Table Statements:

1
2
3
4
5
6
7
8
9


CREATE TABLE test (
 starttime STRING,
 endtime STRING,
 title STRING
)
PARTITIONED BY (username STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
STORED AS TEXTFILE;

1
2
3
4
5
6
7
8
9


CREATE TABLE test2 (
 starttime STRING,
 endtime STRING,
 title STRING
)
PARTITIONED BY (username STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
STORED AS TEXTFILE;

Insert Statement:

1
2


INSERT INTO TABLE test PARTITION(username='admin')
SELECT starttime, endtime, title FROM test2 WHERE username = 'admin';

[Hive] Hive SQL Range-Based Statistics

Wed, 18 Nov 2015 07:45:26 +0000

Requirement:

A table contains a text field. We need to count the number of records whose text length falls into ranges [0,20], [20,40], [40,60], etc. The ranges should automatically extend to cover the maximum value.

Implementation:

Method 1:

1
2
3
4
5
6
7


select count(case when length(text) between 0 and 20 then 1 end) as text1,
count(case when length(text) between 21 and 40 then 1 end) as text2,
count(case when length(text) between 41 and 60 then 1 end) as text3,
count(case when length(text) between 61 and 80 then 1 end) as text4,
count(case when length(text) between 81 and 100 then 1 end) as text5,
count(case when length(text) > 100 then 1 end) as text6
from table;

The problem with this approach is that you need to know the max value first. As the max increases, the SQL statement gets longer and longer, and it requires manual intervention.

Method 2:

1

select floor(length(text)/20) * 20, count(*) from table group by floor(length(text)/20);

This method divides each text’s length by 20, floors the result, then multiplies by 20 to get a grouping key. Using this key for GROUP BY gives us the final result. Without any loops, this simple statement elegantly solves the automatic range-based statistics problem.

[Storm] Installing Storm

Thu, 01 Oct 2015 05:12:26 +0000

A distributed real-time computation system for processing high-speed, large-volume data streams. It adds reliable real-time data processing capabilities to Hadoop.

Home link: http://storm.apache.org/

Download link: http://storm.apache.org/downloads.html

Setup

Java

CentOS comes with JDK 1.7, so this step can be skipped.

Zookeeper

Home link: http://zookeeper.apache.org/releases.html

Download the package, upload, and extract

http://mirror.bit.edu.cn/apache/zookeeper/stable/
1

tar -xf zookeeper-3.4.6.tar.gz

Configure Zookeeper

1
2
3
4
5
6
7
8


tickTime=2000
dataDir=/var/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
server.1=zookeeper1:2888:3888
server.2=zookeeper2:2888:3888
server.3=zookeeper3:2888:3888

Start Zookeeper
1

bin/zkServer.sh start
Test
1

bin/zkCli.sh -server 127.0.0.1:2181
Dependency installation
Java

Pre-installed.
Python

Pre-installed.
Storm
Download the package, upload, and extract

Extract command:
1

tar -xf apache-storm-0.9.5.tar.gz
Configure the Zookeeper address in storm.yaml
1 2

storm.zookeeper.servers: - "127.0.0.1"
Create and configure the workdir
1

storm.local.dir: "/home/admin/storm/workdir"
Note: storm.local.dir is the local disk directory used by Nimbus and Supervisor processes to store a small amount of state (such as jars, confs, etc.). This directory must be created in advance with sufficient access permissions.
1

nimbus.host: "127.0.0.1"
1 2 3 4 5

supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703

Start all Storm services

Nimbus

1

bin/storm nimbus >/dev/null 2>&1 &

Supervisor

1

bin/storm supervisor >/dev/null 2>&1 &

1

bin/storm ui >/dev/null 2>&1 &

Logview

1

bin/storm logviewer > /dev/null 2>&1

Access Storm UI

http://ip:8080

[Hadoop] Setting Up the CDH Environment

Mon, 28 Sep 2015 14:50:42 +0000

1. Connect server

Use CRT or Xshell to connect to the server remotely.

2. Network config

First, configure DNS for the server and check whether it can access the internet.

Enter the following command:

1

ping [www.baidu.com][1]

If you receive a reply from [www.baidu.com][1], it means the server can access the internet. If you get “unknown host,” the DNS configuration is incorrect and needs to be set up.

There are two ways to configure DNS:

Temporary solution: edit the resolv.conf file and add “nameserver 8.8.8.8”.
1 2

vi /etc/resolv.conf nameserver 8.8.8.8
Permanent solution: edit the ifcfg_eth0 file and add “DNS1=8.8.8.8”.
1 2

vi /etc/sysconfig/network-scripts/ifcfg-eth0 DNS1=8.8.8.8

After configuration, run the command “service network restart”.

The second approach is recommended, as the first one will be lost when the server restarts.

Disable the firewall

Enter the command:

1

service iptables stop

3. File transfer

During the installation process, you may need to upload local files to the server. Here is one approach.

Enter the command:

1

sudo yum install lrzsz

After installing the lrzsz package, enter:

1

rz

Then select the file you want to upload.

Yum (Yellow dog Updater Modified) is an RPM-based package manager that can automatically download and install RPM packages from specified servers, handling dependency relationships automatically and installing all dependent packages at once, eliminating the tedious process of downloading and installing them one by one.

4. Install CDH

HDFS: A distributed file storage system, the open-source Java implementation of GFS.

MapReduce: A parallel computing programming model for large-scale datasets.

HBase: NoSQL column database

Hive: Data warehouse

Zookeeper: Distributed lock service

Pig: Big data analysis platform interface

Home link: http://www.cloudera.com/content/cloudera/en/products-and-services/cdh.html

Setup:

1. Java

CentOS 6.7 comes with JDK 1.7, so this step can be skipped. Any JDK-related steps during installation can be ignored.

2. Cloudera Manager Server

Cloudera Manager makes it easy to manage Hadoop deployments of any scale in production. Quickly deploy, configure, and monitor your cluster through an intuitive UI – complete with rolling upgrades, backup and disaster recovery, and customizable alerting.

Download:

http://www.cloudera.com/content/cloudera/en/downloads/cloudera_manager/cm-5-4-7.html

Download the latest version 5.4.7.

After downloading, you will get a file of about 503KB named cloudera-manager-installer.bin. Upload this file (using “rz”) to a designated folder on the server.

Close selinux:

1
2


/etc/selinux/config
selinux=disabled

Hostname

You need to modify the hostname of the machine. To check the current hostname, simply type hostname.

Enter:
1

vi /etc/hosts
Edit the file and add the line “192.168._._ master.com master”.

Note: when adding datanode nodes, do not name them “master” – use names like “datanode1” instead.

Run the “reboot” command to restart the server.

3. CDH

Using cloudera manager

Navigate to the directory containing cloudera-manager-installer.bin and run:

1
2


chmod u+x cloudera-manager-installer.bin
./ cloudera-manager-installer.bin

Keep clicking next. The selected items will be highlighted in bold.

This may take a while, so please be patient…

Choose express free version.
Display the packages to be installed
Specify CDH cluster hosts
Install CDH
Enter username and password
Install the specified Parcel
Select services

4. Storm

A distributed real-time computation system for processing high-speed, large-volume data streams. It adds reliable real-time data processing capabilities to Hadoop.

5. Spark

Spark uses in-memory computing. Starting from multi-iteration batch processing, it allows data to be loaded into memory for repeated queries. It also integrates multiple computing paradigms such as data warehousing, stream processing, and graph computing. Spark is built on top of HDFS and integrates well with Hadoop.

6. Problems

Refreshing metadata packages

1
2


Another app is currently holding the yum lock; waiting for it to exit...
rm -f /var/run/yum.pid

Uninstall Cloudera Manager

1

$ sudo /usr/share/cmf/uninstall-cloudera-manager.sh

Installation failed: Unable to receive heartbeat from Agent

1
2
3
4
5


cat /etc/hosts
ip 域名 主机名
192.168.*.1 master.com master
cat /etc/sysconfig/network
HOSTNAME=master.com

Uninstall Cloudera Manager

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


sudo rm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera*
sudo /usr/share/cmf/uninstall-cloudera-manager.sh
sudo service cloudera-scm-server stop
sudo service cloudera-scm-server-db stop
cloudera-manager-server-db
sudo yum remove cloudera-manager-server
sudo yum remove cloudera-manager-server-db-2
sudo service cloudera-scm-agent hard_stop
sudo yum remove 'cloudera-manager-*'
sudo yum clean all
sudo rm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera* /var/log/cloudera* /var/run/cloudera*
sudo rm -Rf /var/cache/apt/archives/cloudera*
sudo rm /tmp/.scm_prepare_node.lock
sudo rm -Rf /var/lib/flume-ng /var/lib/hadoop* /var/lib/hue /var/lib/navigator /var/lib/oozie /var/lib/solr /var/lib/sqoop* /var/lib/zookeeper
sudo rm -Rf /dfs /mapred /yarn