大数据 on Coinidea的博客

[vsftpd]vsftpd被动模式配置

Wed, 10 Apr 2019 02:56:05 +0000

vsftpd.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140


# Example config file /etc/vsftpd/vsftpd.conf
#
# The default compiled in settings are fairly paranoid. This sample file
# loosens things up a bit, to make the ftp daemon more usable.
# Please see vsftpd.conf.5 for all compiled in defaults.
#
# READ THIS: This example file is NOT an exhaustive list of vsftpd options.
# Please read the vsftpd.conf.5 manual page to get a full idea of vsftpd's
# capabilities.
#
# Allow anonymous FTP? (Beware - allowed by default if you comment this out).
anonymous_enable=NO
#
# Uncomment this to allow local users to log in.
# When SELinux is enforcing check for SE bool ftp_home_dir
local_enable=YES
#
# Uncomment this to enable any form of FTP write command.
write_enable=YES
#
# Default umask for local users is 077. You may wish to change this to 022,
# if your users expect that (022 is used by most other ftpd's)
local_umask=022
#
# Uncomment this to allow the anonymous FTP user to upload files. This only
# has an effect if the above global write enable is activated. Also, you will
# obviously need to create a directory writable by the FTP user.
# When SELinux is enforcing check for SE bool allow_ftpd_anon_write, allow_ftpd_full_access
#anon_upload_enable=YES
#
# Uncomment this if you want the anonymous FTP user to be able to create
# new directories.
#anon_mkdir_write_enable=YES
#
# Activate directory messages - messages given to remote users when they
# go into a certain directory.
dirmessage_enable=YES
#
# Activate logging of uploads/downloads.
xferlog_enable=YES
#
# Make sure PORT transfer connections originate from port 20 (ftp-data).
connect_from_port_20=YES
#
# If you want, you can arrange for uploaded anonymous files to be owned by
# a different user. Note! Using "root" for uploaded files is not
# recommended!
#chown_uploads=YES
#chown_username=whoever
#
# You may override where the log file goes if you like. The default is shown
# below.
#xferlog_file=/var/log/xferlog
#
# If you want, you can have your log file in standard ftpd xferlog format.
# Note that the default log file location is /var/log/xferlog in this case.
xferlog_std_format=YES
#
# You may change the default value for timing out an idle session.
#idle_session_timeout=600
#
# You may change the default value for timing out a data connection.
#data_connection_timeout=120
#
# It is recommended that you define on your system a unique user which the
# ftp server can use as a totally isolated and unprivileged user.
#nopriv_user=ftpsecure
#
# Enable this and the server will recognise asynchronous ABOR requests. Not
# recommended for security (the code is non-trivial). Not enabling it,
# however, may confuse older FTP clients.
#async_abor_enable=YES
#
# By default the server will pretend to allow ASCII mode but in fact ignore
# the request. Turn on the below options to have the server actually do ASCII
# mangling on files when in ASCII mode.
# Beware that on some FTP servers, ASCII support allows a denial of service
# attack (DoS) via the command "SIZE /big/file" in ASCII mode. vsftpd
# predicted this attack and has always been safe, reporting the size of the
# raw file.
# ASCII mangling is a horrible feature of the protocol.
#ascii_upload_enable=YES
#ascii_download_enable=YES
#
# You may fully customise the login banner string:
ftpd_banner=Welcome to blah FTP service.
#
# You may specify a file of disallowed anonymous e-mail addresses. Apparently
# useful for combatting certain DoS attacks.
#deny_email_enable=YES
# (default follows)
#banned_email_file=/etc/vsftpd/banned_emails
#
# You may specify an explicit list of local users to chroot() to their home
# directory. If chroot_local_user is YES, then this list becomes a list of
# users to NOT chroot().
# (Warning! chroot'ing can be very dangerous. If using chroot, make sure that
# the user does not have write access to the top level directory within the
# chroot)
chroot_local_user=YES
chroot_list_enable=NO
# (default follows)
chroot_list_file=/etc/vsftpd/chroot_list
#
# You may activate the "-R" option to the builtin ls. This is disabled by
# default to avoid remote users being able to cause excessive I/O on large
# sites. However, some broken FTP clients such as "ncftp" and "mirror" assume
# the presence of the "-R" option, so there is a strong case for enabling it.
#ls_recurse_enable=YES
#
# When "listen" directive is enabled, vsftpd runs in standalone mode and
# listens on IPv4 sockets. This directive cannot be used in conjunction
# with the listen_ipv6 directive.
listen=YES
#
# This directive enables listening on IPv6 sockets. By default, listening
# on the IPv6 "any" address (::) will accept connections from both IPv6
# and IPv4 clients. It is not necessary to listen on *both* IPv4 and IPv6
# sockets. If you want that (perhaps because you want to listen on specific
# addresses) then you must run two copies of vsftpd with two configuration
# files.
# Make sure, that one of the listen options is commented !!
listen_ipv6=NO

pam_service_name=vsftpd
userlist_enable=YES
tcp_wrappers=YES
local_root=/ftp
anon_root=/ftp
use_localtime=YES
user_config_dir=/etc/vsftpd/userconfig
pasv_enable=yes
pasv_min_port=41000
pasv_max_port=42000
allow_writeable_chroot=YES
listen_port=21
xferlog_file=/var/log/vsftpd.log
pasv_address=your_ip
pasv_addr_resolve=yes
pasv_promiscuous=YES

user_list

1

# 该文件全部注释

chroot_list

1
2


ftpuser
ftpsubuser

/etc/vsftpd/userconfig/ftpsubuser

1

local_root=/ftp/ftpsubuser/

需要执行的命令：

1
2
3
4
5


sudo useradd ftpuser -d /ftp -s /sbin/nologin -m
passwd ftpuser

sudo useradd ftpsubuser -d /ftp/ftpsubuser -s /sbin/nologin -m
passwd ftpsubuser

[Docker]Docker常用命令备忘

Wed, 10 Apr 2019 02:45:13 +0000

最近做的一个项目，涉及到MongoDB集群、Spark集群，Hadoop集群，Python2.7，Python3.5，各种推荐算法，Java Resin环境，LNMP环境、vsftpd等，因为需要部署内网环境、测试环境、生产环境，每次部署就是一次折磨，这个时候Docker的好处就体现出来了，把Python环境、LNMP环境、vsftpd等，全部打入docker中，特别爽歪歪。经过2个月的部署，联调也整理了一些docker的实用命令。
系统环境以CentOS 7为例

下载安装docker并启动服务
https://download.docker.com/linux/centos/7/x86_64/stable/Packages/
下载 docker-ce-18.03.1.ce-1.el7.centos.x86_64.rpm
1 2

yum install docker-ce-18.03.1.ce-1.el7.centos.x86_64.rpm service docker start

docker的常用命令
解决报错WARNING: IPv4 forwarding is disabled. Networking will not work.

第一步：在宿主机上执行 echo "net.ipv4.ip_forward=1" >>/usr/lib/sysctl.d/00-system.conf

第二步：重启network和docker服务

1

[root@localhost /]# systemctl restart network && systemctl restart docker

Failed to get D-Bus connection: Operation not permitted

1

docker run -d --name centos7 --privileged=true --net=host -v /home/code:/home/code centos:mysql_python3_jdk1.8 /usr/sbin/init

长时间运行docker容器，并装载宿主机路径

1

docker run -dit -v /home/code:/home/code python35:latest /bin/bash

进入已经启动的docker容器

1

docker exec -it 容器id /bin/bash

启动某个容器镜像

1

docker run -it python35:latest /bin/bash

Docker导出：

1

docker export -o /home/python35.tar 容器id

docker save *.tar

Docker导入：

1

docker import /home/python35.tar

docker load < *.tar

Docker 打标签

1

docker tag 镜像id REPOSITORY:TAG

Docker 指定网络为宿主机网络

1

docker run -dit --net=host python35:latest /bin/bash

Docker 指定容器和宿主机时间同步

1

docker run -dit -v /etc/localtime:/etc/localtime python35:latest /bin/bash

安装JDK1.8

1

yum install java-1.8.0-openjdk* -y

安装MySQL5.6

1
2
3
4
5
6
7
8


rpm -Uvh http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm
yum -y install mysql-community-server
systemctl enable mysqld
systemctl start mysqld
初始化：
mysql_secure_installation
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '远程登录密码' WITH GRANT OPTION;
FLUSH PRIVILEGES;

Python3编码

1

echo 'export LANG=zh_CN.UTF-8' >> ~/.bashrc

[ElasticSearch]PHP操作ElasticSearch

Thu, 04 Jan 2018 09:04:18 +0000

官方API:
可以下载官方的PHP ElasticSearch API 阅读文档进行相关操作。
示例代码：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


static public function initSearchClient() {
 if (null == BaseFrontController::$client) {
 $hosts['hosts'] = array(
 "host" => '127.0.0.1',
 "port" => '9200',
 'scheme' => 'http'
 );
 BaseFrontController::$client = \Elasticsearch\ClientBuilder::create()
 ->setSSLVerification(false)
 ->setHosts($hosts)
 ->build();
 }
}

// Elastic search
static public function actionSearch($area, $keyword, $start, $end, $size=100, $agg = null) {
 BaseFrontController::initSearchClient();
 $params['index'] = BaseFrontController::getSearchIndex();
 $params['type'] = 'raw';
 $params['body']["size"] = $size;
 // your aggs
 if (null != $agg) {
 $params['body']["aggs"] = $agg;
 }
 $params['body']["query"]["filtered"]['filter']["and"] = [
 ["term"=>['Your area' => $area]], // your area
 ["range"=>["Your time" => array("gte"=>$start, "lte"=>$end)]] // your time
 ];

 if ($keyword != '') {
 $params['body']["query"]["filtered"]['query']['bool']['should']['match_phrase'] = array();// your query word
 }

 $response = BaseFrontController::$client->search($params);
 return $response;
}

[ElasticSearch]ES修改mapping

Sun, 19 Nov 2017 12:23:18 +0000

用太多的MySQL这样的数据库了，直到有一天，用了ES遇到一个大坑。
就是post mapping的时候有一个“字段”analyzed 和 not_analyzed没区分好，一时失误导致该列所有数据全部分词了。数据量大概1.5亿条。
天真的以为能够像MySQL那样修改一下字段的属性即可。ES是基于Lucene的，没有别的办法，通俗一点讲，要么删除索引，重新导入，要么reindex。所谓的reindex就是建立一个新的index，把旧index的数据拷贝过去。这样的教程网上很多。比如：
http://blog.csdn.net/loveyaqin1990/article/details/77684599
https://www.cnblogs.com/wmx3ng/p/4112993.html

目前网上来讲，具体实现代码很少，我找了好久只找到了Python的实现。本文基于ES官方代码的PHP SDK和bulk有一个迁移实现。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58


<?php
require 'vendor/autoload.php';
$hosts['hosts'] = array(
 "host" => '127.0.0.1',
 "port" => '9200',
 'scheme' => 'http'
);
$client = Elasticsearch\ClientBuilder::create()
 ->setSSLVerification(false)
 ->setHosts($hosts)
 ->build();
for ($i = 1; $i <= 10; $i++) {
 if ($i != 10) {
 $params['index'] = 'index-0'.$i;
 }
 else {
 $params['index'] = 'index-'.$i;
 }
 echo $params["index"]."\r\n";
 $params['type'] = 'raw';
 $params['scroll'] = '120s';
 $params["size"] = 50000;
 $params["body"] = array(
 "query" => array(
 "match_all" => array()
 )
 );
 $response = $client->search($params);
 $step = 1;
 while (isset($response['hits']['hits']) && count($response['hits']['hits']) > 0) {
 echo $step++."\t";
 $scroll_id = $response['_scroll_id'];
 unset($response);
 $response = $client->scroll(
 array(
 "scroll_id" => $scroll_id,
 "scroll" => "120s"
 )
 );
 if (count($response['hits']['hits']) > 0) {
 $bulk = array('index'=>$params['index']."-reindex",'type'=>$params['type']);
 foreach ($response["hits"]["hits"] as $key=>$val) {
 $bulk['body'][]=array(
 'index' => array(
 '_id'=>$val['_id']
 ),
 );
 $bulk['body'][] = $val['_source'];
 }
 // insert reindex
 $res = $client->bulk($bulk);
 unset($bulk);
 } else {
 break;
 }
 }
}
?>

[Java]Java通过JDBC连接运行Hive

Thu, 28 Apr 2016 07:11:02 +0000

Hadoop的版本繁多，一旦版本不对Hive的连接和操作可能会报各种各样的错误，这个时候需要用到上一篇博文提到的Maven来管理这些lib。

Hadoop 以及 Hive 的版本查看：

1
2


hadoop version
hive --version

pom 的配置如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>com.coinidea</groupId>
 <artifactId>hive</artifactId>
 <packaging>war</packaging>
 <version>1.0-SNAPSHOT</version>
 <name>hive.web Maven Webapp</name>
 <url>http://maven.apache.org</url>
 <properties>
 <hadoop.version>2.5.0-cdh5.3.2</hadoop.version>
 <hive.version>0.13.1-cdh5.3.2</hive.version>
 </properties>
 <dependencies>
 <!--hadoop和hive-->
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-jdbc</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-common</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-exec</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-metastore</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hive</groupId>
 <artifactId>hive-service</artifactId>
 <version>${hive.version}</version>
 </dependency>
 <dependency>
 <groupId>org.apache.hadoop</groupId>
 <artifactId>hadoop-common</artifactId>
 <version>${hadoop.version}</version>
 <exclusions>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jetty</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jetty-util</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jsp-2.1</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>jsp-api-2.1</artifactId>
 </exclusion>
 <exclusion>
 <groupId>org.mortbay.jetty</groupId>
 <artifactId>servlet-api-2.1</artifactId>
 </exclusion>
 <exclusion>
 <groupId>javax.servlet</groupId>
 <artifactId>servlet-api</artifactId>
 </exclusion>
 <exclusion>
 <groupId>javax.servlet.jsp</groupId>
 <artifactId>jsp-api</artifactId>
 </exclusion>
 <!--tomcat7中自带jasper.jar-->
 <exclusion>
 <groupId>tomcat</groupId>
 <artifactId>jasper-compiler</artifactId>
 </exclusion>
 <exclusion>
 <groupId>tomcat</groupId>
 <artifactId>jasper-runtime</artifactId>
 </exclusion>
 </exclusions>
 </dependency>
 <dependency>
 <groupId>cglib</groupId>
 <artifactId>cglib-nodep</artifactId>
 <version>2.2.2</version>
 </dependency>
 </dependencies>
 <build>
 <finalName>hive</finalName>
 </build>
 <repositories>
 <repository>
 <id>cloudera</id>
 <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
 <releases>
 <enabled>true</enabled>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </repository>
 <repository>
 <id>people.apache.snapshots</id>
 <url>http://repository.apache.org/content/groups/snapshots-group/</url>
 <releases>
 <enabled>false</enabled>
 </releases>
 <snapshots>
 <enabled>true</enabled>
 </snapshots>
 </repository>
 </repositories>
</project>

新建 HiveTest.java

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


package com.coinidea;

import java.sql.*;

public class HiveTest {
 /* Start server: hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000 */
 private static String driverName = "org.apache.hive.jdbc.HiveDriver";

 /**
 * @param args
 * @throws SQLException
 */
 public static void main(String[] args) throws SQLException {
 try {
 Class.forName(driverName);
 } catch (ClassNotFoundException e) {
 e.printStackTrace();
 System.exit(1);
 }
 //replace "hive" here with the name of the user the queries should run as
 Connection con = DriverManager.getConnection("jdbc:hive2://192.168.*.*:10000/default", "hive", "hive");
 Statement stmt = con.createStatement();
 String table_user = "test.user";
 String user_sql = "select uri from " + table_user + " limit 10";
 System.out.println("Running: " + user_sql);
 ResultSet users = stmt.executeQuery(user_sql);
 while (users.next()) {
 System.out.println("USER:" + users.getString(1));
 }
 stmt.close();
 }
}

运行 HiveTest 之前，需要启动 HiveServer2

1

hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000

[Hive]Need to specify partition columns because the destination table is partition

Tue, 23 Feb 2016 13:36:11 +0000

当向已经分区的表插入数据时，有可能会报错：

1

Need to specify partition columns because the destination table is partition

原因是需要在插入的数据中指定分区字段的数值是多少。

比如：

建表语句：

1
2
3
4
5
6
7
8
9


CREATE TABLE test (
 starttime STRING,
 endtime STRING,
 title STRING
)
PARTITIONED BY (username STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
STORED AS TEXTFILE;

1
2
3
4
5
6
7
8
9


CREATE TABLE test2 (
 starttime STRING,
 endtime STRING,
 title STRING
)
PARTITIONED BY (username STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
STORED AS TEXTFILE;

插入语句：

1
2


INSERT INTO TABLE test PARTITION(username='admin')
SELECT starttime, endtime, title FROM test2 WHERE username = 'admin';

[Hive]Hive SQL 分区间统计问题

Wed, 18 Nov 2015 07:45:26 +0000

需求：

表table中存在一个text字段，需要统计出文本长度为[0,20]、[20,40]、[40,60]…按区间划分的数量。其中，需要自动化到包含最大值max的区间。

实现：

第一种方法：

1
2
3
4
5
6
7


select count(case when length(text) between 0 and 20 then 1 end) as text1,
count(case when length(text) between 21 and 40 then 1 end) as text2,
count(case when length(text) between 41 and 60 then 1 end) as text3,
count(case when length(text) between 61 and 80 then 1 end) as text4,
count(case when length(text) between 81 and 100 then 1 end) as text5,
count(case when length(text) > 100 then 1 end) as text6
from table;

这样写有一个问题，就是必须先知道max的值，然后随着max的增大，sql语句会越来越长，而且涉及到人工的干预。

第二种方法：

1

select floor(length(text)/20) * 20, count(*) from table group by floor(length(text)/20);

这种方法，先将每个text的文本长度除以20，向下取整，再乘以20得到一个分组依据，以这个依据进行group by就可以得到最终结果，在规避循环的情况下，用简单的语句，巧妙的解决了自动分区间统计的问题。

[Storm]Storm的安装

Thu, 01 Oct 2015 05:12:26 +0000

用于处理高速、大型数据流的分布式实时计算系统。为Hadoop添加了可靠的实时数据处理功能。

Home link: http://storm.apache.org/

Download link: http://storm.apache.org/downloads.html

Setup

Java

CentOS 自带 JDK 1.7，此处可以忽略该步骤。

Zookeeper

Home link: http://zookeeper.apache.org/releases.html

下载安装包并上传解压

http://mirror.bit.edu.cn/apache/zookeeper/stable/
1

tar -xf zookeeper-3.4.6.tar.gz

配置Zookeeper

1
2
3
4
5
6
7
8


tickTime=2000
dataDir=/var/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
server.1=zookeeper1:2888:3888
server.2=zookeeper2:2888:3888
server.3=zookeeper3:2888:3888

启动Zookeeper
1

bin/zkServer.sh start
测试
1

bin/zkCli.sh -server 127.0.0.1:2181
依赖库安装
Java

自带。
Python

自带。
Storm
下载安装包，上传并解压

解压命令：
1

tar -xf apache-storm-0.9.5.tar.gz
配置Zookeeper地址，在storm.yaml中配置该目录
1 2

storm.zookeeper.servers: - "127.0.0.1"
创建并配置workdir
1

storm.local.dir: "/home/admin/storm/workdir"
备注：storm.local.dir是Nimbus和Supervisor进程用于存储少量状态（如jars、confs等）的本地磁盘目录，需要提前创建该目录并给予足够的访问权限。
1

nimbus.host: "127.0.0.1"
1 2 3 4 5

supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703

启动Storm的所有服务

Nimbus

1

bin/storm nimbus >/dev/null 2>&1 &

Supervisor

1

bin/storm supervisor >/dev/null 2>&1 &

1

bin/storm ui >/dev/null 2>&1 &

Logview

1

bin/storm logviewer > /dev/null 2>&1

访问Storm UI

http://ip:8080

[Hadoop]CDH的环境搭建

Mon, 28 Sep 2015 14:50:42 +0000

1. Connect server

使用CRT或者Xshell远程连接即可。

2. Network config

首先需要给服务器配置DNS，判断服务器是否能连接外网。

输入下列命令行即可：

1

ping [www.baidu.com][1]

如果能收到来自[www.baidu.com][1]应答报表示外网能连接上的，如果是unknown host表示DNS配置不正确需要配置DNS。

配置DNS有两种方式：

临时方案，编辑resolv.conf文件，加入“nameserver 8.8.8.8”文件。
1 2

vi /etc/resolv.conf nameserver 8.8.8.8
长期有效，编辑ifcfg_eth0文件，加入“DNS1=8.8.8.8”文件。
1 2

vi /etc/sysconfig/network-scripts/ifcfg-eth0 DNS1=8.8.8.8

配置好之后，执行命令“service network restart”

建议使用第二种方案，第一种方案当服务器重启的时候就会失效。

关闭防火墙

输入命令：

1

service iptables stop

3. File transfer

在安装的过程中，可能涉及到将本地的文件上传到服务器上。此处提供一种方案。

输入命令：

1

sudo yum install lrzsz

安装lrzsz包之后，输入：

1

rz

选择需要上传的文件即可。

其中Yum (Yellow dog Updater Modified) 是基于RPM包管理，能够从指定的服务器自动下载RPM包并且安装，可以自动处理依赖性关系，并且一次安装所有依赖的软件包，无须繁琐地一次次下载、安装。

4. Install CDH

HDFS：分布式文件存储系统 GFS的Java开源实现。

MapReduce：大规模数据集的并行计算编程模型。

HBase：NoSQL列数据库

Hive：数据仓库

Zookeeper：分布式锁

Pig：大数据分析平台接口

Home link: http://www.cloudera.com/content/cloudera/en/products-and-services/cdh.html

Setup:

1. Java

CentOS 6.7自带了1.7版本的JDK，所以此处省略。安装过程中，涉及到JDK的地方都可以忽略。

2. Cloudera Manager Server

Cloudera Manager makes it easy to manage Hadoop deployments of any scale in production. Quickly deploy, configure, and monitor your cluster through an intuitive UI – complete with rolling upgrades, backup and disaster recovery, and customizable alerting.

Download:

http://www.cloudera.com/content/cloudera/en/downloads/cloudera_manager/cm-5-4-7.html

下载最新的5.4.7版本即可。

下载之后，是一个503kb左右的，名为cloudera-manager-installer.bin的文件，将改文件上传（“rz”）到服务器指定的文件夹上。

Close selinux:

1
2


/etc/selinux/config
selinux=disabled

Hostname

需要修改本机的hostname，查看hostname的方式，直接输入hostname即可。

输入：
1

vi /etc/hosts
即可编辑，加入行“192.168._._ master.com master”

备注，添加datanode节点的时候，请不要命名为master，可以命名为datanode1。

执行“reboot”命令，重启服务器。

3. CDH

Using cloudera manager

到该文件cloudera-manager-installer.bin所在目录下执行：

1
2


chmod u+x cloudera-manager-installer.bin
./ cloudera-manager-installer.bin

一直next向下，其中选中项会加粗的。

速度可能较慢，请耐心等待…

Choose express free version.
显示随后安装的软件包
指定CDH集群主机
安装CDH
输入用户名密码
安装指定Parcel
选择服务

4. Storm

用于处理高速、大型数据流的分布式实时计算系统。为Hadoop添加了可靠的实时数据处理功能。

5. Spark

Spark采用了内存计算。从多迭代批处理出发，允许将数据载入内存作反复查询，此外还融合数据仓库，流处理和图形计算等多种计算范式。Spark构建在HDFS上，能与Hadoop很好的结合。

6. Problems

正在刷新元数据包

1
2


Another app is currently holding the yum lock; waiting for it to exit...
rm -f /var/run/yum.pid

卸载Cloudera Manager

1

$ sudo /usr/share/cmf/uninstall-cloudera-manager.sh

安装失败。无法接收 Agent 发出的检测信号

1
2
3
4
5


cat /etc/hosts
ip 域名 主机名
192.168.*.1 master.com master
cat /etc/sysconfig/network
HOSTNAME=master.com

卸载Cloudera Manager

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


sudo rm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera*
sudo /usr/share/cmf/uninstall-cloudera-manager.sh
sudo service cloudera-scm-server stop
sudo service cloudera-scm-server-db stop
cloudera-manager-server-db
sudo yum remove cloudera-manager-server
sudo yum remove cloudera-manager-server-db-2
sudo service cloudera-scm-agent hard_stop
sudo yum remove 'cloudera-manager-*'
sudo yum clean all
sudo rm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera* /var/log/cloudera* /var/run/cloudera*
sudo rm -Rf /var/cache/apt/archives/cloudera*
sudo rm /tmp/.scm_prepare_node.lock
sudo rm -Rf /var/lib/flume-ng /var/lib/hadoop* /var/lib/hue /var/lib/navigator /var/lib/oozie /var/lib/solr /var/lib/sqoop* /var/lib/zookeeper
sudo rm -Rf /dfs /mapred /yarn