Big Data

[Storm] Installing Storm

A distributed real-time computation system for processing high-speed, large-volume data streams. It adds reliable real-time data processing capabilities to Hadoop.

Home link: http://storm.apache.org/

Download link: http://storm.apache.org/downloads.html

Setup

  1. Java

CentOS comes with JDK 1.7, so this step can be skipped.

  1. Zookeeper

Home link: http://zookeeper.apache.org/releases.html

  1. Download the package, upload, and extract

    http://mirror.bit.edu.cn/apache/zookeeper/stable/

    1

    tar -xf zookeeper-3.4.6.tar.gz

  2. Configure Zookeeper

    1
2
3
4
5
6
7
8

    tickTime=2000
dataDir=/var/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
server.1=zookeeper1:2888:3888
server.2=zookeeper2:2888:3888
server.3=zookeeper3:2888:3888

  3. Start Zookeeper

    1

    bin/zkServer.sh start

  4. Test

    1

    bin/zkCli.sh -server 127.0.0.1:2181

  5. Dependency installation

  6. Java

    Pre-installed.

  7. Python

    Pre-installed.

  8. Storm

  9. Download the package, upload, and extract

    Extract command:

    1

    tar -xf apache-storm-0.9.5.tar.gz

  10. Configure the Zookeeper address in storm.yaml

    1
2

    storm.zookeeper.servers:
  - "127.0.0.1"

    Create and configure the workdir

    1

    storm.local.dir: "/home/admin/storm/workdir"

    Note: storm.local.dir is the local disk directory used by Nimbus and Supervisor processes to store a small amount of state (such as jars, confs, etc.). This directory must be created in advance with sufficient access permissions.

    1

    nimbus.host: "127.0.0.1"

    1
2
3
4
5

    supervisor.slots.ports:
  - 6700
  - 6701
  - 6702
  - 6703

  11. Start all Storm services

    Nimbus

    1

    bin/storm nimbus >/dev/null 2>&1 &

    Supervisor

    1

    bin/storm supervisor >/dev/null 2>&1 &

    UI

    1

    bin/storm ui >/dev/null 2>&1 &

    Logview

    1

    bin/storm logviewer > /dev/null 2>&1

Access Storm UI

http://ip:8080