System Software on Coinidea's Blog

AZKABAN - Open Source Task Scheduling System

Sat, 15 Jun 2019 00:47:43 +0000

Azkaban is an open-source task scheduling system used for scheduling and running tasks (such as data warehouse scheduling), serving as a replacement for crontab in Linux. Official website: https://azkaban.github.io/

Azkaban mainly consists of three components:

MySQL: Azkaban uses MySQL to store projects and execution information.
Azkaban Web Server: Azkaban uses Jetty as the web server, serving as the controller and providing the web interface.
Azkaban Executor Server: The Azkaban executor server executes submitted workflows.

This article primarily uses Azkaban 3.43. If you need help generating compiled tar packages from GitHub source code, feel free to leave a comment, and I can share the installation package files that I have already tested.
The installation package files mainly include:

azkaban-db-3.43.0.tar.gz
azkaban-solo-server-3.43.0.tar.gz
azkaban-exec-server-3.43.0.tar.gz
azkaban-web-server-3.43.0.tar.gz
azkaban-hadoop-security-plugin-3.43.0.tar.gz
Among these, solo is the standalone version. I set up a version with multiple executor nodes and a single web server node.

Note: This article assumes the Azkaban directory is located at /azkaban.

Database
First, you need to install the MySQL database, then create a database called azkaban. Execute the following in MySQL:

source /azkaban/azkaban-db/create-all-sql-0.1.0-SNAPSHOT.sql

Configuration Files
0. Configure the keystore. The keystore is located at /azkaban/azkaban-web/conf/ (the path must match the one in the configuration file).

keytool -keystore keystore -alias jetty -genkey -keyalg RSA
Enter keystore password: azkaban
Re-enter new password: azkaban
What is your first and last name? [Unknown]: Skip
What is the name of your organizational unit? [Unknown]: Skip
What is the name of your organization? [Unknown]: Skip
What is the name of your City or Locality? [Unknown]: Skip
What is the name of your State or Province? [Unknown]: Skip
What is the two-letter country code for this unit? [Unknown]: CN
Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=CN correct?
[no]: Y
Enter key password for  (RETURN if same as keystore password):

/azkaban/azkaban-web/conf/azkaban.properties

# Azkaban Personalization Settings
azkaban.name=Allin
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/azkaban/azkaban-web-server/web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/azkaban/azkaban-web-server/conf/azkaban-users.xml
# Loader for projects
executor.global.properties=/azkaban/azkaban-web-server/conf/global.properties
azkaban.project.dir=/azkaban/azkaban-web-server/projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8443
jetty.keystore=//azkaban/azkaban-web-server/conf/keystore
jetty.password=yourpassword
jetty.keypassword=yourpassword
jetty.truststore=//azkaban/azkaban-web-server/conf/keystore
jetty.trustpassword=yourpassword
# Azkaban Executor settings
executor.port=12321
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=/azkaban/azkaban-web-server/plugins/jobtypes
database.type=mysql
mysql.port=3306
mysql.host=127.0.0.1
mysql.database=azkaban
mysql.user=root
mysql.password=yourmysqlpassword
mysql.numconnections=100

/azkaban/azkaban-exec/conf/azkaban.properties

# Azkaban Personalization Settings
default.timezone.id=Asia/Shanghai
# Loader for projects
executor.global.properties=/azkaban/azkaban-exec-server/conf/global.properties
azkaban.project.dir=/azkaban/azkaban-exec-server/projects
azkaban.jobtype.plugin.dir=/azkaban/azkaban-exec-server/plugins/jobtypes
database.type=mysql
mysql.port=3306
mysql.host=127.0.0.1
mysql.database=azkaban
mysql.user=root
mysql.password=yourpassword
mysql.numconnections=100
# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30

/azkaban/azkaban-web/conf/log4j.properties /azkaban/azkaban-exec/conf/log4j.properties

log4j.rootLogger=INFO,C
log4j.appender.C=org.apache.log4j.ConsoleAppender
log4j.appender.C.Target=System.err
log4j.appender.C.layout=org.apache.log4j.PatternLayout
log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

For multiple executor nodes, you need to add the following configuration to /azkaban/azkaban-web/conf/azkaban.properties, and insert the IP addresses and port numbers of the corresponding executor nodes into the database.

azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

Insert executor node IP addresses and port numbers:

insert into executors(host,port) values("your ip1",12321);
insert into executors(host,port) values("your ip2",12321);

Configure username and password at /azkaban/azkaban-web/conf/azkaban-users.xml
With basic computer knowledge, opening this configuration file is self-explanatory.
Create log directories

mkdir /azkaban/azkaban-web/logs
mkdir /azkaban/azkaban-exec/logs

Start the services

azkaban/azkaban-exec/bin/start-exec.sh
azkaban/azkaban-web/bin/start-web.sh

Common Pitfalls

The keystore location must match the file path set in the configuration file.
Azkaban 3 and above supports multiple executor nodes.
The startup method in step 7 runs in silent mode. During initial testing, it is recommended to use:

azkaban/azkaban-exec/bin/azkaban-web-start.sh
azkaban/azkaban-exec/bin/azkaban-exec-start.sh

This way you can see whether errors occur and what causes them.
4. Based on error messages, you may not have correctly created the logs directory or the path may be incorrect.
5. Since the web server checks the exec server, it is recommended to start the exec server first.
6. If tasks are not executing in a multi-node setup, it may be because the executor nodes have insufficient resources. Check the configuration carefully. The configuration specifies under what memory, CPU, and other resource conditions the server will execute tasks, and how executor nodes are assigned if no specific node is designated. If you do not want to limit resources, modify or comment out the relevant configuration.

azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

How to specify a node to execute a job:
Set “useExecutor” = EXECUTOR_ID in the flow params.
For details, please refer to:
https://www.jianshu.com/p/ffb7bbc1988f

What to Do If You Get Hit by Ransomware

Thu, 22 Nov 2018 07:23:33 +0000

Recently, a friend of mine got hit by ransomware. After some research, I found a good website: the No More Ransom Alliance.

https://www.nomoreransom.org/zh/decryption-tools.html

I’ve always believed that technology is neutral. So if you’ve been hit by ransomware, you might want to check out the website above – there might be some solutions available.

A Great Place to Download RPM Packages

Thu, 22 Nov 2018 07:17:03 +0000

In offline environments (specifically those where you can’t use wget or yum), we often need to compile and install Linux-based operating systems ourselves.

At that point, version dependencies become a real headache. Of course, we could use Docker to solve this.

But if your offline environment also restricts uploads and downloads, and the internal system versions are unpredictable, it becomes an even bigger pain. So I found a great place to download Linux RPM packages:

https://pkgs.org/

It solves the problems of operating system compatibility, system version dependencies, and software dependency management.

Recommending a Great Overseas VPS - Vultr

Thu, 15 Nov 2018 07:17:56 +0000

Introduction to Vultr

Vultr is a well-known US cloud service provided by Choopa.com. Choopa has been providing global support services for gaming companies, so the company has deployed data centers in 14 countries and regions worldwide, including Tokyo (Japan), Singapore, Los Angeles, Seattle (USA), London (UK), Germany, and more. It offers excellent value for money, with plans starting at just $2.5/month. New users receive $25 upon registration. Click the following link to register – new users get $25 (you need to pay $10, which means you pay $10 and get $35 in credit): Promotional offer link

Advantages of Vultr

Numerous data centers: 14 data centers in Japan, the US, Europe, and more.
Excellent architecture: All using KVM architecture, SSD solid-state drives, starting from 1000GB/month traffic.
Powerful images: Besides common Linux distributions, you can also install custom ISO systems and Windows.
Powerful backend: Features including system snapshots, one-click deployment scripts, backups, firewalls, and more.
Promotions: Occasional new user registration bonuses, sometimes with rewards up to $100.
Flexible billing: Uses hourly billing mode, allowing you to add and remove machines at will.
Customer service: Fast response times, usually getting answers within 15 minutes (time zone differences may apply).

Pricing and Configuration

Markdown Anywhere

Tue, 06 Nov 2018 06:29:16 +0000

I’ve been writing articles frequently lately and found that Markdown is a very convenient web typesetting standard. For details, see: http://cesiumcn.org/markdown.html | http://cesium.coinidea.com/markdown.html

Currently, I’ve found that Jianshu and OSChina have excellent Markdown support. However, WordPress doesn’t support it, and surprisingly, Zhihu doesn’t support it either.

WordPress Markdown WordPress has plugins that can provide support. Some people online say the WP Editor.MD plugin is excellent, but after using it on my WordPress version, I found it wasn’t compatible – it crashed directly when activated.

At that moment, I had a flash of inspiration and went directly to the official site https://wordpress.org/plugins/ to look. I found that wp-markdown works quite well.

After activating it, I noticed nothing changed. It turns out you need to configure it. The specific location is: Settings -> Writing.
Zhihu Markdown Zhihu obviously can’t install plugins like WordPress. The talented developers online are truly amazing. There’s a Chrome plugin called Markdown Here. After installing it, when you right-click in a text input box, a Markdown Here menu appears, which automatically converts rich text content to Markdown formatting – just copy and paste.

[CentOS]Upgrading OpenSSH and vsftpd on CentOS

Thu, 23 Nov 2017 08:05:04 +0000

Upgrading OpenSSH to Version 7.4 or Above

Background

Currently, OpenSSH versions below 7.4 on Linux have some critical vulnerabilities, and OpenSSH needs to be upgraded to version 7.4 or above. On a machine without external network access (no yum), this is quite troublesome. Online resources generally recommend backing up SSH and installing Telnet first, so you can still access the machine via Telnet if the upgrade fails.

Common Issues and Solutions

4. Dependencies required for manually compiling OpenSSH

zlib
pam
pam-devel

If you get the error configure: error: PAM headers not found, you can download pam-devel:

http://rpm.pbone.net/index.php3?stat=26&dist=74&size=208492&name=pam-devel-1.1.1-17.el6.x86_64.rpm

5. OpenSSL download

ftp://ftp.openssl.org/source/

6. Installing OpenSSL

References:

Notes

If SecureCRT cannot log in after upgrading OpenSSH, it may be because the SecureCRT version is too old to support certain encryption protocols:

Reference: http://blog.csdn.net/yangg1991/article/details/51755562

Steps to Upgrade OpenSSH

Reference: http://blog.csdn.net/u011080082/article/details/64503534

Upgrading vsftpd from 2.2.2 to 2.3.4

Download the vsftpd-2.3.4.tar.gz package. For 64-bit systems, note:

1
2


sed -i 's/lib\/lib64\/g' vsf_findlibs.sh
sed -i 's/lib\//lib64\//g' vsf_findlibs.sh

Configuring vsftpd

Using active mode:

1

vsftpd /etc/vsftpd/vsftpd.conf &

When starting in standard_alone mode, you need to add the following two lines to the top of the configuration file:

Reference: http://wingjang.blog.163.com/blog/static/47913442200811113104509/

1
2


listen=YES
listen_port=21

After installation, only anonymous users are allowed to log in by default, so you need to modify the configuration:

Reference: http://blog.sina.com.cn/s/blog_7e16680c01018ox1.html

Handling TCP_Wrapper Issues

If there are issues, pay attention to removing spaces, or you can remove it entirely:

References:

vsftpd Download

http://pkgs.fedoraproject.org/repo/pkgs/vsftpd/

[MySQL] Scheduled Backup with MySQL-Front on Windows

Thu, 05 May 2016 03:08:32 +0000

Scheduled backup is an extremely important task in database services.
For most users who manage databases with a GUI (commonly MySQL-Front, Navicat, or MySQL Workbench), interactive backup methods are simple and easy to learn.

Here is a brief guide on how to set up scheduled backups using MySQL-Front (version 5.3):

After connecting to the database, go to “Extras” -> “Jobs” -> “New” -> “Export”

Select the test database and click Next.
Specify the export path “D:\test.sql”. Note that the character set should generally be set to UTF-8.

Next, specify the export content – make sure to check the data option.
Set the backup time and interval.

After completion, a job named “test” will appear in the scheduled tasks section.
If you want to run it immediately, right-click and select Run.

[Git] Configuring and Removing Git Proxy

Sat, 09 Apr 2016 08:02:47 +0000

When configuring Git to use a proxy, you can use the following commands to set and remove the proxy:

Configuring Git Proxy

To configure HTTP and HTTPS proxies for Git, use the following commands:

1
2
3
4
5


# Configure HTTP proxy
git config --global http.proxy http://127.0.0.1:1080

# Configure HTTPS proxy
git config --global https.proxy https://127.0.0.1:1080

Removing Git Proxy Configuration

To remove the proxy settings, use the following commands:

1
2
3
4
5


# Remove HTTP proxy
git config --global --unset http.proxy

# Remove HTTPS proxy
git config --global --unset https.proxy

Explanation

git config --global http.proxy <proxy-url>: Configure the global HTTP proxy.
git config --global https.proxy <proxy-url>: Configure the global HTTPS proxy.
git config --global --unset http.proxy: Remove the global HTTP proxy configuration.
git config --global --unset https.proxy: Remove the global HTTPS proxy configuration.

With these commands, you can easily enable or disable Git proxy settings as needed.

[Linux] Summary of Linux Commands

Sat, 17 Oct 2015 08:24:07 +0000

User Management

Recently I have been setting up a CDH cluster of about 10 nodes. The cluster uses 10 CentOS 6.7 servers, so I frequently need to use Linux user management commands.

Change password
1

passwd
Enter the new password and confirm it. For security, the password should include uppercase and lowercase letters, numbers, and special characters. Linux will also remind you of the password rules.
Switch user
1

su hdfs
This switches to the hdfs user.
Alternatively, without switching users, you can execute a command or access a file with a specific user’s permissions:
1

su hdfs hadoop fs -mkdir /user/hdfs/test
This creates a test directory at /user/hdfs/ on the cluster using hdfs user permissions.

File Management

Create a directory

1

mkdir [folder]

Create a file

1

touch/vi [filename]

Timezone Selection

Follow the prompts to select a timezone, then enter the suggested command.
1

tzselect

Modify the timezone in the clock file.

1
2


vi /etc/sysconfig/clock
ZONE="Asia/Shanghai"

Delete the existing localtime and create a new one.

1
2


rm /etc/localtime
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

Run Commands at Startup

Edit the rc.local file

1

vi /etc/rc.local

Start the NTP Service to Sync with a Time Server on the LAN

1
2


service ntpd start
/usr/sbin/ntpdate 192.168.34.115

[To be continued…]

[Linux] Configuring DNS Addresses on Linux

Wed, 14 Oct 2015 13:28:18 +0000

DNS determines whether a server can reach the internet via domain names. There are two ways to configure DNS server IP addresses on Linux.

Verification method:

1

ping www.baidu.com

Temporary method
a) Edit resolv.conf
1

vi ./etc/resolv.conf
b) Enter
1

nameserver 8.8.8.8
Save and you’re done.

Permanent method
a) Edit ifcfg-eth0

1

vi /etc/sysconfig/network-scripts/ifcfg-eth0

b) Enter

1

DNS1=8.8.8.8

Save and you’re done.

You can use

1

ifconfig

to view the local network interfaces, where you can see the eth0 configuration information.

[Linux] Mounting an NTFS External Hard Drive on CentOS

Thu, 01 Oct 2015 06:26:08 +0000

The server runs CentOS and needs to mount two 2TB external hard drives.

The command is as follows:

1
2


mount -t ntfs /dev/sdn1 /test1
mount： unknown filesystem type 'ntfs'

An error occurred, meaning the system does not recognize “NTFS.” The solution is as follows:

1
2
3
4
5
6
7
8


wget http://tuxera.com/opensource/ntfs-3g_ntfsprogs-2013.1.13.tgz
tar zxvf ntfs-3g_ntfsprogs-2013.1.13.tgz
cd ntfs-3g_ntfsprogs-2013.1.13
./configure
make
make install
mkdir /mnt/window
mount -t ntfs-3g /dev/sdn1 /mnt/window

To unmount:

1

umount /dev/sdn1

[Outlook] Outlook Configuration and General Settings — Outlook 2013

Thu, 01 Oct 2015 03:13:11 +0000

Once you enter the workplace, Outlook becomes an essential tool for office workers. This article explains some configuration and general settings for Outlook (Outlook 2013).

Adding a User Account

Open Outlook 2013, click “File” in the upper left corner, and in the dropdown list, click “+ Add Account.”

Select “Email Account” and click “Next.”

Click “Manual setup or additional server types,” then click “Next.”

Check “POP or IMAP,” then click “Next.” This is generally the option to choose.

Enter your username and password, and configure the incoming and outgoing server addresses. Note that these two addresses are typically pop3 or smtp@corresponding-domain, which you need to confirm with your email service provider. The provider usually has documentation for this.

Click “More Settings” in the image above, configure the outgoing server username and password, and click “Finish.”

Click Next to start the test. After the test completes, the entire process is successful.

Configuring Email Filter Rules

During daily email usage, you may receive a large volume of emails. Efficiently sorting these emails is a very meaningful task.

Open Outlook 2013, click “File” in the upper left corner, and in the dropdown list, click “Manage Rules & Alerts.”

Click “New Rule.”

There are many options in the rules. Configure them according to the rule wizard. For example, you can select “with specific words in the subject” or “from a specific sender,” and then move the emails to a designated folder or specify a particular action.

[Software] Creating GIF Images Locally

Sun, 20 Sep 2015 08:37:13 +0000

Because I open-sourced a WIN32 Snake game, I wanted to create a GIF for demonstration purposes, but figuring out how to actually make one turned out to be tricky.

After some research, here are several methods:

Method 1: Using video software like QQ Player

The prerequisite is that you have a short video clip to convert. You can use QQ Player’s built-in feature to do the conversion.

Detailed instructions:

http://kf.qq.com/faq/120322fu63YV130422iiEbMB.html

Method 2: Using professional software like Photoshop

Prepare multiple frames of images. See this Baidu tutorial for detailed instructions:

http://jingyan.baidu.com/article/7f41ececdb326e593d095c2e.html

Method 3: Using Screen Recording Expert

The image above was created using Screen Recording Expert. First, I recorded a video (in EXE or LXE format), then selected the “Conversion Tools” menu, and converted it from there.

Here is a download link for a portable version of Screen Recording Expert:

Screen Recording Expert.

Cleaning Your Computer with CCleaner

Sun, 21 Jun 2015 07:33:06 +0000

A major life transition is coming - graduation. There’s a computer in the lab that’s been mine to use, and I wanted to give it a thorough clean before I leave. I heard about a tool called CCleaner, tried it out, and it works quite well. It can conveniently clean up Windows | Mac | Android | iPhone devices.

On Windows, you can perform:

Windows and application cleanup
Registry cleanup
Software uninstallation and disk wiping

Adding Images to README.md on GitHub

Thu, 12 Feb 2015 12:42:27 +0000

I had been using SVN before and only truly started using Git a few months ago. I had briefly used GitHub during KDD2013.

Recently, I wanted to polish my programs properly – comments and descriptions are essential and should be written with care.

I encountered a problem: how to add images to a README.md file.

Here is the solution I used:

Create an image folder within your project folder, place the images you want to display in the README.md, and commit that folder to GitHub. Through the preview, you can obtain the image’s URL on GitHub.

README.md supports the following image markup: <img data-original="image-url" src="http://blog.coinidea.com/wp-content/themes/9IPHP/images/lazy_loading.gif" alt="image" />

Example:

https://github.com/hujiulin/WordConverter