MySQL on Coinidea's Blog

[LNMP]Memory Optimization

Wed, 19 Dec 2018 03:01:19 +0000

Problem Description

Recently, my server ran into an issue where it would frequently kill my Tomcat process when memory ran low. Since the overall traffic isn’t very high, LNMP and Tomcat are currently running on the same machine, but this caused Tomcat-related Java applications to become unavailable.

1

free -m

Checking the memory usage, I found the operating system had only about 60MB of memory left – almost completely used up. This situation persisted for about a month, and it was a constant source of frustration. I considered adding more memory but thought it was too expensive. When resources are insufficient, there are generally two approaches: the first is to add resources, and the second is to optimize existing resources. For someone on a budget like me, the second approach is usually my first choice.

Initially, I thought Tomcat itself or some other process was consuming too much memory. After running:

1

top

I found that php-fpm processes were often the ones consuming the most memory.

To view the top 40 memory-consuming processes:

1

ps auxw | head -1; ps auxw | sort -rn -k4 | head -40

I found that php-fpm dominated the top 40 processes, with memory usage ranging from 2% to 7%. The problem was now quite clear – the culprit was php-fpm.

Solution

You can control the number of php-fpm child processes by configuring the pm.max_children property. First, open the php-fpm configuration file:

1

vim /etc/php-fpm.d/www.conf

pm.max_children defaults to 50. Each process uses 1%-2.5% of memory, which adds up to consuming most of the memory. Try reducing the value – I set mine to 25 (also referencing other blogs). At the same time, check the following two properties:

pm.max_spare_servers : This value represents the maximum number of idle processes. If idle processes exceed this value, they will be cleaned up.
pm.min_spare_servers : The minimum number of idle processes. If idle processes fall below this value, new child processes are created.

Neither of these values should exceed the pm.max_children value. Typically, pm.max_spare_servers is set to 60%-80% of the pm.max_children value.

Finally, restart php-fpm:

1

service php-fpm restart

Check the memory again:

1

free -m

I found there was about 600MB of free memory remaining. Even after starting Tomcat, there was still about 300MB to spare.

Reference: https://www.jb51.net/article/129528.htm

1
2

Hope this helps! If you need anything else, feel free to let me know.

[MySQL]Automatic MySQL Backup

Tue, 28 Nov 2017 02:59:06 +0000

Create the backup.sh Script

Create a new backup.sh file with the following content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


db_user="Your database user"
db_passwd="Your password"
db_name="Your database name"

# Backup file storage directory — you need to change this directory
backup_dir="/home/mysqlbackup"

# Date format for backup files (dd-mm-yyyy)
time="$(date +"%Y%m%d%H%M%S")"

# Execute the backup command
mysqldump -h localhost -u $db_user -p$db_passwd --all-databases --sock=/var/lib/mysql/mysql.sock > "$backup_dir/db_$time.sql"

Then, grant execution permissions and run the script:

1
2


chmod +x ./backup.sh
bash -x ./backup.sh

If you get a socket error, you need to specify sock. See: http://blog.csdn.net/seteor/article/details/18356079.

Create a Scheduled Task

Edit the crontab:

1

crontab -e

In the editor, add the following line:

1

00 23 * * * /home/mysqlbackup/backup.sh

This command will execute backup.sh every day at 23:00, backing up the database to /home/mysqlbackup.

View scheduled tasks:

1

crontab -l

[MySQL] is marked as crashed and should be repaired

Fri, 28 Apr 2017 08:45:59 +0000

A website’s articles were not displaying. My best guess was that the database was down. After connecting, the database was accessible, but the articles table could not be opened, with the error:

Table is marked as crashed and should be repaired.

I was startled for a moment. Although I had scheduled backups, could the database really be broken? Or was there a security vulnerability?

Based on the literal meaning, I should repair the table.
After looking it up, Navicat happens to have this feature, so I tried to repair it.
Error:
mysql table is marked as crashed and last (automatic?) repair failed

The automatic repair failed.

Prepared to check the server logs
3.1 MySQL was running. Restarted it – normal, but the articles table still couldn’t be read.
3.2 Since the table uses the ISAM engine, I tried the method found online:

myisamchk -r /your/table/data/path.MYI

Still failed.
3.3 While about to check the logs, I noticed that the disk where MySQL data was stored had run out of space – 0 bytes remaining.

3.4 Obviously, after cleaning up and moving unnecessary files, I used Navicat to repair the table, and it succeeded on the first try.

It’s been a long time since I wrote a blog post. I need to keep at it.

[MySQL] Removing Carriage Returns and Line Feeds from Fields

Sat, 31 Dec 2016 06:32:25 +0000

1

update table set item=replace(replace(item, char(10), ''), char(13), '') where condition;

table: table name
item: field name
condition: query condition

[PHP] Large-Scale Data Import and Export with ThinkPHP, PHP, and MySQL

Thu, 08 Dec 2016 09:37:45 +0000

Large-Scale Data Import and Export

Note: “Large-scale” here refers to importing and exporting 100,000+ records from text files (including Office documents such as docx, xlsx).

MySQL has two very common table types: MyISAM and InnoDB. There are many comparisons of these two types online. The main difference is that InnoDB supports transactions and row-level locking, while MyISAM does not. MyISAM is relatively more efficient, and each has its own advantages and disadvantages.

Use Cases for MyISAM and InnoDB

MyISAM is suitable for:

Frequent COUNT calculations;
Infrequent inserts but frequent queries;
No transactions needed.

InnoDB is suitable for:

High reliability requirements or transaction support needed;
Frequent table updates and queries with high likelihood of table locking.

Differences Between MyISAM and InnoDB

MyISAM does not support advanced features like transactions, while InnoDB does.
MyISAM tables do not support foreign keys.
During write operations (insert, update, delete), MyISAM locks the entire table, while InnoDB locks only the affected rows.
When your database has heavy writes, updates, and deletions with relatively few queries, or when data integrity requirements are high, choose InnoDB. When your database is primarily read-heavy with relatively few updates and writes, and data integrity requirements are less strict, choose MyISAM, as its query performance is faster than InnoDB.

In summary, InnoDB is a good choice for this scenario.

Optimization Methods for Inserting Large Amounts of Data

When PHP needs to insert large amounts of data, inserting one record at a time results in severe I/O performance bottlenecks.

Demo Code:

1
2
3
4
5
6
7


$params = array('value' => '50');
set_time_limit(0);
echo date("H:i:s");
for($i = 0; $i < 2000000; $i++) {
 $connect_mysql->insert($params);
}
echo date("H:i:s");

There are two methods to solve this problem:

1. Using Transactions (InnoDB supported)

For example, you can commit a transaction every 10,000 or 100,000 records, which greatly reduces I/O.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


echo date("H:i:s");
$connect_mysql->query('BEGIN');
$params = array('value' => '50');
for($i = 0; $i < 2000000; $i++) {
 $connect_mysql->insert($params);
 if($i % 100000 == 0) {
 $connect_mysql->query('COMMIT');
 $connect_mysql->query('BEGIN');
 }
}
$connect_mysql->query('COMMIT');
echo date("H:i:s");

2. SQL Statement Concatenation

This is a MySQL feature that uses optimized SQL statements: concatenate SQL statements using insert into table () values (),(),(),() and insert them all at once. If the string is too long, you need to configure MySQL by running:

1

SET GLOBAL max_allowed_packet = 2 * 1024 * 1024 * 10;

1
2
3
4
5
6


$sql = "INSERT INTO twenty_million (value) VALUES";
for($i = 0; $i < 2000000; $i++) {
 $sql .= "('50'),";
}
$sql = substr($sql, 0, strlen($sql) - 1);
$connect_mysql->query($sql);

The first method is more commonly used, while the second is mostly for test data.

By default, PHP has a memory limit of 128MB. When ThinkPHP exceeds this memory limit, you can either increase PHP’s default maximum memory or simply remove the APP_DEBUG constant definition in the entry file or set it to false. This will save a lot of memory. Of course, you can also use transaction mode to commit every n records. ThinkPHP’s addAll function can also be used for batch inserts, though it has quite a few bugs.

Starting and Committing Transactions in ThinkPHP

1
2


$table->startTrans();
$table->commit();

When working with Office files, PHPWord and PHPExcel are recommended.

[SAE] SAE MySQL Cross-Application Authorization

Sat, 09 Apr 2016 07:50:12 +0000

As SAE (Sina App Engine) pricing continues to rise, cost savings become increasingly important. Starting March 2016, shared MySQL also began incurring charges, making it important for multiple applications to share a single MySQL instance.

Assume you have multiple applications (APP1, APP2, APP3): Only APP1 retains the shared MySQL, while APP2 and APP3 will use APP1’s MySQL.
Export APP2 and APP3’s MySQL databases as backups, then delete APP2 and APP3’s MySQL instances. This way, APP2 and APP3’s MySQL will be uninitialized and no longer incur charges.
Import APP2 and APP3’s database files (*.sql) into APP1’s MySQL. Before importing, make sure table names don’t conflict.
In APP1’s MySQL settings, click Cross-Application Authorization to select applications under your account or other accounts. Authorize APP2 and APP3 here.
Write a script to display APP1’s MySQL connection information for the authorized applications APP2 and APP3 to use (this will expose database credentials, so proceed with caution and delete the script promptly).

Related code:

1
2
3
4
5
6
7
8
9


<?php
header("Content-type:text/html;charset=utf-8");
echo "Username :".SAE_MYSQL_USER."<br>";
echo "Password :".SAE_MYSQL_PASS."<br>";
echo "Master Host:".SAE_MYSQL_HOST_M."<br>";
echo "Slave Host :".SAE_MYSQL_HOST_S."<br>";
echo "Port :".SAE_MYSQL_PORT."<br>";
echo "Database :".SAE_MYSQL_DB."<br>";
?>

Run the script: The page will display APP1’s MySQL information. Use this information in your other applications (such as APP2 and APP3) to connect to the database.

Notes

Security: Since the script exposes sensitive information such as database username and password, make sure to delete the script promptly after use to prevent information leakage.
Table name conflicts: When importing APP2 and APP3’s database files into APP1’s MySQL, ensure there are no table name conflicts. If conflicts exist, consider renaming tables before importing.
Authorization management: When performing cross-application authorization on the SAE management platform, only authorize necessary applications to avoid unnecessary permission exposure.

Through these steps, you can effectively reduce SAE MySQL costs while ensuring multiple applications can access the shared database normally.