Yeah, so it’s probably due to misconfiguration on my part but, I ran across an issue where the MySQL binary logs soaked up 83% of the disk space on my server. Of course, at first I didn’t realize that was the issue. After looking through various system logs, I moved on to the MySQL logs since I was having issues shutting down and starting MySQL. In the error log found in /var/lib/mysql, I noted that it said something about not having disk space. Wait a minute, the last time I checked I was using well under 5GB of my 20GB partition on my server. A quick ‘df -h’ confirmed that my root partition was completely full. :/
So to figure out what was using all of my hard drive space, I used this command that will list any files it finds that is over 50MB.
cd /
find . -type f -size +50000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'
And that showed me that over 2 dozen files were in the /var/lib/mysql dir, all with filenames like
mysql-bin.000001
mysql-bin.000002
mysql-bin.000003
mysql-bin.000004
...
mysql-bin.index
After doing some digging (luckily my Seeks node isn’t dependent on MySQL), I found that the log files needed flushed to reclaim this disk space. @kevingranade sent this link which summed up all my search results quite well and after running the FLUSH LOGS; and RESET MASTER; commands from the MySQL command line, I was back to having the majority of my disk space back.
So, now I’ve got to figure out why this happened and how to keep it from happening again. Also, as noted in the above linked blog post, this was just a workaround. A proper solution was needed so, off to the MySQL reference manual, specifically, the Binary Log entry.
Also, at the time of writing, the current version of MySQL in Arch is 5.5.22-1. Seeing that the first binary log files I had were dated February 2nd, I can only imagine that an update had enabled this logging by default. This is just speculation on my part though…
After doing some reading, it looks that the binary log keeps record of every transaction that changes data within your databases. This is useful for 1) larger databases where you have multiple MySQL servers replicating each other and 2) data recovery. After restoring a database, any (UPDATE, INSERT, DELETE, …) transactions that happened after the restoration point, can be executed which will bring the database completely up to date, more detail here. As far as the multiple servers master/slave setup, the slave servers can be synced via the binary log files. This way the master server will keep on working while the slave servers just (attempt to) keep up by performing the same transactions.
Regarding the above commands, the FLUSH LOGS; command, as far as the binary logs is concerned, only closes the current log file and starts a new one. It didn’t remove any files from my hard drive. The RESET MASTER; command actually removed the log files from my hard drive along with resetting the log file count back to 1. I have found that PURGE BINARY LOGS would have been perfect for me to run. This command will remove the files up to a point that you determine, either by date or up to a particular log file. This way you can keep your most recent logs. Check the link for more on using this command. Also, you don’t want to manually remove these files as MySQL keeps record of the files in the mysql-bin.index file. Manually deleting the log files will not update the index file properly. In the future, if I see that my fix below doesn’t work properly, I may set up a cron job to run the PURGE BINARY LOGS command as needed.
Seeing that I have a backup script (see below) that runs nightly, I do not need the bin log to keep data for a long period of time. There is an option to turn this logging off but, I prefer to keep it going. There is a expire_logs_days variable that can be set in your configuration file (/etc/mysql/my.cnf). However,
indicates that there may be issues… :/ I’m not sure what those issues are but, I set it to 3 days anyway. Another option that I’ve added to my configuration is max_binlog_size. The default setting for this is 1GB, which is probably fine but, I want it to rotate a little more often if it is writing that much data to the log file.
What I think may work the best for me, is within my backup script. I am using mysqldump to create the backups. There are two flags that I’ve added to my script, –flush-logs and –delete-master-logs. The flush-logs will reset the logs just as the FLUSH LOGS command above and the delete-master-logs will delete the old files up to the current file being used. The only issue with this I currently see is due to how I prefer to have my backups with the individual backups instead of one master backup. Since my script runs the mysqldump command multiple times, it flushes & deletes multiple times. The script only takes ~20 seconds to complete and I run it at a slow time of night so I doubt that much data is being lost within the logs itself.
So why did I put the #statusnet tag on this and post it to the !feds? Seeing that my StatusNet db is my most active db and when looking at the data within the binary log (phpMyAdmin has a Binary Log tab even…) I mostly see activity from SN. I suspect that other instances may (or may not) see the same. Of course, on our single server setups, the bin log may not be necessary and may even provide a small performance boost if disabled. I think there are some benefits to be had with it running so I’ll leave it going.
In the end, I did learn a little, possibly drew some crap conclusions and probably need to re-write my backup script. And as a ‘proper solution’, I’m not sure this is it but, I’m sticking with it for now. Usual disclaimers (use at your own risk, your server may implode, …) apply.
My database backup script
My script that I have set on a nightly cron job is located here: Link
This script will create individual backups of the databases in my MySQL server instance, create an dated archive of the backups and delete any backups that are older than seven days old.
References:
- http://www.cyberciti.biz/faq/find-large-files-linux/
- http://legroom.net/2008/06/29/flush-and-reset-mysql-binary-logs
- http://dev.mysql.com/doc/refman/5.5/en/binary-log.html
- http://dev.mysql.com/doc/refman/5.5/en/point-in-time-recovery.html
- http://dev.mysql.com/doc/refman/5.5/en/purge-binary-logs.html
- http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_expire_logs_days
- http://dev.mysql.com/doc/refman/5.5/en/replication-options-binary-log.html#sysvar_max_binlog_size
- http://dev.mysql.com/doc/refman/5.5/en/mysqldump.html#option_mysqldump_flush-logs
- http://dev.mysql.com/doc/refman/5.5/en/mysqldump.html#option_mysqldump_delete-master-logs
- http://bin.jpope.org/whirdmysqltookover1
