MySQL: Can't create/write to file '/tmp/#sql_3c6_0.MYI' (Errcode: 2) - What does it even mean?

Question

For some reason my production DB decided to spew out this message. All application calls fail to the DB with the error:

PreparedStatementCallback; SQL [ /*long sql statement here*/ ]; 
Can't create/write to file '/tmp/#sql_3c6_0.MYI' (Errcode: 2); 
nested exception is java.sql.SQLException: Can't create/write to file '/tmp/#sql_3c6_0.MYI' (Errcode: 2)

I have no idea, what this even means. There is no file #sql_3c6_0.MYI in /tmp and I can't create one with a # character for some reason. Has anyone heard about it or seen this error? What could be wrong and some possible things to look at?

The MySQL DB seems to be up and running and can be queried via the console but the application can't seem to get through to it. There was no change to the application code/files. It just happened out the blue. So I'm not even sure where to start look or what resolution tactics to apply. Any ideas?

Answer 1

I meet this error too when I run a wordpress on my Fedora system.

I googled it, and find a way to fix this.

Maybe this will help you too.

1. check mysql config : my.cnf

    cat /etc/my.cnf | grep tmpdir
   

   I can't see anything in my my.cnf

2. add tmpdir=/tmp to my.cnf under [mysqld]

3. restart web/app and mysql server

   /etc/init.d/mysqld restart

Answer 2

Often this means your /tmp partition has run out of space and the file can't be created, or for whatever reason the mysqld process cannot write to that directory because of permission problems. Sometimes this is the case when selinux rains on your parade.

Any operation that requites a "temp file" will go into the /tmp directory by default. The name you're seeing is just some internal random name.

Answer 3

On Fedora with systemd MySQL gets private /tmp directory. In /proc/PID_of_MySQL/mountinfo you will find the line like:

156 129 8:1 /tmp/systemd-namespace-AN7vo9/private /tmp rw,relatime - ext4 /dev/sda1 rw,seclabel,data=ordered

This means a temporary folder /tmp/systemd-namespace-AN7vo9/private is mounted as /tmp in private namespace of MySQL process. Unfortunately this folder is deleted by tmpwatch if not used frequently.

I modified /etc/cron.daily/tmpwatch and inserted the exclude pattern -X '/tmp/systemd-namespace*' like this:

/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \
        -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \
        -X '/tmp/systemd-namespace*' \
        -X '/tmp/hsperfdata_*' 10d /tmp

The side effect is that unused private namespace folders will not be deleted automatically.

Answer 4

The filename looks like a temporary table created by a query in MySQL. These files are often very short-lived, they're created during one specific query and cleaned up immediately afterwards.

Yet they can get very large, depending on the amount of data the query needs to process in a temp table. Or you may have multiple concurrent queries creating temp tables, and if enough of these queries run at the same time, they can exhaust disk space.

I do MySQL consulting, and I helped a customer who had intermittent disk full errors on his root partition, even though every time he looked, he had about 6GB free. After we examined his query logs, we discovered that he sometimes had four or more queries running concurrently, each creating a 1.5GB temp table in /tmp, which was on his root partition. Boom!

Solutions I gave him:

• Increase the MySQL config variables tmp_table_size and max_heap_table_size so MySQL can create really large temp tables in memory. But it's not a good idea to allow MySQL to create 1.5GB temp tables in memory, because there's no way to limit how many of these are created concurrently. You can exhaust your memory pretty quickly this way.

• Set the MySQL config variable tmpdir to a directory on another disk partition with more space.

• Figure out which of your queries is creating such big temp tables, and optimize the query. For example, use indexes to help that query reduce its scan to a smaller slice of the table. Or else archive some of the data in the tale so the query doesn't have so many rows to scan.