MySQL / InnoDB occasionally some updates run very slow, in 'Updating' status

Question

We have been experiencing an occasional massive slow down in UPDATE performance across our database.

For example, the table FooTable we have about 40 columns with a varchar PK in addition there are 10 indexes. The following query took 44 seconds, while at other times it run's virtually instantly. During the slow down the load average on the server is quite low (1.5 for the 5 minute average) and IO as per vmstat is fairly reasonable as well.

Here an an example:

mysql> update FooTable set BarColumn=1349981286086 where varcharPK='e4348411-0fbb-460a-80f7-f1de304a9f7c'
Query OK, 1 row affected (44.94 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> show profile for QUERY 1;
+----------------------+-----------+
| Status               | Duration  |
+----------------------+-----------+
| starting             |  0.000030 |
| checking permissions |  0.000004 |
| Opening tables       |  0.000007 |
| System lock          |  0.000003 |
| Table lock           |  0.000003 |
| init                 |  0.000035 |
| Updating             | 44.949727 |
| end                  |  0.000006 |
| query end            |  0.000003 |
| freeing items        |  0.000115 |
| logging slow query   |  0.000002 |
| logging slow query   |  0.000052 |
| cleaning up          |  0.000003 |
+----------------------+-----------+
13 rows in set (0.02 sec)

For what it's worth the example query above was run on a InnoDB table that was 'rebuilt' (ALTER TABLE FooTable ENGINE=InnoDB;) less then a week ago. I initially suspected that this was related to InnoDB's known performance issues with varchar/non sequential PKs, however we have other tables that use sequential PKs and have seen the same issues.

This is on a production server: Centos 5 2.6.18-238.19.1.el5 x86_64 MySQL/Percona 5.1.57-rel12.8-log 96GB of memory with 58.8G of data spread throughout 87 tables

The relevent InnoDB settings are as follows:

innodb_flush_log_at_trx_commit  = 2
innodb_buffer_pool_size         = 70G
innodb_log_file_size            = 512M
innodb_log_buffer_size          = 64M
innodb_file_per_table           = 1
innodb_thread_concurrency       = 0
innodb_flush_method             = O_DIRECT
innodb_read_io_threads          = 64
innodb_write_io_threads         = 64
optimizer_search_depth          = 0
innodb_file_format              = barracuda

I am not using FORMAT=COMPRESSED on this table, however I am on others.

Any suggestions on how to figure out what is going on in the Updating phase that is taking so long?

Answer 1

The root cause appears to have been long running application transactions. The solution was to break down the large transactions into smaller units of work.