UPDATE / INSERT from time to time takes few seconds

Question

I have problem with really simple INSERT / UPDATES on my server. From time to time it takes more than few second to finish query like this ones:

2.1062s - INSERT INTO `transaction` SET `idUser` = 72, `currency` = 50, `amount` = '10', `action` = 'buyCoins';
11.785s - UPDATE `user` SET `cash` = 10, `crystal` = 10, `expPoints` = 10, `energy` = 10 WHERE idUser = 72;
0.6296s - UPDATE `user` SET `lastEnergyUpdate` = CURRENT_TIMESTAMP WHERE idUser = 72;

It looks like issue isn't depending on specific table. I don't have TRIGGERS on those tables.

Table definitions:

CREATE TABLE `user` (
 `idUser` int(10) unsigned NOT NULL AUTO_INCREMENT,
 `expPoints` int(10) NOT NULL DEFAULT '0',
 `cash` int(10) NOT NULL DEFAULT '1000',
 `crystal` int(10) NOT NULL DEFAULT '10',
 `energy` int(4) NOT NULL DEFAULT '0',
 `name` varchar(50) DEFAULT NULL,
 `surname` varchar(50) DEFAULT NULL,
 `age` int(4) unsigned DEFAULT NULL,
 `sex` enum('men','women','unknown') DEFAULT NULL,
 `lastEnergyUpdate` timestamp NULL DEFAULT NULL,
 `lastLogin` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
 `insertDate` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
 PRIMARY KEY (`idUser`),
 UNIQUE KEY `serviceUnique` (`serviceName`,`serviceId`)
) ENGINE=InnoDB AUTO_INCREMENT=5333 DEFAULT CHARSET=utf8

CREATE TABLE `transaction` (
  `idTransaction` int(10) NOT NULL AUTO_INCREMENT,
  `idUser` int(10) unsigned NOT NULL,
  `currency` enum('crystal','partnerCurrency','cash') DEFAULT NULL,
  `amount` int(5) NOT NULL,
  `action` enum('unlockPlace','buyExtra','collectReleased') NOT NULL,
  `insertDate` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (`idTransaction`),
  KEY `fk_transaction_user1` (`idUser`),
  CONSTRAINT `fk_transaction_user1` FOREIGN KEY (`idUser`) REFERENCES `user` (`idUser`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=156329 DEFAULT CHARSET=utf8

On same server I have more databases (~100) but nothing big. Dump of all databases is around 300MB.

Mysqltunner output:

 >>  MySQLTuner 1.0.1 - Major Hayden <[email protected]>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering

-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.1.66-0ubuntu0.11.10.2-log
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 138M (Tables: 267)
[--] Data in InnoDB tables: 170M (Tables: 327)
[--] Data in MEMORY tables: 0B (Tables: 1)
[!!] Total fragmented tables: 329

-------- Performance Metrics -------------------------------------------------
[--] Up for: 20h 45m 57s (558K q [7.468 qps], 58K conn, TX: 685M, RX: 98M)
[--] Reads / Writes: 66% / 34%
[--] Total buffers: 1.1G global + 6.0M per thread (150 max threads)
[OK] Maximum possible memory usage: 2.0G (12% of installed RAM)
[OK] Slow queries: 0% (54/558K)
[OK] Highest usage of available connections: 6% (10/150)
[OK] Key buffer size / total MyISAM indexes: 16.0M/8.8M
[OK] Key buffer hit rate: 99.9% (245K cached / 258 reads)
[OK] Query cache efficiency: 51.5% (176K cached / 342K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 6% (1K temp sorts / 19K sorts)
[!!] Temporary tables created on disk: 34% (2K on disk / 8K total)
[OK] Thread cache hit rate: 99% (10 created / 58K connections)
[!!] Table cache hit rate: 16% (786 open / 4K opened)
[OK] Open file limit used: 32% (714/2K)
[OK] Table locks acquired immediately: 99% (329K immediate / 329K locks)
[OK] InnoDB data size / buffer pool: 170.3M/512.0M

-------- Recommendations -----------------------------------------------------
General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    MySQL started within last 24 hours - recommendations may be inaccurate
    Temporary table size is already large - reduce result set size
    Reduce your SELECT DISTINCT queries without LIMIT clauses
    Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
    table_cache (> 1024)

Of course it happens only for ~1% of that queries (99% works fine), and then HDD is really bussy (13% - 20% wa on 8 cores server)

Should I keep increasing table_cache? Any other ideas what's going on? How can I improve it?

My MySQL Server is 5.1.66. I tried to upgrade to 5.5.x but that didn't help me, so I downgrade it back.

Answer 1

Of course it happens only for ~1% of that queries (99% works fine), and then HDD is really bussy (13% - 20% was on 8 cores server)

1. You are hitting a problem with your disk - I bet this occurs while your innodb log is flushing to disk - I would experiment with both smaller (for flushes more often) and larger log sizes (for less frequent flushes) - I expect you get more from more frequent flushes.

2. The other thing to see is what is eating your IO - run iotop during this time if you can to capture the culprit.

3. Otherwise, ensure your tmp, data, and log partitions are physically separate if you can.