MYSQL, very slow order by

Question

I have got two tables. One is a User table with a primary key on the userid and the other table references the user table with a foreign key.

The User table has only one entry (for now) and the other table has one million entrys.

The following join drives me mad:

 SELECT p0_.*, p1_.*
 FROM photo p0_, User p1_
 WHERE p0_.user_id = p1_.user_id
 ORDER BY p0_.uploaddate DESC Limit 10 OFFSET 100000

The query takes 12sec on a very fast machine with the order by and 0.0005 sec without the order by.

I've got an index on user_id (IDX_14B78418A76ED395) and a composite index ("search2") on user_id and uploaddate.

EXPLAIN shows the following:

+----+-------------+-------+------+------------------------------+----------------------+---------+---------------------+-------+---------------------------------+
| id | select_type | table | type | possible_keys                | key                  | key_len | ref                 | rows  | Extra                           |
+----+-------------+-------+------+------------------------------+----------------------+---------+---------------------+-------+---------------------------------+
|  1 | SIMPLE      | p1_   | ALL  | PRIMARY                      | NULL                 | NULL    | NULL                |     1 | Using temporary; Using filesort |
|  1 | SIMPLE      | p0_   | ref  | IDX_14B78418A76ED395,search2 | IDX_14B78418A76ED395 | 4       | odsfoto.p1_.user_id | 58520 |                                 |
+----+-------------+-------+------+------------------------------+----------------------+---------+---------------------+-------+---------------------------------+

Table definitions:

CREATE TABLE `photo` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`album_id` int(11) DEFAULT NULL,
`exif_id` int(11) DEFAULT NULL,
`title` varchar(50) COLLATE utf8_unicode_ci NOT NULL,
`width` int(11) NOT NULL,
`height` int(11) NOT NULL,
`uploaddate` datetime NOT NULL,
`filesize` int(11) DEFAULT NULL,
`path` varchar(200) COLLATE utf8_unicode_ci NOT NULL,
`originalFilename` varchar(200) COLLATE utf8_unicode_ci NOT NULL,
`mimeType` varchar(200) COLLATE utf8_unicode_ci NOT NULL,
`description` longtext COLLATE utf8_unicode_ci,
`gpsData_id` int(11) DEFAULT NULL,
`views` int(11) DEFAULT NULL,
`likes` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UNIQ_14B78418B0FC9251` (`exif_id`),
UNIQUE KEY `UNIQ_14B7841867E96507` (`gpsData_id`),
KEY `IDX_14B78418A76ED395` (`user_id`),
KEY `IDX_14B784181137ABCF` (`album_id`),
KEY `search_idx` (`uploaddate`),
KEY `search2` (`user_id`,`uploaddate`),
KEY `search3` (`uploaddate`,`user_id`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;


CREATE TABLE `user` (
`user_id` int(11) NOT NULL,
`photoCount` int(11) NOT NULL,
`photoViews` int(11) NOT NULL,
`photoComments` int(11) NOT NULL,
`photoLikes` int(11) NOT NULL,
`username` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

What can I do to speed up this query?

Answer 1

Seems you're suffering from MySQL's inability to do late row lookups:

• MySQL ORDER BY / LIMIT performance: late row lookups
• Late row lookups: InnoDB

Try this:

SELECT  p.*, u.*
FROM    (
        SELECT  id
        FROM    photo
        ORDER BY
                uploaddate DESC, id DESC
        LIMIT   10
        OFFSET  100000
        ) pi
JOIN    photo p
ON      p.id = pi.id
JOIN    user u
ON      u.user_id = p.user_id

Answer 2

You have two problems:

1. You need to create an INDEX(user_id, uploaddate) which will greatly increase the efficiency of the query.

2. You need to find a workaround to using LIMIT 10 OFFSET 100000. MySQL is creating a recordset with 100,000 records in it, then it pulls the last 10 records off the end... that is extremely inefficient.

https://www.percona.com/blog/2006/09/01/mysql-order-by-limit-performance-optimization/

Answer 3

You need a separate index on uploaddate. This sort will take advantage of composite index only if uploaddate is first column in it. You can also try to add user_id to ORDER BY:

    ....      
    ORDER BY p0_.user_id, p0_.uploaddate