Microsoft SQL Server Paging

Question

There are a number of sql server paging questions on stackoverflow and many of them talk about using ROW_NUMBER() OVER (ORDER BY ...) AND CTE. Once you get into the hundreds of thousands of rows and start adding sorting on non-primary key values and adding custom WHERE clauses, these methods become very inneficient. I have a dataset of several million rows I am trying to page through with custom sorting and filtering, but I am getting poor performance, even with indexes on all the fields that I sort by and filter by. I even went as far as to include my SELECT columns in each of the indexes, but this barely helped and severely bloated my database.

I noticed the stackoverflow paging only takes about 500 milliseconds no matter what sorting criteria or page number you click on. Anyone know how to make paging work efficiently in SQL Server 2008 with millions of rows? This would include getting the total rows as efficiently as possible.

My current query has the exact same logic as this stackoverflow question about paging: Best paging solution using SQL Server 2005?

Answer 1

Anyone know how to make paging work efficiently in SQL Server 2008 with millions of rows?

If you want accurate perfect paging, there is no substitute for building an index key (position row number) for each record. However, there are alternatives.

(1) total number of pages (records)

• You can use an approximation from sysindexes.rows (almost instant) assuming the rate of change is small.
• You can use triggers to maintain a completely accurate, to the second, table row count

(2) paging

(a)
You can show page jumps within say the next five pages to either side of a record. These need to scan at most {page size} x 5 on each side. If your underlying query lends itself to travelling along the sort order quickly, this should not be slow. So given a record X, you can go to the previous page using (assuming sort order is a asc, b desc

select top(@pagesize) t.*
from tbl x
inner join tbl t on (t.a = x.a and t.b > x.b) OR
                    (t.a < a.x)
where x.id = @X
order by t.a asc, t.b desc

(i.e. the last {page size} of records prior to X)

To go five pages back, you increase it to TOP(@pagesize*5) then further TOP(@pagesize) from that subquery.

Downside: This option requires that you cannot directly jump to a particular location, your options are only FIRST (easy), LAST (easy), NEXT/PRIOR, <5 pages either side

(b)
If the paging is always going to be quite specific and predictable, maintain an INDEXED view or trigger-updated table that does not contain gaps in the row number. This may be an option if the tables normally only see updates at one end of the spectrum, with gaps from deletes easily filled quickly by shifting not-so-many records.

This approach gives you a rowcount (last row) and also direct access to any page.