why Postgres prefer seq scan to partial index with explicit where condition?

Question

I have a simple query like select * from xxx where col is not null limit 10. I don't know why Postgres prefer seq scan which is much slower than partial index (I have analyzed the table). How to debug problem like this?

The table has more than 4 millions rows. And about 350,000 rows satisfied pid is not null.

I think there may be something wrong with the cost estimation. Cost of seq scan is lower than index scan. But how to dig into this?

I have a guess but not sure about it. The not null rows occupy about 10% of total rows. It means it may get 10 rows that not null when seq scan 100 rows. And it think the cost of seq scan 100 rows is lower than index scan 10 rows and then random fetch 10 full rows. Is it?

> \d data_import
+--------------------+--------------------------+----------------------------------------------------------------------------+
| Column             | Type                     | Modifiers                                                                  |
|--------------------+--------------------------+----------------------------------------------------------------------------|
| id                 | integer                  |  not null default nextval('data_import_id_seq'::regclass) |
| name               | character varying(64)    |                                                                            |
| market_activity_id | integer                  |  not null                                                                  |
| hmsr_id            | integer                  |  not null default (-1)                                                     |
| site_id            | integer                  |  not null default (-1)                                                     |
| hmpl_id            | integer                  |  not null default (-1)                                                     |
| hmmd_id            | integer                  |  not null default (-1)                                                     |
| hmci_id            | integer                  |  not null default (-1)                                                     |
| hmkw_id            | integer                  |  not null default (-1)                                                     |
| creator_id         | integer                  |                                                                            |
| created_at         | timestamp with time zone |                                                                            |
| updated_at         | timestamp with time zone |                                                                            |
| bias               | integer                  |                                                                            |
| pid                | character varying(128)   |  default NULL::character varying                                           |
+--------------------+--------------------------+----------------------------------------------------------------------------+
Indexes:
    "data_import_pkey" PRIMARY KEY, btree (id)
    "unique_hmxx" UNIQUE, btree (site_id, hmsr_id, hmpl_id, hmmd_id, hmci_id, hmkw_id) WHERE pid IS NULL
    "data_import_pid_idx" UNIQUE, btree (pid) WHERE pid IS NOT NULL
    "data_import_created_at_idx" btree (created_at)
    "data_import_hmsr_id" btree (hmsr_id)
    "data_import_updated_at_idx" btree (updated_at)



> set enable_seqscan to false;
apollon> explain (analyse, verbose)  select * from data_import where pid is not null limit 10
+-------------------------------------------------------------------------------------------------------------------------------------------------------------
| QUERY PLAN
|-------------------------------------------------------------------------------------------------------------------------------------------------------------
| Limit  (cost=0.42..5.68 rows=10 width=84) (actual time=0.059..0.142 rows=10 loops=1)
|   Output: id, name, market_activity_id, hmsr_id, site_id, hmpl_id, hmmd_id, hmci_id, hmkw_id, creator_id, created_at, updated_at, bias, pid
|   ->  Index Scan using data_import_pid_idx on public.data_import  (cost=0.42..184158.08 rows=350584 width=84) (actual time
|         Output: id, name, market_activity_id, hmsr_id, site_id, hmpl_id, hmmd_id, hmci_id, hmkw_id, creator_id, created_at, updated_at, bias, pid
|         Index Cond: (data_import.pid IS NOT NULL)
| Planning time: 0.126 ms
| Execution time: 0.177 ms
+-------------------------------------------------------------------------------------------------------------------------------------------------------------
EXPLAIN
Time: 0.054s

> set enable_seqscan to true;
> explain (analyse, verbose)  select * from data_import where pid is not null limit 10
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| QUERY PLAN                                                                                                                                        |
|---------------------------------------------------------------------------------------------------------------------------------------------------|
| Limit  (cost=0.00..2.37 rows=10 width=84) (actual time=407.042..407.046 rows=10 loops=1)                                                          |
|   Output: id, name, market_activity_id, hmsr_id, site_id, hmpl_id, hmmd_id, hmci_id, hmkw_id, creator_id, created_at, updated_at, bias, pid       |
|   ->  Seq Scan on public.data_import  (cost=0.00..83016.60 rows=350584 width=84) (actual time=407.041..407.045 rows=10 loops=1)  |
|         Output: id, name, market_activity_id, hmsr_id, site_id, hmpl_id, hmmd_id, hmci_id, hmkw_id, creator_id, created_at, updated_at, bias, pid |
|         Filter: (data_import.pid IS NOT NULL)                                                                                    |
|         Rows Removed by Filter: 3672502                                                                                                           |
| Planning time: 0.116 ms                                                                                                                           |
| Execution time: 407.078 ms                                                                                                                        |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
EXPLAIN
Time: 0.426s

Answer 1

Your problem are the

Rows Removed by Filter: 3672502

PostgreSQL knows the distribution of the values and how they are correlated with the physical table layout, but it does not know that the rows at the beginning of the table all have NULL for pid.

If the NULLs were evenly distributed, the sequential scan would quickly find 10 hits and stop, but as it is, it has to read 3672512 rows to find 10 matching ones.

If you add ORDER BY pid (even though you don't need it) before the LIMIT, the optimizer will do the right thing.