Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Making sense of Postgres row sizes

Tags:

I got a large (>100M rows) Postgres table with structure {integer, integer, integer, timestamp without time zone}. I expected the size of a row to be 3*integer + 1*timestamp = 3*4 + 1*8 = 20 bytes.

In reality the row size is pg_relation_size(tbl) / count(*) = 52 bytes. Why?

(No deletes are done against the table: pg_relation_size(tbl, 'fsm') ~= 0)

like image 408
Arman Avatar asked Nov 26 '12 18:11

Arman


People also ask

How big is too big for a PostgreSQL table?

PostgreSQL normally stores its table data in chunks of 8KB. The number of these blocks is limited to a 32-bit signed integer (just over two billion), giving a maximum table size of 16TB.

Does number of columns affect performance in Postgres?

Yes the number of columns will - indirectly - influence the performance. The data in the columns will also affect the speed.

How many columns should a Postgres table have?

There is a limit on how many columns a table can contain. Depending on the column types, it is between 250 and 1600.

How to calculate row size of postgress table row?

For example a table has 50 rows and total table size is 10gb but i want to know each row size rather than complete table size. Function to calculate row size of postgress table row is something like this SELECT sum (pg_column_size (t.*)) as filesize, count (*) as filerow FROM TABLE_NAME as t;

What are the limits of PostgreSQL?

Appendix K. PostgreSQL Limits Item Upper Limit Comment database size unlimited number of databases 4,294,950,911 relations per database 1,431,650,303 relation size 32 TB with the default BLCKSZ of 8192 bytes 7 more rows ...

How does PostgreSQL store large chunks of data?

PostgreSQL uses a fixed page size (commonly 8 kB), and does not allow tuples to span multiple pages. Therefore, it is not possible to store very large field values directly. As the documentation explains, PostgreSQL can't store rows (tuples) in multiple pages. So how does the database store large chunks of data?

What are the disadvantages of using medium-size text in PostgreSQL?

The main problem with medium-size texts is that they make the rows very wide. This is a problem because PostgreSQL, as well as other OLTP oriented databases, are storing values in rows. When we ask the database to execute a query with only a few columns, the values of these columns are most likely spread across many blocks.


2 Answers

Calculation of row size is much more complex than that.

Storage is typically partitioned in 8 kB data pages. There is a small fixed overhead per page, possible remainders not big enough to fit another tuple, and more importantly dead rows or a percentage initially reserved with the FILLFACTOR setting.

And there is even more overhead per row (tuple): an item identifier of 4 bytes at the start of the page, the HeapTupleHeader of 23 bytes and alignment padding. The start of the tuple header as well as the start of tuple data are aligned at a multiple of MAXALIGN, which is 8 bytes on a typical 64-bit machine. Some data types require alignment to the next multiple of 2, 4 or 8 bytes.

Quoting the manual on the system table pg_tpye:

typalign is the alignment required when storing a value of this type. It applies to storage on disk as well as most representations of the value inside PostgreSQL. When multiple values are stored consecutively, such as in the representation of a complete row on disk, padding is inserted before a datum of this type so that it begins on the specified boundary. The alignment reference is the beginning of the first datum in the sequence.

Possible values are:

  • c = char alignment, i.e., no alignment needed.

  • s = short alignment (2 bytes on most machines).

  • i = int alignment (4 bytes on most machines).

  • d = double alignment (8 bytes on many machines, but by no means all).

Read about the basics in the manual here.

Your example

This results in 4 bytes of padding after your 3 integer columns, because the timestamp column requires double alignment and needs to start at the next multiple of 8 bytes.

So, one row occupies:

   23   -- heaptupleheader  +  1   -- padding or NULL bitmap  + 12   -- 3 * integer (no alignment padding here)  +  4   -- padding after 3rd integer  +  8   -- timestamp  +  0   -- no padding since tuple ends at multiple of MAXALIGN 

Plus item identifier per tuple in the page header (as pointed out by @A.H. in the comment):

 +  4   -- item identifier in page header ------  = 52 bytes 

So we arrive at the observed 52 bytes.

The calculation pg_relation_size(tbl) / count(*) is a pessimistic estimation. pg_relation_size(tbl) includes bloat (dead rows) and space reserved by fillfactor, as well as overhead per data page and per table. (And we didn't even mention compression for long varlena data in TOAST tables, since it doesn't apply here.)

You can install the additional module pgstattuple and call SELECT * FROM pgstattuple('tbl_name'); for more information on table and tuple size.

Related:

  • Table size with page layout
  • Calculating and saving space in PostgreSQL
like image 69
Erwin Brandstetter Avatar answered Sep 21 '22 03:09

Erwin Brandstetter


Each row has metadata associated with it. The correct formula is (assuming naïve alignment):

3 * 4 + 1 * 8 == your data 24 bytes == row overhead total size per row: 23 + 20 

Or roughly 53 bytes. I actually wrote postgresql-varint specifically to help with this problem with this exact use case. You may want to look at a similar post for additional details re: tuple overhead.

like image 37
Sean Avatar answered Sep 23 '22 03:09

Sean