PostgreSQL statistics issue - could not rename temporary statistics file

Question

I am running PotgreSQL 9.4 on Windows, and constantly get the error,

2015-06-15 09:35:36 EDT LOG could not rename temporary statistics file "pg_stat_tmp/global.tmp" to "pg_stat_tmp/global.stat": Permission denied 

I also see constant 200-800k writes to global.stat and global.tmp. I have seen other users with the same issue, but no solution. It is a big database server, with 300g of data, and 6,000 databases.

I tried setting,

track_activities=off

In the config file, but it did not seem to have any affect.

Any help for the error, or reducing the write?

Answer 1

After my initial answer, I decided to research the operation of the stats collector and in particular what it is doing with the files in pg_stat_tmp. I've substantially re-written the answer as a result.

What are the global.stat / global.tmp files used for?

Postgresql contains functionality to collect statistics and status information about its operation. The function is described in Section 27.2 of the manual.

This information is collated by the stats collector process. It is made available to the other postgresql processes via the global.stat file. The first time you run a query that accesses this data within a transaction, the backend which you are connected to will read the global.stat file and cache the result, using it until the end of the transaction.

To keep this file up to date, the stats collector process periodically re-writes it with updated information. It typically does this several times a second. The process is as follows:

1. Create a new file global.tmp
2. Write data to this file
3. Rename global.tmp as global.stat, overwriting the previous global.stat

The global.tmp and global.stats files are written into the directory configured by the stats_temp_directory configuration parameter. Normally this is set to $PGDATA/pg_stat_tmp.

On shutdown, the stats file is written into the file $PGDATA/global/pgstat.stat, and the files in the tmp dir above are removed. This file is then read and removed when the database is started up again.

Why is the stats collector processor creating so much I/O load?

Normally, the amount of data written to the global.stats is relatively modest and writing it does not generate that much I/O traffic. However under some circumstances it does seem to get very bloated. When this happens the amount of load generated can start to get excessive as the entire file is rewritten more than once a second.

I have had one experience where it grew by a factor or 10 or more, compared to other similar servers. This machine did have an unusually large number of databases (for our application at least - 30-40 databases - but nothing like the 6000 you say you have). It is possible that having a large number of databases exacerbates this.

Some of the references below talk about a pattern of creating / dropping lots of tables causing bloat in these files, and that perhaps autovacuum is not running aggressively enough to remove the associated bloat. You may wish to consider your autovac settings.

Why do I get 'Permission Denied' errors on Windows?

After examining the postgresql source code I think there may be a race condition in accessing the global.stats file which could happen at any time, but is exacerbated by the size of the file.

The default mode of operation in Windows is that it is not possible to rename or remove a file while another process has it open. This is different to Linux (or Unix) where a file can be renamed or removed while other processes are accessing it.

In the sequence above you can see that if one of the backend processes is reading the file at the same time as the stats collector is rewriting it, then the backend process may still have the file open at the time the rename is attempted. That leads to the 'Permission Denied' error you are seeing.

Naturally when the file becomes very large, then the amount of time taken to read it becomes more significant, therefore the probability of the stats collector process attempting a rename while a backend still has it open increases.

However, since the file is frequently being rewritten, the impact of these errors is relatively mild. It just means that this particular update fails, leading the the backends getting slightly out of date statistics. The next update will probably succeed.

Note that Windows does offer a file opening mode which does allow files to be deleted or renamed while they are opened by another process, however as far as I could tell, this mode is not used by Postgresql. I could not find any bug report on this - seems like it should be reported.

In summary, these errors are a side effect of the main problem, which is the excessive size of the global.stat file.

I've turned track_activities off but the file is still being written - Why?

From what I can see, track_activites affects only one of the sets of information that the stats collector is collecting.

In addition, it looks as though the stats collector process is started regardless of these settings, and will continue to re-write the file. The settings appear to control only the collection of fresh data.

My conclusion is that once the file has become bloated, it will remain so and continue to be re-written, even once all of the stats collection options are turned off.

What can I do to avoid this problem?

Once the file has become bloated, it seems that the easiest way to get the database back into a good working state is to remove the file, using the following steps:

1. Stop the database

2. When the DB is stopped, the pg_stat_tmp directory is empty and a file $PGDATA/global/pgstat.stat is written. We renamed this file to pgstat.stat.old.

3. Start the database. It creates a fresh set of pgstat files. After confirming the server was operating correctly you can remove the old file you have renamed.

This is the process we used when one of our servers suffered from this problem.

Needless to say be very careful when manually manipulating any files under the Postgresql Data directory.

After this you may want to monitor the server to see if it the file becomes bloated again. If it does then here are some additional ideas to consider:

• As mentioned above I have seen some references to this file becoming bloated if autovacuum is not running aggressively enough. You may wish to tune the autovacuum settings

• Disabling any of the track_xxx options described in the Section 18.9.1 of the manual which are not required may help

• It is possible to place the pg_stats_tmp directory in a tmpfs filesystem (or whatever equivalent RAM based filesystem is available in windows). Doing so should eliminate I/O as a concern for these files.

References:

• Postgres stats collector showing high disk I/O
• Too much I/O generated by postgres stats collector process
• stats collector suddenly causing lots of IO

Answer 2

Here might be a solution for your problem. https://wiki.postgresql.org/wiki/May_2015_Fsync_Permissions_Bug

Another possibility could be antivirus settings. Try to turn it off temporarily.