"FATAL: the database system is shutting down" while creating connection to PostgreSQL

Question

I am getting FATAL: the database system is shutting down errors while creating PostgreSQL JDBC connections to a PostgreSQL 9.2 server. The specific exception path I'm getting from JDBC is here:

Caused by: org.postgresql.util.PSQLException: FATAL: the database system is shutting down
    at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:398)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173)
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
    at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136)
    at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
    at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
    at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
    at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
    at org.postgresql.Driver.makeConnection(Driver.java:393)
    at org.postgresql.Driver.connect(Driver.java:267)

From various log files (from PostgreSQL, from our management layer, from the app using PostgreSQL), I do not see any database shutdown actually happening (other connections are created to PostgreSQL as usual, no shutdown was initiated from our management layer, etc.), however in the PostgreSQL server logs I do see the error message with the same timestamp:

2014-06-16 12:30:00.736 GMT LOG:  connection received: host=127.0.0.1 port=38530
2014-06-16 12:30:00.737 GMT FATAL:  the database system is shutting down

Researching online, I learned that this error message is used whenever PostgreSQL shuts down connections.

Why would PostgreSQL refuse to give me a new connection? Could this be caused by some sort of resource contention? How can I get more information about the error from PostgreSQL itself?

Answer 1

This issue turned out to be caused by a very bad misuse of PostgreSQL where our server had booted two PostgreSQL instances on the same data directory (it had deleted postmaster.pid and used a new port number, so the normal safeguards against this behavior were bypassed) which explains why the logs did not hold any useful information -- they were being overwritten by the PostgreSQL instance which wasn't refusing connections. The actual issue was caused by a complex interaction between the competing PostgreSQL instances, and I hope nobody else runs into this issue as well!

Answer 2

Maybe there is not enough space on hard drive.