Postgresql database randomly crashes

Question

I really have no idea what the problem is.

The logs read

FATAL: sorry, too many clients already

Over and over again. At first I thought that sometimes the connections lingered or were not closed properly so I tested that by connecting to the database and checking how many opened connections there were at any given time, and the answer has always been 1.

I tried to connect to the website that's using the DB and I managed to glimpse at 2 or 3 opened connections that were promptly closed when the page was done loading.

My remaining guess is that sometimes there are spikes in concurrent connections to the website and that causes the database to stop accepting new connections and somehow doesn't allow the current connections to be dropped.

I DID NOT WRITE ANY CODE THAT CONNECTS TO THE DATABASE, I'm using a pretty vanilla Django (1.7) backend that handles all of the connections.

I couldn't find anything while searching google, has anyone experienced any problems?

EDIT:

Database configuration is here(PasteBin)

Essential part:

port = 26445                # (change requires restart)
max_connections = 500           # (change requires restart)
unix_socket_directory = '/home/clearintent/webapps/norr2_db/run'        # (change requires restart)

shared_buffers = 32MB           # min 128kB
                    # (change requires restart)
log_destination = 'stderr'      # Valid values are combinations of
logging_collector = on          # Enable capturing of stderr and csvlog
log_directory = 'pg_log'        # directory where log files are written,
log_filename = 'postgresql-%a.log'  # log file name pattern,
log_truncate_on_rotation = on       # If on, an existing log file with the
log_rotation_age = 1d           # Automatic rotation of logfiles will
log_rotation_size = 0           # Automatic rotation of logfiles will
datestyle = 'iso, mdy'
lc_messages = 'C'           # locale for system error message
lc_monetary = 'C'           # locale for monetary formatting
lc_numeric = 'C'            # locale for number formatting
lc_time = 'C'               # locale for time formatting
default_text_search_config = 'pg_catalog.english'

Answer 1

Sounds like something is bashing your PostgreSQL but that error alone does not crash the DB.
It simply means the latest connection attempt exceeded the amount of allowed parallel connections to the DB and it was rejected.

But, if you want to dump the amount of connections every minute, you can use this script

#!/bin/bash

function spew_connections() {
# Run psql on local if trust or auth is set.
# Change dbadmin and dbname accordingly.
/usr/bin/psql -d dbname -U dbadmin -w -t -c "SELECT localtimestamp(2), count(*) FROM pg_stat_activity;"
}

echo -n `spew_connections` >> /tmp/connections
echo >> /tmp/connections

And then, to execute it using crontab every minute

crontab -e
*/1 * * * * /path/to/executable/script