Scoreboard is full,not at MaxRequestWorkers

Question

I an using Apache and Tomcat with mod-jk in my project. My web application is in Tomcat instances. I am using Apache jmeter for testing http load on tomcat. But I'm getting the error Scoreboard is full,not at MaxRequestWorkers even with 1000 threads.

[mpm_event:error] [pid 24313:tid 3075319488] AH00485: scoreboard is full, not at MaxRequestWorkers

[mpm_event:error] [pid 24313:tid 3075319488] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

I've configured http-mpm for approximately 5000 threads.

httd-mpm.conf file is:

<IfModule mpm_event_module>
    ServerLimit              200
    StartServers             3
    MinSpareThreads          75
    MaxSpareThreads          250
    MaxClients               5000 
    ThreadsPerChild          25
    MaxRequestWorkers        5000
    MaxConnectionsPerChild   5000
</IfModule>

Can anybody please guide me what can be the problem in this configuration?

Answer 1

It seems that Apache tries to gracefully finish some of the processes and hangs while doing so. It might be a problem of Apache (https://bz.apache.org/bugzilla/show_bug.cgi?id=53555) or a problem of the application itself.

You can easily check it by looking at the server-status page of apache. In order to do that you need to enable the status.conf apache module. Then go to your-server.org/server-status. You will see all the processes and threads currently being used or gracefully finishing ("G"-state). If you have to many G's that don't go away, you found your problem. It would look sth like this:

Too many threads gracefully finishing...

The solution is pretty simple. Just set MaxConnectionsPerChild to 0 or comment it out (mpm_event.conf). What this does is just let the process continue it's work without restarting it. That's a good thing in our case, since restarting the process let to the problem of having to many G's filling up the scoreboard and eventually crashing Apache.

Hope that helped.