ConcurrentModificationException in unmodified ArrayList in Apache HttpClient

Question

Had ConcurrentModificationException thrown in production, the list which throws is a vanilla java7 ArrayList, the List is not modified in the loop and is local to the method above - not being passed anywhere else.

The Exception was thrown consistently every time the api call was made until server restart - then it stopped.

java.util.ConcurrentModificationException
    at java.util.ArrayList$Itr.next(ArrayList.java:837)
    at org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:175)
    at org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:174)
    at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:131)

Code is below but difficult to read outside of IDE, here's GrepCode where the call is made, see that matchedCookies is not passed out of the thread scope.

BestMatchSpec.formatCookies does not modify the cookies list:

public List<Header> formatCookies(final List<Cookie> cookies) {
    Args.notNull(cookies, "List of cookies");
    int version = Integer.MAX_VALUE;
    boolean isSetCookie2 = true;
    for (final Cookie cookie: cookies) {
        if (!(cookie instanceof SetCookie2)) {
            isSetCookie2 = false;
        }
        if (cookie.getVersion() < version) {
            version = cookie.getVersion();
        }
    }

The calling method RequestAddCookies.process constructs the list and doesn't make the list of cookies available to another thread.

public void process(final HttpRequest request, final HttpContext context)
        throws HttpException, IOException {
...
    // Find cookies matching the given origin
    final List<Cookie> matchedCookies = new ArrayList<Cookie>();
    final Date now = new Date();
    for (final Cookie cookie : cookies) {
        if (!cookie.isExpired(now)) {
            if (cookieSpec.match(cookie, cookieOrigin)) {
                if (this.log.isDebugEnabled()) {
                    this.log.debug("Cookie " + cookie + " match " + cookieOrigin);
                }
                matchedCookies.add(cookie);
            }
        } else {
            if (this.log.isDebugEnabled()) {
                this.log.debug("Cookie " + cookie + " expired");
            }
        }
    }
    // Generate Cookie request headers
    if (!matchedCookies.isEmpty()) {
        final List<Header> headers = cookieSpec.formatCookies(matchedCookies);
...
}

Am I missing something?

Answer 1

Thanks to @guido for pointing to this issue on apache. It is due to a bug with the JIT compiler fixed in java7u60. The code should not be able to produce the ConcurrentModificationException.

HttpClient report: https://issues.apache.org/jira/browse/HTTPCLIENT-1173

Java bug: http://bugs.java.com/view_bug.do?bug_id=8021898

Test to reproduce bug: https://github.com/rholder/jvm-loop-unswitching-bug