GetQueuedCompletionStatus can't dequeue IO from IOCP if the thread which originally issued the IO is blocking in ReadFile under windows 8

Question

My app stop working after switching to windows 8. I spend hours to debug the problem, found out IOCP behave differently between windows 8 and previous versions. I extract the necessary code to demonstrate and reproduce the problem.

SOCKET sListen;

DWORD WINAPI WorkerProc(LPVOID lpParam)
{
    ULONG_PTR dwKey;
    DWORD dwTrans;
    LPOVERLAPPED lpol;
    while(true)
    {
        GetQueuedCompletionStatus((HANDLE)lpParam, &dwTrans, &dwKey, (LPOVERLAPPED*)&lpol, WSA_INFINITE);
        printf("dequeued an IO\n");
    }
}
DWORD WINAPI StartProc(LPVOID lpParam)
{
    WSADATA WsaData;
    if (WSAStartup(0x202,&WsaData)!=0) return 1;
    sListen = WSASocket(AF_INET, SOCK_STREAM, 0, NULL, 0, WSA_FLAG_OVERLAPPED);
    SOCKADDR_IN si;
    ZeroMemory(&si,sizeof(si));
    si.sin_family = AF_INET;
    si.sin_port = ntohs(1999);
    si.sin_addr.S_un.S_addr = INADDR_ANY;
    if(bind(sListen, (sockaddr*)&si, sizeof(si)) == SOCKET_ERROR) return 1;
    listen(sListen, SOMAXCONN);
    HANDLE hCompletion = CreateIoCompletionPort(INVALID_HANDLE_VALUE, 0, 0, 0);
    CreateIoCompletionPort((HANDLE)sListen, hCompletion, (DWORD)0, 0);
    CreateThread(NULL, 0, WorkerProc, hCompletion, 0, NULL);
    return 0;
}
DWORD WINAPI AcceptProc(LPVOID lpParam)
{
    DWORD dwBytes;
    LPOVERLAPPED pol=(LPOVERLAPPED)malloc(sizeof(OVERLAPPED));
    ZeroMemory(pol,sizeof(OVERLAPPED));
    SOCKET sClient = WSASocket(AF_INET, SOCK_STREAM, 0, NULL, 0, WSA_FLAG_OVERLAPPED);
    BOOL b = AcceptEx(sListen, 
        sClient,
        malloc ((sizeof(sockaddr_in) + 16) * 2), 
        0,
        sizeof(sockaddr_in) + 16, 
        sizeof(sockaddr_in) + 16, 
        &dwBytes, 
        pol);
    if(!b && WSAGetLastError() != WSA_IO_PENDING)   return 1;
    HANDLE hPipe=CreateNamedPipeA("\\\\.\\pipe\\testpipe",PIPE_ACCESS_DUPLEX,PIPE_TYPE_BYTE | PIPE_READMODE_BYTE | PIPE_WAIT,PIPE_UNLIMITED_INSTANCES,4096,4096,999999999,NULL);
    BYTE chBuf[1024]; 
    DWORD  cbRead; 
    CreateFileA("\\\\.\\pipe\\testpipe", GENERIC_READ |GENERIC_WRITE,  0,NULL, OPEN_EXISTING, 0, NULL);
    ReadFile(hPipe,chBuf,1024, &cbRead,NULL);
    return 0;
}

int main()
{
    printf ("Starting server on port 1999...");
    WaitForSingleObject(CreateThread(NULL, 0, StartProc, NULL, 0, NULL),INFINITE);
    CreateThread(NULL, 0,AcceptProc, NULL, 0, NULL);
    printf ("done\n");
    Sleep(10000000);
    return 0;
}

This program listen on port 1999 and issue an async accpet then reading a blocking pipe. I have tested this program on Windows 7, 8, XP, 2003, 2008, after "telnet 127.0.0.1 1999", "dequeued an IO\n" will printed on console except windows 8.

The point is the thread which originally issued the async operation must not blocking in ReadFile or GetQueuedCompletionStatus will never dequeue that IO until ReadFile returns on windows 8.

I also tested using "scanf" instead of reading pipe, the results are same since "scanf" will call ReadFile to read console eventually. I don't know if ReadFile is the only function affected or there may be other functions.

What I can think of is using a dedicated thread to issue async operations, and all business logic communicate with that dedicated thread to perform accept/send/recv. But extra layer means extra overhead, is there any way to achieve the same performance as previous versions of windows on windows 8?

Answer 1

See https://connect.microsoft.com/WindowsServer/feedback/details/760161/breaking-change-to-acceptex-and-iocp-in-server-2012-and-windows-8

This is a bug and the official MS response is "We've passed this to the base OS team and they will consider this for a future update. I'm resolving this postponed."

Note: I ran this test on a fully patched version of Windows 8 today (12th Sep 2013) in preparation to testing Windows 8.1 and found that the problem appears to be fixed on Windows 8 now. I've no idea WHEN it was fixed.