IOCP C++ TCP client

Question

I am having some trouble implementing TCP IOCP client. I have implemented kqueue on Mac OSX so was looking to do something similar on windows and my understanding is that IOCP is the closest thing. The main problem is that GetCompetetionStatus is never returning and always timeouts out. I assume I am missing something when creating the handle to monitor, but not sure what. This is where I have gotten so far:

My connect routine: (remove some error handling for clarity )

struct sockaddr_in server;
struct hostent *hp;
SOCKET sckfd;
WSADATA wsaData;

int iResult = WSAStartup( MAKEWORD(2,2), &wsaData );


if ((hp = gethostbyname(host)) == NULL)
    return NULL;
WSASocket(AF_INET,SOCK_STREAM,0,NULL,0,WSA_FLAG_OVERLAPPED)
if ((sckfd = WSASocket(AF_INET,SOCK_STREAM,0, NULL, 0, WSA_FLAG_OVERLAPPED)) == INVALID_SOCKET)
{
    printf("Error at socket(): Socket\n");
    WSACleanup();
    return NULL;
}

server.sin_family = AF_INET;
server.sin_port = htons(port);
server.sin_addr = *((struct in_addr *)hp->h_addr);
memset(&(server.sin_zero), 0, 8);

//non zero means non blocking. 0 is blocking.
u_long iMode = -1;
iResult = ioctlsocket(sckfd, FIONBIO, &iMode);
if (iResult != NO_ERROR)
    printf("ioctlsocket failed with error: %ld\n", iResult);


HANDLE hNewIOCP = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, ulKey, 0);
CreateIoCompletionPort((HANDLE)sckfd, hNewIOCP , ulKey, 0);

connect(sckfd, (struct sockaddr *)&server, sizeof(struct sockaddr));

//WSAConnect(sckfd, (struct sockaddr *)&server, sizeof(struct sockaddr),NULL,NULL,NULL,NULL);

return sckfd;   

Here is the send routine: ( also remove some error handling for clarity )

IOPortConnect(int ServerSocket,int timeout,string& data){

char buf[BUFSIZE];
strcpy(buf,data.c_str());
WSABUF buffer = { BUFSIZE,buf };
DWORD bytes_recvd;
int r;
ULONG_PTR ulKey = 0;
OVERLAPPED overlapped;
 OVERLAPPED* pov = NULL;
HANDLE port;

HANDLE hNewIOCP = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, ulKey, 0);
CreateIoCompletionPort((HANDLE)ServerSocket, hNewIOCP , ulKey, 0);


BOOL get = GetQueuedCompletionStatus(hNewIOCP,&bytes_recvd,&ulKey,&pov,timeout*1000);

if(!get)
    printf("waiton server failed. Error: %d\n",WSAGetLastError());
if(!pov)
    printf("waiton server failed. Error: %d\n",WSAGetLastError());

port = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, (u_long)0, 0);

SecureZeroMemory((PVOID) & overlapped, sizeof (WSAOVERLAPPED));

r = WSASend(ServerSocket, &buffer, 1, &bytes_recvd, NULL, &overlapped, NULL);
printf("WSA returned: %d WSALastError: %d\n",r,WSAGetLastError());
if(r != 0)
{
    printf("WSASend failed %d\n",GetLastError());
    printf("Bytes transfered: %d\n",bytes_recvd);
}
if (WSAGetLastError() == WSA_IO_PENDING)
    printf("we are async.\n");
CreateIoCompletionPort(port, &overlapped.hEvent,ulKey, 0);

BOOL test = GetQueuedCompletionStatus(port,&bytes_recvd,&ulKey,&pov,timeout*1000); 

CloseHandle(port);
return true;

}

Any insight would be appreciated.

Answer 1

You are associating the same socket with multiple IOCompletionPorts. I'm sure thats not valid. In your IOPortConnect function (Where you do the write) you call CreateIOCompletionPort 4 times passing in one shot handles.

My advice:

• Create a single IOCompletion Port (that, ultimately, you associate numerous sockets with).
• Create a pool of worker threads (by calling CreateThread) that each then block on the IOCompletionPort handle by calling GetQueuedCompletionStatus in a loop.
• Create one or more WSA_OVERLAPPED sockets, and associate each one with the IOCompletionPort.
• Use the WSA socket functions that take an OVERLAPPED* to trigger overlapped operations.
• Process the completion of the issued requests as the worker threads return from GetQueuedCompletionStatus with the OVERLAPPED* you passed in to start the operation.

Note: WSASend returns both 0, and SOCKET_ERROR with WSAGetLastError() as WSA_IO_PENDING as codes to indicate that you will get an IO Completion Packet arriving at GetQueuedCompletionStatus. Any other error code means you should process the error immediately as an IO operation was not queued so there will be no further callbacks.

Note2: The OVERLAPPED* passed to the WSASend (or whatever) function is the OVERLAPPED* returned from GetQueuedCompletionStatus. You can use this fact to pass more context information with the call:

struct MYOVERLAPPED {
  OVERLAPPED ovl;
};
MYOVERLAPPED ctx;
WSASend(...,&ctx.ovl);
...
OVERLAPPED* pov;
if(GetQueuedCompletionStatus(...,&pov,...)){
  MYOVERLAPPED* pCtx = (MYOVERLAPPED*)pov;

Answer 2

Chris has dealt with most of the issues and you've probably already looked at plenty of example code, but...

I've got some free IOCP code that's available here: http://www.serverframework.com/products---the-free-framework.html

There are also several of my CodeProject articles on the subject linked from that page.