too many WSAEWOULDBLOCK with send() function (NB TCP)

[hbs24]

Hi,

introduction:
My project is to porting c code (proxy server) for WIN env.
The goal is to minimize modification like using select() etc.., so im try use the WEPOLL API and test how it works.
For some reason im got too many WSAEWOULDBLOCK messages when use the send()..and that happen after EPOLLOUT is set. I know the meaning but after all the epoll_wait function say Im ready to send.
When im test the CPU usage (task manger) send and recv goto 40%.
Im send http post to remote web server (that just reply with the same data...)
total data is 1MB with chunck of 10KB.
the avg message error is about 1300+- for two tcp sockets (sending the same http post...)

**when i put sleep(5) after getting WSAEWOULDBLOCK the CPU usage drop around 15%!
There is anything Im do worng? because its seems to me like the epoll mechanism differents in term of optimization from linux.

thanks a lot.

[Bert Belder]

@hbs24

• Is your server multi threaded?
• If it is multi threaded, are all threads polling the same epoll port, or do they all have a port of their own?

[hbs24]

my test code is not multi threaded but finally the server will be...now just make poc for the wepoll API validate its work like at linux...

[Bert Belder]

If epoll_wait() is reporting that the socket is writable, and yet on the next attempt to call send() it reports WSAEWOULDBLOCK, that'd be a bug.
Please post a test case that demonstrates the problem, otherwise there's not much I can do.

[hbs24]

#include<stdio.h>
#include<winsock2.h>

#pragma comment(lib,"ws2_32.lib") //Winsock Library

#include "wepoll.h"
#pragma warning(disable:4996)

WSADATA wsa;
SOCKET s;
struct sockaddr_in server;
char *message, server_reply[2001];
char sendbuf[24910];
int recv_size;
HANDLE epoll_hnd;
int i, r, state = 0, count = 0;
struct epoll_event ev, ev_out;
u_long one = 1;



typedef struct my_data {
	unsigned int total2send;
	unsigned int remain2send;
	unsigned int winsize;
	unsigned int from;
	SOCKET soc;
	char *msg;
}soc_t;


static SOCKET create_and_add_socket(HANDLE epoll_hnd) {
	SOCKET sock;
	u_long one = 1;
	int r;
	struct epoll_event ev;

	sock = socket(AF_INET, SOCK_STREAM, 0);
	if (sock == INVALID_SOCKET) {
		printf("Could not create socket : %d \n", WSAGetLastError());
	};

	r = ioctlsocket(sock, (long)FIONBIO, &one);
	if (r != NO_ERROR) {
		printf("ioctlsocket failed with error: %d\n", r);
	}

	ev.events = EPOLLOUT | EPOLLIN;
	ev.data.sock = sock;
	r = epoll_ctl(epoll_hnd, EPOLL_CTL_ADD, sock, &ev);
	printf("r = epoll_ctl =%d\n", r);
	return sock;
}



int main(int argc, char *argv[])
{
	epoll_hnd = epoll_create1(0);//here im create epoll instance
	if (epoll_hnd == -1) {
		printf("err create the epoll!\n");
		return 0;
	}
	
	s = create_and_add_socket(epoll_hnd);

	if (WSAStartup(MAKEWORD(2, 2), &wsa) != 0)
	{
		printf("Failed. Error Code : %d", WSAGetLastError());
		return 1;
	}
	server.sin_addr.s_addr = inet_addr("212.76.76.76");
	server.sin_family = AF_INET;
	server.sin_port = htons(80);

		if (r = connect(s, (struct sockaddr *)&server, sizeof(server)) < 0)
		{
			printf("connect error=%d\n", WSAGetLastError());
			if (WSAGetLastError() != WSAEWOULDBLOCK) {
				return 1;
			}
	}
	soc_t s1 = (soc_t) { 1000090, 1000090,25000, 0, s, message};//init the params
	
	memset(sendbuf, '$', 24910);//generate the header and data to send
	char message1[25000] = "POST / HTTP/1.1\r\nHost: 212.76.76.76\r\nContent-Type: text/html\r\nContent-Length: 1000000\r\n\r\n";
	memcpy(message1 + 90, sendbuf, 24909);
	char message2[25000];
	message2[24999] = '\0';
	memset(message2, '$', 24999);
	int  r2 = 0;


	//main epoll loop
	while (1) {
		r2 = epoll_wait(epoll_hnd, &ev_out, 1, -1);
		printf("WSApoll=%u \n", r2);
		if (r2 != SOCKET_ERROR) {
				if (s1.remain2send) {
					if (ev_out.events & EPOLLOUT) {
						printf("ready to write!\n");
						s1.msg = (s1.total2send - s1.remain2send > sizeof(message1)) ? message2 + s1.from : message1 + s1.from;
						s1.winsize = (s1.remain2send > s1.winsize) ? s1.winsize : s1.remain2send;
						if ((r = send(s1.soc, s1.msg, s1.winsize, 0)) < 0)
						{
							printf("send error=%d\n", WSAGetLastError());
							if (WSAGetLastError() != WSAEWOULDBLOCK) {
								return 1;
							}
						}
						else {
							s1.remain2send -= r;
							s1.from = (r == s1.winsize) ? 0 : r;
							printf("send %d bytes by %d..remain to send more %d bytes!\n", r, ev_out.events, s1.remain2send);
						}
					}
				}
				else if (!s1.remain2send) {
					if (ev_out.events & EPOLLIN) {
						printf("event2=%u\n", ev_out.events);
						if ((recv_size = recv(s1.soc, server_reply, 2000, 0)) == SOCKET_ERROR)
						{
							puts("recv failed");
						}
						else {
							if (!recv_size) {
								printf("connection close\n");
							}
							puts("Reply received\n");
							server_reply[recv_size] = '\0';
							puts(server_reply);
							printf("recv_size=%d\n by %d", recv_size,ev_out.events);
						}
					}
					else if (ev_out.events & POLLHUP) {
						printf("event3=%u\n", ev_out.events);
						break;
					}
				}
		}
		else {
			printf("socket error: %d \n", WSAGetLastError());
			break;
		}
	}
	getchar();
	return 0;
}

[hbs24]

[Bert Belder]

@hbs24 It does indeed look like there's a bug where EPOLLOUT is always reported for outgoing connections. Thanks for reporting it. I'll push a fix shortly.

[Bert Belder]

@hbs24
The bug has been fixed in version 1.5.4.
I'm going to close this issue now - feel free to open another one if you run into other bugs.
Thanks again for reporting this one.