Internal TCP/IP send buffer?

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

From: Marton Anka (marton_at_03am.com)
Date: 10/22/04 

Date: Fri, 22 Oct 2004 03:01:31 +0200

Hello,

I'm trying to make two applications talk to each other over TCP/IP. The
sender has small blocks of data that need to be transmitted to the receiver
as soon as possible. Untransmitted packets might become obsolete quickly
therefore I need to minimize buffering on both the sender and the receiver
side.

In theory you can do this by disabling Nagle and setting SO_RCVBUF and so
SO_SNDBUF to a small size (let's suppose one kilobyte for now). Tests show
however that there's an internal operating system buffer that is 63836
(65536-1700) bytes in size. What is this buffer, and what can I do to work
around it?

Here's some quick and (very) dirty sample code.

Both the sender and the receiver start with initializing a socket:

 WSADATA wsad;
 WSAStartup(MAKEWORD(2,0), &wsad);
 SOCKET s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
 sockaddr_in sa;
 int len = sizeof(sa);
 ZeroMemory(&sa, sizeof(sa));
 sa.sin_family = AF_INET;
 sa.sin_port = htons(3333);
 sa.sin_addr.s_addr = inet_addr ("127.0.0.1");
 int nTotal = 0;

The receiver then accepts a connection and displays the bytes available for
read:

  bind(s, (SOCKADDR*)&sa, len);
  listen(s, SOMAXCONN);
  SOCKET sock = accept(s, (SOCKADDR*)&sa, &len);
  SetSocketOptions (sock);
  while(TRUE) {
   ioctlsocket(sock, FIONREAD, (u_long*)&nTotal);
   printf("Receivable: %d bytes\n", nTotal);
   Sleep(1000);
  }
  closesocket(sock);

The sender connects to the receiver and pushes down data one byte at a time:

  connect(s, (SOCKADDR*)&sa, len);
  SetSocketOptions (s);
  while(TRUE) {
   char sz[1];
   send(s, sz, sizeof(sz), 0);
   nTotal += sizeof(sz);
   printf("Sent %d bytes (total:%d)\n", sizeof(sz), nTotal);
  }

SetSocketOptions simply disables Nagle and sets both the receive and the
send buffers to one K:

void SetSocketOptions(SOCKET s)
{
 DWORD dwPacketSize = 1024;
 BOOL bTrue = TRUE;
 setsockopt(s, SOL_SOCKET, SO_RCVBUF, (char*)&dwPacketSize,
sizeof(dwPacketSize));
 setsockopt(s, SOL_SOCKET, SO_SNDBUF, (char*)&dwPacketSize,
sizeof(dwPacketSize));
 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, (PSTR)&bTrue, sizeof(bTrue));
}

Running the receiver produces the following (expected) output after the
sender connects:

Receivable: 1024 bytes
Receivable: 1024 bytes
.... etc

Running the sender produces the following:

....
Sent 1 bytes (total:65881)
Sent 1 bytes (total:65882)
Sent 1 bytes (total:65883)
Sent 1 bytes (total:65884)

...Then it stops here due to blocking on a send() that has no matching
recv()

The sender has 1k of buffers to work with that we set in SO_SNDBUF. The
receiver has another 1k of buffers due to SO_RCVBUF. We've managed to cram
65884 bytes down send()'s throat without blocking. We know where 2kbytes are
buffered, so where do the other 63836 bytes live? Note that 64836 is a
suspicious number - one would expect an even 64kbytes (IPv4 datagram?). The
difference is exactly 1700 bytes, which is close to an ethernet payload size
(1500 bytes) plus some change (headers, etc).

Can this internal buffer be disabled or limited in size somehow?

Thanks,

Marton Anka

Relevant Pages

  • Re: Neutral Format as a Coupling reduction idea that doesnt work.
    ... There are potentially unlimited and unexpected side effects for both sender and receiver. ... That definition defines the semantics of the just the data packet. ... That represents a solution level coupling but at least it is very narrowly focused on the message definition rather than what each side does with the information. ...
    (comp.object)
  • Re: PostMessage and unprocessed messages
    ... The sender has no way to know when to delete the message, so making it the owner has no ... Since the receiver would have no way to know the message is deleted, ... collection, note that as long as something is in the queue, it is owned by the queue ... so adding memory management complexity to the ...
    (microsoft.public.vc.mfc)
  • Re: PostMessage and unprocessed messages
    ... No need to keep a linked list, because the PostMessage queue is already ... The sender has no way to know when to delete the message, ... Since the receiver would have no way to know the message is deleted, ... so adding memory management complexity to the ...
    (microsoft.public.vc.mfc)
  • Re: PostMessage and unprocessed messages
    ... No need to keep a linked list, because the PostMessage queue is already ... The sender has no way to know when to delete the message, ... Since the receiver would have no way to know the message is deleted, ... so adding memory management complexity to the ...
    (microsoft.public.vc.mfc)
  • Re: Interface complexity problem in game
    ... one does not pass objects across the subsystem boundary that are ... data packet object) has to exist already in the sender subsystem implementation. ... the receiver can't do anything except manipulate the values in its own context and that will be explicitly understood by the receiver. ... one should never pass object references across subsystem boundaries, regardless of what some interoperability infrastructures allow one to do. ...
    (comp.object)