Re: Consistent performance issues at high bandwidths, UDP.

"Volodymyr M. Shcherbyna" wrote:
Why are you using more then one (1) socket to send data to different nodes
over UDP?

That's the way the software was designed, however I've been able to get the
best performance with one socket per device (59 sockets), I can't explain why
nor will I try to, but it outperforms a single socket to all devices by a
long shot.

You are wasting system resources using more then 1 UDP socket in this case.

I do have the benefit of not having to use system resources for anything
else besides this application, so that's always a plus.

Also, since you are sending UDP packets quite intensivly I recommend per
each send(...) send close to MSS packet, i.e. for Ethernet 1472 bytes.
Since you are a multimedia application its better to compress your data, say
deflate will allow you to compress 12k of data into 1k (of course this
depends on kind of data, but this was my best result I got using deflate).

Compressing the data won't help, as the hardware on the receiving end
doesn't understand compression of any form. Sending larger packets won't
help, as the hardware on the receiving end relies on UDP packet boundaries
for it's own protocol packets. We have to send the data that the hardware is
expecting, therefore changing packet sizes or data formats is not something
that can be done.


Anyways, we've found the problem. It turned out to be unrelated to software.
There were issues with network topology and low quality switches at the root
of it. The reason for inconsistent application performance turned out to be
ethernet flow control (triggered because of other network hardware issues)
throttling the application frame rate back. We've moved to better network
switches, changed to a new network layout, and everything is working great.


Thanks for your replies,
Jason



--
Volodymyr, blog: http://www.shcherbyna.com/
(This posting is provided "AS IS" with no warranties, and confers no
rights)
"Jason Cipriani" <JasonCipriani@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:FB744993-F574-40FF-86DC-B7F8FF38663C@xxxxxxxxxxxxxxxx
I have an application that streams data over the network at high bandwidths
that is having a lot of performance issues. I narrowed it down to a very
minimal case, and am leaving out all the rest of the details here:

The application is a multimedia application that must stream frames of
data
using UDP at consistent frame rates. Each frame is about 160kB split into
927
UDP packets (of 174 bytes each), with each packet sent to one of 59
destination devices (max 16 packets per destination). It is a dedicated
gigabit LAN and I've verified that a gigabit connection was established
between all network devices. The performance issue I'm experiencing is
highly
inconsistent frame rates caused by a periodic delay in the call to
sendto()
at extremely regular intervals. For the purposes of testing I've turned
off
the frame rate limiter and am sending data as fast as possible (with the
limiter set to 60FPS average bandwidth is roughly 75mbps).

I have two configurations that I've tested:

1) 927 sockets. Each frame sends 174 bytes over each of these sockets.

2) 59 sockets. Each frame sends 2436 to 2784 bytes over each of these
sockets.

At gigabit speed each frame *should* take about 1.2ms to send (160kB /
1000mb ~= 0.0012, mind your bits and bytes). In practice I can't even get
close to 1.2ms per frame (see below).

In configuration 1, when sending frames as fast as possible, each frame
generally takes about 10ms to send. The real problem is every 0.7 seconds
(~65 frames, ~10MB), a frame takes a whopping 1.8 seconds to send. I can't
explain this massive delay, but it is at extremely regular intervals.

In configuration 2, the same problem exists except each frame generally
takes about 20ms to send, and every 0.1 seconds (~6 frames, ~1MB) , a
frame
takes 0.13 seconds to send. Again, it is at extremely regular intervals.

I had thought that something else on the machine was interfering with
network communications (checking usual culprits, making sure wireless
networking disabled, etc.) but the thing is configuration 1 and 2
experience
distinctly different patterns of delay. One is 1.8 seconds every 0.7
seconds,
the other is 0.13 seconds every 0.1 seconds.

Another piece of information that may be important is the delay does not
seem to happen when the network cable is unplugged. This, of course,
strongly
suggests some hardware issues at some point in the network. However, I
have
not been able to verify this yet as the software is for a client running
things remotely, and communication is sometimes difficult (unfortunately,
I
can't be at the site to witness the actual problem, which does not occur
on
the development machine :-( ).

My test application does not do anything exotic. It simply creates the
sockets like so:

socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);

And sends data packets to their appropriate destinations:

sendto(p.sock, data, datalen, 0, paddr, addrlen);

All calls to sendto() are succeeding, all data is sent and I have verified
that it is correctly received by the remote devices.

1. What is causing the delays at regular intervals?
2. Why does the delay time and interval length depend on how many sockets
I
have open and/or how much data I'm sending per socket (note that in both
configurations each frame is the same amount of data total)?
3. How can I troubleshoot / solve this? Are there some kind of socket
options I can set to improve performance and consistency?

It's frustrating because the software goes live soon, and these problems
didn't occur until the software was run on site. Any advice, hints, info
would be greatly appreciated.

Thanks a lot,
Jason



.

Relevant Pages

  • Re: Consistent performance issues at high bandwidths, UDP.
    ... all createing multiple sockets should do is use more non-paged memory ... The only difference is the number of sockets and how many packets ... and every 3.1 seconds a frame took 3.8 seconds to send. ... inconsistent frame rates caused by a periodic delay in the call ...
    (microsoft.public.win32.programmer.networks)
  • Re: Consistent performance issues at high bandwidths, UDP.
    ... The only difference is the number of sockets and how many packets I'm ... In that test each frame took 40ms to send, ... explain this massive delay, but it is at extremely regular intervals. ...
    (microsoft.public.win32.programmer.networks)
  • Re: Consistent performance issues at high bandwidths, UDP.
    ... SENDING side when sending to the same host from all sockets? ... The only difference is the number of sockets and how many packets I'm ... and every 3.1 seconds a frame took 3.8 seconds to send. ... explain this massive delay, but it is at extremely regular intervals. ...
    (microsoft.public.win32.programmer.networks)
  • Consistent performance issues at high bandwidths, UDP.
    ... I have an application that streams data over the network at high bandwidths ... Each frame is about 160kB split into 927 ... Each frame sends 174 bytes over each of these sockets. ... explain this massive delay, but it is at extremely regular intervals. ...
    (microsoft.public.win32.programmer.networks)
  • Re: Overhead, UDP: Many packets on 1 socket vs. one packet and many ?sockets.
    ... a gigabit LAN network using UDP. ... into "frames" and each device frame is a single UDP packet. ... information in the data packets. ... over each of those sockets. ...
    (comp.unix.programmer)
Loading