Re: Consistent performance issues at high bandwidths, UDP.


"m" wrote:

Your seeing a performance impact from changing the number of sockets on the
SENDING side when sending to the same host from all sockets?

Almost. Yes, I am seeing a distinct performance impact from changing the
number of sockets on the sending side. However, I am not sending to the same
host from all sockets. There are 59 devices on the LAN, each with their own
IP address.

In the configuration with 927 sockets on the sending side, there are 14-16
sockets per destination. No socket sends to more than one destination.

In the configuration with 59 sockets on the sending side, there is 1 socket
per destination. No socket sends to more than one destination.

In the configuration with 1 socket on the sending side, that 1 socket is
used to send to all 59 destinations.


Thanks again for your replies,
Jason





"Jason Cipriani" <JasonCipriani@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:A6BD1237-1258-4EEB-B581-D3C7FA8A3373@xxxxxxxxxxxxxxxx
Thanks for your reply.

"m" wrote:
What is different between config 1 & 2?

The only difference is the number of sockets and how many packets I'm
sending with each socket. I also tried a test with a single socket,
sending
all data with just that socket. In that test each frame took 40ms to send,
and every 3.1 seconds a frame took 3.8 seconds to send (wow).

So, with the same frame size (~160kB), the same number of packets per
frame
(927 UDP packets, 174 bytes each), I can see distinctly different patterns
depending on how many sockets I split those packets between:

927 sockets x 1 packet: 10ms/frame, 0.7 seconds normal, 1.8 second delays

59 sockets x ~16 packets: 20ms/frame, 0.1 seconds normal, 0.13 second
delays

1 socket x 927 packets: 40ms/frame, 3.1 seconds normal, 3.8 second delays.

I also found something interesting, if I increase the packet size to 512
bytes (over a single socket, still testing the same amount of data), while
the frame rate was slow (took 150ms to send 160kB for some reason), it was
consistent and there were no regular delays. I can't explain that. Also
that
test is not that useful since increasing packet sizes is not an option...
the
protocol used to talk to the network devices relies on UDP packet
boundaries.


The most lickly cause is the NIC driver / hardware is not able to keep up
with your sending rate and the IO is stalling until it can catch up.
Lots
of gigabit NICs, espectially eary ones, cannot achive full gigabit speed.
Another strong possibilty is that the bus / bridge that feeds the NIC is
being saturated.

That is how I feel, too. I can't think of any other good explanation.
Testing with other hardware can help verify this.

In either case, test with alternate hardware and see what happens.

Tests on other hardware should be run today or tomorrow, when I get the
results back hopefully it will answer some questions.

The are
also some requistry parameters that control the IP stack behaviour in
some
situations that might affect you and you can reduce host load by using
overalpped IO & scatter / gather IO

I may have to start digging in to registry tweaks but I'd rather get as
close as I can through less hacky means first. With luck we'll find out
it's just unique to the machine we were running it on.

Machine causing problems was a MacBook Pro, Core 2 Duo T7700 @ 2.6 (i
think)
GHz. Not sure what NIC.

Next tests will be on a Mac Mini and an as-of-yet unidentified PC. I have
to
get the hardware specs from the people on site.

Development machine, which never showed any problems, was a Thinkpad T60
Core Duo T2600 @ 2.16GHz, but the network setup was significantly
different
(Intel 3945ABG Wireless, not really relevant). If none of the other
machines
work I'll have to take the Thinkpad there and test with that, I guess.
Remote
work is such a pain.

Thanks,
Jason

"Jason Cipriani" <JasonCipriani@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in
message
news:29C16DAE-BFB7-44C1-A97B-D2E3E60C4295@xxxxxxxxxxxxxxxx
I wrote:
In configuration 1, when sending frames as fast as possible, each
frame
generally takes about 10ms to send. The real problem is every 0.7
seconds
(~65 frames, ~10MB), a frame takes a whopping 1.8 seconds to send. I
can't
explain this massive delay, but it is at extremely regular intervals.

I should note that I sent data as fast as possible in the test
application,
but in the real application even when I limit as low as 30FPS, I still
see
the same problem, except the intervals are longer. For example, at
30FPS
(approx average bandwidth 30-40mbps) it takes 10-20ms to send a frame
and
I
still see the 1.8 second delays, but it happens every 3 or 4 seconds
rather
than every 0.7 seconds.

Thanks,
Jason


Original message:

"Jason Cipriani" wrote:
I have an application that streams data over the network at high
bandwidths
that is having a lot of performance issues. I narrowed it down to a
very
minimal case, and am leaving out all the rest of the details here:

The application is a multimedia application that must stream frames of
data
using UDP at consistent frame rates. Each frame is about 160kB split
into
927
UDP packets (of 174 bytes each), with each packet sent to one of 59
destination devices (max 16 packets per destination). It is a
dedicated
gigabit LAN and I've verified that a gigabit connection was
established
between all network devices. The performance issue I'm experiencing is
highly
inconsistent frame rates caused by a periodic delay in the call to
sendto()
at extremely regular intervals. For the purposes of testing I've
turned
off
the frame rate limiter and am sending data as fast as possible (with
the
limiter set to 60FPS average bandwidth is roughly 75mbps).

I have two configurations that I've tested:

1) 927 sockets. Each frame sends 174 bytes over each of these sockets.

2) 59 sockets. Each frame sends 2436 to 2784 bytes over each of these
sockets.

At gigabit speed each frame *should* take about 1.2ms to send (160kB /
1000mb ~= 0.0012, mind your bits and bytes). In practice I can't even
get
close to 1.2ms per frame (see below).

In configuration 1, when sending frames as fast as possible, each
frame
generally takes about 10ms to send. The real problem is every 0.7
seconds
(~65 frames, ~10MB), a frame takes a whopping 1.8 seconds to send. I
can't
explain this massive delay, but it is at extremely regular intervals.

In configuration 2, the same problem exists except each frame
generally
takes about 20ms to send, and every 0.1 seconds (~6 frames, ~1MB) , a
frame
takes 0.13 seconds to send. Again, it is at extremely regular
intervals.

I had thought that something else on the machine was interfering with
network communications (checking usual culprits, making sure wireless
networking disabled, etc.) but the thing is configuration 1 and 2
experience
distinctly different patterns of delay. One is 1.8 seconds every 0.7
seconds,
the other is 0.13 seconds every 0.1 seconds.

Another piece of information that may be important is the delay does
not
seem to happen when the network cable is unplugged. This, of course,
strongly
suggests some hardware issues at some point in the network. However, I
have
not been able to verify this yet as the software is for a client
running
things remotely, and communication is sometimes difficult
(unfortunately,
I
can't be at the site to witness the actual problem, which does not
occur
on
the development machine :-( ).

My test application does not do anything exotic. It simply creates the
sockets like so:

socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);

And sends data packets to their appropriate destinations:

sendto(p.sock, data, datalen, 0, paddr, addrlen);

All calls to sendto() are succeeding, all data is sent and I have
verified
that it is correctly received by the remote devices.

1. What is causing the delays at regular intervals?
2. Why does the delay time and interval length depend on how many
sockets
I
have open and/or how much data I'm sending per socket (note that in
both
configurations each frame is the same amount of data total)?
3. How can I troubleshoot / solve this? Are there some kind of socket
options I can set to improve performance and consistency?

It's frustrating because the software goes live soon, and these
problems
didn't occur until the software was run on site. Any advice, hints,
info
would be greatly appreciated.

Thanks a lot,
Jason






.

Relevant Pages

  • RE: Consistent performance issues at high bandwidths, UDP.
    ... explain this massive delay, but it is at extremely regular intervals. ... Each frame is about 160kB split into 927 ... Each frame sends 174 bytes over each of these sockets. ...
    (microsoft.public.win32.programmer.networks)
  • Re: Consistent performance issues at high bandwidths, UDP.
    ... all createing multiple sockets should do is use more non-paged memory since ... The only difference is the number of sockets and how many packets I'm ... and every 3.1 seconds a frame took 3.8 seconds to send. ...
    (microsoft.public.win32.programmer.networks)
  • Re: Consistent performance issues at high bandwidths, UDP.
    ... SENDING side when sending to the same host from all sockets? ... The only difference is the number of sockets and how many packets I'm ... and every 3.1 seconds a frame took 3.8 seconds to send. ... explain this massive delay, but it is at extremely regular intervals. ...
    (microsoft.public.win32.programmer.networks)
  • Consistent performance issues at high bandwidths, UDP.
    ... I have an application that streams data over the network at high bandwidths ... Each frame is about 160kB split into 927 ... Each frame sends 174 bytes over each of these sockets. ... explain this massive delay, but it is at extremely regular intervals. ...
    (microsoft.public.win32.programmer.networks)
  • Re: Consistent performance issues at high bandwidths, UDP.
    ... best performance with one socket per device (59 sockets), ... for it's own protocol packets. ... There were issues with network topology and low quality switches at the root ... throttling the application frame rate back. ...
    (microsoft.public.win32.programmer.networks)
Loading