Re: Consistent performance issues at high bandwidths, UDP.
- From: Jason Cipriani <JasonCipriani@xxxxxxxxxxxxxxxxxxxxxxxxx>
- Date: Wed, 29 Oct 2008 16:58:01 -0700
Thanks for your reply.
"m" wrote:
What is different between config 1 & 2?
The only difference is the number of sockets and how many packets I'm
sending with each socket. I also tried a test with a single socket, sending
all data with just that socket. In that test each frame took 40ms to send,
and every 3.1 seconds a frame took 3.8 seconds to send (wow).
So, with the same frame size (~160kB), the same number of packets per frame
(927 UDP packets, 174 bytes each), I can see distinctly different patterns
depending on how many sockets I split those packets between:
927 sockets x 1 packet: 10ms/frame, 0.7 seconds normal, 1.8 second delays
59 sockets x ~16 packets: 20ms/frame, 0.1 seconds normal, 0.13 second delays
1 socket x 927 packets: 40ms/frame, 3.1 seconds normal, 3.8 second delays.
I also found something interesting, if I increase the packet size to 512
bytes (over a single socket, still testing the same amount of data), while
the frame rate was slow (took 150ms to send 160kB for some reason), it was
consistent and there were no regular delays. I can't explain that. Also that
test is not that useful since increasing packet sizes is not an option... the
protocol used to talk to the network devices relies on UDP packet boundaries.
The most lickly cause is the NIC driver / hardware is not able to keep up
with your sending rate and the IO is stalling until it can catch up. Lots
of gigabit NICs, espectially eary ones, cannot achive full gigabit speed.
Another strong possibilty is that the bus / bridge that feeds the NIC is
being saturated.
That is how I feel, too. I can't think of any other good explanation.
Testing with other hardware can help verify this.
In either case, test with alternate hardware and see what happens.
Tests on other hardware should be run today or tomorrow, when I get the
results back hopefully it will answer some questions.
The are
also some requistry parameters that control the IP stack behaviour in some
situations that might affect you and you can reduce host load by using
overalpped IO & scatter / gather IO
I may have to start digging in to registry tweaks but I'd rather get as
close as I can through less hacky means first. With luck we'll find out
it's just unique to the machine we were running it on.
Machine causing problems was a MacBook Pro, Core 2 Duo T7700 @ 2.6 (i think)
GHz. Not sure what NIC.
Next tests will be on a Mac Mini and an as-of-yet unidentified PC. I have to
get the hardware specs from the people on site.
Development machine, which never showed any problems, was a Thinkpad T60
Core Duo T2600 @ 2.16GHz, but the network setup was significantly different
(Intel 3945ABG Wireless, not really relevant). If none of the other machines
work I'll have to take the Thinkpad there and test with that, I guess. Remote
work is such a pain.
Thanks,
Jason
"Jason Cipriani" <JasonCipriani@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message.
news:29C16DAE-BFB7-44C1-A97B-D2E3E60C4295@xxxxxxxxxxxxxxxx
I wrote:
In configuration 1, when sending frames as fast as possible, each frame
generally takes about 10ms to send. The real problem is every 0.7 seconds
(~65 frames, ~10MB), a frame takes a whopping 1.8 seconds to send. I
can't
explain this massive delay, but it is at extremely regular intervals.
I should note that I sent data as fast as possible in the test
application,
but in the real application even when I limit as low as 30FPS, I still see
the same problem, except the intervals are longer. For example, at 30FPS
(approx average bandwidth 30-40mbps) it takes 10-20ms to send a frame and
I
still see the 1.8 second delays, but it happens every 3 or 4 seconds
rather
than every 0.7 seconds.
Thanks,
Jason
Original message:
"Jason Cipriani" wrote:
I have an application that streams data over the network at high
bandwidths
that is having a lot of performance issues. I narrowed it down to a very
minimal case, and am leaving out all the rest of the details here:
The application is a multimedia application that must stream frames of
data
using UDP at consistent frame rates. Each frame is about 160kB split into
927
UDP packets (of 174 bytes each), with each packet sent to one of 59
destination devices (max 16 packets per destination). It is a dedicated
gigabit LAN and I've verified that a gigabit connection was established
between all network devices. The performance issue I'm experiencing is
highly
inconsistent frame rates caused by a periodic delay in the call to
sendto()
at extremely regular intervals. For the purposes of testing I've turned
off
the frame rate limiter and am sending data as fast as possible (with the
limiter set to 60FPS average bandwidth is roughly 75mbps).
I have two configurations that I've tested:
1) 927 sockets. Each frame sends 174 bytes over each of these sockets.
2) 59 sockets. Each frame sends 2436 to 2784 bytes over each of these
sockets.
At gigabit speed each frame *should* take about 1.2ms to send (160kB /
1000mb ~= 0.0012, mind your bits and bytes). In practice I can't even get
close to 1.2ms per frame (see below).
In configuration 1, when sending frames as fast as possible, each frame
generally takes about 10ms to send. The real problem is every 0.7 seconds
(~65 frames, ~10MB), a frame takes a whopping 1.8 seconds to send. I
can't
explain this massive delay, but it is at extremely regular intervals.
In configuration 2, the same problem exists except each frame generally
takes about 20ms to send, and every 0.1 seconds (~6 frames, ~1MB) , a
frame
takes 0.13 seconds to send. Again, it is at extremely regular intervals.
I had thought that something else on the machine was interfering with
network communications (checking usual culprits, making sure wireless
networking disabled, etc.) but the thing is configuration 1 and 2
experience
distinctly different patterns of delay. One is 1.8 seconds every 0.7
seconds,
the other is 0.13 seconds every 0.1 seconds.
Another piece of information that may be important is the delay does not
seem to happen when the network cable is unplugged. This, of course,
strongly
suggests some hardware issues at some point in the network. However, I
have
not been able to verify this yet as the software is for a client running
things remotely, and communication is sometimes difficult (unfortunately,
I
can't be at the site to witness the actual problem, which does not occur
on
the development machine :-( ).
My test application does not do anything exotic. It simply creates the
sockets like so:
socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
And sends data packets to their appropriate destinations:
sendto(p.sock, data, datalen, 0, paddr, addrlen);
All calls to sendto() are succeeding, all data is sent and I have
verified
that it is correctly received by the remote devices.
1. What is causing the delays at regular intervals?
2. Why does the delay time and interval length depend on how many sockets
I
have open and/or how much data I'm sending per socket (note that in both
configurations each frame is the same amount of data total)?
3. How can I troubleshoot / solve this? Are there some kind of socket
options I can set to improve performance and consistency?
It's frustrating because the software goes live soon, and these problems
didn't occur until the software was run on site. Any advice, hints, info
would be greatly appreciated.
Thanks a lot,
Jason
- Follow-Ups:
- References:
- Consistent performance issues at high bandwidths, UDP.
- From: Jason Cipriani
- RE: Consistent performance issues at high bandwidths, UDP.
- From: Jason Cipriani
- Re: Consistent performance issues at high bandwidths, UDP.
- From: m
- Consistent performance issues at high bandwidths, UDP.
- Prev by Date: Re: WinInet DNS cache issue
- Next by Date: Re: Consistent performance issues at high bandwidths, UDP.
- Previous by thread: Re: Consistent performance issues at high bandwidths, UDP.
- Next by thread: Re: Consistent performance issues at high bandwidths, UDP.
- Index(es):
Relevant Pages
|
