Re: Consistent performance issues at high bandwidths, UDP.
- From: "m" <m@xxx>
- Date: Wed, 29 Oct 2008 19:18:18 -0400
What is different between config 1 & 2?
The most lickly cause is the NIC driver / hardware is not able to keep up
with your sending rate and the IO is stalling until it can catch up. Lots
of gigabit NICs, espectially eary ones, cannot achive full gigabit speed.
Another strong possibilty is that the bus / bridge that feeds the NIC is
being saturated.
In either case, test with alternate hardware and see what happens. The are
also some requistry parameters that control the IP stack behaviour in some
situations that might affect you and you can reduce host load by using
overalpped IO & scatter / gather IO
"Jason Cipriani" <JasonCipriani@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:29C16DAE-BFB7-44C1-A97B-D2E3E60C4295@xxxxxxxxxxxxxxxx
I wrote:
In configuration 1, when sending frames as fast as possible, each frame
generally takes about 10ms to send. The real problem is every 0.7 seconds
(~65 frames, ~10MB), a frame takes a whopping 1.8 seconds to send. I
can't
explain this massive delay, but it is at extremely regular intervals.
I should note that I sent data as fast as possible in the test
application,
but in the real application even when I limit as low as 30FPS, I still see
the same problem, except the intervals are longer. For example, at 30FPS
(approx average bandwidth 30-40mbps) it takes 10-20ms to send a frame and
I
still see the 1.8 second delays, but it happens every 3 or 4 seconds
rather
than every 0.7 seconds.
Thanks,
Jason
Original message:
"Jason Cipriani" wrote:
I have an application that streams data over the network at high
bandwidths
that is having a lot of performance issues. I narrowed it down to a very
minimal case, and am leaving out all the rest of the details here:
The application is a multimedia application that must stream frames of
data
using UDP at consistent frame rates. Each frame is about 160kB split into
927
UDP packets (of 174 bytes each), with each packet sent to one of 59
destination devices (max 16 packets per destination). It is a dedicated
gigabit LAN and I've verified that a gigabit connection was established
between all network devices. The performance issue I'm experiencing is
highly
inconsistent frame rates caused by a periodic delay in the call to
sendto()
at extremely regular intervals. For the purposes of testing I've turned
off
the frame rate limiter and am sending data as fast as possible (with the
limiter set to 60FPS average bandwidth is roughly 75mbps).
I have two configurations that I've tested:
1) 927 sockets. Each frame sends 174 bytes over each of these sockets.
2) 59 sockets. Each frame sends 2436 to 2784 bytes over each of these
sockets.
At gigabit speed each frame *should* take about 1.2ms to send (160kB /
1000mb ~= 0.0012, mind your bits and bytes). In practice I can't even get
close to 1.2ms per frame (see below).
In configuration 1, when sending frames as fast as possible, each frame
generally takes about 10ms to send. The real problem is every 0.7 seconds
(~65 frames, ~10MB), a frame takes a whopping 1.8 seconds to send. I
can't
explain this massive delay, but it is at extremely regular intervals.
In configuration 2, the same problem exists except each frame generally
takes about 20ms to send, and every 0.1 seconds (~6 frames, ~1MB) , a
frame
takes 0.13 seconds to send. Again, it is at extremely regular intervals.
I had thought that something else on the machine was interfering with
network communications (checking usual culprits, making sure wireless
networking disabled, etc.) but the thing is configuration 1 and 2
experience
distinctly different patterns of delay. One is 1.8 seconds every 0.7
seconds,
the other is 0.13 seconds every 0.1 seconds.
Another piece of information that may be important is the delay does not
seem to happen when the network cable is unplugged. This, of course,
strongly
suggests some hardware issues at some point in the network. However, I
have
not been able to verify this yet as the software is for a client running
things remotely, and communication is sometimes difficult (unfortunately,
I
can't be at the site to witness the actual problem, which does not occur
on
the development machine :-( ).
My test application does not do anything exotic. It simply creates the
sockets like so:
socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
And sends data packets to their appropriate destinations:
sendto(p.sock, data, datalen, 0, paddr, addrlen);
All calls to sendto() are succeeding, all data is sent and I have
verified
that it is correctly received by the remote devices.
1. What is causing the delays at regular intervals?
2. Why does the delay time and interval length depend on how many sockets
I
have open and/or how much data I'm sending per socket (note that in both
configurations each frame is the same amount of data total)?
3. How can I troubleshoot / solve this? Are there some kind of socket
options I can set to improve performance and consistency?
It's frustrating because the software goes live soon, and these problems
didn't occur until the software was run on site. Any advice, hints, info
would be greatly appreciated.
Thanks a lot,
Jason
.
- Follow-Ups:
- Re: Consistent performance issues at high bandwidths, UDP.
- From: Jason Cipriani
- Re: Consistent performance issues at high bandwidths, UDP.
- References:
- Consistent performance issues at high bandwidths, UDP.
- From: Jason Cipriani
- RE: Consistent performance issues at high bandwidths, UDP.
- From: Jason Cipriani
- Consistent performance issues at high bandwidths, UDP.
- Prev by Date: RE: TCP communication hang
- Next by Date: Re: WinInet DNS cache issue
- Previous by thread: RE: Consistent performance issues at high bandwidths, UDP.
- Next by thread: Re: Consistent performance issues at high bandwidths, UDP.
- Index(es):
Relevant Pages
|
