Re: A question regarding MTU: how it can effect TCP performance + other queries

Hi all,

Thanks for all the replies.

Again, my question is not regarding TCP. I have identified reasons why TCP
throughput is less in case of virtual NIC.

My concern is about UDP throughput that why UDP Iperf is reporting UDP
throughpu on virtual NIC greater than that on in physical NIC. I am using
Iperf with default options for UDP. Below is the output of Iperf UDP client
for physical NIC.

------------------------------------------------------------
Client connecting to 192.168.1.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 8.00 KByte (default)
------------------------------------------------------------
[1184] local 192.168.1.1 port 2281 connected with 192.168.1.2 port 5001
[1564] local 192.168.1.1 port 2262 connected with 192.168.1.2 port 5001
[1524] local 192.168.1.1 port 2264 connected with 192.168.1.2 port 5001
[1444] local 192.168.1.1 port 2268 connected with 192.168.1.2 port 5001
[1364] local 192.168.1.1 port 2272 connected with 192.168.1.2 port 5001
[1384] local 192.168.1.1 port 2271 connected with 192.168.1.2 port 5001
[1324] local 192.168.1.1 port 2274 connected with 192.168.1.2 port 5001
[1404] local 192.168.1.1 port 2270 connected with 192.168.1.2 port 5001
[1284] local 192.168.1.1 port 2276 connected with 192.168.1.2 port 5001
[1244] local 192.168.1.1 port 2278 connected with 192.168.1.2 port 5001
[1204] local 192.168.1.1 port 2280 connected with 192.168.1.2 port 5001
[1264] local 192.168.1.1 port 2277 connected with 192.168.1.2 port 5001
[1684] local 192.168.1.1 port 2257 connected with 192.168.1.2 port 5001
[1424] local 192.168.1.1 port 2269 connected with 192.168.1.2 port 5001
[1544] local 192.168.1.1 port 2263 connected with 192.168.1.2 port 5001
[1224] local 192.168.1.1 port 2279 connected with 192.168.1.2 port 5001
[1464] local 192.168.1.1 port 2267 connected with 192.168.1.2 port 5001
[1624] local 192.168.1.1 port 2259 connected with 192.168.1.2 port 5001
[1304] local 192.168.1.1 port 2275 connected with 192.168.1.2 port 5001
[1504] local 192.168.1.1 port 2265 connected with 192.168.1.2 port 5001
[1584] local 192.168.1.1 port 2261 connected with 192.168.1.2 port 5001
[1344] local 192.168.1.1 port 2273 connected with 192.168.1.2 port 5001
[1484] local 192.168.1.1 port 2266 connected with 192.168.1.2 port 5001
[1604] local 192.168.1.1 port 2260 connected with 192.168.1.2 port 5001
[1644] local 192.168.1.1 port 2258 connected with 192.168.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[1564] 0.0-10.0 sec 21.8 MBytes 18.2 Mbits/sec
[1184] 0.0-10.0 sec 21.7 MBytes 18.2 Mbits/sec
[1444] 0.0-10.0 sec 21.8 MBytes 18.2 Mbits/sec
[1524] 0.0-10.0 sec 21.8 MBytes 18.3 Mbits/sec
[1364] 0.0-10.0 sec 21.9 MBytes 18.4 Mbits/sec
[1384] 0.0-10.0 sec 21.7 MBytes 18.2 Mbits/sec
[1404] 0.0-10.0 sec 21.8 MBytes 18.3 Mbits/sec
[1204] 0.0-10.0 sec 22.0 MBytes 18.4 Mbits/sec
[1244] 0.0-10.0 sec 22.1 MBytes 18.5 Mbits/sec
[1324] 0.0-10.0 sec 21.9 MBytes 18.4 Mbits/sec
[1284] 0.0-10.0 sec 22.0 MBytes 18.4 Mbits/sec
[1264] 0.0-10.0 sec 21.9 MBytes 18.4 Mbits/sec
[1464] 0.0-10.0 sec 22.1 MBytes 18.5 Mbits/sec
[1224] 0.0-10.0 sec 22.1 MBytes 18.5 Mbits/sec
[1424] 0.0-10.0 sec 22.0 MBytes 18.5 Mbits/sec
[1684] 0.0-10.0 sec 22.0 MBytes 18.5 Mbits/sec
[1544] 0.0-10.0 sec 22.0 MBytes 18.4 Mbits/sec
[1504] 0.0-10.0 sec 22.2 MBytes 18.6 Mbits/sec
[1624] 0.0-10.0 sec 22.2 MBytes 18.6 Mbits/sec
[1304] 0.0-10.0 sec 22.3 MBytes 18.6 Mbits/sec
[ ID] Interval Transfer Bandwidth
[1584] 0.0-10.0 sec 22.5 MBytes 18.8 Mbits/sec
[1344] 0.0-10.0 sec 22.7 MBytes 19.0 Mbits/sec
[1484] 0.0-10.0 sec 22.7 MBytes 19.0 Mbits/sec
[1604] 0.0-10.0 sec 23.0 MBytes 19.3 Mbits/sec
[1644] 0.0-10.0 sec 23.0 MBytes 19.3 Mbits/sec
[1564] WARNING: did not receive ack of last datagram after 10 tries.
[1184] WARNING: did not receive ack of last datagram after 10 tries.
[1564] Sent 15516 datagrams
[1184] Sent 15474 datagrams
[1524] WARNING: did not receive ack of last datagram after 10 tries.
[1524] Sent 15582 datagrams
[1444] WARNING: did not receive ack of last datagram after 10 tries.
[1444] Sent 15535 datagrams
[1384] WARNING: did not receive ack of last datagram after 10 tries.
[1364] WARNING: did not receive ack of last datagram after 10 tries.
[1384] Sent 15459 datagrams
[1364] Sent 15647 datagrams
[1264] WARNING: did not receive ack of last datagram after 10 tries.
[1204] WARNING: did not receive ack of last datagram after 10 tries.
[1264] Sent 15638 datagrams
[1204] Sent 15675 datagrams
[1284] WARNING: did not receive ack of last datagram after 10 tries.
[1284] Sent 15684 datagrams
[1244] WARNING: did not receive ack of last datagram after 10 tries.
[1244] Sent 15734 datagrams
[1324] WARNING: did not receive ack of last datagram after 10 tries.
[1324] Sent 15630 datagrams
[1404] WARNING: did not receive ack of last datagram after 10 tries.
[1404] Sent 15580 datagrams
[1464] WARNING: did not receive ack of last datagram after 10 tries.
[1464] Sent 15763 datagrams
[1424] WARNING: did not receive ack of last datagram after 10 tries.
[1424] Sent 15717 datagrams
[1544] WARNING: did not receive ack of last datagram after 10 tries.
[1684] WARNING: did not receive ack of last datagram after 10 tries.
[1544] Sent 15704 datagrams
[1684] Sent 15714 datagrams
[1224] WARNING: did not receive ack of last datagram after 10 tries.
[1224] Sent 15773 datagrams
[1504] WARNING: did not receive ack of last datagram after 10 tries.
[1624] WARNING: did not receive ack of last datagram after 10 tries.
[1504] Sent 15842 datagrams
[1624] Sent 15844 datagrams
[1304] WARNING: did not receive ack of last datagram after 10 tries.
[1304] Sent 15875 datagrams
[1584] WARNING: did not receive ack of last datagram after 10 tries.
[1584] Sent 16050 datagrams
[1484] WARNING: did not receive ack of last datagram after 10 tries.
[1344] WARNING: did not receive ack of last datagram after 10 tries.
[1484] Sent 16209 datagrams
[1344] Sent 16180 datagrams
[1604] WARNING: did not receive ack of last datagram after 10 tries.
[1604] Sent 16398 datagrams
[1644] WARNING: did not receive ack of last datagram after 10 tries.
[1644] Sent 16399 datagrams
[SUM] 0.0-10.2 sec 553 MBytes 456 Mbits/sec

The output of Iperf UDP client for virtual NIC is as follows:

------------------------------------------------------------
Client connecting to 192.168.100.16, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 8.00 KByte (default)
------------------------------------------------------------
[1196] local 192.168.100.17 port 2579 connected with 192.168.100.16 port
5001
[1684] local 192.168.100.17 port 2558 connected with 192.168.100.16 port
5001
[1580] local 192.168.100.17 port 2560 connected with 192.168.100.16 port
5001
[1560] local 192.168.100.17 port 2561 connected with 192.168.100.16 port
5001
[1540] local 192.168.100.17 port 2562 connected with 192.168.100.16 port
5001
[1520] local 192.168.100.17 port 2563 connected with 192.168.100.16 port
5001
[1500] local 192.168.100.17 port 2564 connected with 192.168.100.16 port
5001
[1480] local 192.168.100.17 port 2565 connected with 192.168.100.16 port
5001
[1460] local 192.168.100.17 port 2566 connected with 192.168.100.16 port
5001
[1440] local 192.168.100.17 port 2567 connected with 192.168.100.16 port
5001
[1420] local 192.168.100.17 port 2568 connected with 192.168.100.16 port
5001
[1400] local 192.168.100.17 port 2569 connected with 192.168.100.16 port
5001
[1380] local 192.168.100.17 port 2570 connected with 192.168.100.16 port
5001
[1360] local 192.168.100.17 port 2571 connected with 192.168.100.16 port
5001
[1340] local 192.168.100.17 port 2572 connected with 192.168.100.16 port
5001
[1320] local 192.168.100.17 port 2573 connected with 192.168.100.16 port
5001
[1300] local 192.168.100.17 port 2574 connected with 192.168.100.16 port
5001
[1276] local 192.168.100.17 port 2575 connected with 192.168.100.16 port
5001
[1236] local 192.168.100.17 port 2577 connected with 192.168.100.16 port
5001
[1156] local 192.168.100.17 port 2581 connected with 192.168.100.16 port
5001
[1260] local 192.168.100.17 port 2576 connected with 192.168.100.16 port
5001
[1216] local 192.168.100.17 port 2578 connected with 192.168.100.16 port
5001
[1176] local 192.168.100.17 port 2580 connected with 192.168.100.16 port
5001
[1604] local 192.168.100.17 port 2559 connected with 192.168.100.16 port
5001
[1136] local 192.168.100.17 port 2582 connected with 192.168.100.16 port
5001
[ ID] Interval Transfer Bandwidth
[1196] 0.0-10.0 sec 47.8 MBytes 40.0 Mbits/sec
[1684] 0.0-10.0 sec 30.7 MBytes 25.7 Mbits/sec
[1560] 0.0-10.0 sec 48.3 MBytes 40.4 Mbits/sec
[1580] 0.0-10.0 sec 30.8 MBytes 25.8 Mbits/sec
[1176] 0.0-10.0 sec 30.7 MBytes 25.7 Mbits/sec
[1320] 0.0-10.0 sec 47.9 MBytes 40.1 Mbits/sec
[1276] 0.0-10.0 sec 48.1 MBytes 40.3 Mbits/sec
[1400] 0.0-10.0 sec 48.0 MBytes 40.2 Mbits/sec
[1216] 0.0-10.0 sec 30.6 MBytes 25.6 Mbits/sec
[1540] 0.0-10.0 sec 30.4 MBytes 25.5 Mbits/sec
[1300] 0.0-10.0 sec 31.3 MBytes 26.2 Mbits/sec
[1480] 0.0-10.0 sec 47.9 MBytes 40.2 Mbits/sec
[1440] 0.0-10.0 sec 47.9 MBytes 40.1 Mbits/sec
[1236] 0.0-10.0 sec 47.9 MBytes 40.1 Mbits/sec
[1360] 0.0-10.0 sec 47.7 MBytes 39.9 Mbits/sec
[1156] 0.0-10.0 sec 48.5 MBytes 40.6 Mbits/sec
[1520] 0.0-10.0 sec 48.2 MBytes 40.3 Mbits/sec
[1420] 0.0-10.0 sec 31.4 MBytes 26.3 Mbits/sec
[1260] 0.0-10.0 sec 30.0 MBytes 25.2 Mbits/sec
[1380] 0.0-10.0 sec 31.2 MBytes 26.1 Mbits/sec
[ ID] Interval Transfer Bandwidth
[1500] 0.0-10.0 sec 29.9 MBytes 25.0 Mbits/sec
[1460] 0.0-10.0 sec 30.4 MBytes 25.5 Mbits/sec
[1340] 0.0-10.0 sec 30.7 MBytes 25.7 Mbits/sec
[1136] 0.0-10.0 sec 34.3 MBytes 28.7 Mbits/sec
[1604] 0.0-10.0 sec 52.0 MBytes 43.5 Mbits/sec
[1196] WARNING: did not receive ack of last datagram after 10 tries.
[1196] Sent 34096 datagrams

So any idea why is it like that?

Thanks,

Arsalan

"Pankaj Garg" <pankajg@xxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:Pine.WNT.4.64.0611292332321.2364@xxxxxxxxxxxxxxxxx

*Removing multiple newsgroups as cross posting usually generates tons of
extra traffic.*

What packet size IO are you doing in case of TCP? Can you check if your
physical NIC has TCP large send offload enabled?

The reason for the large difference in TCP performance may be due to the
offload happening in case of physical NIC.

I can't think of anything for the UDP case however, that just seems
strange to me. You probably need to dig deeper and tell us what exactly is
happening here. How many packets are being sent, what was the size of
these packets? Are you grouping multiple UDP packets in one TCP packet?

--
Pankaj Garg
2006-11-29 at 11:32pm
http://www.intellectualheaven.com


On Wed, 29 Nov 2006, Steve Dispensa wrote:


I'm sorry, I misread the UDP part of your question. You're right,
something seems off there. Are you buffering somewhere somehow? Are some
packets hitting the bit bucket after being delivered to the virtual NIC,
perhaps?

As for TCP case, you said:

What I found that for TCP, throughput that I am getting on physical
interface is around 900 Mbps and on virtual interface i am getting
around 90 Mbps (only 10% of the physcial one). All the tests were
performed with Iperf with 15 parallel streams.

and

So, for TCP packets over virtual interface, I am sending a TCP packet
encapsulated within another TCP packet when passed to physical
interface, while for UDP I am sending UDP packet encapsulated within
TCP packet when passed to physical interface.

Nagle would matter for UDP packets encapsulated within TCP packets,
depending (a great deal) on the characteristics of the traffic you are
testing with. Nagle's has an adverse affect on transmission speed,
because it introduces an artificial queuing delay, regardless of what
it's carrying. I suggested turning on NODELAY for the "outside" TCP
connection, not (necessarily) the "inside" connection.

At any rate, your results do seem strange to me; not a whole lot else
comes to mind though. You might want to instrument your miniport and
track where the packets spend their time between receipt at your send
handler and final send() to the other system.

Good luck.

-sd


On 2006-11-29 07:37:48 -0600, "Arsalan Ahmad" <arsal__@xxxxxxxxxxx> said:

My question was that Iperf is showing less UDP throughput over physical
interface (around 550Mbps for 15 parallel streams) while for virtual NIC
it is showing greater throughput (around 1Gbps for 15 parallel streams).
So why is this? because what I thought that UDP thoughput should be high
in case of physical interface which is not I am getting. And how
Nagle's. Turn on NODELAY for TCP can effect UDP throughput performance?

Thanks,

Arsalan


"Steve Dispensa" <dispensa@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote in
message
news:2006112900064250073-dispensa@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
On 2006-11-28 04:03:23 -0600, "Arsalan Ahmad" <arsal__@xxxxxxxxxxx>
said:

Hello all,

I have been doing some throughput tests with Iperf for my VNIC and
physical NIC. I connect two P-4 3.4 GHz computers with 1Gbps NIC cards
together via cross-over cable and then performed my experiments.

What I found that for TCP, throughput that I am getting on physical
interface is around 900 Mbps and on virtual interface i am getting
around 90 Mbps (only 10% of the physcial one). All the tests were
performed with Iperf with 15 parallel streams.

However, for UDP its totally different. With Iperf with 15 parallel
streams, I am getting only around 500 Mbps throughput on physical
interface while approximately 1000Mbps (1Gbps) on virtual interface. I
am unable to figure out why am I getting this strange behaviour. Can
you guess?

Yes, I can guess; are you tunneling the packets? Do you have Nagle's
turned on? Did you do the math on the bandwidth-delay product and make
your TCP window was set appropriately?


My virtual NIC pass all the packets to a user-level application which
then send all the IP packets of virtual interface to user-level
application running on the other computer. The communication between
the two user-level application is via TCP.

Windows is sensitive to the exact structure of your sockets code. To
get the best possible performance from the OS, you may want to look
into using I/O Completion for your tunnel code on both ends, and make
sure you keep enough buffers pending such that the protocol is never
waiting on you.


So, for TCP packets over virtual interface, I am sending a TCP packet
encapsulated within another TCP packet when passed to physical
interface, while for UDP I am sending UDP packet encapsulated within
TCP packet when passed to physical interface.

Given this, I'm voting for Nagle's. Turn on NODELAY and re-test.

-sd






.

Relevant Pages

  • [Full-disclosure] Strange repeating probes to port 80
    ... for unknown reason I decided to create very lame honeypot. ... enabled IIS and forwarded ports 80 and 135 (both TCP and UDP). ... Then after ACK from remote host TCP data is sent ...
    (Full-Disclosure)
  • Strange repeating probes to port 80
    ... for unknown reason I decided to create very lame honeypot. ... enabled IIS and forwarded ports 80 and 135 (both TCP and UDP). ... Then after ACK from remote host TCP data is sent ...
    (Security-Basics)
  • Performance on udp socket
    ... (UDP 150 byte) ... with bad checksum ... broadcast/multicast datagrams dropped due to no socket ...
    (freebsd-performance)
  • Re: Loosing UDP packets...
    ... The applications I support send lots of UDP via ... Sometimes all the packets hit the wire, ... Perhaps that is only for a non-blocking socket - per my stuff above? ... as I don't send> MTU datagrams. ...
    (comp.os.linux.networking)
  • Re: UDP datagram loss in my C# application level, what is wrong?
    ... it must be in my application where the UDP datagrams are lost. ... data irregularities are _always_ a possibility with UDP. ... You may want to consider buffer pooling and threaded work scheduling ...
    (microsoft.public.dotnet.languages.csharp)
Quantcast