Re: C# System.String Memory Usage [TCP IP Connections]

Hi Shawn,

If you're getting 5x the TCPIP performance by calling GC.Collect, you're
suffering very badly from heap fragmentation due to all the pinning that's
going on when you send/receive data.

GC.Collect isn't really the right answer there, although I understand how it
could look that way. What you really need is a buffer pool that's all
allocated together, and that you check byte[] buffers into and out of. This
way you don't allocate new byte[] each time you make a socket call - this
causes continual fragmentation, and you'll eventually run out of memory.

The problem is described in detail here:
https://blogs.msdn.com/yunjin/archive/2004/01/27/63642.aspx

A good pair of articles on the subject is here:
http://www.coversant.net/dotnetnuke/Default.aspx?tabid=88&EntryID=9
http://blogs.msdn.com/adarshk/archive/2005/08/20/454022.aspx

.... and the limits of how far you can go is talked about here:
http://www.coversant.net/Default.aspx?tabid=88&EntryID=10

In short (re: your original assertion) the only time you really want to call
"GC.Collect" is just before you're forced to grow your buffer pool. When you
do grow, you want to grow by a large amount (1k, 5k, 10k instances of
whatever your buffer size is) so that any holes in your heap are all close
together. This leverages the GC.Collect mostly to compact the heap, so that
your buffers are allocated as low as possible.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise
http://www.coversant.net/blogs/cmullins


"Shawn B." <leabre@xxxxxxxx> wrote in message
news:u8HogwtIHHA.2456@xxxxxxxxxxxxxxxxxxxxxxx
calling GC.Collect is not a good solution. The .NET Framework calls those
methods when necessary, calling it manually will hardly do anything
useful (but some bad things...).

While this might be true (in general) I have a project that actually
improves scalability and performance by calling GC.Collect at the right
place. Without doing so, we can only get about N number of connections
(TCP/IP) on a single server. By doing so, we get about N*5 without a
single noticeable degradation of performance. There's also another place
where we have lots of string handling where placing a GC.Collect in the
right place drastically increases our perceivable performance.

But I agree with you, in general, there probly is a better way. But that
doesn't mean to rule out the possibility. Just because everyone says not
to, doesn't mean you shouldn't see for yourself whether doing so might
help your situation. Of course, this also assumes you understand more
about the GC/JIT/CLR internals than most others do. It also asummes you
are a master of profiling so you can determine whether it helps or not.

YMMV.


Thanks,
Shawn



.

Relevant Pages

  • Re: C# System.String Memory Usage [TCP IP Connections]
    ... If you're getting 5x the TCPIP performance by calling GC.Collect, ... suffering very badly from heap fragmentation due to all the pinning that's ... call "GC.Collect" is just before you're forced to grow your buffer pool. ...
    (microsoft.public.dotnet.framework.performance)
  • [EXPL] Smail preparse_address_1() Heap Overflow
    ... The following security advisory is sent to the securiteam mailing list, and can be found at the SecuriTeam web site: http://www.securiteam.com ... There is a heap buffer overflow, ... ssize_t Send(int s, const void *buf, size_t len, int flags) ...
    (Securiteam)
  • smail remote and local root holes
    ... I've been trying to send an email to greg woods, the maintainer of smail, to 3 ... There is a heap buffer overflow, and a signal handling related vulnerability. ...
    (Bugtraq)
  • Re: AWR Sample Report
    ... Shared Pool Size: 1,264M 1,264M Log Buffer: 14,344K ... Cached Commit SCN referenced 39,329 1,254.55 3,575.36 ... IMU Redo allocation size 30,844 983.89 2,804.00 ... java call heap collected bytes 0 0.00 0.00 ...
    (comp.databases.oracle.server)
  • AWR Sample Report
    ... Shared Pool Size: 1,264M 1,264M Log Buffer: 14,344K ... Cached Commit SCN referenced 39,329 1,254.55 3,575.36 ... IMU Redo allocation size 30,844 983.89 2,804.00 ... java call heap collected bytes 0 0.00 0.00 ...
    (comp.databases.oracle.server)