Memory problems - WinDbg and SOS: Who recognizes this pattern?

Yesterday I posted a message about OutOfMemory exceptions in our production
environment.

[http://msdn.microsoft.com/newsgroups/managed/default.aspx?dg=microsoft.public.dotnet.framework.webservices.enhancements&mid=93a987c9-14e2-4fb2-9c59-1e47e87e49e9&sloc=en-us]

Last night I managed to get a memory dump using ADPlus and I analyzed it
using Windbg and the SOS extension. The dump contains large blocks of free
memory: actually over 99% of the used head is “free”. The memory seems to be
severely fragmented. Large free block of free memory exist.

We run several webservices and one of these webservices, the one that seems
to cause the problem, makes several (up to 30) asynchronous webservice calls.
Below some (summarized) results from windebug. A major part of the heap
consists of the following pattern:

0x04ee8e80 0x0c8de4dc 36 2 System.Net.CoreResponseData
0x04ee8ea4 0x0c8dcf5c 84 2 System.Net.NestedSingleAsyncResult
0x04ee8ef8 0x01a9292c 20 2 System.Byte[]
0x04ee8f0c 0x79bf6f1c 28 2 System.AsyncCallback
0x04ee8f28 0x0c8dd2a0 120 2 System.Net.Sockets.OverlappedAsyncResult
0x04ee8fa0 0x79c0d470 32 2 System.Threading.Overlapped

Followed and preceded by a huge (10mb +) free space. Does anyone recognize
this pattern?

Regards,
Martijn Kaag
________________________________________________________________________
0:000> !eeheap -gc
Number of GC Heaps: 2
------------------------------
Heap 0 (0x000d4a68)
generation 0 starts at 0x8f88c8e0
generation 1 starts at 0x8f88c8d4
generation 2 starts at 0x020d0030
ephemeral segment allocation context: none
segment begin allocated size
0x020d0000 0x020d0030 0x05b9fcc4 0x03acfc94(61,668,500)
0x15810000 0x15810030 0x166e5930 0x00ed5900(15,554,816)
0x19810000 0x19810030 0x1c3cee14 0x02bbede4(45,870,564)
0x21810000 0x21810030 0x24aaf7b4 0x0329f784(53,081,988)
0x39810000 0x39810030 0x3c1dd570 0x029cd540(43,832,640)
0x31810000 0x31810030 0x32ff9934 0x017e9904(25,073,924)
0x45810000 0x45810030 0x4915be80 0x0394be50(60,079,696)
0x685c0000 0x685c0030 0x6c4136f0 0x03e536c0(65,353,408)
0x80010000 0x80010030 0x83c24060 0x03c14030(62,996,528)
0x88010000 0x88010030 0x8941baec 0x0140babc(21,019,324)
0x94010000 0x94010030 0x97b24994 0x03b14964(61,950,308)
0x90010000 0x90010030 0x9264202c 0x02631ffc(40,050,684)
0xa0010000 0xa0010030 0xa36ef1e0 0x036df1b0(57,536,944)
0xa8010000 0xa8010030 0xab969328 0x039592f8(60,134,136)
0xb0010000 0xb0010030 0xb3d1f20c 0x03d0f1dc(64,025,052)
0xb4010000 0xb4010030 0xb800fee4 0x03fffeb4(67,108,532)
0x8c010000 0x8c010030 0x8fab474c 0x03aa471c(61,490,972)
Large object heap starts at 0x0a0d0030
segment begin allocated size
0x0a0d0000 0x0a0d0030 0x0a166078 0x00096048(614,472)
Heap Size 0x33b41f38(867,442,488)
------------------------------
Heap 1 (0x000d4e50)
generation 0 starts at 0x9c00fd18
generation 1 starts at 0x9c00fbd0
generation 2 starts at 0x060d0030
ephemeral segment allocation context: none
segment begin allocated size
0x060d0000 0x060d0030 0x09eaeb80 0x03ddeb50(64,875,344)
0x11810000 0x11810030 0x15552780 0x03d42750(64,235,344)
0x1d810000 0x1d810030 0x1faf9230 0x022e9200(36,606,464)
0x25810000 0x25810030 0x26be7400 0x013d73d0(20,804,560)
0x2d810000 0x2d810030 0x31336100 0x03b260d0(62,021,840)
0x35810000 0x35810030 0x35d9a00c 0x00589fdc(5,808,092)
0x3d810000 0x3d810030 0x414457f4 0x03c357c4(63,133,636)
0x41810000 0x41810030 0x45641ad8 0x03e31aa8(65,215,144)
0x49810000 0x49810030 0x4d7b6604 0x03fa65d4(66,741,716)
0x4d810000 0x4d810030 0x4e17b214 0x0096b1e4(9,875,940)
0x71f70000 0x71f70030 0x74a483c0 0x02ad8390(44,925,840)
0x84010000 0x84010030 0x854cd988 0x014bd958(21,748,056)
0x29810000 0x29810030 0x2b71a080 0x01f0a050(32,546,896)
0x9c010000 0x9c010030 0x9f4a780c 0x034977dc(55,146,460)
0xa4010000 0xa4010030 0xa7d34f94 0x03d24f64(64,114,532)
0xac010000 0xac010030 0xaff250f4 0x03f150c4(66,146,500)
0xb8010000 0xb8010030 0xbba59670 0x03a49640(61,118,016)
0x98010000 0x98010030 0x9c00fd24 0x03fffcf4(67,108,084)
Large object heap starts at 0x0b0d0030
segment begin allocated size
0x0b0d0000 0x0b0d0030 0x0b0d0030 0x00000000(0)
Heap Size 0x33fc4bb0(872,172,464)
------------------------------
GC Heap Size 0x67b06ae8(1,739,614,952)


0:000> !dumpheap -stat
------------------------------
Heap 0
total 92,553 objects
------------------------------
Heap 1
total 81,563 objects
------------------------------
total 174,116 objects
Statistics:
MT Count TotalSize Class Name
(..;.)
0x79b94638 36,506 2,604,180 System.String
0x01a9209c 24,857 2,938,600 System.Object[]
0x01a9292c 3,441 3,498,892 System.Byte[]
0x000d4728 265 1,724,763,556 Free
Total 174,116 objects, Total size: 1,739,608,168

Fragmented blocks larger than 0.5MB:
Addr Size Followed by
0x027c9088 11.8MB large free object followed by 0x0339da1c System.Byte[]
0x033a72c8 8.8MB large free object followed by 0x03c710b8
System.Net.CoreResponseData
0x03cdc4bc 0.7MB large free object followed by 0x03d925b8 System.Byte[]
0x03d92dd4 3.5MB large free object followed by 0x0410aaac System.Byte[]
0x0410afbc 0.8MB large free object followed by 0x041d7a28 System.Byte[]
0x04244490 8.7MB large free object followed by 0x04afb01c
System.Net.NestedSingleAsyncResult
0x04afb138 3.9MB large free object followed by 0x04ee8e80
System.Net.CoreResponseData
0x04fc7f68 0.7MB large free object followed by 0x0508767c System.Byte[]
0x05087714 0.5MB large free object followed by 0x0510ea2c System.Byte[]
0x05111a98 3.2MB large free object followed by 0x054400a4 System.Byte[]
0x054405a8 7.4MB large free object followed by 0x05b9fb84
System.Net.CoreResponseData
0x15810030 5.8MB large free object followed by 0x15dd69ac System.Byte[]
0x15dd9a18 0.9MB large free object followed by 0x15eb3704 System.Byte[]
0x15eb6770 8.2MB large free object followed by 0x166e5050 System.Byte[]
0x19810030 31.2MB large free object followed by 0x1b73ff14 System.Byte[]
0x1b7a8a54 11.9MB large free object followed by 0x1c3974a8
System.Net.NestedSingleAsyncResult
0x21810030 37.1MB large free object followed by 0x23d30000 System.Byte[]
0x23d3306c 13.5MB large free object followed by 0x24aace48 System.Byte[]
0x39810030 41.0MB large free object followed by 0x3c10a284
System.Net.NestedSingleAsyncResult
[…and hundreds more!…]


0:000> !dumpheap -fix 0x027c9088 0x39810030
------------------------------
Heap 0
Address MT Size Gen
0x027c9088 0x000d4728 12,405,140 2 Free
0x0339da1c 0x01a9292c 116 2 System.Byte[]
0x0339da90 0x0c8d8e14 16 2 System.Net.ProtocolToken
0x0339daa0 0x01a9292c 20 2 System.Byte[]
0x0339dab4 0x79b94638 84 2 System.String text/xml; charset=utf-8
0x0339db08 0x79ba2ee4 24 2 System.Collections.ArrayList
0x0339db20 0x01a9209c 80 2 System.Object[]
0x0339db70 0x79ba2ee4 24 2 System.Collections.ArrayList
0x0339db88 0x01a9209c 20 2 System.Object[]
0x0339db9c 0x02040914 16 2
System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry
0x0339dbac 0x000d4728 38,532 2 Free
0x033a7230 0x01a9292c 116 2 System.Byte[]
0x033a72a4 0x0c8d8e14 16 2 System.Net.ProtocolToken
0x033a72b4 0x01a9292c 20 2 System.Byte[]
0x033a72c8 0x000d4728 9,215,472 2 Free
0x03c710b8 0x0c8de4dc 36 2 System.Net.CoreResponseData
0x03c710dc 0x0c8dcf5c 84 2 System.Net.NestedSingleAsyncResult
0x03c71130 0x01a9292c 20 2 System.Byte[]
0x03c71144 0x79bf6f1c 28 2 System.AsyncCallback
0x03c71160 0x0c8dd2a0 120 2 System.Net.Sockets.OverlappedAsyncResult
0x03c711d8 0x79c0d470 32 2 System.Threading.Overlapped
0x03c711f8 0x000d4728 436,540 2 Free
0x03cdbb34 0x01a9292c 2,440 2 System.Byte[]
0x03cdc4bc 0x000d4728 745,724 2 Free
0x03d925b8 0x01a9292c 2,076 2 System.Byte[]
0x03d92dd4 0x000d4728 3,636,440 2 Free
0x0410aaac 0x01a9292c 1,296 2 System.Byte[]
0x0410afbc 0x000d4728 838,252 2 Free
0x041d7a28 0x01a9292c 116 2 System.Byte[]
0x041d7a9c 0x0c8d8e14 16 2 System.Net.ProtocolToken
0x041d7aac 0x01a9292c 20 2 System.Byte[]
0x041d7ac0 0x000d4728 432,484 2 Free
0x04241424 0x01a9292c 12,396 2 System.Byte[]
0x04244490 0x000d4728 9,137,036 2 Free
0x04afb01c 0x0c8dcf5c 84 2 System.Net.NestedSingleAsyncResult
0x04afb070 0x01a9292c 20 2 System.Byte[]
0x04afb084 0x79bf6f1c 28 2 System.AsyncCallback
0x04afb0a0 0x0c8dd2a0 120 2 System.Net.Sockets.OverlappedAsyncResult
0x04afb118 0x79c0d470 32 2 System.Threading.Overlapped
0x04afb138 0x000d4728 4,119,880 2 Free
0x04ee8e80 0x0c8de4dc 36 2 System.Net.CoreResponseData
0x04ee8ea4 0x0c8dcf5c 84 2 System.Net.NestedSingleAsyncResult
0x04ee8ef8 0x01a9292c 20 2 System.Byte[]
0x04ee8f0c 0x79bf6f1c 28 2 System.AsyncCallback
0x04ee8f28 0x0c8dd2a0 120 2 System.Net.Sockets.OverlappedAsyncResult
0x04ee8fa0 0x79c0d470 32 2 System.Threading.Overlapped
0x04ee8fc0 0x000d4728 2,152 2 Free
0x04ee9828 0x0c8de4dc 36 2 System.Net.CoreResponseData
0x04ee984c 0x0c8dcf5c 84 2 System.Net.NestedSingleAsyncResult
0x04ee98a0 0x01a9292c 20 2 System.Byte[]
0x04ee98b4 0x79bf6f1c 28 2 System.AsyncCallback
0x04ee98d0 0x0c8dd2a0 120 2 System.Net.Sockets.OverlappedAsyncResult
0x04ee9948 0x79c0d470 32 2 System.Threading.Overlapped
0x04ee9968 0x000d4728 405,384 2 Free
0x04f4c8f0 0x01a9292c 688 2 System.Byte[]
0x04f4cba0 0x000d4728 504,456 2 Free
0x04fc7e28 0x0c8de4dc 36 2 System.Net.CoreResponseData
0x04fc7e4c 0x0c8dcf5c 84 2 System.Net.NestedSingleAsyncResult
0x04fc7ea0 0x01a9292c 20 2 System.Byte[]
0x04fc7eb4 0x79bf6f1c 28 2 System.AsyncCallback
0x04fc7ed0 0x0c8dd2a0 120 2 System.Net.Sockets.OverlappedAsyncResult
0x04fc7f48 0x79c0d470 32 2 System.Threading.Overlapped
0x04fc7f68 0x000d4728 784,148 2 Free
0x0508767c 0x01a9292c 116 2 System.Byte[]
0x050876f0 0x0c8d8e14 16 2 System.Net.ProtocolToken
0x05087700 0x01a9292c 20 2 System.Byte[]
0x05087714 0x000d4728 553,752 2 Free
0x0510ea2c 0x01a9292c 12,396 2 System.Byte[]
0x05111a98 0x000d4728 3,335,692 2 Free
0x054400a4 0x01a9292c 1,284 2 System.Byte[]
0x054405a8 0x000d4728 7,730,652 2 Free
0x05b9fb84 0x0c8de4dc 36 2 System.Net.CoreResponseData
0x05b9fba8 0x0c8dcf5c 84 2 System.Net.NestedSingleAsyncResult
0x05b9fbfc 0x01a9292c 20 2 System.Byte[]
0x05b9fc10 0x79bf6f1c 28 2 System.AsyncCallback
0x05b9fc2c 0x0c8dd2a0 120 2 System.Net.Sockets.OverlappedAsyncResult
0x05b9fca4 0x79c0d470 32 2 System.Threading.Overlapped
0x15810030 0x000d4728 6,056,316 2 Free
0x15dd69ac 0x01a9292c 12,396 2 System.Byte[]
0x15dd9a18 0x000d4728 892,140 2 Free
0x15eb3704 0x01a9292c 12,396 2 System.Byte[]
0x15eb6770 0x000d4728 8,579,296 2 Free
0x166e5050 0x01a9292c 2,272 2 System.Byte[]
0x19810030 0x000d4728 32,702,180 2 Free
0x1b73ff14 0x01a9292c 2,272 2 System.Byte[]
0x1b7407f4 0x000d4728 425,904 2 Free
0x1b7a87a4 0x01a9292c 688 2 System.Byte[]
0x1b7a8a54 0x000d4728 12,511,828 2 Free
0x1c3974a8 0x0c8dcf5c 84 2 System.Net.NestedSingleAsyncResult
0x1c3974fc 0x01a9292c 20 2 System.Byte[]
0x1c397510 0x79bf6f1c 28 2 System.AsyncCallback
0x1c39752c 0x0c8dd2a0 120 2 System.Net.Sockets.OverlappedAsyncResult
0x1c3975a4 0x79c0d470 32 2 System.Threading.Overlapped
0x1c3975c4 0x000d4728 227,124 2 Free
0x1c3cecf8 0x0c8dcf5c 84 2 System.Net.NestedSingleAsyncResult
0x1c3ced4c 0x01a9292c 20 2 System.Byte[]
0x1c3ced60 0x79bf6f1c 28 2 System.AsyncCallback
0x1c3ced7c 0x0c8dd2a0 120 2 System.Net.Sockets.OverlappedAsyncResult
0x1c3cedf4 0x79c0d470 32 2 System.Threading.Overlapped
0x21810030 0x000d4728 38,928,336 2 Free
0x23d30000 0x01a9292c 12,396 2 System.Byte[]
0x23d3306c 0x000d4728 14,130,652 2 Free
0x24aace48 0x01a9292c 10,604 2 System.Byte[]
0x39810030 0x000d4728 42,967,636 2 Free
0x31810030 0x000d4728 20,313,856 2 Free
0x32b6f730 0x01a9292c 688 2 System.Byte[]
0x32b6f9e0 0x000d4728 4,577,056 2 Free
0x32fcd100 0x01a9292c 12,396 2 System.Byte[]
0x32fd016c 0x000d4728 169,608 2 Free
0x32ff97f4 0x0c8de4dc 36 2 System.Net.CoreResponseData
0x32ff9818 0x0c8dcf5c 84 2 System.Net.NestedSingleAsyncResult
0x32ff986c 0x01a9292c 20 2 System.Byte[]
0x32ff9880 0x79bf6f1c 28 2 System.AsyncCallback
0x32ff989c 0x0c8dd2a0 120 2 System.Net.Sockets.OverlappedAsyncResult
0x32ff9914 0x79c0d470 32 2 System.Threading.Overlapped
0x0a0d0030 0x01a9209c 2,064 3 System.Object[]
0x0a0d0840 0x000d4728 1,984 3 Free
0x0a0d1000 0x01a9209c 4,096 3 System.Object[]
0x0a0d2000 0x01a9209c 2,064 3 System.Object[]
0x0a0d2810 0x000d4728 2,032 3 Free
0x0a0d3000 0x01a9209c 4,096 3 System.Object[]
0x0a0d4000 0x01a9209c 2,064 3 System.Object[]
0x0a0d4810 0x000d4728 2,032 3 Free
0x0a0d5000 0x01a9209c 4,096 3 System.Object[]
0x0a0d6000 0x01a9209c 2,064 3 System.Object[]
0x0a0d6810 0x000d4728 2,032 3 Free
0x0a0d7000 0x01a9209c 2,064 3 System.Object[]
0x0a0d7810 0x01a9209c 4,096 3 System.Object[]
0x0a0d8810 0x000d4728 2,032 3 Free
0x0a0d9000 0x01a9209c 2,064 3 System.Object[]
0x0a0d9810 0x01a9209c 4,096 3 System.Object[]
0x0a0da810 0x000d4728 2,032 3 Free
0x0a0db000 0x01a9209c 2,064 3 System.Object[]
0x0a0db810 0x01a9209c 4,096 3 System.Object[]
0x0a0dc810 0x000d4728 2,032 3 Free
0x0a0dd000 0x01a9209c 2,064 3 System.Object[]
0x0a0dd810 0x000d4728 6,128 3 Free
0x0a0df000 0x01a9209c 2,064 3 System.Object[]
0x0a0df810 0x000d4728 6,128 3 Free
0x0a0e1000 0x01a9209c 4,096 3 System.Object[]
0x0a0e2000 0x000d4728 4,096 3 Free
0x0a0e3000 0x01a9209c 4,096 3 System.Object[]
0x0a0e4000 0x000d4728 4,096 3 Free
0x0a0e5000 0x01a9209c 2,064 3 System.Object[]
0x0a0e5810 0x000d4728 6,128 3 Free
0x0a0e7000 0x01a9209c 4,096 3 System.Object[]
0x0a0e8000 0x000d4728 4,096 3 Free
0x0a0e9000 0x01a9209c 2,064 3 System.Object[]
0x0a0e9810 0x000d4728 6,128 3 Free
0x0a0eb000 0x01a9209c 4,096 3 System.Object[]
0x0a0ec000 0x000d4728 245,840 3 Free
0x0a128050 0x01a9209c 4,096 3 System.Object[]
0x0a129050 0x000d4728 245,800 3 Free
0x0a165078 0x01a9209c 4,096 3 System.Object[]
total 154 objects
------------------------------
Heap 1
Address MT Size Gen
total 0 objects
------------------------------
total 154 objects

______________________________
www.VECOZO.nl

.

Relevant Pages

  • Re: Whats the difference between the heap and the freestore?
    ... >> In general, when discussing dynamically allocated memory, I hear people ... As of a couple of days ago, I though the heap ... > When you use the term heap in the context of memory allocation it annoys ... > without saying that they have to be allocated on a stack or anything else. ...
    (comp.lang.cpp)
  • Re: Tip Required on Dynamic Memory Allocation in Server Applications ...
    ... a memory allocation failure after 300 usages of the string ... Suggestions where given to pre-reserve the size of the heap, ... My program works untill 200-300 requests are ...
    (microsoft.public.win32.programmer.kernel)
  • Re: C++ Garbage Collector on VMS?
    ... case of a reference counting system, have a non-zero reference count) as ... to the beginning of the heap, updating the pointers to the objects as it ... (segregating allocation areas by type/size can help a lot there, ... any way with GC-controlled memory). ...
    (comp.os.vms)
  • Re: Wonder why GC has only 2 generations ... here is the answer.
    ... > native allocation, and can help your application improve performance. ... > The GC maintains a list of roots, which point into the GC heap. ... of moving large areas of memory outweighs the amount of memory that would be ...
    (microsoft.public.dotnet.faqs)
  • Re: Wonder why GC has only 2 generations ... here is the answer.
    ... > native allocation, and can help your application improve performance. ... > The GC maintains a list of roots, which point into the GC heap. ... of moving large areas of memory outweighs the amount of memory that would be ...
    (microsoft.public.dotnet.framework)