RE: Memory problems - WinDbg and SOS: Who recognizes this pattern?

Thanks for your reply Martijn,

yes, generally configuration error will cause a rapid & direct exception or
prevent the application from running. However, sometimes, we may configure
the application inappropriately. e.g: for ASP.NET deployed application, we
have to set compilation/@debug = false so as to ensure pages or dynamic
resources are batch compiled so as to avoid potential heap memory
fragment... Also, when using outbound webrequest calls frequently in
asp.net application, we need to take care of the connection limit setting
(by default 2 connection to a give remote server...) ..... Anyway, such
critical exception like OOM are not easy to troubleshooting since they're
not caused by some obvious or direct incorrect setting or coding...

Thanks,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)









--------------------
| Thread-Topic: Memory problems - WinDbg and SOS: Who recognizes this
pattern?
| thread-index: AcYSL55d4bdeek8TQyWi2I+oQ84gSg==
| X-WBNR-Posting-Host: 193.108.210.227
| From: "=?Utf-8?B?dmVjb3pvQG9ubGluZS5ub3NwYW0=?=" <vecozo@xxxxxxxxxxxxx>
| References: <7D97A85E-4C9E-48A7-8FCA-4623769D46C2@xxxxxxxxxxxxx>
<VuUgpUaEGHA.2660@xxxxxxxxxxxxxxxxxxxxx>
| Subject: RE: Memory problems - WinDbg and SOS: Who recognizes this
pattern?
| Date: Thu, 5 Jan 2006 11:38:35 -0800
| Lines: 321
| Message-ID: <320822FF-41B3-42E9-AA47-0BA0012E5D46@xxxxxxxxxxxxx>
| MIME-Version: 1.0
| Content-Type: text/plain;
| charset="Utf-8"
| Content-Transfer-Encoding: 8bit
| X-Newsreader: Microsoft CDO for Windows 2000
| Content-Class: urn:content-classes:message
| Importance: normal
| Priority: normal
| X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.0
| Newsgroups: microsoft.public.dotnet.framework.performance
| NNTP-Posting-Host: TK2MSFTNGXA03.phx.gbl 10.40.2.250
| Path: TK2MSFTNGXA02.phx.gbl!TK2MSFTNGXA03.phx.gbl
| Xref: TK2MSFTNGXA02.phx.gbl
microsoft.public.dotnet.framework.performance:11174
| X-Tomcat-NG: microsoft.public.dotnet.framework.performance
|
| Thanx Steven,
|
| I'll look into these articles on my way home. One question though. I
assumed
| that the framework and asp.net are "foolproof" in the way that a wrong
config
| should never result in an OOM; at most in a 500 or 503 error back to the
| client.
|
| In other words: I assumed that only applications that are wrongly
| implemented can result in a OOM even if the load on the machine is too
high
| or the machine is wrongly configured. Is this assumption incorrect?
|
| Regards,
| Martijn Kaag
|
| ______________________________
| www.VECOZO.nl
|
|
|
| "Steven Cheng[MSFT]" wrote:
|
| > Hi vecozo,
| >
| > Seems you're encountering series of problems in your production
webservice.
| > For such memory or performance issue in ASP.NET web application, here
are
| > some general settings you can check first:
| >
| > 1. is the "debug" attribute in the web.config's <compilation> element
set
| > to "false" in your webservices application? If not, please set it to
false
| > to ensure the asp.net runtime compile the dynamic assemblies in release
| > mode so as to avoid memory fragment...
| >
| > 2. Is there many COM interop or PInvoke code in your webservice, this
is
| > also possible cause of memory leak or other performance issue....
| >
| > 3. Since you mentioned that your webservice's code will also make many
| > additional outbound webservice calls, have you configured the proper
| > maxConnections setting and max threading number for your asp.net
| > application? You can find the related setting described in the below
msdn
| > article:
| >
| >
https://msdn.microsoft.com/library/en-us/dnpag/html/scalenetchapt06.asp?fram
| > e=true
| >
| >
https://msdn.microsoft.com/library/en-us/dnpag/html/scalenetchapt10.asp?fram
| > e=true#scalenetchapt10_topic9
| >
| > In addition, as I've mentioned in the other thread, you can contact PSS
or
| > global support center for assistance through the following address when
you
| > feel necessary:
| >
| > http://support.microsoft.com/gp/assistsupport
| >
| > Thanks,
| >
| > Steven Cheng
| > Microsoft Online Support
| >
| > Get Secure! www.microsoft.com/security
| > (This posting is provided "AS IS", with no warranties, and confers no
| > rights.)
| >
| >
| > --------------------
| > | Thread-Topic: Memory problems - WinDbg and SOS: Who recognizes this
| > pattern?
| > | thread-index: AcYRGvFxVw75f/xxQjm6kSsKypLtYA==
| > | X-WBNR-Posting-Host: 84.117.15.17
| > | From: "=?Utf-8?B?dmVjb3pvQG9ubGluZS5ub3NwYW0=?="
<vecozo@xxxxxxxxxxxxx>
| > | Subject: Memory problems - WinDbg and SOS: Who recognizes this
pattern?
| > | Date: Wed, 4 Jan 2006 02:38:04 -0800
| > | Lines: 309
| > | Message-ID: <7D97A85E-4C9E-48A7-8FCA-4623769D46C2@xxxxxxxxxxxxx>
| > | MIME-Version: 1.0
| > | Content-Type: text/plain;
| > | charset="Utf-8"
| > | Content-Transfer-Encoding: 8bit
| > | X-Newsreader: Microsoft CDO for Windows 2000
| > | Content-Class: urn:content-classes:message
| > | Importance: normal
| > | Priority: normal
| > | X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.0
| > | Newsgroups: microsoft.public.dotnet.framework.performance
| > | NNTP-Posting-Host: TK2MSFTNGXA03.phx.gbl 10.40.2.250
| > | Path: TK2MSFTNGXA02.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGXA03.phx.gbl
| > | Xref: TK2MSFTNGXA02.phx.gbl
| > microsoft.public.dotnet.framework.performance:11165
| > | X-Tomcat-NG: microsoft.public.dotnet.framework.performance
| > |
| > | Yesterday I posted a message about OutOfMemory exceptions in our
| > production
| > | environment.
| > |
| >
[http://msdn.microsoft.com/newsgroups/managed/default.aspx?dg=microsoft.publ
| >
ic.dotnet.framework.webservices.enhancements&mid=93a987c9-14e2-4fb2-9c59-1e4
| > 7e87e49e9&sloc=en-us]
| > |
| > | Last night I managed to get a memory dump using ADPlus and I analyzed
it
| > | using Windbg and the SOS extension. The dump contains large blocks of
| > free
| > | memory: actually over 99% of the used head is ââ?¬Å?freeââ?? The
memory seems
| > to be
| > | severely fragmented. Large free block of free memory exist.
| > |
| > | We run several webservices and one of these webservices, the one that
| > seems
| > | to cause the problem, makes several (up to 30) asynchronous
webservice
| > calls.
| > | Below some (summarized) results from windebug. A major part of the
heap
| > | consists of the following pattern:
| > |
| > | 0x04ee8e80 0x0c8de4dc 36 2 System.Net.CoreResponseData
| > | 0x04ee8ea4 0x0c8dcf5c 84 2
System.Net.NestedSingleAsyncResult
| > | 0x04ee8ef8 0x01a9292c 20 2 System.Byte[]
| > | 0x04ee8f0c 0x79bf6f1c 28 2 System.AsyncCallback
| > | 0x04ee8f28 0x0c8dd2a0 120 2
| > System.Net.Sockets.OverlappedAsyncResult
| > | 0x04ee8fa0 0x79c0d470 32 2 System.Threading.Overlapped
| > |
| > | Followed and preceded by a huge (10mb +) free space. Does anyone
| > recognize
| > | this pattern?
| > |
| > | Regards,
| > | Martijn Kaag
| > |
________________________________________________________________________
| > | 0:000> !eeheap -gc
| > | Number of GC Heaps: 2
| > | ------------------------------
| > | Heap 0 (0x000d4a68)
| > | generation 0 starts at 0x8f88c8e0
| > | generation 1 starts at 0x8f88c8d4
| > | generation 2 starts at 0x020d0030
| > | ephemeral segment allocation context: none
| > | segment begin allocated size
| > | 0x020d0000 0x020d0030 0x05b9fcc4 0x03acfc94(61,668,500)
| > | 0x15810000 0x15810030 0x166e5930 0x00ed5900(15,554,816)
| > | 0x19810000 0x19810030 0x1c3cee14 0x02bbede4(45,870,564)
| > | 0x21810000 0x21810030 0x24aaf7b4 0x0329f784(53,081,988)
| > | 0x39810000 0x39810030 0x3c1dd570 0x029cd540(43,832,640)
| > | 0x31810000 0x31810030 0x32ff9934 0x017e9904(25,073,924)
| > | 0x45810000 0x45810030 0x4915be80 0x0394be50(60,079,696)
| > | 0x685c0000 0x685c0030 0x6c4136f0 0x03e536c0(65,353,408)
| > | 0x80010000 0x80010030 0x83c24060 0x03c14030(62,996,528)
| > | 0x88010000 0x88010030 0x8941baec 0x0140babc(21,019,324)
| > | 0x94010000 0x94010030 0x97b24994 0x03b14964(61,950,308)
| > | 0x90010000 0x90010030 0x9264202c 0x02631ffc(40,050,684)
| > | 0xa0010000 0xa0010030 0xa36ef1e0 0x036df1b0(57,536,944)
| > | 0xa8010000 0xa8010030 0xab969328 0x039592f8(60,134,136)
| > | 0xb0010000 0xb0010030 0xb3d1f20c 0x03d0f1dc(64,025,052)
| > | 0xb4010000 0xb4010030 0xb800fee4 0x03fffeb4(67,108,532)
| > | 0x8c010000 0x8c010030 0x8fab474c 0x03aa471c(61,490,972)
| > | Large object heap starts at 0x0a0d0030
| > | segment begin allocated size
| > | 0x0a0d0000 0x0a0d0030 0x0a166078 0x00096048(614,472)
| > | Heap Size 0x33b41f38(867,442,488)
| > | ------------------------------
| > | Heap 1 (0x000d4e50)
| > | generation 0 starts at 0x9c00fd18
| > | generation 1 starts at 0x9c00fbd0
| > | generation 2 starts at 0x060d0030
| > | ephemeral segment allocation context: none
| > | segment begin allocated size
| > | 0x060d0000 0x060d0030 0x09eaeb80 0x03ddeb50(64,875,344)
| > | 0x11810000 0x11810030 0x15552780 0x03d42750(64,235,344)
| > | 0x1d810000 0x1d810030 0x1faf9230 0x022e9200(36,606,464)
| > | 0x25810000 0x25810030 0x26be7400 0x013d73d0(20,804,560)
| > | 0x2d810000 0x2d810030 0x31336100 0x03b260d0(62,021,840)
| > | 0x35810000 0x35810030 0x35d9a00c 0x00589fdc(5,808,092)
| > | 0x3d810000 0x3d810030 0x414457f4 0x03c357c4(63,133,636)
| > | 0x41810000 0x41810030 0x45641ad8 0x03e31aa8(65,215,144)
| > | 0x49810000 0x49810030 0x4d7b6604 0x03fa65d4(66,741,716)
| > | 0x4d810000 0x4d810030 0x4e17b214 0x0096b1e4(9,875,940)
| > | 0x71f70000 0x71f70030 0x74a483c0 0x02ad8390(44,925,840)
| > | 0x84010000 0x84010030 0x854cd988 0x014bd958(21,748,056)
| > | 0x29810000 0x29810030 0x2b71a080 0x01f0a050(32,546,896)
| > | 0x9c010000 0x9c010030 0x9f4a780c 0x034977dc(55,146,460)
| > | 0xa4010000 0xa4010030 0xa7d34f94 0x03d24f64(64,114,532)
| > | 0xac010000 0xac010030 0xaff250f4 0x03f150c4(66,146,500)
| > | 0xb8010000 0xb8010030 0xbba59670 0x03a49640(61,118,016)
| > | 0x98010000 0x98010030 0x9c00fd24 0x03fffcf4(67,108,084)
| > | Large object heap starts at 0x0b0d0030
| > | segment begin allocated size
| > | 0x0b0d0000 0x0b0d0030 0x0b0d0030 0x00000000(0)
| > | Heap Size 0x33fc4bb0(872,172,464)
| > | ------------------------------
| > | GC Heap Size 0x67b06ae8(1,739,614,952)
| > |
| > |
| > | 0:000> !dumpheap -stat
| > | ------------------------------
| > | Heap 0
| > | total 92,553 objects
| > | ------------------------------
| > | Heap 1
| > | total 81,563 objects
| > | ------------------------------
| > | total 174,116 objects
| > | Statistics:
| > | MT Count TotalSize Class Name
| > | (..;.)
| > | 0x79b94638 36,506 2,604,180 System.String
| > | 0x01a9209c 24,857 2,938,600 System.Object[]
| > | 0x01a9292c 3,441 3,498,892 System.Byte[]
| > | 0x000d4728 265 1,724,763,556 Free
| > | Total 174,116 objects, Total size: 1,739,608,168
| > |
| > | Fragmented blocks larger than 0.5MB:
| > | Addr Size Followed by
| > | 0x027c9088 11.8MB large free object followed by 0x0339da1c
System.Byte[]
| > | 0x033a72c8 8.8MB large free object followed by 0x03c710b8
| > | System.Net.CoreResponseData
| > | 0x03cdc4bc 0.7MB large free object followed by 0x03d925b8
System.Byte[]
| > | 0x03d92dd4 3.5MB large free object followed by 0x0410aaac
System.Byte[]
| > | 0x0410afbc 0.8MB large free object followed by 0x041d7a28
System.Byte[]
| > | 0x04244490 8.7MB large free object followed by 0x04afb01c
| > | System.Net.NestedSingleAsyncResult
| > | 0x04afb138 3.9MB large free object followed by 0x04ee8e80
| > | System.Net.CoreResponseData
| > | 0x04fc7f68 0.7MB large free object followed by 0x0508767c
System.Byte[]
| > | 0x05087714 0.5MB large free object followed by 0x0510ea2c
System.Byte[]
| > | 0x05111a98 3.2MB large free object followed by 0x054400a4
System.Byte[]
| > | 0x054405a8 7.4MB large free object followed by 0x05b9fb84
| > | System.Net.CoreResponseData
| > | 0x15810030 5.8MB large free object followed by 0x15dd69ac
System.Byte[]
| > | 0x15dd9a18 0.9MB large free object followed by 0x15eb3704
System.Byte[]
| > | 0x15eb6770 8.2MB large free object followed by 0x166e5050
System.Byte[]
| > | 0x19810030 31.2MB large free object followed by 0x1b73ff14
System.Byte[]
| > | 0x1b7a8a54 11.9MB large free object followed by 0x1c3974a8
| > | System.Net.NestedSingleAsyncResult
| > | 0x21810030 37.1MB large free object followed by 0x23d30000
System.Byte[]
| > | 0x23d3306c 13.5MB large free object followed by 0x24aace48
System.Byte[]
| > | 0x39810030 41.0MB large free object followed by 0x3c10a284
| > | System.Net.NestedSingleAsyncResult
| > | [�and hundreds more!�]
| > |
| > |
| > | 0:000> !dumpheap -fix 0x027c9088 0x39810030
| > | ------------------------------
| > | Heap 0
| > | Address MT Size Gen
| > | 0x027c9088 0x000d4728 12,405,140 2 Free
| > | 0x0339da1c 0x01a9292c 116 2 System.Byte[]
| > | 0x0339da90 0x0c8d8e14 16 2 System.Net.ProtocolToken
| > | 0x0339daa0 0x01a9292c 20 2 System.Byte[]
| > | 0x0339dab4 0x79b94638 84 2 System.String text/xml;
charset=utf-8
| > | 0x0339db08 0x79ba2ee4 24 2 System.Collections.ArrayList
| > | 0x0339db20 0x01a9209c 80 2 System.Object[]
| > | 0x0339db70 0x79ba2ee4 24 2 System.Collections.ArrayList
| > | 0x0339db88 0x01a9209c 20 2 System.Object[]
| > | 0x0339db9c 0x02040914 16 2
| > |
System.Collections.Specialized.NameObjectCollectionBase/NameObjectEntry
| > | 0x0339dbac 0x000d4728 38,532 2 Free
| > | 0x033a7230 0x01a9292c 116 2 System.Byte[]
| > | 0x033a72a4 0x0c8d8e14 16 2 System.Net.ProtocolToken
| > | 0x033a72b4 0x01a9292c 20 2 System.Byte[]
| > | 0x033a72c8 0x000d4728 9,215,472 2 Free
| > | 0x03c710b8 0x0c8de4dc 36 2 System.Net.CoreResponseData
| > | 0x03c710dc 0x0c8dcf5c 84 2
System.Net.NestedSingleAsyncResult
| > | 0x03c71130 0x01a9292c 20 2 System.Byte[]
| > | 0x03c71144 0x79bf6f1c 28 2 System.AsyncCallback
| > | 0x03c71160 0x0c8dd2a0 120 2
| > System.Net.Sockets.OverlappedAsyncResult
| > | 0x03c711d8 0x79c0d470 32 2 System.Threading.Overlapped
| > | 0x03c711f8 0x000d4728 436,540 2 Free
| > | 0x03cdbb34 0x01a9292c 2,440 2 System.Byte[]
| > | 0x03cdc4bc 0x000d4728 745,724 2 Free
| > | 0x03d925b8 0x01a9292c 2,076 2 System.Byte[]
| > | 0x03d92dd4 0x000d4728 3,636,440 2 Free
| > | 0x0410aaac 0x01a9292c 1,296 2 System.Byte[]
| > | 0x0410afbc 0x000d4728 838,252 2 Free
| > | 0x041d7a28 0x01a9292c 116 2 System.Byte[]
| > | 0x041d7a9c 0x0c8d8e14 16 2 System.Net.ProtocolToken
| > | 0x041d7aac 0x01a9292c 20 2 System.Byte[]
| > | 0x041d7ac0 0x000d4728 432,484 2 Free
| > | 0x04241424 0x01a9292c 12,396 2 System.Byte[]
| > | 0x04244490 0x000d4728 9,137,036 2 Free
| > | 0x04afb01c 0x0c8dcf5c 84 2
System.Net.NestedSingleAsyncResult
| > | 0x04afb070 0x01a9292c 20 2 System.Byte[]
| > | 0x04afb084 0x79bf6f1c 28 2 System.AsyncCallback
| > | 0x04afb0a0 0x0c8dd2a0 120 2
| > System.Net.Sockets.OverlappedAsyncResult
| > | 0x04afb118 0x79c0d470 32 2 System.Threading.Overlapped
| > | 0x04afb138 0x000d4728 4,119,880 2 Free
| > | 0x04ee8e80 0x0c8de4dc 36 2 System.Net.CoreResponseData
| > | 0x04ee8ea4 0x0c8dcf5c 84 2
System.Net.NestedSingleAsyncResult
| > | 0x04ee8ef8 0x01a9292c 20 2 System.Byte[]
| > | 0x04ee8f0c 0x79bf6f1c 28 2 System.AsyncCallback
| > | 0x04ee8f28 0x0c8dd2a0 120 2
| > System.Net.Sockets.OverlappedAsyncResult
| > | 0x04ee8fa0 0x79c0d470 32 2 System.Threading.Overlapped
| > | 0x04ee8fc0 0x000d4728 2,152 2 Free
| > | 0x04ee9828 0x0c8de4dc 36 2 System.Net.CoreResponseData
| > | 0x04ee984c 0x0c8dcf5c 84 2
System.Net.NestedSingleAsyncResult
| > | 0x04ee98a0 0x01a9292c 20 2 System.Byte[]
| > | 0x04ee98b4 0x79bf6f1c 28 2 System.AsyncCallback
| > | 0x04ee98d0 0x0c8dd2a0 120 2
| > System.Net.Sockets.OverlappedAsyncResult
| > | 0x04ee9948 0x79c0d470 32 2 System.Threading.Overlapped
| > | 0x04ee9968 0x000d4728 405,384 2 Free
| > | 0x04f4c8f0 0x01a9292c 688 2 System.Byte[]
| > | 0x04f4cba0 0x000d4728 504,456 2 Free
| > | 0x04fc7e28 0x0c8de4dc 36 2 System.Net.CoreResponseData
| > | 0x04fc7e4c 0x0c8dcf5c 84 2
System.Net.NestedSingleAsyncResult
| > | 0x04fc7ea0 0x01a9292c 20 2 System.Byte[]
| > | 0x04fc7eb4 0x79bf6f1c 28 2 System.AsyncCallback
| > | 0x04fc7ed0 0x0c8dd2a0 120 2
| > System.Net.Sockets.OverlappedAsyncResult
| > | 0x04fc7f48 0x79c0d470 32 2 System.Threading.Overlapped
| > | 0x04fc7f68 0x000d4728 784,148 2 Free
| > | 0x0508767c 0x01a9292c 116 2 System.Byte[]
| > | 0x050876f0 0x0c8d8e14 16 2 System.Net.ProtocolToken
| > | 0x05087700 0x01a9292c 20 2 System.Byte[]
| > | 0x05087714 0x000d4728 553,752 2 Free
| > | 0x0510ea2c 0x01a9292c 12,396 2 System.Byte[]
| > | 0x05111a98 0x000d4728 3,335,692 2 Free
| > | 0x054400a4 0x01a9292c 1,284 2 System.Byte[]
| > | 0x054405a8 0x000d4728 7,730,652 2 Free
| > | 0x05b9fb84 0x0c8de4dc 36 2 System.Net.CoreResponseData
| > | 0x05b9fba8 0x0c8dcf5c 84 2
System.Net.NestedSingleAsyncResult
| > | 0x05b9fbfc 0x01a9292c 20 2 System.Byte[]
|

.

Relevant Pages

  • RE: Memory problems - WinDbg and SOS: Who recognizes this pattern?
    ... Seems you're encountering series of problems in your production webservice. ... For such memory or performance issue in ASP.NET web application, ... A major part of the heap ... | ephemeral segment allocation context: ...
    (microsoft.public.dotnet.framework.performance)
  • Re: Alocated Memory Error (SQLSERVR)
    ... memory for the process dropped from 990 to 330. ... to grab available RAM for performance reasons, ... configured alert threshold rather than an error in MSDE configuration. ... Server 2000 SP4, ...
    (microsoft.public.windows.server.sbs)
  • Virtex4+PPC+ext. RAM: Problems generating ACE files (solved!?)
    ... containing both the FPGA and the software configuration. ... resulting ACE file loaded fine. ... Which is unfortunate as the start vector points to that memory type. ... something wrong but I am having trouble accessing the instruction ...
    (comp.arch.fpga)
  • Summary: problems with 4GB DIMMS in T5120
    ... specifically dealt with Crucial and ended up returning his memory. ... operating at reduced capacity ... Oct 12 17:41:38 ERROR: Operating with a degraded memory configuration. ...
    (SunManagers)
  • Re: Alocated Memory Error (SQLSERVR)
    ... I had the same problem except, my memory hog was SBSMONITORING. ... critical and you don't want it grabbing all that RAM. ... configured alert threshold rather than an error in MSDE configuration. ... Server 2000 SP4, ...
    (microsoft.public.windows.server.sbs)
Quantcast