Re: Operator new failing in windows after several days of operation

Tech Tip: Click here to run a free scan for Windows Errors and optimize PC performance

>>compiler and linker settings. The old non-conformant behavior for
>>operator new is to return NULL when there's out-of-memory.
>>
>
> I am compiling under Visual Studio 6.0, Service Pack 5.
>
Unfortunately, that is not enough to tell from here.
You may be using MFC, that has it's own style of exception and new operator.
You may be miximg modules compiled against different versions of the
C-Runtime.
You may have provided an implementation of operator new that the linker
will prefer over the one from the libraries.
On average, people underestimate the fact that proper usage
of SC++L containers require a throwing new operator,
and they get confused by badly set-up build environments.

>
> '!address -summary' generates not output. Windbg says it can't find
> symbols for the following type:
> ntdll!_NT_TIB
>
Are you on Windows 2000 ? In that case, the public symbols will not help.
WinXp and above is well supported. Unfortunately the fall back for the
lack of `!address` is `!vadump` that is not very friendly to use and to
interpret.
There was a thred titled "CreateThread Fails with ERROR_NOT_ENOUGH_MEMORY"
in the microsoft.public.win32.programmer.kernel forum where some better
instructions were provided.
However, if you could repro this in a more debuggable environment, that
would be best.

> '!heap -p -all' generated a 68MB log file. I've attached the first
> few lines and the last few lines at the end of this post. The gist of
> the file is that it the heap appears to have entries starting at
> address 0x130680, and continues with incrementing addresses all the
> way to 0x16476f88.

The overall idea behind reading those logs would be to spot address
space fragmentation. From `!heap`, you should look for the HEAP_SEGMENT.
>From the address of the heap segment and the output of `!vadump`,
you should be able to figure out if the virtual address space for heap
segments
have been exhausted.
Other things the output od `!heap` can tell you is the busy-free-busy
pattern,
that is very common with SC++L containers that grows by doubling the size
of the previous storage. This causes internal heap fragmentation.

> I have not been successful in finding documentation of the !heap and
> !address commands in windbg. Is there a place where all the commands
> in windbg are documented? A URL would be absolutely fantastic.
>

the new set of commands for `!heap` and `!address` are self documenting
(`!heap -p` and `!address -?` are your friends). Actually, !address should
have
been documented recently, only 18 months after it was created.

The stack you have posted are better.
The first one seems a plain null pointer.
can you check exactly if `std::allocator<char>::_Allocate` is throwing of
not ?
I would expect it is not throwing, if the new operator goes to msvcrt.dll,
but, it may be throwing is the new operator goes against mfc42.dll

the second stack is the standard compliant behavior for an unexpected
exception
MSVCRT!abort() is being called, and the process is expected to die soon.
You should review your exception strategy in your threads.

> Yes, 70MB is the application's current working set (determined by
> reading the "Mem Usage" column in Task Manager).
>

The working set is probably the worst indicator of what is really going on
in your process. I strongly belive that your process has exhausted the
virtual address space, and, the heap manager is failing.
The spew from `!vadump` should tell you how much address space you are
really using.
`!address -summary` would have told you that in one simple shot.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm


"Adrian Vacaliuc" <adrian.vacaliuc@xxxxxxxxxxxxxxx> wrote in message
news:n75if19edd2en0h29o87c7td73ufuqa1ju@xxxxxxxxxx
> Thank you very much for the quick reply. Please allow me to try to
> provide more information:
>
>>> I think there is very strong evidence that operator new is returning a
>>> NULL pointer on me.
>>
>>compiler and linker settings. The old non-conformant behavior for
>>operator new is to return NULL when there's out-of-memory.
>>
>
> I am compiling under Visual Studio 6.0, Service Pack 5.
>
>>Memory fragmentation is known to cause Out-Of-Memory failure.
>>Normally, this happens around the 1.2 - 1.7 Gig of used address space.
>>To diagnose these issues, you may consider using `!address -summary`
>>or `!heap -p -all` to inspect the heap and address space layout.
>
> I have very little experience with WinDbg. I have never needed
> anything more than the debugger included with VS before now, so please
> bear with me.
>
> '!address -summary' generates not output. Windbg says it can't find
> symbols for the following type:
> ntdll!_NT_TIB
>
> I HAVE used the microsoft symbol server to download the ntdll.pdb on
> the machine I'm debugging on. How do I get Windbg to use it?
>
> '!heap -p -all' generated a 68MB log file. I've attached the first
> few lines and the last few lines at the end of this post. The gist of
> the file is that it the heap appears to have entries starting at
> address 0x130680, and continues with incrementing addresses all the
> way to 0x16476f88.
>
> I have not been successful in finding documentation of the !heap and
> !address commands in windbg. Is there a place where all the commands
> in windbg are documented? A URL would be absolutely fantastic.
>
>>Is 70Megs your working set or your address space usage ?
>
> Yes, 70MB is the application's current working set (determined by
> reading the "Mem Usage" column in Task Manager).
>
>>If you google for posts in the newsgroup
>>microsoft.public.win32.programmer.kernel and
>>
>>microsoft.public.windbg
>>
>>you may be able to see how similar issues were investigated in the past.
>
> Thank you, I will investigate this next.
>
>>> Am I dealing with a heap corruption issue? I don't think that's
>>> what's happening here, because a heap corruption type of problem
>>> usually throws an exception that your application can catch. That
>>> isn't happening here.
>>
>>Hard to tell from the scarce information provided. Without a stack trace
>>of an exception (the output of `r;kb` from cdb/ntsd/windbg)
>>not much progress can be made.
>>It's absolutely false that a heap corruption throws and exception
>>that an application can catch.
>>The heap manager deoes not throw. It may happen that certain heap
>>corruptions
>>end up causign an access vuiolation, but, what the application can do
>>with that exception is completely undefined behavior.
>
> Our application has 11 threads. Here is the stack trace at the time
> of the crash in the thread that throws the exception:
>
>>0:006> r;kb
>>eax=00000001 ebx=00000000 ecx=00fbfd60 edx=ffffffff esi=00fbfd60
>>edi=0000003f
>>eip=780c1baa esp=00fbf9a0 ebp=00fbf9c4 iopl=0 nv up ei pl nz na pe
>>nc
>>cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000
>>efl=00010202
>>MSVCP60!std::basic_string<char,std::char_traits<char>,std::allocator<char>
>> >::_Copy+0x99:
>>780c1baa 8060ff00 and byte ptr [eax-0x1],0x0
>>ds:0023:00000000=??
>>ChildEBP RetAddr Args to Child
>>00fbf9c4 780c1b35 00000000 00000027 00fbfd60
>>MSVCP60!std::basic_string<char,std::char_traits<char>,std::allocator<char>
>> >::_Copy+0x99
>>00fbf9d4 780c1ac7 00000027 00000001 0012cc28
>>MSVCP60!std::basic_string<char,std::char_traits<char>,std::allocator<char>
>> >::_Grow+0x7e
>>00fbf9e8 780c2516 0286f0e0 00000027 00fbfe58
>>MSVCP60!std::basic_string<char,std::char_traits<char>,std::allocator<char>
>> >::assign+0x10
>>00fbf9f8 0402d580 0286f0e0 00fbfa0c 00fbfd58
>>MSVCP60!std::basic_string<char,std::char_traits<char>,std::allocator<char>
>> >::basic_string<char,std::char_traits<char>,std::allocator<char> >+0x28
>>00fbfa1c 02801a14 0286f0e0 008a0000 0696e450 utils!fstring::fstring+0x32
>>[C:\cvs\stsgui\utils\fstring.cpp @ 61]
>>00fbfd7c 0283c85d 00fbfdf0 0092efc8 0092f110
>>PsrsXml!CommandMessageXML::ToXML+0x43
>>[C:\cvs\stsgui\PsrsXmlDll\CommandMessageXML.cpp @ 88]
>>00fbfe14 040138ae 00fbfe30 00fbfe30 06aa6548
>>PsrsXml!XmlWorkItem::WorkItemRun+0x6d
>>[C:\cvs\stsgui\PsrsXmlDll\XmlSocketHandler.cpp @ 42]
>>00fbfe68 04015775 0091fe30 00000001 04f00430 utils!EventThread::Run+0x100
>>[C:\cvs\stsgui\utils\basethread.cpp @ 131]
>>00fbfed0 04014d7f 0091f990 0012cc28 0091fe30
>>utils!BaseThread<EventThread>::RunWithCatchDisabled+0x115
>>[../include\internal/BaseThread.h @ 426]
>>00fbff0c 6c380185 0091f990 0091fea0 0091feb8
>>utils!BaseThread<EventThread>::BaseThreadRun+0x2f
>>[../include\internal/BaseThread.h @ 434]
>>00fbff7c 780085bc 0012cc28 0091fea0 0091fe28 MFC42!_AfxThreadEntry+0xe6
>>WARNING: Stack unwind information not available. Following frames may be
>>wrong.
>>00fbffb4 7c57b388 0091feb8 0091fea0 0091fe28 MSVCRT!endthreadex+0xbc
>>00fbffec 00000000 78008532 0091feb8 00000000 KERNEL32!lstrcmpiW+0xb7
>
> Incidentially, at the time of the exception, another thread looks like
> it was in the midst of failing a memory allocation as well:
>
>>0:001> r;kb
>>eax=00000000 ebx=00abfd9c ecx=04069459 edx=77fbb286 esi=0407a760
>>edi=19930520
>>eip=78006972 esp=00abf83c ebp=00abf844 iopl=0 nv up ei pl zr na po
>>nc
>>cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000
>>efl=00010246
>>MSVCRT!abort+0x24:
>>78006972 393e cmp [esi],edi
>>ds:0023:0407a760=19930520
>>ChildEBP RetAddr Args to Child
>>WARNING: Stack unwind information not available. Following frames may be
>>wrong.
>>00abf844 780071b7 00abf940 00abfd9c 00abf960 MSVCRT!abort+0x24
>>00abf87c 77fbb272 00abf940 00abfd9c 00abf960 MSVCRT!_CxxFrameHandler+0x26
>>00abf8a0 77facc28 00abf940 00abfd9c 00abf960 ntdll!ExecuteHandler+0x26
>>00abf928 77f91bc6 00abf940 00abf960 00abf940
>>ntdll!RtlDispatchException+0x76
>>00abf928 7c59bc3f 00abf940 00abf960 00abf940
>>ntdll!KiUserExceptionDispatcher+0xe
>>00abfc84 78007108 e06d7363 00000001 00000003 KERNEL32!RaiseException+0x56
>>00abfcc4 6c3c3a11 00abfcd4 6c423f50 6c443840 MSVCRT!CxxThrowException+0x34
>>00abfcd8 6c3ca66d 6c3ca694 00000050 00000000
>>MFC42!AfxThrowNotSupportedException
>>00abfcdc 6c3ca694 00000050 00000000 2e2bc18a MFC42!AfxNewHandler+0x5
>>00abfcf4 0284ef4a 00000050 04051795 00abfde8 MFC42!operator new+0x35
>>00abfd7c 02850113 0091f144 00abfda8 0404d37b
>>PsrsXml!XmlPsrsSensors::HandleTimeToSendIntruders+0x25
>>[C:\cvs\stsgui\PsrsXmlDll\XmlPsrsSensors.cpp @ 136]
>>00abfd88 0404d37b 0091f4b0 0091f4b0 04001b01
>>PsrsXml!SinkFunctorBase<XmlPsrsSensors>::Run+0x13
>>[..\include\utils/sourcesink.h @ 149]
>>00abfda8 0404cc98 00000000 2e2bc18a 00000b4c
>>utils!ISink::Dispatch::Run+0x4b [../include\utils/sourcesink.h @ 60]
>>00abfe20 04054b8e 0012ca6c 00abfe58 052ad278
>>utils!BaseSource::TellSinks+0x141 [C:\cvs\stsgui\utils\sourcesink.cpp @
>>82]
>>00abfe68 04056cef 009138e8 00000001 00913a50
>>utils!TimerScheduler::SchedulerThread+0x15c
>>[C:\cvs\stsgui\utils\timesink.cpp @ 261]
>>00abfed0 0405600f 009136a0 0012ca6c 009138e8
>>utils!BaseThread<TimerScheduler>::RunWithCatchDisabled+0x10f
>>[../include\internal/BaseThread.h @ 426]
>>00abff0c 6c380185 009136a0 00913958 00913970
>>utils!BaseThread<TimerScheduler>::BaseThreadRun+0x2f
>>[../include\internal/BaseThread.h @ 434]
>>00abff7c 780085bc 0012ca6c 00913958 009138e0 MFC42!_AfxThreadEntry+0xe6
>>00abffb4 7c57b388 00913970 00913958 009138e0 MSVCRT!endthreadex+0xbc
>>00abffec 00000000 78008532 00913970 00000000 KERNEL32!lstrcmpiW+0xb7
>
> Doesn't this stack trace imply a failing memory allocation?
>
> Here's the head and tail of the !heap -p -all command:
>
>> _HEAP @ 130000
>> _HEAP_LOOKASIDE @ 130688
>> _HEAP_SEGMENT @ 130640
>> CommittedRange @ 130680
>> HEAP_ENTRY: Size : Prev Flags - UserPtr UserSize - state
>> * 130680: 0303 : N/A [N/A] - 130688 (1800) - (busy)
>> 131e98: 0008 : N/A [N/A] - 131ea0 (24) - (busy)
>> 131ed8: 000c : N/A [N/A] - 131ee0 (48) - (busy)
>> 131f38: 0005 : N/A [N/A] - 131f40 (10) - (busy)
>> 131f60: 000c : N/A [N/A] - 131f68 (48) - (busy)
>> 131fc0: 000a : N/A [N/A] - 131fc8 (38) - (busy)
>> 132010: 0006 : N/A [N/A] - 132018 (14) - (busy)
>>*** WARNING: Unable to verify checksum for C:\cvs\stsgui\Release\Sts.exe
>> ? Sts!s_ImportExportFileHeader+aab
>> 132040: 000c : N/A [N/A] - 132048 (48) - (busy)
>> 1320a0: 0006 : N/A [N/A] - 1320a8 (16) - (busy)
>> 1320d0: 000b : N/A [N/A] - 1320d8 (3a) - (busy)
>> 132128: 000c : N/A [N/A] - 132130 (48) - (busy)
>> 132188: 0006 : N/A [N/A] - 132190 (14) - (busy)
>> ? Sts!NameBookmarkDlg::OnOK+455
>> 1321b8: 000a : N/A [N/A] - 1321c0 (38) - (busy)
>> 132208: 0007 : N/A [N/A] - 132210 (1a) - (busy)
>> ? Sts!ShellDisplay::ShellDisplay+2eb
>> 132240: 000a : N/A [N/A] - 132248 (38) - (busy)
>> 132290: 000c : N/A [N/A] - 132298 (48) - (busy)
>> 1322f0: 000b : N/A [N/A] - 1322f8 (3e) - (busy)
>> 132348: 000c : N/A [N/A] - 132350 (48) - (busy)
>> 1323a8: 0007 : N/A [N/A] - 1323b0 (1a) - (busy)
>> ? Sts!NameBookmarkDlg::OnOK+44f
>> 1323e0: 000b : N/A [N/A] - 1323e8 (3e) - (busy)
>> 132438: 000c : N/A [N/A] - 132440 (48) - (busy)
>> 132498: 0006 : N/A [N/A] - 1324a0 (16) - (busy)
>> 1324c8: 000b : N/A [N/A] - 1324d0 (3a) - (busy)
>> 132520: 000c : N/A [N/A] - 132528 (48) - (busy)
>> 132580: 0008 : N/A [N/A] - 132588 (22) - (busy)
>> 1325c0: 000c : N/A [N/A] - 1325c8 (48) - (busy)
>> 132620: 000c : N/A [N/A] - 132628 (44) - (busy)
>> 132680: 0006 : N/A [N/A] - 132688 (18) - (busy)
>> 1326b0: 000c : N/A [N/A] - 1326b8 (48) - (busy)
>> 132710: 0006 : N/A [N/A] - 132718 (16) - (busy)
>> 132740: 000b : N/A [N/A] - 132748 (3a) - (busy)
>> 132798: 0003 : N/A [N/A] - 1327a0 (10) - (free)
>> 1327b0: 000d : N/A [N/A] - 1327b8 (4e) - (busy)
>
> ... skipping to end ...
>
>> 164765d0: 0007 : N/A [N/A] - 164765d8 (20) - (busy)
>> 16476608: 0007 : N/A [N/A] - 16476610 (20) - (busy)
>> 16476640: 0007 : N/A [N/A] - 16476648 (20) - (busy)
>> 16476678: 0009 : N/A [N/A] - 16476680 (30) - (busy)
>> 164766c0: 0007 : N/A [N/A] - 164766c8 (20) - (busy)
>> 164766f8: 0049 : N/A [N/A] - 16476700 (230) - (busy)
>> Psrs!PsrsTrack::`vftable'
>> 16476940: 0013 : N/A [N/A] - 16476948 (80) - (busy)
>> Psrs!Psr***::`vftable'
>> 164769d8: 0009 : N/A [N/A] - 164769e0 (30) - (busy)
>> 16476a20: 0009 : N/A [N/A] - 16476a28 (30) - (busy)
>> 16476a68: 0004 : N/A [N/A] - 16476a70 (18) - (free)
>> 16476a88: 0009 : N/A [N/A] - 16476a90 (30) - (busy)
>> 16476ad0: 0007 : N/A [N/A] - 16476ad8 (20) - (busy)
>> 16476b08: 0009 : N/A [N/A] - 16476b10 (30) - (busy)
>> 16476b50: 0007 : N/A [N/A] - 16476b58 (20) - (busy)
>> 16476b88: 0049 : N/A [N/A] - 16476b90 (230) - (busy)
>> Psrs!PsrsTrack::`vftable'
>> 16476dd0: 0005 : N/A [N/A] - 16476dd8 (10) - (busy)
>> 16476df8: 0005 : N/A [N/A] - 16476e00 (10) - (busy)
>> 16476e20: 0005 : N/A [N/A] - 16476e28 (10) - (busy)
>> 16476e48: 0009 : N/A [N/A] - 16476e50 (30) - (busy)
>> 16476e90: 0006 : N/A [N/A] - 16476e98 (10) - (busy)
>> 16476ec0: 0007 : N/A [N/A] - 16476ec8 (20) - (busy)
>> 16476ef8: 000b : N/A [N/A] - 16476f00 (50) - (free)
>> 16476f50: 0007 : N/A [N/A] - 16476f58 (20) - (busy)
>> 16476f88: 0007 : N/A [N/A] - 16476f90 (20) - (busy)
>> Psrs!boost::detail::sp_counted_base_impl<PsrsTrack
>> *,boost::checked_deleter<PsrsTrack> >::`vftable'
>> 16476fc0: 0008 : N/A [N/A] - 16476fc8 (20) - (busy)
>> Psrs!boost::detail::sp_counted_base_impl<PsrsTrack
>> *,boost::checked_deleter<PsrsTrack> >::`vftable'
>> VirtualAllocdBlocks @ 8a0050
>> 2130018: 0180 : N/A [N/A] - 2130020 (1c0e80) - (busy
>> VirtualAlloc)
>> 2300018: 02e0 : N/A [N/A] - 2300020 (19dd20) - (busy
>> VirtualAlloc)
>> 2660018: 02e0 : N/A [N/A] - 2660020 (19dd20) - (busy
>> VirtualAlloc)
>> 2a30018: 02e0 : N/A [N/A] - 2a30020 (19dd20) - (busy
>> VirtualAlloc)
>> 1860018: 0760 : N/A [N/A] - 1860020 (1938a0) - (busy
>> VirtualAlloc)
>> 12e0018: 0760 : N/A [N/A] - 12e0020 (1938a0) - (busy
>> VirtualAlloc)
>> 1b10018: 02e0 : N/A [N/A] - 1b10020 (19dd20) - (busy
>> VirtualAlloc)
>>Closing open log file c:\windbg.log
>
>
> Thanks again for any help.
>
> -Adrian


.

Quantcast