Re: Tip Required on Dynamic Memory Allocation in Server Applications ...

From: Ivan Brugiolo [MSFT] (ivanbrug_at_online.microsoft.com)
Date: 03/22/04 

Date: Mon, 22 Mar 2004 09:06:52 -0800

If you are NOT re-defining the "operator new" in global scope,
or you are NOT using somthing like
std::basic_string<WCHAR,char_traits<WCHAR>,YourOwnGoodAllocator>
then it's not clear to me who is using your private heap.

On top of that, a memory allocation failure after 300 usages of the string
class
(that is expected by standard and by design to throw exceptions)
seems "too-soon" to me.

I had a forum discussion with a developr that was making heavy usage
of SC++L containers and strings, and he ended-up encountering
memory pressure in his program at about 500-800 Megs
of memory used by his program, over a 2Gig availalbe address space.
It turned out that he had run into a "corner" case of the heap-expansion,
exacerbated by virtual-address-space depletion.
Suggestions where given to pre-reserve the size of the heap,
and to possibly use a private allocator based on virtual-alloc for blocks
over 0xFFFE * 0x8,
plus the suggestion to use LowFragHeap.

For you to run into a similar problem with just 300 std::strings,
you must have either huge strings, or a memory leak somewhere else,
or your application runs as part of a JobObject with a cap on the used
memory.

Can you run `!heap -s' when your application crashes,
so that we can have an idea of what is exactly going on ? 

-- 
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
"Shashank Welankar" <welankars@rediffmail.com> wrote in message
news:#gjzy78DEHA.3568@tk2msftngp13.phx.gbl...
> Hello Ivan,
>
> I am using std::string just by declaring the variable of string type, like
> this -
>          std::string  sText ;
> I am not doing any special thing like allocator ... and other stuff.
> No operator new(size_t) .... is defined by me.
> All is by default.
>
> Also, now I am not even using my Heap pool class. I have remove
> the Heap pool class and simply using new/delete now to perform
> memory allocation. My program works untill 200-300 requests are
> served and then it raise the "bad_alloc" exception simultaneously.
>
> Please Help.
>
> -Vipul Pathak ;
> Indore (MP) India.
> [Win32 And C++ : Deadly Combination]
> ------------------------------------------------------
>
>
>
>
>
> "Ivan Brugiolo [MSFT]" <ivanbrug@online.microsoft.com> wrote in message
> news:uI5BQtFDEHA.3784@TK2MSFTNGP10.phx.gbl...
> > One of the stacks below seems a quite old bug
> > in the copy constructor of the std::exception class,
> > but it's hard to tell without the full context around.
> > Depending of the C-Runtime you are using this may or may not be fixed
> >
> > I hope that you are catching by reference and not by value,
> > just to avoid un-necessary memory allocations in a code-path that
> > is anyway critical and that would cause abort() to be called.
> >
> > Othet than that, you may want to tell us how you are
> > hooking your allocator that defaults on a private heap.
> > Are you creating a std::Allocator for your SC++L classes and containers
?
> > Are you rewriting "void * __cdecl operator new(size_t s)" in global
scope
> ?
> > If yes, do you respect the throwing semantics ?
> > And the "new-handler" semantics ?
> > Do you use more than one C-runtime in the same executable ?
> > How do you link the whole thing ?
> >
> > -- 
> > This posting is provided "AS IS" with no warranties, and confers no
> rights.
> > Use of any included script samples are subject to the terms specified at
> > http://www.microsoft.com/info/cpyright.htm
> >
> >
> > "Shashank Welankar" <welankars@rediffmail.com> wrote in message
> > news:#gGAtbBDEHA.3888@TK2MSFTNGP10.phx.gbl...
> > > Hello Gurus ....
> > >
> > > I am expecting a Useful Tip here .....
> > >
> > > Here is the description of my problem:
> > > I have written a Server Application, which is in a broad sense a
Router.
> > It
> > > Routes requests from
> > > clients to another process and response from that process to
appropriate
> > > clients.
> > > Because of heavy use of new/delete (for every connection and every
> request
> > > being processed, I call
> > > it "Per Connection Memory" and "Per IO Memory"), eventually the
> > Application
> > > goes Slower (As
> > > expected- This was Fragmentation of Heap).
> > >
> > > To Rescue, this I build a Heap Pool Manager Class.
> > > What this does is- Creates a private Heap using HeapCreate() API, the
> > > Initial size in committed
> > > pages could be any (I tried much- from 8KB of Init. Memory, to 128
MB).
> > This
> > > class allocate memory
> > > blocks on requests and maintain a vector of Addresses and Allocated
> blocks
> > > are marked as
> > > "CK_BLOCK_IS_LOCKED". When threads in my Application returns the block
> by
> > > calling "ResignBlock()",
> > > the block is then marked as "CK_BLOCK_IS_UNLOCKED", but the block is
not
> > > physically delete/free()ed.
> > > There fore when the request for same sized block is Queued again, the
> heap
> > > class will find this
> > > "Unblocked" maked Block and assign it to the request maker.
> > >
> > > Every thing is OK with these, and I am using it but now a very strange
> > > problem is coming ...
> > > All STL class are causing Dynamic Memory Allocation Failure. Every
time
> > the
> > > Error popups up from a
> > > different place- Some times the "std::string" class cause the
exception
> > > "std::bad_alloc", some times
> > > "std::basic_fstream", and some times the COM class "_bstr_t" causes
> memory
> > > Allocation Failure. One
> > > thing is common is all of these is that they allocate memory from
heap.
> > >
> > > The error I got are as bolow-
> > >
> > > 1) Unhandled exception at 0x004607c0 in MTRouter.exe:  0xC0000005:
> Access
> > > violation reading location 0x00000000.
> > > 2) Unhandled exception at 0x7c59ba9d in MTRouter.exe: Microsoft C++
> > > exception: std::bad_alloc @ 0x00e6e6a0.
> > > 3) Unhandled exception at 0x77fa144b in MTRouter.exe: User breakpoint.
> > >
> > >
> > > The top 6 entries in Call Stack are as Listed below -
> > > 6) MTRouter.exe!strlen()  Line 78 Asm
> > > 5) MTRouter.exe!exception::exception(const exception & that={...})
> > > 4) MTRouter.exe!std::bad_alloc::bad_alloc(const std::bad_alloc &
> > > __that={...})
> > > 3) MTRouter.exe!std::_Nomemory()  Line 9 + 0xd
> > > 2) MTRouter.exe!operator new(unsigned int sz=4, const
> std::_DebugHeapTag_t
> > &
> > > tag={...}, char *
> > >
> > > file=0x0048b610, int line=131)
> > > 1) MTRouter.exe!std::ios_base::_Init()  Line 131 + 0x16
> > > .
> > > .
> > > .
> > >
> > > I have tested the Heap class throughly as its not causing any memory
> Leak.
> > > The Private Heap is also
> > > Growable and doesn't cause allocation requests to fail when it crosses
a
> > > some simple X limit. Also,
> > > I hope the STL Class are not using my Private Heap and they must be
> using
> > > the CRT Heap or some thing
> > > like that.
> > >
> > > I am tired ..... :-(
> > > Can Experts put Light on the Situation ....
> > >
> > > Thankyou.
> > >
> > > -Vipul Pathak ;
> > > Indore (MP) India.
> > > (Win32 API and C++: Deadly Combination)
> > >
> > >
> >
> >
>
>

Relevant Pages

  • Re: Tip Required on Dynamic Memory Allocation in Server Applications ...
    ... hooking your allocator that defaults on a private heap. ... > it "Per Connection Memory" and "Per IO Memory"), ... > blocks on requests and maintain a vector of Addresses and Allocated blocks ... > All STL class are causing Dynamic Memory Allocation Failure. ...
    (microsoft.public.win32.programmer.kernel)
  • Re: Whats the difference between the heap and the freestore?
    ... >> In general, when discussing dynamically allocated memory, I hear people ... As of a couple of days ago, I though the heap ... > When you use the term heap in the context of memory allocation it annoys ... > without saying that they have to be allocated on a stack or anything else. ...
    (comp.lang.cpp)
  • Memory problems - WinDbg and SOS: Who recognizes this pattern?
    ... Last night I managed to get a memory dump using ADPlus and I analyzed it ... ephemeral segment allocation context: none ... Large object heap starts at 0x0a0d0030 ...
    (microsoft.public.dotnet.framework.performance)
  • Re: Windows Update & System restore
    ... history" on the windows update page still lists the update. ... "phantom usage" relating to the unused portions of memory allocation ... the memory allocation requests that are issued by Windows components, ...
    (microsoft.public.windowsxp.general)
  • Re: C++ Garbage Collector on VMS?
    ... case of a reference counting system, have a non-zero reference count) as ... to the beginning of the heap, updating the pointers to the objects as it ... (segregating allocation areas by type/size can help a lot there, ... any way with GC-controlled memory). ...
    (comp.os.vms)
Loading