Re: High Memory Usage Garbage Collection Question
- From: "Willy Denoyette [MVP]" <willy.denoyette@xxxxxxxxxx>
- Date: Fri, 22 Jul 2005 23:46:13 +0200
Inline
Willy.
"Ian Taite" <ian.taite@xxxxxxxxx> wrote in message
news:1122027505.589356.180460@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Hello,
>
> I'm exploring why one of my C# .NET apps has "high" memory usage, and
> whether I can reduce the memory usage.
>
> I have an app that wakes up and processes text files into a database
> periodically. What happens, is that the app reads the contents of a
> text file line by line into an ArrayList. Each element of the ArrayList
> is a string representing a record from the file. The ArrayList is then
> processed, and the arraylist goes out of scope.
>
> As the file is read, and the ArrayList is extended, memory usage goes
> from about 6Mb up to about 200Mb in some cases, depending on the size
> of the file.
>
> What is peculiar is that after the file has been read and closed, and
> the ArrayList has gone out of scope, the memory usage does not not
> reduce even after waiting several minutes.
>
> My expectation is that most of the 200Mb would be garbage collected
> soon after the ArrayList goes out of scope, and released back to the
> system after several minutes, but this does not happen.
>
After the functions ends with 200Mb string data in the Gen2 heap (yes, all
your string objects are aged objects) , the program returns to the main loop
and waits for a command (quit, gc, ...). But there is no reason for the GC
to run and clean-up the garbage, he quite happy with all the memory free in
the system ;-). Things would be different if you didn't have that much
memory available, the system would request the GC to clean-up after the
ArrayList became eligible for collection.
But wait, If you didn't have that memory free the system would start paging
because you need that memory badly because of your design which is somewhat
flawned.
It's flawned because I see no valid reason to store a complete text file in
a ArrayList (or array that's the same) one line per entry. Each entry has a
size of 16 + (line length in character * 2) (16 is the object overhead and,
I suppose the file is ANSI, while strings are unicode in .NET). That's a
reason not to store the whole file in memory, but only read small chuncks at
a time and process these. The small chunks (let's say 1000 lines at a time)
could be stored in a fixed length array, after which you can start
processing the lines stored in the array. Once this is done you refile the
array with the next chunk until the whole file is processd.
> Another peculiar behaviour is that when the app wakes to process a
> second file, memory usage remains at about 200Mb. After the second file
> is processed, memory is still 200Mb. Note that when the second file
> processed, memory does not jump from 200mb to 400mb back to 200mb.
>
> I have tried using ArrayList's Clear() method to empty the arraylist
> before it goes out of scope but that does not make any difference.
>
> To try and make the problem reproducable, I wrote a test app that
> simply reads a file into an arraylist, calls ArrayList.Clear() then
> allows the ArrayList to fall out of scope, and sure enough the problem
> is still there. Memory usage climbs in rough proportion to the file
> size and remains there until the app ends, even if the file is read
> several times, and even when the array list is out of cope, i.e. not
> referenced.
>
> Explicitly calling CG.Collect() made hardly any difference, although
> calling GC.GetTotalMemory(true) shortly after processing each file did
> reduce the amount of memory shown by task manager for the process.
>
> I've re-written my app not to read the whole file into memory before
> processing it (not a particularly good idea in the first place) and the
> maximum memory usage has changed from 200mb to 35mb.
>
> So what is really happening here?
>
> In C# there's no way to explicitly release resources, so I do not have
> direct control over releasing memory.
>
> Assuming that the objects in the arraylist and the arraylist itself are
> no longer referenced they should be candidates for garbage collection,
> and should eventually be garbage collected, at which point the memory
> they used should be released back to the system.
>
> Perhaps these objects are no longer referenced, and the GC has decided
> that as there's insufficient contention for memory on the server, it
> won't bother garbage collecting them, resulting in apparently high
> memory usage by the application.
>
> I don't really know how to measure what's happening, any offers would
> be greatfully received. I've included some test code below in the form
> of a console app; you'll have to make the code process a suitably large
> text file of your own though.
>
> Regards,
> Ian.
>
> <!-- Code Starts here -->
>
> using System;
> using System.IO;
> using System.Collections;
>
> namespace SimpleFileReader
> {
> class SimpleFileReader
> {
> public static string readLineResponse;
>
> [STAThread]
> static void Main(string[] args)
> {
>
> Console.WriteLine( "Press return to create an app object, and run
> the app." );
> readLineResponse = Console.ReadLine();
>
> SimpleFileReader app = new SimpleFileReader();
> app.Execute();
>
> Console.WriteLine( "Type quit to terminate the app." );
>
> readLineResponse = "continue";
>
> while ( true )
> {
> Console.WriteLine(
> "\nHit return to continue, \n" +
> " or type 'gc' to garbage collect, \n" +
> " or type 'gtm' to call GC.GetTotalMemory(true), \n" +
> " or type 'quit' and hit return.\n\n" );
>
> readLineResponse = Console.ReadLine();
>
> if ( "gc" == readLineResponse ) GC.Collect();
> if ( "gtm" == readLineResponse ) GC.GetTotalMemory( true );
> if ( "quit" == readLineResponse ) break;
> }
> }
>
>
> public void Execute()
> {
> GetRecords( @"C:\SimpleFileReader\SLEXTRACTFILE050701.DAT" );
> }
>
>
> public void GetRecords( string fileName )
> {
> int recordCounter = 0;
>
> ArrayList al = new ArrayList();
>
> try
> {
> FileStream fs = File.OpenRead( fileName );
> StreamReader sr = new StreamReader( fs );
>
> string aLine = string.Empty;
>
> Console.WriteLine( "Press return to read the file." );
> readLineResponse = Console.ReadLine();
>
> while( ( aLine = sr.ReadLine() )!= null )
> {
> if ( aLine.Length > 0 )
> {
> al.Add( aLine );
> ++recordCounter;
> }
> }
>
> Console.WriteLine( recordCounter.ToString() + " record(s) read from
> " + fileName );
> Console.WriteLine( "File read. Press return to continue." );
> readLineResponse = Console.ReadLine();
>
> sr.Close();
> fs.Close();
> }
> catch( System.Exception ex )
> {
> string extractFailedMessage =
> "An exception occurred whilst extracting the contents of " +
> fileName;
>
> Console.WriteLine( extractFailedMessage + "\n" + ex.ToString() );
> }
>
> Console.WriteLine( "Press return to clear the arraylist" );
> readLineResponse = Console.ReadLine();
> al.Clear();
> Console.WriteLine( "al.Clear() called." );
>
> Console.WriteLine( "Press return to call GC.GetTotalMemory" );
> readLineResponse = Console.ReadLine();
> Console.WriteLine( "GC.GetTotalMemory just returned: " +
> GC.GetTotalMemory( true ).ToString() );
> }
> }
> }
>
>
> <!-- Code Ends here -->
>
.
- Follow-Ups:
- Re: High Memory Usage Garbage Collection Question
- From: Ian Taite
- Re: High Memory Usage Garbage Collection Question
- References:
- High Memory Usage Garbage Collection Question
- From: Ian Taite
- High Memory Usage Garbage Collection Question
- Prev by Date: Re: For assembly, C# or VB or C++ ?
- Next by Date: Re: Save file from the Server to User's Desktop
- Previous by thread: High Memory Usage Garbage Collection Question
- Next by thread: Re: High Memory Usage Garbage Collection Question
- Index(es):
Relevant Pages
|
|
