Re: Memory Management extremely poor in C# when manipulating string..

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance


"Segfahlt" <Segfahlt@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:0D2A26C4-903F-4A15-BAEE-B45EC295F7AA@xxxxxxxxxxxxxxxx
>I have a fairly simple C# program that just needs to open up a fixed width
> file, convert each record to tab delimited and append a field to the end
> of
> it.
>
> The input files are between 300M and 600M. I've tried every memory
> conservation trick I know in my conversion program, and a bunch I picked
> up
> from reading some of the MSDN C# blogs, but still my program ends up using
> hundreds and hundreds of megs of ram. It is also taking excessively long
> to
> process the files. (between 10 and 25 minutes). Also, with each
> successive
> file I process in the same program, performance goes way down, so that by
> the
> 3rd file, the program comes to a complete halt and never completes.
>
> I ended up rewriting the process in perl which takes only a couple minutes
> and never really gets above a 40 M footprint.
>
> What gives?
>
> I'm noticing this very poor memory handling in all my programs that need
> to
> do any kind of intensive string processing.
>
> I have a 2nd program that just implements the LZW decompression
> algorithm(pretty much copied straight out of the manuals.) It works great
> on
> files less than 100K, but if I try to run it on a file that's just 4.5M
> compressed, it runs up to 200+ Megs footprint and then starts throwing Out
> of
> Memory exceptions.
>
> I was wondering if somebody could look at what I've got down and see if
> I'm
> missing something important? I'm an old school C programmer, so I may be
> doing something that is bad.
>
> Would appreciate any help anybody can give.
>
> Regards,
>
> Seg

That's really hard to answer such broad question without a clear description
of the algorithm used or by seeing any code, so I'll have to guess:
1. You read the whole input file into memory.
2. You store each modified record into an array of strings or into a
StringArray, and write it to the file when done with the input file.
3. 1 + 2
.....
Anyway you seem to hold too much strings in memory before writing to the
output file.

Willy.


.

Relevant Pages

  • Re: .NET SUCKS --- READ FOLLOWING. MICROSOFT IS A SUCKY CO
    ... > system just doesn't cut it in high memory load situations. ... Garbage Collection is not an excuse for poor memory management. ... One excellent example is the use of strings. ... because many "professional" programmers know very little about what is ...
    (microsoft.public.dotnet.framework)
  • Re: Secure Credentials pwd handling
    ... strings that are produced when you read the properties. ... Co-author of "The .NET Developer's Guide to Directory Services Programming" ... so at some point the value will be in memory. ... SecureString is added to .NET to support this use case. ...
    (microsoft.public.dotnet.security)
  • Re: CStrings and memory
    ... Then you allocate a few more strings ... Now you need to allocate another string, but there is no more memory. ... In addition to strings, there are other things that get allocated: ... have perhaps confused "address space" with "working set". ...
    (microsoft.public.vc.mfc)
  • Re: Verbose functional languages?
    ... whereas a memory leak due to too much laziness in the wrong place can be. ... If you declare your data structure elements as strict, ... I think size matters, for the optimization heuristics. ... strings for symbol names. ...
    (comp.lang.functional)
  • Re: mex: cell (of strings) allocation
    ... strings and numeric data, and returns the numeric data in a ... and the strings in a cell array of strings. ... cannot seem to use the same idiom for the cell of strings. ... *exactly* the same amount of memory you originally ...
    (comp.soft-sys.matlab)