Re: Large text file - in memory ( > 60mb)

Greg,

The text-file consists of records of 80 characters seperated by a NewLine.
These records all have a record type 1 thru 9, a set of records start with
record 1 and end with record 9 at wich point the next set will start with
record type 1.

I search the contents of the file for the search criteria as entered by the
user f.i. 2742281, when I find this sequence I have to make sure it's found
in exactly the right position within the record to make sure I have compared
it to the right field. Then I have to show this record found (wich should be
record type 1) and show all records until I find recordtype 9 (of EOF).

I could create an index but that would complicate the app, I also thought of
maybe creating a datatable to ease the search but I'm pretty sure memory
consumption would be even worse...

I was just wondering why the current app is consuming so much memory wich is
now clear to me. I guess my client will have to make the decision, cheap app
wich will use much memory, little more expensive app using less memory.


Regards,
Jurjen.


"Greg Young" <druckdruckREMOVEgoose@xxxxxxxxxxx> wrote in message
news:egRx8uJ7GHA.4604@xxxxxxxxxxxxxxxxxxxxxxx
As CD has said this is expected as strings are UTF-16 .. my question would
be how you are searching this file.

Are you just doing keyword searches? Depending on the type of search you
might be much better off doing something like building an index of the
file and loading the index into memory.

Cheers,

Greg

"Jurjen de Groot" <Jurjen.de.Groot@xxxxxxxxx> wrote in message
news:OCVR7iI7GHA.4552@xxxxxxxxxxxxxxxxxxxxxxx
My program needs to search a large textfile (>60MB).

At this time I'm using a streamreader to read the file into a
string-variable (objString = sr.ReadToEnd). Before reading the file the
proces running my programm uses about 10mb, after reading the text-file
into the string, it uses over 200mb. I would expect the program to use
between 70 and 100mb.

Is there a more efficient way of storing this data in-memory and still be
able to search through it ... ?


TIA,
Jurjen.





.

Relevant Pages

  • Re: Large text file - in memory ( > 60mb)
    ... The file is over 64mb in size, reading it line by line to do a search ... while running the app, it would mean reading/searching the>64mb file many ... Then I have to show this record found (wich ... maybe creating a datatable to ease the search but I'm pretty sure memory ...
    (microsoft.public.dotnet.framework)
  • Re: Large text file - in memory ( > 60mb)
    ... You would need some logic added to keep track of an entire recordset, which can be a string[] of length 9 ... Then I have to show this record found (wich should be ... I was just wondering why the current app is consuming so much memory wich is ...
    (microsoft.public.dotnet.framework)
  • Re: Large text file - in memory ( > 60mb)
    ... The file is over 64mb in size, reading it line by line to do a search seems ... that's why I opted to keep the file in memory which might not be the best ... Then I have to show this record found (wich should ... I could create an index but that would complicate the app, ...
    (microsoft.public.dotnet.framework)
  • Re: Packed Records
    ... Every record type is either packed or not; this is part of its type ... difference to it is the exact memory layout. ... Any speed difference results only from the memory architecture that ... two full word accesses are done and only the ...
    (alt.comp.lang.borland-delphi)
Loading