Re: Large text file - in memory ( > 60mb)

Hi Jurjen,

my napkin calculations show that you have a 750,000 record table that
doesn't change often and that you need to search many times by each user.

Dude... is SQL Server really so bad of an option? c'mon! Why write this
capability into your app when it is available to you for free?


--
--- Nick Malik [Microsoft]
MCSD, CFPS, Certified Scrummaster
http://blogs.msdn.com/nickmalik

Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
I do not answer questions on behalf of my employer. I'm just a
programmer helping programmers.
--
"Jurjen de Groot" <info@xxxxxxxxxxxxxx> wrote in message
news:OvpwWCT7GHA.1256@xxxxxxxxxxxxxxxxxxxxxxx
Morten,

Thanks for your reply, I understand what you're doing in the code, but
isn't reading line by line slow ?
The file is over 64mb in size, reading it line by line to do a search
seems like a lot of overhead, especially when the user does many searches
while running the app, it would mean reading/searching the >64mb file many
times, that's why I opted to keep the file in memory which might not be
the best idea.
I'm currently trying to get some more time from my client to try and
optimize by creating an index of the file (which doesn't change that
often) and searching through that and retrieving part of the text-file
corresponding to the index...

Jurjen.



"Morten Wennevik" <MortenWennevik@xxxxxxxxxxx> wrote in message
news:op.tg8x7ygjklbvpo@xxxxxxxxxxxxxxxxxx
Hi Jurjen,

Sounds to me like you could just use a ReadLine() and do a search per
record. You should use the encoding used in the file. If you don't
specify an encoding, UTF-8 is used. You would need some logic added to
keep track of an entire recordset, which can be a string[] of length 9

StreamReader sr = new StreamReader("", Encoding.Default);
string s = null;
string[] recordset = new string[9];
int index = 0;

while ((s = sr.ReadLine()) != null)
{
int i = GetRecordNumber(s);

if (i >= (index + 1))
;// missing record

recordset[index] = s;

index++;

if (i == 9) // complete record
{
if (SearchRecordSet(recordset))
return true;

Array.Clear(recordset, 0, 9);
index = 0;
}
}


PS! Your system clock is a bit too fast


On Wed, 11 Oct 2006 09:01:00 +0200, Jurjen de Groot <info@xxxxxxxxxxxxxx>
wrote:

Greg,

The text-file consists of records of 80 characters seperated by a
NewLine.
These records all have a record type 1 thru 9, a set of records start
with
record 1 and end with record 9 at wich point the next set will start with
record type 1.

I search the contents of the file for the search criteria as entered by
the
user f.i. 2742281, when I find this sequence I have to make sure it's
found
in exactly the right position within the record to make sure I have
compared
it to the right field. Then I have to show this record found (wich
should be
record type 1) and show all records until I find recordtype 9 (of EOF).

I could create an index but that would complicate the app, I also
thought of
maybe creating a datatable to ease the search but I'm pretty sure memory
consumption would be even worse...

I was just wondering why the current app is consuming so much memory
wich is
now clear to me. I guess my client will have to make the decision, cheap
app
wich will use much memory, little more expensive app using less memory.


Regards,
Jurjen.


"Greg Young" <druckdruckREMOVEgoose@xxxxxxxxxxx> wrote in message
news:egRx8uJ7GHA.4604@xxxxxxxxxxxxxxxxxxxxxxx
As CD has said this is expected as strings are UTF-16 .. my question
would
be how you are searching this file.

Are you just doing keyword searches? Depending on the type of search you
might be much better off doing something like building an index of the
file and loading the index into memory.

Cheers,

Greg

"Jurjen de Groot" <Jurjen.de.Groot@xxxxxxxxx> wrote in message
news:OCVR7iI7GHA.4552@xxxxxxxxxxxxxxxxxxxxxxx
My program needs to search a large textfile (>60MB).

At this time I'm using a streamreader to read the file into a
string-variable (objString = sr.ReadToEnd). Before reading the file the
proces running my programm uses about 10mb, after reading the text-file
into the string, it uses over 200mb. I would expect the program to use
between 70 and 100mb.

Is there a more efficient way of storing this data in-memory and still
be
able to search through it ... ?


TIA,
Jurjen.








--
Happy Coding!
Morten Wennevik [C# MVP]



.

Relevant Pages

  • Re: How to purge file cache
    ... What I'm after is an app which does reading, of a file using FSD- hence i'm ... assuming that the file won't be memory-mapped into the memory. ... and do a re-copy would take shorter time as the reading is done via cache... ... >> megs) are cached in the file system. ...
    (microsoft.public.development.device.drivers)
  • Re: Large text file - in memory ( > 60mb)
    ... You would need some logic added to keep track of an entire recordset, which can be a string[] of length 9 ... Then I have to show this record found (wich should be ... I was just wondering why the current app is consuming so much memory wich is ...
    (microsoft.public.dotnet.framework)
  • Re: Large text file - in memory ( > 60mb)
    ... The file is over 64mb in size, reading it line by line to do a search seems ... that's why I opted to keep the file in memory which might not be the best ... Then I have to show this record found (wich should ... I could create an index but that would complicate the app, ...
    (microsoft.public.dotnet.framework)
  • Re: Large text file - in memory ( > 60mb)
    ... These records all have a record type 1 thru 9, a set of records start with ... Then I have to show this record found (wich should be ... I was just wondering why the current app is consuming so much memory wich is ...
    (microsoft.public.dotnet.framework)
  • Strange .NET garbage collector problem
    ... and put them in a ListView control. ... After reading the data, my app ... Now, if a minimize the app, the used memory drops to 1 or 2 ...
    (microsoft.public.dotnet.framework.aspnet.buildingcontrols)