Re: File Stream - Performance is getting slower and slower - why?



colRawDataFile

This is the key point in this routine - it's basically the entire
file, loaded into a collection - therefore, this collection has
334,386 entries in it.

And why do you load the entire file into a collection??? This doesn't
make sense to me (but maybe I don't have a complete understanding of
your solution)... I would try an approach like this (pseudo code):

FileReader input = ...;
FileWriter output = ...;
string data;
while (input is not EOF)
{
if (i = 50000)
{
// close old output
// create new output
}

output.write(input.ReadLine());
}

no need to fetch all lines into memory (a collection).

hth
Markus
.



Relevant Pages

  • Re: Report Sum If <=
    ... If I'm understanding what you are trying to do, do it in the query. ... create the new expression (your IIF statement). ... I would like to display only those entries who meet the criteria or flag ...
    (microsoft.public.access.reports)
  • Re: Request for Comments: libarchive, bsdtar
    ... but my understanding is that it has to pull in a lot of ... Since tar files are a well ... > with ownership/permissions different than is in the unpriv'd build ... ustar/gzip archive to stdout with all entries owned by 'root'. ...
    (freebsd-arch)
  • IPMON Format ???
    ... A little help please with understanding the format of ipmon.log. ... I have entries like this (mind line-wrap): ...
    (freebsd-questions)
  • Re: NameObjectCollectionBase
    ... I would like to confirm my understanding of your issue. ... duplicated keys into a NameObjectCollectionBase. ... When adding entries with duplicated ... "This posting is provided "AS IS" with no warranties, ...
    (microsoft.public.dotnet.languages.csharp)