Re: File writes in a loop

Here the un-vague version of the question.

I have a 112 GB log file. I need to parse out this very large file to
smaller files. We've decided for business reasons to parse it by
date. I won't know the date, which is the name of the output file,
until after I read the record in the log. The records are supposed to
be in date order but this is NOT guaranteed. When I read a record I
then open the output file for writing. If I already have the output
file open I don't want to close it and then re-open it. The preceived
performance hit of closing and opening the same file thousands of
times doesn't make sense.

My original design was to do the following...

Open Log
Loop
Read Record
Open/Create Ouput
Write Record
Next
Close Log

All this works fine until it does the Open/Create Output on the second
interation. I was hoping to learn of a way to check if the file is
already open and wrap the Open/Create in a conditional statement.

The dictionary suggestions seem like a good solution for mulitple open
files. I will give this a try.

Thanks for the help.
.

Relevant Pages

  • Re: File writes in a loop
    ... We've decided for business reasons to parse it by ... I won't know the date, which is the name of the output file, ... performance hit of closing and opening the same file thousands of ... All this works fine until it does the Open/Create Output on the second ...
    (microsoft.public.dotnet.languages.csharp)
  • RE: How to parse UNIX commands ...
    ... awk -F"" ' {print ... $6}'` to check for the mount points. ... into some output file and then parse it using OPEN-CLOSE ...
    (perl.beginners)
  • Re: Problem with InputStream & Process
    ... nor consuming the stdout and stderr etc. I'd simply parse ... then delete the temporary output file. ... output and the output file was not growing over time. ... But I don't think it will provide the real time effect ...
    (comp.lang.java.programmer)
  • Re: lmutil extraction
    ... output from the lmutil call. ... Use matlab or another program to parse ... the output file and put it against your mapping of users. ...
    (comp.soft-sys.matlab)