Re: Performance issue: Writing multiple files vs single file using fopen/fwrite

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

On May 14, 9:09 pm, "Igor Tandetnik" <itandet...@xxxxxxxx> wrote:
Rahul <rahulsha...@xxxxxxxxxx> wrote:
On May 14, 6:40 pm, "Carl Daniel [VC++ MVP]"
<cpdaniel_remove_this_and_nos...@xxxxxxxxxxxxxxx> wrote:
Rahul wrote:
I want to know why is writing into multiple files slow. for the
single file also we are opening and closing it 400 times so why is
this running 80 seconds faster.

First try a simple test: Make your program create and close 400
files, writing nothing to each file. By your description, you'd
expect that to take 80 seconds. If it does, then you have your
answer: file creation is slow. If it doesn't, then you know that
there's something else different between the two configurations of
your program that accounts for the difference.

As a second experiment, you might try using CreateFile/WriteFile
instead of fopen/fwrite to see if that makes a difference.

The application behaves differently on different systems. on one
computer I saw the fopen time (total for all 400 files) varying from
20 seconds to 100 seconds on different run's, while on the other the
fopen time remained same but the fwrite time (total for all files)
changed from 80 to 140 seconds.

Do these machines have different brands of antivirus software running,
by any chance? Do the numbers change if you temporarily disable such
software?
--
With best wishes,
    Igor Tandetnik

With sufficient thrust, pigs fly just fine. However, this is not
necessarily a good idea. It is hard to be sure where they are going to
land, and it could be dangerous sitting under them as they fly
overhead. -- RFC 1925- Hide quoted text -

- Show quoted text -

There is no antivirus software on the systems as they are performance
testing machines. and I had killed all other network drivers also, so
the only proceses running were windows essential services e.g.
svchost, exloprer, etc.
I also killed the other non essential drivers (e.g. java updater,
sound manager ect. )

But there is a mojor difference in the machines, one has windows 2003
SP2 and other has windows XP
Though the area of slowdown was different in both machine (fopen vs
fwrite) yet both showed a slowdown in case of different files.

I did 2 quick tests
1) using a simple cpp program for writing data into same file (20 MB
data 200 times, opening and closing the file in each of the 200
iteration)
2) writing 20 MB in 200 different files
and got following results

same_File.exe
Total time_ fopen for 200 files in seconds = 1.829000
Total time_ fwrite for 20480 Kbytes per file in seconds = 11.123000

different_File.exe
Total time_ fopen for 200 files in seconds = 17.984000
Total time_ fwrite for 20480 Kbytes per file in seconds = 86.746000

on multiple runs i got tsimilar timings on different machines.

Is there any way to make make multiple file write as efficient as
single file write. By explicitly caching some directory structure or
something??

.

Relevant Pages

  • Re: Binary files, little&big endian setting bits
    ... > was thinking of just writing it as chars (afaik chars are the only ... to the storage or transmission medium. ... signals or the holes in a punched card and to reconstitute ... byte data exchange is if the machines use different byte ...
    (comp.lang.c)
  • Re: rsfc SEDITION!!11!!!
    ... into the street and beaten by an angry mob for writing.. ... :know a bit about the many ways such machines *could* be rigged. ... Nixon and Kennedy both got away with it in Illinois... ...
    (rec.sport.football.college)
  • Re: Word automation version problem.
    ... how much data and are you writing it cell ... On the Word 2000 machines this works fast and nice, ... Recently a warning message, showed up in the error list: ...
    (microsoft.public.dotnet.languages.vb)
  • Re: rsfc SEDITION!!11!!!
    ... into the street and beaten by an angry mob for writing.. ... It:seems to me that there is only one attitude that could support rejecting:out of hand that such cheating could have occurred: "Nobody would ever:do such a thing." ... I can't believe how lax the rules are over making sure:those machines are honest. ...
    (rec.sport.football.college)
  • Re: Why 100% of all Mactards have Windows experience....
    ... Total Time: 4:59 ... should be irrelevant unless those apps are writing to disk. ... (probably SATA given the age of the drive). ...
    (comp.sys.mac.advocacy)