Re: Disk access performance



huayra wrote:

I am working in an application, where multiple threads have parallel disk
access to process each one file. The threads are independent, no critical
section or syncronization between them exist nor is necesary. They do not
process the same file.

In my experience, disk I/O on windows parralelizes very poorly -- try
copying 2 files from or to the same disk at a time (let the OS wars
begin :).

I need some advice about how to get the best performance out of this
scenario. Also, if somebody can suggest a tool to look at the times each
thread expends waiting for a disk access system call, and stuff like that.

Getting a diagnostics tool is definatly the way to go -- otherwise you
may end up chasing ghosts.

A specific tool would be best, unfortunatly I don't know of any.

A sampling profiler should be able to show you roughly how much time is
spent waiting for disk-IO, but beware the profiler might affect the
scenario here, where there is real parallelism between the CPU and I/O.

You can modify your program to access files through adapters that
measure time spent in I/O. This may not be that complicated since you
can inject the adapter where the files are opened.

If you have a virus-scanner with "on-access" scan, disable it.
Virus-scanners with on-access are som of the worst viruses you can get
-- they steal your productivity little by little.

If you have enough memory you could linearize the reading of the files
between threads, and read them into memory in their entirity. I know
this is not a nice solution, but it *may* just work without a whole lot
of development effort.

For a more complicated solution you could read the files into memory in
large chunks via a Stream adapter, and linearize that between threads.
This may allow you to more effectively utilize the CPU.

--
Helge Jensen
mailto:helge.jensen@xxxxxxx
sip:helge.jensen@xxxxxxx
-=> Sebastian cover-music: http://ungdomshus.nu <=-
.

Relevant Pages

  • Re: Disk access performance
    ... disk I/O on windows parralelizes very poorly -- try ... where there is real parallelism between the CPU and I/O. ... If you have enough memory you could linearize the reading of the files ... large chunks via a Stream adapter, ...
    (microsoft.public.dotnet.framework.performance)
  • Re: Ram Card Design Help
    ... A board that emulated the two board altair controller and ... USB mass storage devices, like a floppy disk, memory stick, hard ... through some I/O ports and readable either by I/O port or by a common ...
    (comp.os.cpm)
  • Re: Caching control
    ... |> | invalidate/unmap them in order to discard the data from memory. ... |> writing out to disk. ... | easy to discard as clean disk cache. ... stating that a specific amount of RAM can be used only for I/O ...
    (comp.os.linux.development.system)
  • Re: Virtual Memory Settings?
    ... > your virtual memory set to at least 1.2xRAM. ... (Disk I/O is generally at least 1,000 times slower than memory I/O!) ... where the single disk holding the jfs log can't keep up with all the I/O ...
    (comp.unix.aix)
  • Re: OT:PC problem
    ... I am rapidly losing disc space. ... tab and click on the Mem Usage column header twice to sort by memory usage ... Performance' tab and click both I/O Read Bytes and I/O Write Bytes tick ... they are writing to disk by clicking the I/O Write Bytes column twice. ...
    (uk.sport.football.clubs.celtic)