Re: High-performance IO

Hi mate


As I can see, things are not as seamless as I thought - it looks like
you are bound by some logical constraints of your program, so that you
need asynchronous IO and multiple threads. In such case the only thing
you can optimize is memory buffer

That's why I want to use AWE -- physically contiguous and non paged.

First of all, memory does not have to be physically contigious - the
only thing you need is to make it resident and locked

But I don't know whether the disk drivers are supposed to support 64-bit
(36, in fact) physical addressing.

Drivers are not concerned about things like that - after all, they deal
only with linear addresses. Furthermore, AWE can utilize up to 4 G of
memory even without PAE. In such case, as far as your program is
concerned, memory that has been mapped with
MapUserPhysicalPages() is just non-paged one, with couple of limitation
that AWE imposes (i.e. non-sharable and always read/write). The only
problem is that AWE requires SeLockMemory privilege

Anton Bassov






Piotr Wyderski wrote:
Thank for your answers,

anton bassov wrote:

First of all, "one file-one thread" strategy is not the best one in
your situation.

Obviously, but it is the existing architecture and I cannot change it.

These files may be physically located in totally different parts of the
disk. Therefore, every time context switch occur, the system has to
update the position of the head, i.e. you have to search the disk all
the time.

Doesn't Windows sort queued request to minimize seek time
(the elevator algoritm etc.)? Anyway, it is the reason I want to
make IO transfers in large chunks. Reasonably bounded latency
does not matter, throughput does.

I think the best thing to do is to process files
sequentially, i.e. one by one

Cannot be done, the files contain only partial information. Only a specified
constelation of them provide a complete set of data one can call record.

Second, as I have already mentioned in my previous message, you should
try to avoid page faults.

That's why I want to use AWE -- physically contiguous and non paged.
But I don't know whether the disk drivers are supposed to support 64-bit
(36, in fact) physical addressing.

To summarize, the fact that you are about to access files only once and
do so sequentially gives you a HUGE potential for optimization in terms
of minimizing the number of disk accesses and disk searching.

Yes, I want to make use of this scenario.

Best regards
Piotr Wyderski

.

Relevant Pages

  • Proposed Assembler Commands
    ... ACM Automatically Clear Memory ... BKCRDR Backspace Card Reader ... BKSPD Backspace Disk ... EIAO Execute In Any Order ...
    (sci.electronics.design)
  • Re: Att. Alex Nichol -VM cont.
    ... I find documentation that says a 4kb page in memory is written to the hard ... manner that a 4kb memory page is equal to a 4kb cluster written on the disk". ... So where is the basis for saying since 4kb paging in memory that 4kb clusters ...
    (microsoft.public.windowsxp.general)
  • Re: Slow performance
    ... Leaving your computer on 24/7 means that if any programme has a memory ... The Disk Defragmenter report shows you have minimal free disk space. ... select Properties, General, Advanced and check the box before Compress ... Volume fragmentation ...
    (microsoft.public.windowsxp.perform_maintain)
  • Re: Slow performance
    ... I realize that the pagefile usage is excessive and I need to reduce memory ... I've never messed with those settings before. ... Peak are greatly in excess of the installed RAM. ... The Disk Defragmenter report shows you have minimal free disk space. ...
    (microsoft.public.windowsxp.perform_maintain)
  • Re: teaching a child - console or GUI
    ... I use a RDBMS to reuse existing code and optimizations. ... >>And loading them in a system under load (with constant disk io) is worse ... quality stuff), and quality power supply, quality memory etc. ... > lists of pointers on disk, extract files of frequent search fields in ...
    (comp.lang.pascal.delphi.misc)