Re: ReadFileScatter and WriteFileGather

Tech Tip: Click here to run a free scan for Windows Errors and optimize PC performance

"Beverly Brown" <beverly.brown@xxxxxxxxx> wrote in message
news:%23Pmjry7RGHA.4688@xxxxxxxxxxxxxxxxxxxxxxx
Looking at the docs for these functions, I see that the buffers in the
array must be page-sized and page-aligned. I haven't quite figured out how
this is useful in a real-world example. Why would an application want
discontiguous page-sized, page-aligned virtual buffers?

Page-aligned page-size buffers can be passed down to bare metal without any
transformations, which is required for maximum efficiency. If you don't care
about this, you should not care about scatter-gather IO, either.

The usefulness of discontiguous buffers cannot be seen in a example that can
be posted here.

One example is a DB server that must cache disk data without using the OS
cache. While the cache is relatively free, you can read contiguous disk data
into contiguous buffers. But eventually the cache will fill up, and then
parts of it will be purged, and you suddenly cannot read contiguous disk
data into contiguous buffers, unless you grow the cache, or de-fragment the
cache, or issue multiple IO requests, or just read contiguous data into
discontiguous buffers. Do I need to explain why the latter option is the
best?

In order to get such an array, VirutalAlloc would need to be called
repeatedly requesting single pages each time.

This can be done, but this is exactly what should not be done. Instead, you
allocate a big contiguous chunk and maintain a free page list/bitmap and use
whatever free pages that happen to be in the list. You do not re-allocate
each time you do IO.

[...]

Also, if the buffers need to be page-aligned as the docs say, why bother
having a member specifying Alignment in the FILE_SEGMENT_ELEMENT
structure?

It is not a structure, it is a union. This element ensures that each element
is 64 bit aligned.

S


.

Relevant Pages

  • Re: Increased Buffers due to patch 56e49d (vmscan: evict use-once pages first), but why exactly?
    ... All is about the increased amount of "Buffers" accounted as active while ... But I didn't get how this prefers buffers compared to cache pages (I ... This indeed sounds like the kind of workload that would only ... filesystem metadata all the time. ...
    (Linux-Kernel)
  • Re: Increased Buffers due to patch 56e49d (vmscan: evict use-once pages first), but why exactly?
    ... All is about the increased amount of "Buffers" accounted as active while ... But I didn't get how this prefers buffers compared to cache pages (I ... This indeed sounds like the kind of workload that would only ... it might makes stream I/O benchmark score a bit because such workload ...
    (Linux-Kernel)
  • Re: Top - Mem: 119212k used
    ... AC>> meant "60+megs after excluding cache", ... AC>> This indicates 54360K in use, after excluding buffers and cache. ... AC> is the smallest window manager I could find. ... (/usr/bin/X11/xterm, not the 'fancy' xterms), and Mozilla. ...
    (comp.os.linux.misc)
  • Re: free (command) memory question
    ... > I just ran the free command and it shows as follows: ... Of the used memory 105MB is buffers and 213MB ... would assume Linux always keeps a little physical memory for buffers and ... cache, so I would expect it to start swapping before all of the additional ...
    (alt.os.linux.suse)
  • Re: Is Greenspun enough?
    ... >Ulrich isn't wrong, George. ... The "global page cache" has pretty much ... but not to swappable data because the page in RAM ... is not, in general, a copy of the disk data. ...
    (comp.lang.lisp)