Re: Optimal Buffer Size To Read A File

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

From: Jon Skeet [C# MVP] (skeet_at_pobox.com)
Date: 12/19/04


Date: Sun, 19 Dec 2004 08:49:01 -0000

Amy L. <amyl@paxemail.com> wrote:
> Is there a Buffer size that is optimial based on the framework or OS that is
> optimal when working with chunks of data? Also, is there a point where the
> buffer size might not be optimal (too large)? I am considering an 8K or 16K
> Buffer. The files sizes are random but range between 8K - 100K with the
> occasional files being several megs.
>
> Example:
>
> int _READBUFFER_ = 1024 ;
> fi = new FileInfo( args[ 0 ] ) ;
> FileStream fs = fi.OpenRead() ;
> while ( fs.Read( ByteArray, 0, _READBUFFER_ ) > 0 )
> {
> myStringBuilder.Append( textConverter.GetString( ByteArray ) ) ;
> }

"Optimal" depends on where you draw the line between time and memory.
The larger the buffer, the faster - but the more expensive in terms of
memory.

Note that your code above is broken, as it doesn't use the value
returned by Read to make sure that only that amount of data is
converted by the text converter.

-- 
Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too


Relevant Pages

  • Re: Expat problems
    ... > buffer is nul-terminated. ... Not a standard header. ... was read in by "chunks" instead of the whole file into memory. ... I say read in the data in chunks. ...
    (comp.lang.c)
  • block move clarification - System.Move()
    ... time, with chunks capable of being expressed in 8 bit, 16 bit and 32 bit ... I've been working on pooling memory and working out ways to make the memory ... each buffer already in existence. ... But I do not know ASM, which is why I am here. ...
    (borland.public.delphi.language.basm)
  • Re: [RFC 1/3 v3] mm: iommu: An API to unify IOMMU, CPU and device memory management
    ... But given a bunch of large chunks of memory, is there any API that can ... What is the problem about mapping a 1MB buffer with the DMA API? ... We don't need a new abstraction to reserve some memory. ...
    (Linux-Kernel)
  • Re: [RFC 1/3 v3] mm: iommu: An API to unify IOMMU, CPU and device memory management
    ... But given a bunch of large chunks of memory, is there any API that can ... What is the problem about mapping a 1MB buffer with the DMA API? ... We don't need a new abstraction to reserve some memory. ...
    (Linux-Kernel)
  • Re: Discovering variable types...
    ... >- but I suppose MS expect us to use wrappers ... memory allocations for your variables from disk as well. ... >They most certainly are of fixed size, changing the size of a String ... >>me to keep buffer size and current postion right in the memory block. ...
    (comp.lang.pascal.delphi.misc)