Re: File System as a cache mechanism

Tech-Archive recommends: Fix windows errors by optimizing your registry

From: Alvin Bruney [MVP] (vapor)
Date: 08/02/04 

Date: Mon, 2 Aug 2004 12:23:14 -0500

you should look into fragment caching for the static stuff. basically, it
re-implements internally what you have coded by hand without using I/O
access.

> 3) Under a couple of occurances of heavy load, we've gotten some File
> IO "file is in use" expections when trying to remove a file to be deleted.

right, because of concurrency you may trying to massage a file that is in
use.
One technique to over come this delete problem is to first try to delete the
file, if it fails because it is in use - hot, then you rename the file and
then delete both instances - hot delete.

finally, is this a bonafide webfarm, or you mean that it is a bunch of
servers hooked together? 

-- 
Regards,
Alvin Bruney
[ASP.NET MVP http://mvp.support.microsoft.com/default.aspx]
Got tidbits? Get it here... http://tinyurl.com/27***
"John Linn" <JohnLinn@discussions.microsoft.com> wrote in message 
news:38EFEFDE-83DA-41D2-B15B-DD98956DE613@microsoft.com...
> In the journey of finding ways to scale our application with my current 
> hardware, I've been playing around with using our servers hard disk as a 
> means of caching information.
>
> We have a LOT of content that doesn't change very much and is guarenteed 
> to be used every single page view (template information for hundreds of 
> templates, 'whose online' counts,  custom navigation, etc).  Under load, 
> there is way too much % of CPU going to these "framework" items and as we 
> add web machines to the farm, the overall RPS for each machine drops.  Up 
> until now, we've used Cache for these items, but we are now having 
> problems with memory consumption over these hundreds of templates (and I 
> want to scale to thousands of templates).
>
> So I've worked up a system where the unchanging information is written out 
> to disc after the application is first loaded, and then subsequent reads 
> come from the disk.  If a change is made to any of the disk cached 
> information, I simply delete the file through UI tools.  Each machine is 
> on DFS, so when the text file is deleted/updated, it gets copied over to 
> the other web servers.  So far, it's proven to work very nicely, because 
> on simple pages there is a slightly increased RPS rate.  On more complex 
> pages there is a much greater RPS rate.  And most importantly, the RPS 
> doesn't decrease as we add servers to the farm.
>
> I understand that disk IO still isn't as fast as memory cache and there 
> will be bottlenecks.  But the way I see it, I am eliminating a single 
> point of contention in the DB server and can scale the overall RPS of my 
> app by simply adding smaller servers to the web farm and using their 
> disk's IO.
>
> My questions:
> 1) Is this a sound design?
> 2) Can I rely on DFS to control the removal/updating of the cached files?
> 3) Under a couple of occurances of heavy load, we've gotten some File IO 
> "file is in use" expections when trying to remove a file to be deleted. 
> For removal, we use the static methods of File.Delete and 
> Directory.Delete.  To create the files we use File.CreateText and to read 
> the files, we use StreamReader.  Is there a way to lock a file that is to 
> be manipulated eliminate this issue (like application.lock)?
>
> Thanks a lot!
>
> JL
Quantcast