Re: W3WP memory usage

From: Paul (a_at_b.com)
Date: 03/11/04 

Date: Thu, 11 Mar 2004 16:08:28 -0500

Kevin, thank you for your response.

I am quite aware that Sharepoint files are stored in the database. When I
said "use the file system", I meant that I will not store the files in
Sharepoint at all, but instead store them in the file system, effectively
giving up on Sharepoint altogether.

When you say "whole lot of hits to that database", I assume you mean the set
of queries the sharepoint classes send to the SQL db to request the site
names and document names. My experience is with SQL server, and I can
assure you that querying a couple dozen rows from it is NOT causing a 50M
memory request. Besides, the memory usage is in W3WP, and not SQL Server.
The SQL server process never goes over 20Meg of usage!

No, the Sharepoint libraries are hogging the memory, and for some reason are
not releasing it, and then dying under their own weight (bringing IIS with
it).

Now, if you are telling me that the SPFile class, when instanciated, stores
the entire file in memory, then I could believe that it might need 10 or so
meg on a request (the complete size of all the files I am storing in
sharepoint at this time). But nothing in the (hopeless) Sharepoint
documentation would indicate that. I am simply requesting the names and
urls to the different files.

And so, my question is basically the same: What is causing the Sharepoint
Class Library to eat 50M of memory on a request of what is approximately 20K
worth of data?

"Kevin Spencer" <kevin@takempis.com> wrote in message
news:OtGFVQ6BEHA.3404@TK2MSFTNGP10.phx.gbl...
> You have a bigger problem than you realize. Check your file system. The
> pages aren't there. They are all stored in the database, and your app is
> making a whole lot of hits to that database, which is the reason for all
the
> excess overhead. So, you won't be able to use the file system to enumerate
> sites/pages and/or gather information about them. Your best bet would be
to
> run this utility as a Service and store the data in a database. The
Service
> only has to "catalogue" the content maybe once a day, and your Web Service
> can hit the database to get the "catalogue" of pages.
>
> --
> HTH,
> Kevin Spencer
> .Net Developer
> Microsoft MVP
> Big things are made up
> of lots of little things.
>
> "Paul" <a@b.com> wrote in message
> news:#BTs3U5BEHA.2804@tk2msftngp13.phx.gbl...
> > I wrote a web service that uses the Sharepoint Class Library to navigate
a
> > sharepoint site collecting the names of all the Webs, their shared
> > documents, and the versions, and returns it to a client. Memory usage
of
> > the w3wp process goes insane, grabbing 50 Meg chunks just about every
time
> I
> > traverse the tree (but sometimes only 5M). Sometimes the w3wp process
> gives
> > the memory back, and other times it heads on up to around 700Meg, but it
> > always eventually craps out, at which point I get out of memory errors
on
> > the server (on any IIS web site defined) and I have to stop and start
the
> > world wide web publishing service. Some web sites return a page with
> > "-2146893056 (0x80090300)" on them, and nothing else.
> >
> > The code is fairly simple (just loop through each web and collect its
> Shared
> > Documents files and their version's names, and recursively call for sub
> > webs). The whole collection is pretty small, about 10 sites, with a few
> > nested sites no more than 3 deep. I would guess the resulting data
> > structure that gets returned through the web service is no more than 20K
> or
> > so. My server is running SBS2003 and has a Gig of RAM.
> >
> > I am wondering if there is a resource that I am not closing properly and
> > somehow all the sharepoint libraries are getting loaded every time. If
> > there is no fix, We're going to have to chuck out Sharepoint altogether,
> and
> > just use the File System.
> >
> > Here is the code, in case you are interested. Sorry about the tabbing.
> Any
> > help would be appreciated. Thanks, Paul
> >
> > public class SPWSTree
> > {
> > public static SPTree GetTree(string i_server, string i_site, int
i_depth)
> > {
> > if (i_depth > 5)
> > return new SPTree();
> > // create the new Tree Node
> > SPTree a_returnNode = new SPTree();
> > // get a handle on the site.
> > SPSite a_site = new SPSite(i_server);
> > if (i_site == "/")
> > {
> > a_returnNode.m_url = "";
> > a_returnNode.m_displayName = "XESP";
> > }
> > else
> > {
> > a_returnNode.m_url = i_site.Substring(i_site.LastIndexOf("/"));
> > a_returnNode.m_displayName = a_site.AllWebs[i_site].Title;
> > }
> > ArrayList a_folders = new ArrayList();
> > foreach (SPWeb a_web in
a_site.AllWebs[i_site].GetSubwebsForCurrentUser())
> > {
> > a_folders.Add(GetTree(i_server, a_web.ServerRelativeUrl, i_depth + 1));
> > }
> > if (a_folders.Count > 0)
> > a_returnNode.m_folders = (SPTree [])a_folders.ToArray(typeof(SPTree));
> > // now get all the files in this site's shared documents folder
> > SPFileCollection a_fileCollection = null;
> > try
> > {
> > SPFolder a_sharedDocuments = a_site.AllWebs[i_site].GetFolder("Shared
> > Documents");
> > a_fileCollection = a_sharedDocuments.Files;
> > }
> > catch (Exception /* Ex */)
> > {
> > // a_returnNode.m_displayName = "1:" + Ex.Message;
> > }
> > if (a_fileCollection != null)
> > {
> > ArrayList a_files = new ArrayList();
> > foreach (SPFile a_file in a_fileCollection)
> > {
> > SPTree a_fileNode = new SPTree();
> > a_fileNode.m_url = "/" + a_file.Url;
> > a_fileNode.m_displayName =
> a_file.Url.Substring(a_file.Url.LastIndexOf("/")
> > + 1);
> > if (a_file.CheckedOutBy != null)
> > a_fileNode.m_checkedOutBy = a_file.CheckedOutBy.LoginName;
> > a_fileNode.m_currentVersion = a_file.UIVersion.ToString();
> > ArrayList a_versions = new ArrayList();
> > foreach (SPFileVersion a_fileVersion in a_file.Versions)
> > {
> > SPVersion a_version = new SPVersion();
> > a_version.m_comment = a_fileVersion.CheckInComment;
> > a_version.m_createdBy = a_fileVersion.CreatedBy.LoginName;
> > a_version.m_createdOn = a_fileVersion.Created.ToString("r");
> > a_version.m_size = a_fileVersion.Size.ToString();
> > a_version.m_id = a_fileVersion.ID.ToString();
> > a_versions.Add(a_version);
> > }
> > SPVersion a_currentVersion = new SPVersion();
> > a_currentVersion.m_comment = a_file.CheckInComment;
> > a_currentVersion.m_createdBy = a_file.ModifiedBy.LoginName;
> > a_currentVersion.m_createdOn = a_file.TimeLastModified.ToString("r");
> > a_currentVersion.m_size = a_file.Length.ToString();
> > a_currentVersion.m_id = a_file.UIVersion.ToString();
> > a_versions.Add(a_currentVersion);
> > a_fileNode.m_versions = (SPVersion
> [])a_versions.ToArray(typeof(SPVersion));
> > a_files.Add(a_fileNode);
> > }
> > if (a_files.Count > 0)
> > a_returnNode.m_files = (SPTree [])a_files.ToArray(typeof(SPTree));
> > }
> > return a_returnNode;
> > }
> > }
> > public class SPVersion
> > {
> > [XmlAttribute()] public string m_id;
> > [XmlAttribute()] public string m_createdBy;
> > [XmlAttribute()] public string m_createdOn;
> > [XmlAttribute()] public string m_size;
> > [XmlText()] public string m_comment;
> > public SPVersion()
> > {
> > }
> > }
> > public class SPTree
> > {
> > [XmlAttribute()] public string m_url;
> > [XmlAttribute()] public string m_checkedOutBy;
> > [XmlTextAttribute()] public string m_displayName;
> > public string m_currentVersion;
> > public SPVersion [] m_versions;
> > public SPTree [] m_folders;
> > public SPTree [] m_files;
> > public SPTree()
> > {
> > }
> > }
> >
> >
>
>

Relevant Pages

  • Re: Filesystem with Sharepoint
    ... It was SharePoint Team Services. ... There were often sync problems between the database data and the file system data and so Microsoft moved on to using the same place for everything in the follow-up product WSS. ... The DB approach actually penalises you in the sense that a file may not be worth the disk its stored on - ie can be moved to older, slower storage and the metadata reflecting this change. ...
    (microsoft.public.sharepoint.portalserver)
  • RE: transfer companyweb config db
    ... configuration database for sharepoint services to a new location on the sql ... You may mistake SQL server conception. ... Move the Monitoring Database ... >project web access not working correctly is to use a config database on ...
    (microsoft.public.windows.server.sbs)
  • Re: w3wp.exe memory issue.
    ... also maybe you can deactivate the cache of sharepoint. ... the "Database" option store serializable ... values to the database instead-of the memory, ...
    (microsoft.public.sharepoint.portalserver)
  • Re: WSS MSDE DB Location Change Nightmare
    ... STS_servername_1 database. ... able to do your reinstall of MSDE and restore over top of the new Content DB ... Sharepoint Services was installed. ... Pool Is the Account That Has the Required Permissions to the SQL Server ...
    (microsoft.public.sharepoint.windowsservices)
  • Re: Where are files stored?
    ... All files stored within the SharePoint systems are stored in databases. ... WSS 3.0 indexes PDF files provided you have installed an IFilter for PDF files. ... Is it possible to organize storage of files on a file system rather than database? ... I use SQL Server express edition for development purposes, and if I add many pdfs/images I will exceed 4 gb limit of database size. ...
    (microsoft.public.sharepoint.windowsservices)
Loading