Loading large file in memory or not?

From: Gustaf Liljegren (gustaf.liljegren_at_bredband.net)
Date: 02/12/04 

Date: Thu, 12 Feb 2004 13:56:40 +0100

I'm writing a program that will query an XML file with the XmlTextReader
class (the question is not specifically about XML, however). This file is
very large (maybe 20 MB at most), and during extreme conditions, it'll have
to be queried thousands of times (maybe 20,000 times at most). Small
fragments of this file are used to construct another file.

What is most economical, in terms of system resources? Opening and closing
the file + initializing and destroying the XmlTextReader thousands of times,
or loading all the contents in the file in memory, and pick data from a tree
in memory? Also, would it be more efficient to query a table in SQL Server
thousands of times?

Thanks,

Gustaf

Relevant Pages

  • Re: Loading large file in memory or not?
    ... > I'm writing a program that will query an XML file with the XmlTextReader ... > in memory? ... would it be more efficient to query a table in SQL Server ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: xmltextreader
    ... does not store data in memory. ... in my xml file I have ... I have to load departments in a different table and rest of it in a ... Copying data you read using the XmlTextReader to two different tables depending on what the XML is should not be a problem. ...
    (microsoft.public.dotnet.languages.csharp)
  • XML - traversing in VB
    ... I have an XML file that looks something like this: ... Can someone refresh my memory? ... BTW, I want to use XMLTextReader to do this. ...
    (microsoft.public.dotnet.framework.aspnet)
  • XmlDocument.ReadNode() breaking - why?
    ... XmlTextReader and XmlDocument.ReadNode. ... his program just creates a new dom using new document like this: ... What's happening is that for this weird xml file he gets, ... are valid iso-8859-1 (the document's declared encoding) in a CDATA section. ...
    (microsoft.public.dotnet.xml)
  • Parsing XML with ElementTree (unicode problem?)
    ... (this question was also posted in the devshed python forum: ... I'm trying to parse an XML file ... where query is a query to the AWS, and this specific query has the ... some of the content with 'bla bla' string in order to make it fit ...
    (comp.lang.python)
Loading