RE: System.OutOfMemory Exception

Tech-Archive recommends: Fix windows errors by optimizing your registry

From: Cowboy (Gregory A. Beamer) - MVP (NoSpamMgbworld_at_comcast.netNoSpamM)
Date: 01/21/05 

Date: Fri, 21 Jan 2005 12:35:06 -0800

Consider creating a view in SQL Server that gets the data in the correct
format and use a Process object to fire BCP to pull from this view. You will
have to assemble a proper BCP line, but the advantages are as follows:

1. Reduced load on SQL Server
2. No memory overhead issues
3. Much faster than anything you can write in VB.NET

If you are comfortable with running a batch script, you can simply put the
BCP line in a .bat file and run it. Less control, less overhead.

The problem with your application is most likely the 2 GB virtual memory
allocation per application in Windows. You can registry tweak this, but
realize that it could have adverse effects on programs other than yours. BCP
is a better option on a variety of fronts and quite easy to set up from the
command line (using SQL Server books online). 

---
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA
***************************
Think Outside the Box!
***************************
"LP" wrote:
> Hello,
> 
> We running VB.NET application which gets massive amounts of data from SQL
> Server, loads data into DataTables, then re-arranges data into tabular
> structure and outputs it to a flat file. This is just the nature of
> requirements for this application. Business analyst here want certain data
> to be "re-horizontalize" and served in a single flat file.
> We ran this app many times before on data from different quarters, that
> worked without any problems. However, now they want to run this program
> against data for the whole last year. We keep getting System.OutOfMemory
> Exception. I realize that it's a huge amount of data that has to be
> processed in a dataset..., but is there any quick fixes for this kind of
> error, system tweaks (besides more memory). We really don't have time to
> redesign this application to handle data any differently at this time. They
> need this file yesterday!!! Please help.
> 
> By the way; the machine that it's running on has 2gb of RAM,
> 
> 
>

Relevant Pages

  • RE: System.OutOfMemory Exception
    ... Consider creating a view in SQL Server that gets the data in the correct ... format and use a Process object to fire BCP to pull from this view. ... > to be "re-horizontalize" and served in a single flat file. ...
    (microsoft.public.dotnet.framework)
  • RE: System.OutOfMemory Exception
    ... Consider creating a view in SQL Server that gets the data in the correct ... format and use a Process object to fire BCP to pull from this view. ... > to be "re-horizontalize" and served in a single flat file. ...
    (microsoft.public.dotnet.languages.vb)
  • Re: Flat file parsing into SQL Server 2000
    ... parsing a flat file into SQL Server 2000 using C#. ... I do something similar with a DTS package. ... I did not tried BCP though, so you should also explore that path. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: How to transfer data to another site and update data later on
    ... If you would like to do it by hand, I would suggest that you use bcp to move ... the data out to a flat file, WinZip the file and ship it. ... Before performing your updates, you should probably back up the TX ... Also our SQL Server has ...
    (microsoft.public.sqlserver.server)
  • Re: SQL Server 2005 - Simple package - Integration Services
    ... SSIS is an excellent way to move flat file data into SQL Server. ... Destination" failed with error code 0xC0209029. ...
    (microsoft.public.sqlserver.dts)