Re: Splitting a 31GB text file into smaller files
Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance
web33 wrote:
I have a 31gb file that was exported from a unix box.
Eeek!
I was trying to bulkinsert it into sql but there seems to be something wrong
with the file. In order to isolate the issue I have tried splitting it into
smaller files. I get an out of memory on the 7th chunk of 10,000,000 lines
using the code below. The exe only seems to be consuming about 20Mb of memory.
I have tried several techniques but all yield the same results.
Any suggestions?
1. Ask .Net questions in a *.dotnet.* newsgroup.
2. Most .Net applications will consume 20Mb of memory, if not more -
that's the size of the runtime library, a.k.a. the .Not Framework.
(and, more usefully)
3. Use the bcp utility to load tha data, not bulk insert (which is just
a wrapper for bcp, anyway). bcp can tell you about errors as it goes
along and can [probably] deal with a file this big all by itself.
Regards,
Phill W.
.
Relevant Pages
- Re: Splitting a 31gb text file
... In order to isolate the issue I have tried splitting it into smaller files. ... The exe only seems to be consuming about 20Mb of memory. ... Try writing a loop which *just* reads the lines, ...
(microsoft.public.dotnet.framework) - Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets
... > set of CPUs isolated in child cpusets, ... I did think of the user adding cpus to this map from the ... > kernel code to adapt to new cpu or memory settings. ... > purpose of your patch is not so much to isolate cpus, ...
(Linux-Kernel) - RE: DLLHOST.EXE Memory Leak?
... isolate all of your COM+ components into their own packages. ... (instead of 5 sites running in the same DLLHOST). ... PerfMon, use the ASP, Process, Processor, Memory, Thread, and Web Service ... A Memory Leak will almost never recover on it's own, ...
(microsoft.public.inetserver.iis.security) - Re: IIS Out of Memory
... Create a COM+ Server package, put your COM objects in the package. ... Pat ... Can you point me to a reference on how to isolate the COM objects? ... >> It probably means that you have a Memory leak that is leading to memory ...
(microsoft.public.inetserver.iis) - Re: Inserts so slow using _Connection.Execute
... If I use bcp, don't I have to save the records to a text file first? ... five you a decent Delphi example. ... you point a BCP API function to the array and say DO IT for as ... If you can't fit all of the rows into memory at one time, ...
(borland.public.delphi.database.ado) |
|
