Re: 200MB and bigger flat files
- From: "Paul Somers" <paul@xxxxxxxxxx>
- Date: Mon, 8 Jan 2007 11:38:27 +0200
Are you using BizTalk 2006?
I have been able to process 200 mb flat files through BizTalk 2006
successfully, it did take a bit of time I recall, I did need to ensure that
I had sufficient memory on the BizTalk server, even though with 2006 it
caches things to disk.
"Sinisa Catic" <SinisaCatic@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:CEEBC2FC-9115-4672-B615-81CFE60DBE3B@xxxxxxxxxxxxxxxx
Hi, I am doing some proof of concept tests to find best solution to
process
flat files that are very large (from BizTalk perspective): 200MB and
bigger.
One solution is to split file into smaller chunks in the receive pipeline
by
inserting carriage return (before its converted to XML). Right now I am
doing
such test with the 100MB flat file that will be split into 29 pieces each
with the size around 3.5MB 9I am gradualy increasing file size). I am
doing
tests with various batch sizes and have found that bigger batch size
generally cuts processing time. I am doing splitting only and routing
outgoing XML into separate send folder. As I have expected processing is
slow
for such large input file but I wonder why is not faster because I have
noticed that CPU is almost not used after file is taken and split into
chunks. Memory also stays flat all the time (1.4GB from the total of 2GB,
4
logical processors). It looks processing is IO bound and not CPU or memory
bound. I wonder if processing could be shortened by adding more messagebox
databases or maybe tuning some parameters. It does not look even that SQL
Server is bottleneck. More looks like the problem is how BizTalk is using
SQL
Server. Any suggestion would be welcome. Thanks.
Sinisa Catic
.
- Prev by Date: Re: Sendport for Webservice
- Next by Date: Oracle Adapter Error
- Previous by thread: Re: Sendport for Webservice
- Next by thread: Oracle Adapter Error
- Index(es):
Relevant Pages
|
