Re: Huge data needs to be transfer from Fixed width Text File to S
- From: "Massimo" <barone@xxxxxxxxx>
- Date: Wed, 22 Jun 2005 23:24:17 +0200
"Permood" <Permood@xxxxxxxxxxxxxxxxxxxxxxxxx> ha scritto nel messaggio news:56C3E79A-83B8-423A-BC6E-584F634BD4C1@xxxxxxxxxxxxxxxx
For 2 reasons i am going to develop the Application, 1- Transform the data, because I have to change data into Date and money from Char and I need to perform certain validation which is very hard to implement in TQSL as i am using SQL Server 2000.
If you find TSQL is not the right choice for your job, you can use Windows Scripting (VBScript or else) inside DTS packages; I'm not a VBScript developer, but I'm quite sure it's powerful enough to convert text strings into dates ;-)
2- There are 35 scripts, which currently perform this operation, and this is
done manually, I need to make a application where i can schulede them and
make it more rebust and conditional as compare to TSQL functionality.
You can run a DTS package with any scheduling you like, and you can manage the package workflow as accurately as you need; i.e., you can do a task (like a data validation), then do other tasks (like mailing an administrator, or executing a query on another DB, or beginning a data transfer from DB A to DB B) based on the outcome of the first one.
Oh, and you can also write store stored procedures and call them from the package, of course.
I am able to do so in VB.Net but the only problem i having now, Import time
increased 2-3 times as compare to bulk Insert in SQL Server.
Of course; bulk insert (like DTS) uses its own bulk insertion APIs, which are *way* faster than any ADO/ADO.NET/ODBC query-based application.
I suggest you have a better look inside SQL Server's DTS. They're exactly what you're looking for, but lots of people wrongly think they can be used only to import an Access DB into SQL Server; there's a lot more to them than this.
By the way, have you set the database recovery mode to "bulk-logged"? This can *really* save you time (and transaction logs size) on big data import operations, as it makes the DB log only the whole operation in the transaction logs, as opposed to logging every single row inserted.
Massimo
.
- References:
- Prev by Date: Re: FYI - Alternative DB engine for .net
- Next by Date: RE: DataGrid control scrolling
- Previous by thread: Re: Huge data needs to be transfer from Fixed width Text File to S
- Next by thread: Re: Huge data needs to be transfer from Fixed width Text File to S
- Index(es):
Relevant Pages
|
