Re: Fastest Method to add rows to a dataset?

Tech-Archive recommends: Fix windows errors by optimizing your registry

If speeding up view time for your test data is your goal, then here are some options.

For speeding up a DataSet, you end up turning off all constraints, which it sounds like you have already hit apon. I am not 100% sure the best way to turn off all type checking, other than perhaps make everything a string. There is a danger in going that direction, however, as you have no type checking if you approve the test data.

I do not know of any other way to speed things up tremendously with DataSets. Other directions to try are LINQ and the EntityFramework. I can tell you, from experience, that LINQ may load faster, but it will certainly bog down on putting thousands of rows into the database. It also requires an open context the entire time, which may or may not be a no go for you. The EntityFramework is promising, but I have not played with it for this type of work.

One other option is to create custom objects that you load and then your biggest worry is putting them into the database. You can load them back into a dataset and perhaps put up with the slowness at that point (already approved data) or you can set up a DAL of your own. You might also choose to bulk load from the objects at this point.

For speed, I you might also consider putting up a temp table and creating a bulk load file. This is likely to have low impact on your current design. You can then examine the contents from the database rather quickly or migrate directly into the main table from there. Creating the file should take a second or two at max, bulk loading a few thousand rows is very fast as well. Not sure on load of the DataSet from the database, but that should be fairly fast.

--
Gregory A. Beamer
MVP, MCP: +I, SE, SD, DBA

Subscribe to my blog
http://feeds.feedburner.com/GregoryBeamer#

or just read it:
http://feeds.feedburner.com/GregoryBeamer

********************************************
| Think outside the box! |
********************************************
"Leon_Amirreza" <r_rahmaty@xxxxxxxxxxx> wrote in message news:%23nAaYuOHJHA.3884@xxxxxxxxxxxxxxxxxxxxxxx
Because I have a DataGenerator (or whatever u want to call it) = some kind of math or simulation program
that produces data in a FOR statement i need somewhere to store it in memory (no backing store at all)
I have 4 Gb Ram to store temporary data (no problem with that)
But i want to bowse the result (output) at one see if it is usable then persist it or in other words store it in a file!

"Cowboy (Gregory A. Beamer)" <NoSpamMgbworld@xxxxxxxxxxxxxxxxxx> wrote in message news:O5SvLINHJHA.4884@xxxxxxxxxxxxxxxxxxxxxxx
Why?

Serious question. Why are you adding 1000s of rows to a DataSet? Is this for display purposes (report) or so you can add them to the database? If the latter, I suggest not using the DataSet, as it is not the best way to bulk load records.

If for a Report, you probably have the option of linking hte two persistant stores to get one DataSet. Just a thought.

Back to why? Answers to the question of which is faster are meaningless until one understands the context.

--
Gregory A. Beamer
MVP, MCP: +I, SE, SD, DBA

Subscribe to my blog
http://feeds.feedburner.com/GregoryBeamer#

or just read it:
http://feeds.feedburner.com/GregoryBeamer

********************************************
| Think outside the box! |
********************************************
"Leon_Amirreza" <r_rahmaty@xxxxxxxxxxx> wrote in message news:uDF2JaMHJHA.1156@xxxxxxxxxxxxxxxxxxxxxxx
Hi,
I need to add thousands of rows to a Dataset:

1- I have heard that some mechanism like EnforceConstraints = false would make it Faster;
any more options on the Dataset to make it even faster?

2- the Dataset is bound to SOME BindingSources that they are also bound to many components on the form (simple and complex);
any other options on the BindingSources to make it even faster?

3- Even more options?

Thank you very much



.

Relevant Pages

  • Re: Storing images in postgreSQL (with PHP)
    ... > I am wanting to learn how to store images in a postgreSQL database. ... Some code to load a picture might be: ...
    (comp.lang.php)
  • Re: concurrency with DBI questions
    ... or update statements into a database table. ... I can only speak from a MySQL perspective, but the fastest way to load ... better ways to accomplish this. ... by dropping all the indices before a bulk load and remaking them after. ...
    (comp.lang.perl.misc)
  • Re: Import large text file to a MS Access database
    ... a MS Acccess Database, of course i don't want to use Access, but do it ... than 4 hours to add all lines/records to the database. ... For a bulk load of a database, such as you describe, it is often ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: storing PublicKey in MS Access
    ... I'm using MS Access for a database to store a public key. ... >> load the bytes later in the application where the bytes are converted ...
    (comp.lang.java.databases)
  • Re: Generating one table with a terabyte of data
    ... I didn't know we could split a database file group over multiple HDs. ... > Run perfmon counters to a log and analyze them after a bulk load. ... Another thing is to split the updates across file sets. ... >> loading of daily data in the fastest time. ...
    (microsoft.public.sqlserver.dts)