RE: scaling out data access
- From: "Kerry Moorman" <KerryMoorman@xxxxxxxxxxxxxxxxxxxxxxxxx>
- Date: Mon, 19 Sep 2005 07:12:05 -0700
tin,
Trying to use a dataset as an in-memory database with hundreds of thousasnds
of rows in some of the tables is a mistake.
The first and most important thing you can do for data access scalability is
to be sure you are opening connections as late as possible and closing them
as soon as possible. Then let .Net's connection pooling manage the
connections.
Kerry Moorman
"tin" wrote:
> requirment: scaling out data access to increase overall system capacity while
> decreasing individual request execution time, without server clusters <-- is
> that possible??
>
> a bit trade off in idv. request execution time is acceptable if the whole
> system would scale. expensive server cluster is out of the equation. i can
> think of 2 solutions:
>
> #1 - data replication (transactional model), 2 concerns:
>
> a) all testing i have done, shows that there's at least 10 seconds delay in
> replicating the data? would sql 2005 improves the latency?
>
> b) the data is still accessed from hard disk, so it doesn't really improve
> performance? i know in linux/unix, you can mount the memory to a drive, and
> use the drive for data access?? is there something like this in windows
> 2003??
>
> #2 - use ado.net (most likely 2.0) to load all the data from the database to
> a dataset, this way data can be accessed from the memory, and i can load the
> data to multiple servers to balance load? a few concerns:
>
> a) is ado.net 2.0 designed for heavy massive concurrent read/write?
>
> b) would the DataTable.Select() really selects faster than the SQL Select
> against the database?? i have half a million business records in a table,
> let's say to match a business name? i heard in 2.0, the indexing engine is
> improved? does it also benefits the datatable.select method?
>
> c) i know i need to handle synchonizing the data between the different
> servers holding the dataset, and the sql database itself, is there anything
> in ado.net 2.0 and sql 2005 that would help, or is there anything i should
> pay more attention to? i guess the query notification and ado.net row change
> events would help?
>
> d) this is a big problem. the dataset would be loading half a million
> business data, which is a few hundreds MB big. and the dataset is going to
> be hosted in an asp.net web application/service. that means any file change
> in the appl, would cause a recompile/restart, and the dataset needs to be
> loaded again, is there a way to avoid that??
.
- Follow-Ups:
- RE: scaling out data access
- From: tin
- RE: scaling out data access
- References:
- scaling out data access
- From: tin
- scaling out data access
- Prev by Date: Re: How can I sort a datetime column in a dataset.
- Next by Date: Re: Object Relational Mapper with deep copy ability?
- Previous by thread: scaling out data access
- Next by thread: RE: scaling out data access
- Index(es):
Relevant Pages
|
