Re: Spltting large datasets for parallel processing
- From: "William \(Bill\) Vaughn" <billvaRemoveThis@xxxxxxxxxx>
- Date: Thu, 20 Apr 2006 09:27:39 -0700
I would investigate Notification services or SQL Dependency.
--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
INETA Speaker
www.betav.com/blog/billva
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________
"Andy Furnival" <AndyFurnival@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:D9BD77F1-FB08-4E0E-AEBF-DAB95266EB71@xxxxxxxxxxxxxxxx
Hi people,
I have a problem, where I'm trying to find a fast effective means of
continually polling a database for messages to process which are grouped
based on content and then processing these groups in multiple threads and
avoid any waiting.
The processing is essentially, pick up all messages from the database,
split
into batches and post via http call to and end service that will deal with
the messages. Each post will return a unique result code, which I will
then
store against all messages in that batch for tracking purposes.
The recorsets returned from the database at anyone time can be from 1 to
1,000,000 messages and processing at a maximum in batches of 250.
My problem is that I dont want to update a recordset everytime I load new
data to give it a processing state so it will not be loaded more than
once,
I've tried this before and can be a lengthy process, also if my service
terminated I have then lost which actual messages have been processed or
not.
Thanks for any help
.
- Prev by Date: Re: Create my own db connection wizard
- Next by Date: Re: Updating two table through data adapter.
- Previous by thread: Streaming SQL Data
- Next by thread: Re: Spltting large datasets for parallel processing
- Index(es):
Relevant Pages
|
|
