Re: large data in memory

Hello, bp,

Re:
> Or do people code stuff like this all the time?

Yes (unfortunately). ;-)

But "Large" is a relative term, and it's a difficult to be definitive without knowing precisely what you are trying to do. It sounds like you will be making inefficient use of a significant amount of space. But there are circumstances where this could still be a reasonable thing to do. Questions like:

How often will the application be used?
How quickly must the data be accessed?
How much time can we devote to development?

can all influence this. For example, for a "little application" that will only be used once, the approach mentioned might be quite reasonable. (However, watch out for those applications that will only be used "once". They have a tendency to get reused, recycled, enhanced, and expanded until they become monsters.)

If you need to, you can find several ways to improve efficiency. For example (depending on the type of "database" you are using) you may find that you can access only the records actually required with a single query by using an "inner join". If you are using only a few elements from a very large array, a "hash table" might provide a better storage solution.

Cheers,
Randy


bp wrote:
Hello,

We are building a little application to do a large amount of data manipulation. We will query from a database a large amount of rows. From that will pull an id out. From each id we will query from another table more information. We were thinking of dimming an array when we know how many ids we have. Then for the array have an associated dictionary for each id.

We might end up with 10,000 to 100,000 array entries. And the associated dictionary that would contain a few values. The id is only a 27 char string. And the other values are strings only 10 to 20 chars.

Does this seem like a large amount of data to keep in storage? To manipulate?

Or do people code stuff like this all the time?



Thanks for any help




.

Relevant Pages

  • large data in memory
    ... manipulation. ... We will query from a database a large amount of rows. ... Then for the array have an associated dictionary for each ...
    (microsoft.public.vb.general.discussion)
  • Re: Copying a table to an array
    ... You probably don't need a query, ... Dim rstRSet As Recordset ... table to enable data from the latter to be read into an array with GetRows. ... exercise has therefore to be repeated prior to each interrogation. ...
    (microsoft.public.access.formscoding)
  • Re: Mix/Max/Avg Help based on dynamic ranges
    ... Did you enter Biff's formula as an array? ... The query I gave you is dependent on your breaking out the ranges as Biff ... Where Nums are the values to avg based on the head count range. ... deemed optimal for building device profiles based on locn size and device ...
    (microsoft.public.excel.worksheet.functions)
  • Re: Copying a table to an array
    ... "Peter Hibbs" wrote: ... You probably don't need a query, ... Dim rstRSet As Recordset ... table to enable data from the latter to be read into an array with GetRows. ...
    (microsoft.public.access.formscoding)
  • Re: Imitating Mail Merge on server
    ... given moment we know which data (in this case, customers) is being worked ... and concantenate each query together. ... loop the array, plug in the current recordset value. ... We have close to fifty tokens that we use in the application correspondence, ...
    (microsoft.public.inetserver.asp.general)