what is the best way to store this data

From: D (Dave_at_nothing.net)
Date: 02/20/05 

Date: Sun, 20 Feb 2005 12:12:29 -0500

I have about a thousand variable length arrays of numeric data that I need
to store, some can be as small as 14 numbers others can be as large as 1500
numbers. In your opinion, what would be the best way to do that?

I'm going to read them in as I need them and save them occasionally as they
don't change much.

I have thought about

1. sql table - Would that be a waste of space for the records that may have
only 14 numbers, to have 1486 empty columns? what is the max columns for a
table in sql btw?

2. xml files - Not sure if I want 1000 xml files around

3. I could store them vertically in a table, the design would look like

    long lGroupIndex - id's which data group the row belongs too
    long lDataIndex - index within the group
    long lData - actual data value.

 if I did this and a worse case scenario of 1500 arrays of 1500 number would
produce 2250000 , that wouldn't be a problem with sql would it ?

Feel free to offer your suggestions.

Thanks

Relevant Pages

  • RE: adding data to a file before it gets regexed
    ... The reasion I am trying to add the string to the line, is so that I can use ... it is not a good idea to store entire files in memory and it is really ... the arrays are the contents of the files broken down by lines. ... Any opinion expressed in this e-mail is personal to the sender ...
    (perl.beginners)
  • Re: writing at the end of the lines
    ... > Now due to the size of the arrays and the large number of time steps ... Memory tends to be ... the cost of programmer time. ... And if you can't store the whole file, ...
    (comp.lang.fortran)
  • Re: understanding data types
    ... what data types are supported by MatLab and its arrays. ... It seems that MatLab ... I am trying to store data from SQL Server tables into a MatLab array ...
    (comp.soft-sys.matlab)
  • Re: Cells and saving as ASCII
    ... i used this method to save text and numeric data to a ascii tab delimited file. ... %store as ASCII-format ... idx = find ... The result file must be in ASCII-format. ...
    (comp.soft-sys.matlab)
  • Re: Xml Data Storage
    ... If the number is quite large then you may want to cache portions in memory, ... If the number is small then just store the contents in memory. ... I need to use theese informations in my ping method. ... arrays to store IP Adresses and SiteNames.The arrays was created at runtime just when the ...
    (microsoft.public.dotnet.languages.csharp)