Re: Serialization/Compression


I wrote an smart client multiuser application that handles much less data, but it does serialize individual data on the client as XML.

I find that a more balanced approach, architecturally.

Bala Nagarajan wrote: 
Hello,

We are developing an application (Windows forms) that allows users to take a snapshot of the some database tables and save them another set set of tables (called Model tables)and work with them . Since the data that goes into model tables are huge (in the order of 30000 records for each table) we envisaged that we are going to run out of database space if more users start hitting our application database. To solve this problem i suggetsed to split the records that go into the model tables in chunks of say 5000 ,binary serialize the data, compress it and store the compressed form of the data in a blob field in the database. Of course the application will have to the reverse : decompress, deserialize and then render data to GUI. This proces does incur overhead because of the intermediate operations but i thought it is worth implementing since it can save us atleast 60-70% of space which i guess is pretty signifcant. Also the time taken to retrieve 6 records (instead of 30000) from the database which contains 30000 records in serialized format will be much efficient and faster.

I just want to know if this approach is a good solution. Please let me know if there is a better way of resolving this issue. The downside if this approach is when the user modifies the data. The problems are as follows.

1. If the user has edited the data i will have to find out which chunk he has modified, serialize and compress only that portion of the data. I don't want to serialize all the chunks of data if the user just modifies only one chunk of the data. Though i can use some kind of identifier to identify the chunk the process may be cumbersome.

2. Even If the user just modifies one record i will have to serialize and compress 5000 records not matter what, which is kind of bad.

I am not sure as to how to tackle these problem and will greatly apperciate if you help me out.

Thanks a lot for the help.

Bala







.

Relevant Pages

  • Re: Self-generating Number
    ... I built a database to track reprocessing of parts that were fixed ... We are required by instruction to serialize these parts ...
    (microsoft.public.access.queries)
  • Re: [PHP] Writing my own web based user forum
    ... Since it seems you are using a database there is no need to serialize. ... In your original post you did not mention a "picture upload site" so I ... I am looking at writing my own web based user forum somewhat for my own ...
    (php.general)
  • Re: Whats a good database for a really simple app?
    ... you could also create the stuff as a object database and just serialize ... > if it's on disk at startup, create it if not, and serialize it as the ... > anyone needing even a two user app, but for a single user app it ...
    (microsoft.public.dotnet.languages.vb)
  • Serialization/Compression
    ... envisaged that we are going to run out of database space if more users start ... approach is when the user modifies the data. ... If the user has edited the data i will have to find out which chunk he ... serialize and compress only that portion of the data. ...
    (microsoft.public.dotnet.general)
  • Re: Serialization/Compression
    ... Will the snapshots exists forever or only while they are being worked ... > this approach is when the user modifies the data. ... If the user has edited the data i will have to find out which chunk he ... serialize and compress only that portion of the data. ...
    (microsoft.public.dotnet.general)