Re: Serialization/Compression

Here are some questions.

1) Will the snapshots exists forever or only while they are being worked
on? Will you be able to delete the snapshots at some point in time?
2) How many users are expected to make snapshots at around the same time?
3) Are the snapshots only for viewing or will the user be making changes as
well?
4) If the user can make changes to a snapshot, where will those changes
persist? Will they be pushed back to the main take that the snapshot came
from?

"Bala Nagarajan" <baladotnet@xxxxxxxxxxxxxxxxx> wrote in message
news:%23G$lI92IGHA.3120@xxxxxxxxxxxxxxxxxxxxxxx
> Hello,
>
> We are developing an application (Windows forms) that allows users to take
> a snapshot of the some database tables and save them another set set of
> tables (called Model tables)and work with them . Since the data that goes
> into model tables are huge (in the order of 30000 records for each table)
> we envisaged that we are going to run out of database space if more users
> start hitting our application database. To solve this problem i suggetsed
> to split the records that go into the model tables in chunks of say 5000
> ,binary serialize the data, compress it and store the compressed form of
> the data in a blob field in the database. Of course the application will
> have to the reverse : decompress, deserialize and then render data to GUI.
> This proces does incur overhead because of the intermediate operations
> but i thought it is worth implementing since it can save us atleast 60-70%
> of space which i guess is pretty signifcant. Also the time taken to
> retrieve 6 records (instead of 30000) from the database which contains
> 30000 records in serialized format will be much efficient and faster.
>
> I just want to know if this approach is a good solution. Please let me
> know if there is a better way of resolving this issue. The downside if
> this approach is when the user modifies the data. The problems are as
> follows.
>
> 1. If the user has edited the data i will have to find out which chunk he
> has modified, serialize and compress only that portion of the data. I
> don't want to serialize all the chunks of data if the user just modifies
> only one chunk of the data. Though i can use some kind of identifier to
> identify the chunk the process may be cumbersome.
>
> 2. Even If the user just modifies one record i will have to serialize and
> compress 5000 records not matter what, which is kind of bad.
>
> I am not sure as to how to tackle these problem and will greatly
> apperciate if you help me out.
>
> Thanks a lot for the help.
>
> Bala
>
>
>
>
>
>
>


.

Relevant Pages

  • [19/99] dm snapshot: sort by chunk size to fix race
    ... Avoid a race causing corruption when snapshots of the same origin have ... different chunk sizes by sorting the internal list of snapshots by chunk ... triggers reallocation of A, then reallocation of C and links them ... there is no link with the pending exception for C it does not know to ...
    (Linux-Kernel)
  • Re: [PATCH v4 06/13] dm snapshot: merge target should not allocate new exceptions
    ... Mike Snitzer wrote: ... directly to the origin (because the chunk is not remapped or was already ... snapshot_merge_mapmust make exceptions in other snapshots. ...
    (Linux-Kernel)
  • Re: Serialization/Compression
    ... We are developing an application that allows users to take a snapshot of the some database tables and save them another set set of tables and work with them. ... To solve this problem i suggetsed to split the records that go into the model tables in chunks of say 5000,binary serialize the data, compress it and store the compressed form of the data in a blob field in the database. ... If the user has edited the data i will have to find out which chunk he has modified, serialize and compress only that portion of the data. ... I don't want to serialize all the chunks of data if the user just modifies only one chunk of the data. ...
    (microsoft.public.dotnet.general)
  • Serialization/Compression
    ... envisaged that we are going to run out of database space if more users start ... approach is when the user modifies the data. ... If the user has edited the data i will have to find out which chunk he ... serialize and compress only that portion of the data. ...
    (microsoft.public.dotnet.general)
Loading