Re: Idea for ECMA/C# Standard - compile time hash for performance

Just to give an idea of scale of the problem imagine I have an object that
contains 100 fields that I do hash lookups on to get and set the values. Now
imagine I need to populate, read and process 3000-6000 of these per second.
So for populating it is 300,000 operations per second (hashing) and an
equivalent number of reads. So we'll say 600,000 operations. On a 1.8GHZ box
600,000 calls to GetHashCode is about 1/4 of a second in and of itself not
counting the lookup then and processing. So when all is said and done the
hashing in this case in and of itself accounts for 25-35% of processor time
or more depending on how many times those members need to be accessed. If
this could be done with a straight array lookup using the minimal memory
possible we could probably cut this down to 1% processor time or so. A huge
savings since we do much more than just populate and read the data, we do
calculations and comparisons so having that extra 24-34% of CPU would help us
quite a bit.

"WXS" wrote:

> The example I gave was just that an example and the reality is those enums
> can be almost anything in an integer range so sparse array really isn't an
> answer, especially allocating the whole array. And if done as a lookup tree
> hash or map requires that lookup time, which in the financial industry I work
> in with thousands of market data messages (with lots of data in each), or
> more comming in, it is becoming less and less viable by the day (although
> increasing processor speeds help mitigate a bit - when a customer is willing
> to upgrade).
>
> The key is I don't want to burn a huge amount of memory and I want to incur
> the minimal amount of lookup time possible. In this case the only way to
> handle this is with a language change I believe. Because having the compiler
> do the mapping at compile time saves me from burning huge memory and
> virtually any lookup time at run time.
>
>
> Thanks,
> Dave
>
>
> "Bruce Wood" wrote:
>
> > If your original enum type does not have values up into four digits or
> > more, why not just use an array directly, even if it's a sparse array?
> > What difference does it make, on a machine with 500 MB of memory, if
> > you waste 80 entries in an array of 100?
> >
> > It seems hardly worth a change to the language (or a hash table for
> > that matter) to save such a small amount of memory. (The hash table
> > would probably consume as much overhead as that anyway.)
> >
> >
.

Relevant Pages

  • Re: Collections of structured-data objects: what approach?
    ... you don't need 4 (lookup by key) an Array is perfect. ... then it's desirable to have an index for the array on the ... But if you are going to implement an index using a helper hash, ... Microsoft Visio MVP ...
    (comp.lang.ruby)
  • Re: Collections of structured-data objects: what approach?
    ... by doing this you can chain methods. ... you don't need 4 (lookup by key) an Array is perfect. ... For lookup, then it's desirable to have an index for the array on the key field, which can be a ruby hash. ...
    (comp.lang.ruby)
  • Re: Associative Array in C
    ... OK so my current idea for implementing this is using an array of C ... If you provided the errors in sorted order, ... do the lookup. ... I guess that a hash table will work perfectly well and be the best ...
    (comp.lang.c)
  • Re: Mutable objects which define __hash__ (was Re: Why are tuples immutable?)
    ... It means you only have to compute the hash for any given object once, ... >> of nested tuples as keys and reused them frequently for lookup. ... is entered into the dict state. ... Once the bucket is selected, a number of keys may be found there. ...
    (comp.lang.python)
  • Re: universality of codons to amino acids
    ... >> When doing bits of bioinformatic software, a hash table for codons ... >> that converts them to amino acids is actually a common theme. ... mapping the hash key to an index in a lookup table that has ... > For hash-tables see: ...
    (talk.origins)