Re: Singles to Doubles


"Karl E. Peterson" <karl@xxxxxxxx> wrote in message
news:uldbmN%23xHHA.4652@xxxxxxxxxxxxxxxxxxxxxxx
Hi Bob --

Which gives me these results:

0.3026 0.302599996328354 0.3026
True True
0 -3.67164609826887E-09

And, of course, the value I'm after *is* 0.3026 -- not the one that's off
0.00000000367... Seems like some sort of register garbage that I'm picking up
in the conversion?


The one time science/math teacher in me has to beg to differ, on grounds of
significant digits.

If you have some data, such as a measurement, recorded as 0.3026, what you have
(what you know, that is), is really more like 0.30255 < x < 0.30265, recorded
to 4 significant figures. You don't have *exactly* 0.3026 at all.

In fact, .30259999 is just as likely correct as 0.30260000.

The various statements about accuracy, precision, and exactness are really
irrelevant. What is the data about, and how precisely was it measured?

If you are averaging it, or taking other such statistics, the tiny variations
are irrelevant. If you are totalling them, your result contains an error of
..0001 * n anyway, so fussing over micro-decimals won't matter at all.


.

Relevant Pages

  • Re: Initializing constants
    ... then the number of significant digits could be predetermined. ... device with greater precision were used, ... require adjustment. ... I'm pretty sure the measurement 1.053 would have ...
    (comp.lang.c)
  • Re: frequency resolution issue
    ... resolution is in 0.01Hz? ... Precision is about the variation in the measured or calculated quantities ... measurement of the same thing from measurement to measurement. ... If you applied the same 4 measures to another data set and the resulting ...
    (comp.dsp)
  • Re: Now the UK Has its own date in the history of al quiada attacks
    ... That will affect the measurement. ... > to the precision indicated. ... though I suspect that's not a big worry in this case. ... The mathematician said, "If a beautiful woman was standing across the room, ...
    (comp.os.vms)
  • Re: Measuring Our Absolute Velocity
    ... The L/R and lower detectors feed into a coincidence detector. ... What exactly do the switches do? ... What is the measurement precision required for the lightpaths? ...
    (sci.physics)
  • Re: Whats a boy to do?
    ... its a matter of measurement precision. ... the same distance, especially in the theoretical "math puzzle" sense, ... The probability of two darts ...
    (rec.outdoors.fishing.fly)