Re: 15 Significant Digits Limitation a Mistake for Spatial Information

"Matthew" <Matthew@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:98365604-FD37-43AC-BB50-2FC52C4072FB@xxxxxxxxxxxxxxxx
The IEEE Standard of limiting 15 significant digits of precision is a
practical mistake for Microsoft to follow. Especially for marking
Latitude &
Longitude coordinates for geocodes and establishing and storing boundaries
for maps at zoomed levels. Take for instance, a latitude value of
39.729831646409565 in Denver Colorado. Excel only stores it as
39.7298316464095. The longitude of -105.02543449401855 is stored as
-105.025434494018. You loose two significant digits of accuracy for this
spacial information.

Can the decision be reevaluated for practical purposes?


No.

Due to hardware limitations, computers always have used very specic formats
for numeric data.
In general, there are integer (8-bit, 16-bit, 32-bit, 64-bit, etc. fixed
types), and floating-point types.

For integer types, you have signed and unsigned types, and ones complement
or twos complement format.
Each is subject to a range of values determined by the number of bits used
to represent the data type.

For floating-point, there used to be more diifferent formats han one could
easily count.
IEEE (actually I was a participant in the early meetings 30 years ago) did
us a favor by standardiszing the format.

If you need to do any calculations that exceed the ranges of these standard
data types, then you either have to try
Decimal data type or roll your own extended precision data types.

A place to start might be "The Definitive Guide To How Computers Do Math".
ISBN 0-471-73278-8.

Or a good numerical analysis book.


.

Relevant Pages

  • Re: Formatting large numbers
    ... (approximately 14-15 significant figures), so the result is being rounded. ... Assuming these numbers are stored in a table, change the data type of the ... Graham Mandeno [Access MVP] ... > and I have played around with many different Formats, ...
    (microsoft.public.access.formscoding)
  • Re: Between dates
    ... FromDOS and ToDOS are Date data type ... What type of strings? ... Some formats ... What Excel does with type conversion is magic and not ...
    (microsoft.public.access.modulesdaovba)
  • Re: Summing number wrong????
    ... > Do you think it may have to with the data type of the field SalesTax? ... >>> tried all number formats with no luck yet. ... >> Well currency has no problems due to imprecision, ... the sum of a column of those values is almost never going to agree with the sum ...
    (microsoft.public.access.formscoding)
  • Re: Internal representation of a TimeStamp
    ... I wish IBM would give us a choice of formats. ... representation of a timestamp on the AS/400 is not the only one. ... the Z data type is an aging version of the ISO standard. ...
    (comp.sys.ibm.as400.misc)