precision of double data

Jul 23, 2013 at 8:55 PM
Hi Keith,

it would be very kind if you can comment on this issue that i observe. say i have in the excel cell numbe like .34567. When i read the number in i observe in the c++ debugger the number has changed to .34566999999999999. This i guess is a precision issue between excell and c++ but is there some flag which can be switched on/off to automatically handle the issue(which is read the data as is maybe padding with zeros at the end).

Best
Coordinator
Jul 23, 2013 at 9:01 PM
This is due to how floating point numbers are represented. See https://xllfloat.codeplex.com for the gory details.
In particular https://xllfloat.codeplex.com/SourceControl/latest#trunk/float.cpp.