|
Title: value change mystery in WinOls Post by: userpike on May 21, 2013, 12:02:28 PM I have been moving some maps over to a newer software revision and in TVUB particularly the values don't stay what I type in! They are close but not exactly what I typed in. For instance: I typed 1.5015 and go to the next value and it saves as 1.4912. WTF? Does this have something to do with the checksums being corrected or something? I'm confused.
Title: Re: value change mystery in WinOls Post by: ddillenger on May 21, 2013, 12:17:30 PM Normal. There is no exact hex match for the value you entered, it rounds it to the nearest possibility.
Title: Re: value change mystery in WinOls Post by: k0mpresd on May 21, 2013, 01:36:36 PM copy the actual hex and not the map value in decimal. see if that helps.
sometimes what i enter will round down to FE for example, instead of FF. have to actually physically change the map to FF. Title: Re: value change mystery in WinOls Post by: nyet on May 21, 2013, 02:37:02 PM This is a HUGE problem for 8-bit values with obnoxious scalars...
if i see a scalar <.01 for an 8-bit value, occasionally i will multiply it by 100 or 1000 (depending on what it is), or even different units entirely, and make a duplicate table definition for it. Title: Re: value change mystery in WinOls Post by: userpike on May 22, 2013, 12:34:28 PM This is a HUGE problem for 8-bit values with obnoxious scalars... if i see a scalar <.01 for an 8-bit value, occasionally i will multiply it by 100 or 1000 (depending on what it is), or even different units entirely, and make a duplicate table definition for it. Will you go into further detail please? What is the advantage of multiplying the scalar? To be honest, I don't even know what the scalars are there for... I've seen them, but haven't messed with them. I assume they will change what is seen on the maps but don't know why you would want to do that. Title: Re: value change mystery in WinOls Post by: nyet on May 22, 2013, 12:57:43 PM Because for some reason WinOLS does precision by "digits right of the decimal point" rather than significant figures, and for all 8 bit values, the precision should always be 2-3 sigfigs.
However, there is no way to specify "digits right of the decimal point" such that it is correct for all scalars. I don't really feel like going into an explanation of sig figs vs. decimal points.. science 101 and all.. Title: Re: value change mystery in WinOls Post by: Axis on May 22, 2013, 02:04:39 PM Userpike: Read this
http://mathforum.org/library/drmath/view/58335.html Title: Re: value change mystery in WinOls Post by: Acki on May 22, 2013, 03:19:46 PM Sounds like the very old excel bug.
Incorrect calc of floating point. Without using x.xx you have integer and not the problem. Title: Re: value change mystery in WinOls Post by: userpike on May 22, 2013, 10:25:02 PM Because for some reason WinOLS does precision by "digits right of the decimal point" rather than significant figures, and for all 8 bit values, the precision should always be 2-3 sigfigs. However, there is no way to specify "digits right of the decimal point" such that it is correct for all scalars. I don't really feel like going into an explanation of sig figs vs. decimal points.. science 101 and all.. I expect they did it like that because it's more accurate, but I think I get the problem in doing this with the 8 bit values. Anything with higher than 2-3 sig figs would take up too much space being that 8 bits is the max for each value so the value gets rounded to something that will fit? Thanks for the link Axis, it was a good refresher. |