Archive. This is an archive of the Apple Support Community. Discussions are being restored and soon you will be able to read and post here again.

Roundoff error in iPhone Calculator app when using x^2 function

I was helping my granddaughter with calculus homework. She evaluated 40*t - 16*t^2 at t=2.5 using the calculator on her iPhone (despite me asking her to use fractions for simplicity and accuracy):

40*2.5 - 16*2.5^2

To my surprise, there was a roundoff error as a result: 2e-32 instead of zero.

I realize that subtraction of nearly-equal numbers is the best way to get round-off errors, but I'm surprised that the answer was not 0.


Using the following on the same calculator, gives the proper answer:

40*2.5 - 16*2.5*2.5


Is this a problem with the x^2 function being used by Apple being different precision than the rest of the numerics?


Whatever the answer to the above, I don't think it's normal in programming languages, since the usual knock against x^2 is that it is slower than x*x, not that it's somehow less precise.


FWIW, all of the following yield 0 as the result: the calculator app on macOS, whether in reverse polish mode or not, the freeware calculator app on my iPhone (cmpxRPN), Stata (a statistics package) returns 0 whether using quad precision or floating-point (24-bit) precision.


Thanks for any insight.

0 replies

There are no replies.