Archive. This is an archive of the Apple Support Community. Discussions are being restored and soon you will be able to read and post here again.

How to limit precision in the Mac Calculator app?

How can I limit the precision of answers on the Mac Calculator (OS 26.7.2)?

22 replies

Then why didn't you say this:


"In the Apple Calculator app on Mac, choose View > Decimal Places in the menu bar, then pick how many digits to show after the decimal point."



...and now the additional complexity:


the articles indicate that displayed numbers will be rounded, but only for display purposes. The underlying calculations are still done with the un-rounded values.


This could cause your calculations to drift in an unexpected way.

As I see it, the calculator uses all available digits, unless the user has limited the display via the "Decimal Places" submenu.

If I wanted to increase precision, I'd simply enter more digits in the entry values, although the calculator apparently ignores zeros at the right end of data values. 1.5000 * 1.5000 returns 2.25, when I've specified 4 digits past the decimal point.

...


Round results or increase precision


When you round your results, Calculator stores the full value and displays the rounded value.


...


Note: Use caution when rounding, as your results may not be as precise as they should be. For example, the calculator returns 10 (rather than 9.9) when you enter “99 ÷ 10 =” with decimal places set to zero.


https://support.apple.com/guide/calculator/use-the-basic-calculator-calc2c05fe23/mac



DickWarner wrote:
As I see it, the calculator uses all available digits, unless the user has limited the display via the "Decimal Places" submenu.
If I wanted to increase precision, I'd simply enter more digits in the entry values, although the calculator apparently ignores zeros at the right end of data values. 1.5000 * 1.5000 returns 2.25, when I've specified 4 digits past the decimal point.

No. The calculator app uses floating point math. Any "precision" or "digits" that you see are simply eye candy. They don't really exist in the calculations.


And this is true of most such consumer apps. I mentioned Excel, but Excel does the same thing. It's just that in Excel you can add your own column and do the math correctly. And because it's 2026, there's always a chance that a recent release of an app is going to have some critical bug involving some decades-old operation that was changed and never tested. So let all those people addicted to updates test this software for you.


I don't use Calculator for important calculations. But I have seen this kind of error in my financial spreadsheets, especially with currency conversions.

Using View in calculator does not limit precision!


It only adds Rounding at the level you specify, (not necessarily at the level you need).


In the case of money calculations where you enforce rounding to whole cents, the calculations proceed at the maximum resolution of the computer, fractional cents may occasionally break the wrong way, and a column of numbers calculated (but presented as whole cents) may not appear to add up correctly. ( Off by cents).


Why would you want to Limit Precision?


What is it exactly that you want to accomplish?


Doing numerical calculations with a computer (or calculator) is a subject that has quite a bit of complexity to it, so there may some surprising answers.

DickWarner wrote:
How can I limit the precision of answers on the Mac Calculator (OS 26.7.2)?

Don't. If you're working with values that require more precision (in various respects) than what the built-in calculator provides, then you're better off doing it yourself in (and older version of) Excel.


The calculator and most ordinary computer-based math does not use decimal digits at all.


Instead, most numbers you enter are converted and calculated in binary floating point notation, something similar to scientific notation, except it's using a powers of 2 rather than a powers of 10.


$67.13 is recorded and calculated as something more like 0.6713 e10, except the radix used is a power of 2, not 10. {where e means 'times ten to the power'}


After many calculations, especially multiplications and divisions, more inaccuracy creeps in, and if it were not for Rounding, the answers would drift to numbers that were obviously WRONG. Some answers computed as powers of 2 do not convert easily to powers of 10, and may no longer be precise integer values.


in most software, this is generally done regardless of whether the numbers involved could easily be converted and calculated as Integer values, with no loss of precision, but magnitudes might be limited to on the order of 32 bits, or about 4 billion, depending on implementation.

I created an Excel spreadsheet to help a friend appropriately distribute the charges to the several offices in her building. The electric company uses a single meter and delivers a single bill, yet there are separate meters for each office. She uses that spreadsheet, but likes to double-check using the calculator.


FWIW, I've been using floating-point math (Scientific Notation) for about 68 years!

"Any "precision" or "digits" that you see are simply eye candy. They don't really exist in the calculations." is false! Even if the actual calculations are done in binary math, the conversions from decimal to binary and back should make use of all the data the user enters.


As noted early in this discussion, internal calculations do indeed carry many digits of precision. (But not necessarily the number of digits seen in the user presentation.)


After calculations, some values calculated may no longer be exact integer values, and if it were not for ROUNDING before user presentation, the inaccuracies generated can sometimes make 'correct' answers appear to be wrong. For example, Rounding to the wrong decimal place and make answers wrong.


Calculating with computer math has a lot of nuances to it.

Thanks for taking the time to reply!


...but I can't quite agree with: "The calculator and most ordinary computer-based math does not use decimal digits at all." The computer may well convert the decimal input into binary before doing any calculations, and do the calculations in binary, but it MUST use the user's input as a starting point.


It's probably been about 30 years since I forgot how to deal with decimal fractions in binary.


Thanks to all who replied. I got the needed answer; no further answers required.



Grant Bennet-Alder wrote:
$67.13 is recorded and calculated as something more like 0.6713 e10, except the radix used is a power of 2, not 10. {where e means 'times ten to the power'}

Hey! You changed the numbers! It was $56.12. I'm not re-doing my calculations.


Anyway, it's worse than that. $56.12 cannot be represented in either single or double-precision arithmetic. Same with $67.13.


After many calculations, especially multiplications and divisions, more inaccuracy creeps in, and if it were not for Rounding, the answers would drift to numbers that were obviously WRONG.

Again, it's worse than that. If one is purely dealing with floating-point data, then 52 bits of precision is highly accurate. In most cases, that's well within the expected inaccuracy of a sensor or wherever the data is originally coming from.


The problem is government forms and financial statements. People expect those to add up. But unless floating point data is properly handled, then this inaccuracy can easily show up with only 2 decimal places of precision. I've seen it myself.

DickWarner wrote:
I created an Excel spreadsheet to help a friend appropriately distribute the charges to the several offices in her building. The electric company uses a single meter and delivers a single bill, yet there are separate meters for each office. She uses that spreadsheet, but likes to double-check using the calculator.

Yes. That's exactly where you'll see the problem. Introducing any non-currency numbers requires careful consideration, and often manual rounding.


FWIW, I've been using floating-point math (Scientific Notation) for about 68 years!
"Any "precision" or "digits" that you see are simply eye candy. They don't really exist in the calculations." is false!

It is true.


Your example wasn't a great one for showing the problem. "1.5000" is treated no differently than "1.5" internally. Unfortunately, .5 is a power of 2. But as Grant Bennet-Adler pointed out, it does get complicated. Because the computer doesn't use either 1.5000 or 1.5. The computer uses 0.15. And it doesn't even do that, as the first bit is one and can be treated as a special case with the exponent to get a little bit more precision. Alas, I forget the details. It's been a long time since I dug into floating point arithmetic.


Usually people never notice these things because accountants do it all. And it's not that accountants do it correctly, they don't. Their acceptable error margin is way, way higher than you would like.

Older computers had extensive support for decimal integer math. The IBM 360 and 370 had extensive decimal (read binary-coded decimal digits) machine instructions for direct decimal arithmetic -- NO conversion to binary at all.


As floating point became more common and faster, the default for computer arithmetic became floating point binary. if you use floating point binary, you often get numbers like .6712499999999 e 10 when you expected 67.13.


If it were not for extensive (behind your back) Rounding, default computer math using floating point binary would be unusable.


And depending on exactly what you are doing, you may need to know that Rounding is taking place to fully trust your calculations.

Grant Bennet-Alder wrote:
Using View in calculator does not limit precision!

Where did I say it does? I was just pointing out that in certain types of financial calculations, namely, the exact financial calculations the OP is asking about, floating point problems aren't merely an academic exercise. It can show up in the dollars and sense, literally.


The OP has lost interest in this thread, as I have too.

With decimal places set to zero, 10 is indeed the correct answer to "99 ÷ 10 =".

If the user has intentionally/knowingly set the number of decimal places (more correctly the number of digits or places after the decimal point) to zero, then 10 is as precise as it should be.


Of course there is more than one method of rounding, each considered correct in a given field.

for example, I've noticed that Excel uses a different rounding algorithm than my CAD program.

That way of thinking usually works fine as long as all you are doing is additional and subtraction.


When you use multiplication and especially division, hiding the underlying precision and depending on rounding could introduce small errors that could be larger than expected.


I am not trying to start a fight, just to make sure you understand that there could be small errors introduced, and WHY.


My computer architecture teacher used to say this about floating point numbers:

"Working with floating point number representation is like moving a pile of sand. Every time you manipulate it, you lose a little sand and pick up a little DIRT."

FWIW, I took his request to be:


“How to set / limit the number of digits displayed to the ‘right’ of the decimal point”

Because I did not have access to my Macs to test it myself.


Pointing the OP to the Apple Guides seemed the next best available answer.



Thanks! I have to confess that I hadn't even noticed that the Calculator had menus!

I'm helping a friend who is working with $, so wants to limit answers to 2 digits after the decimal point.

Correct.