Almost every calculator, spreadsheet and programming language stores numbers the same way: as 64-bit binary floating point, defined by the IEEE-754 standard. It's a brilliant compromise that handles numbers from 10⁻³⁰⁸ to 10³⁰⁸ in a fixed amount of memory, but it can't represent most decimal fractions exactly. Usually that doesn't matter at all. Occasionally it matters a lot.
How a double-precision number is stored
The 64 bits are split into three fields, and the value is
x = (−1)^sign × 1.fraction × 2^(exponent − 1023)
- Sign (1 bit): 0 for positive, 1 for negative. This is why there's a separate −0.
- Exponent (11 bits): stored with a bias of 1023, giving powers of 2 from 2⁻¹⁰²² to 2¹⁰²³ for normal numbers. The all-zeros and all-ones patterns are reserved for zero, tiny "subnormal" numbers, infinity and NaN (not a number).
- Fraction (52 bits): the digits after the binary point. Normal numbers always start with a 1 before the point, so that bit isn't stored, giving 53 bits of precision. That's about 15.95 decimal digits.
So any double has 15 to 17 significant decimal digits, no matter how big or small it is. The gaps between neighbouring doubles grow with the size of the number.
Why 0.1 + 0.2 isn't exactly 0.3
A fraction has a finite decimal expansion only if its denominator (in lowest terms) has no prime factors other than 2 and 5, the factors of 10. That's why ¼ = 0.25 terminates but ⅓ = 0.333… doesn't.
Binary is stricter: a fraction terminates only if its denominator is a power of 2. One-tenth has a factor of 5 in its denominator, so in binary it repeats for ever:
0.1 (decimal) = 0.000110011001100110011… (binary)
The computer has to round that to 53 significant bits, and the stored "0.1" is actually 0.1000000000000000055511151231257827… Likewise "0.2" and "0.3" are each a tiny bit off. When the rounded 0.1 and 0.2 are added, the result lands on 0.3000000000000000444…, which is about 4.4 × 10⁻¹⁷ above the true 0.3, and happens to be the next double up from the one closest to 0.3. Print all 17 digits and you see 0.30000000000000004.
The error is in the 17th significant figure. For any measurement you could actually make, that's irrelevant. It only shows up when software prints more digits than the number really has, or when you test two results for exact equality.
Type 0.1 + 0.2 in the scientific calculator. Kalkuli shows 0.3 because it displays 12 significant figures, which is enough to hide representation error but far more than any real-world input needs.
Machine epsilon and ULPs
Machine epsilon is the gap between 1 and the next larger double:
ε = 2⁻⁵² ≈ 2.220446049250313 × 10⁻¹⁶
Rounding a real number to the nearest double introduces a relative error of at most ε/2, about 1.1 × 10⁻¹⁶. Each arithmetic operation does the same, so a sensible calculation of a few hundred steps is still accurate to around 13 significant figures.
The absolute gap between neighbouring doubles, called a ULP (unit in the last place), scales with the number. Near 1 it's 2.2 × 10⁻¹⁶; near one million it's about 1.2 × 10⁻¹⁰; near 10¹⁶ it's 2, which means 10¹⁶ + 1 can't be stored at all.
The error that actually bites: catastrophic cancellation
Rounding error is harmless while it stays relative to the size of the numbers. It becomes a problem when you subtract two numbers that are almost equal. The leading digits cancel, and what's left is mostly the rounding error from before.
Take x² + 100,000,000x + 1 = 0. The textbook formula gives the smaller root as (−b + √(b² − 4ac)) / 2a. Here √(b² − 4) is so close to b that the subtraction wipes out almost every significant digit.
In double precision the textbook formula gives −7.45 × 10⁻⁹. The true root is −1.00 × 10⁻⁸. That's a 25% error from arithmetic that looks perfectly innocent.
The fix is to avoid the subtraction. Compute the large root with the sign that adds magnitudes, then use the fact that the roots multiply to c/a:
q = −½ (b + sign(b)·√(b² − 4ac)) x₁ = q / a x₂ = c / q
This gives both roots to full precision, and it's what the quadratic solver uses.
Cancellation hides in other familiar formulas too: computing variance as (Σx²)/n − x̄² for data with a large mean, or 1 − cos x for small x. The standard cures are to rearrange the formula (Kalkuli's statistics calculator subtracts the mean first, then squares) or use an identity (1 − cos x = 2 sin²(x/2)).
What Kalkuli does about it
- Displays 12 significant figures. Enough for any practical purpose, and it hides representation noise such as 0.30000000000000004.
- Exact trig values in degree mode. The angle is reduced to 0–360° first, and results within 10⁻¹⁴ of 0, ±½ or ±1 are returned exactly. So sin 180° = 0 rather than 1.2 × 10⁻¹⁶, and cos 60° = 0.5 rather than 0.5000000000000001. tan 90° is reported as undefined instead of 1.6 × 10¹⁶.
- Stable formulas in the quadratic and statistics tools, as described above.
- Exact integers for counting. nPr and nCr use whole-number arithmetic with no size limit up to n = 1000, so 59C6 = 45,057,474 exactly.
- Singularity check for matrices. A determinant smaller than 10⁻¹² times the scale of the entries is treated as zero, because an inverse computed from it would be mostly rounding error.
- Clear errors instead of NaN. Division by zero, logs of non-positive numbers, square roots of negatives and overflow beyond 1.8 × 10³⁰⁸ all give a message saying what went wrong.
So can you trust the answer?
For anything you'd type into a calculator by hand, yes. Floating-point error is at the level of 1 part in 10¹⁵, and your inputs are rarely known to better than 1 part in 10⁴. The digits you should actually report are set by your measurements, not by the computer; significant figures and rounding explains how to decide.
Where you do need care is in long automated calculations, subtractions of nearly equal quantities, and comparisons like "is this exactly zero?". Those are problems of method, not of the calculator.