← Back to Archive

Resolution is not accuracy

Why more decimal places do not mean a better measurement.

An instrument that displays more decimal places is reporting a finer resolution, the smallest increment it can show, and that has no necessary connection to accuracy, whether the displayed number is close to the true value, so a beautifully fine display on a poorly calibrated or badly used instrument reports a wrong answer with far more apparent confidence than a coarser one would.

Two separate properties

Resolution describes how finely an instrument divides its scale, to the nearest millimetre or the nearest thousandth of one, and it is fixed by the instrument's design regardless of the part being measured. Accuracy describes how close the displayed number is to the part's true dimension, and it depends on calibration, technique, and the scatter and bias the previous articles in this set described. A finely resolved instrument can be badly inaccurate if it was never calibrated, and a coarser one can be trustworthy within its limits if it has been checked properly. Casual talk of a "precise" instrument bundles the two together.

The clock in someone else's kitchen

A digital clock showing the time to the second looks far more precise than an analogue clock readable only to the nearest minute, and in the narrow sense of dividing time finely, it is. That says nothing about whether it is right. A digital clock that was never set properly will display a wrong time to the exact second, while the coarser analogue clock, set against a reliable source, makes a smaller claim and keeps it. Anyone who has trusted a crisp digital display in an unfamiliar kitchen and later found it fifteen minutes off has learned the lesson already. A smartphone manages both, because it keeps checking itself against network time and corrects drift before it shows, precision paired with a mechanism for staying accurate.

Digits the caliper cannot back up

Every extra digit carries an implicit claim that it is meaningful, and the claim is often false. A typical digital caliper displays to a hundredth of a millimetre, yet its maker's own accuracy figure is usually about two hundredths either way, so the last digit on the screen already sits inside the instrument's stated uncertainty before variable finger pressure, calibration drift and ordinary scatter add their share. A sensitive caliper left untouched on a stable bench shows this literally, its last digit flickering between two adjacent values with nothing about the setup changing, the display registering noise.

The display's arithmetic is honest. The process around it simply cannot support every digit it shows.

Buying for accuracy

Two calipers can show the same number of decimal places while differing enormously in the calibration, build quality and accuracy specification behind them. A specification sheet that quotes accuracy as its own figure, ideally backed by a calibration certificate traceable to a recognised standard, tells a buyer what resolution never will, and paying extra for finer resolution while ignoring that figure buys the one property that was never in doubt.

A well-calibrated, well-maintained instrument used with consistent technique can have accuracy that matches its resolution, and then every digit earns its place. Some manufacturers deliberately cap a display below what the electronics could show, a quiet admission that more digits would outrun the accuracy specification. The only way to know whether the extra digits deserve trust is to check the instrument against a known reference, and a cheap instrument recently calibrated is often more trustworthy than an expensive one nobody has checked in years, whichever shows more digits.

More on Measurement