Rounding, and if your numbers don't match

Rounding — what you'll actually see on screen

The raw numbers in each worked example are what Open Gauge's calculation engine produces internally (and what's stored in the database, already rounded server-side to 8 decimal places — not the same thing as the display rule below). Two different rounding rules apply on top of that in the UI:

1. Combined and expanded uncertainty are rounded to at most 2 significant figures for display, per GUM §7.2.6 — in the wizard, the historical calibration view, and the certificate PDF. This is the "shown rounded: …" value in each worked example.
2. Every other stat (R², RMSE, max error, hysteresis, repeatability, ...) goes through a general-purpose fixed-decimal formatter that strips trailing zeros. The wizard's live analysis defaults to 6 decimals; the historical-calibration view on the asset page defaults to 4 — so the same stored value can display with one fewer decimal digit once a calibration is saved and you're looking at its history entry versus its live analysis. Either way, 0.99999993 rounds to 1 at both 6 and 4 decimals, and 0.43588989 shows as 0.43589 in the wizard but 0.4359 in the history view. Seeing R² display as exactly 1 for a near-perfect fit is this formatter rounding up, not a bug — the stored value is still 0.99999993, not exactly 1.

The "Expected results" table in each worked example gives the full raw value throughout (for verifying the underlying math) and calls out the rounded value only where a rule actually changes what's displayed (combined/expanded uncertainty).

If your numbers don't match

• Off by a unit-conversion-sized factor (10, 100, 1000...) — check that the channel's Accuracy unit matches the reference/measured unit you used when entering points. Open Gauge does not currently validate that these agree.
• "% FS" doesn't appear in the Accuracy/Resolution/Uncertainty unit dropdown — it only shows up once both Range min and Range max are filled in on that channel; see The "% FS" convention.
• "Reference standard" row missing (Example 1.2) — confirm Asset A's calibration (1.1) was actually saved, not just analyzed, before starting Asset B's calibration, and that Asset A's channel has Calibration role checked (the reference-asset picker only lists assets with at least one reference-role channel).
• Different polynomial degree (Example 2) — if you get a different degree than 2, try re-entering the points exactly as listed; small transcription errors shift which degree AIC prefers. Polynomial fitting is sensitive to exact input values in a way the other checks aren't.
• No CONFORMS/DOES NOT CONFORM badge (Example 4) — this is expected, not a bug. A coefficients-only calibration has no raw data to compute an assessed error from, so there's nothing to compare against a spec. Use a regular (non-coefficients-only) calibration if you want a conformity statement.