Curve fitting & regression statistics

Open Gauge fits a polynomial calibration curve to your (reference, measured) data points, then reports a set of standard regression statistics describing how good that fit is.

The fit

Open Gauge fits:

$$\text{reference} = f(\text{measured}) = c_0 \cdot \text{measured}^n + c_1 \cdot \text{measured}^{n-1} + \dots + c_n$$

via least-squares (np.polyfit), degree 1 through 5. By default the degree is auto-selected via AIC (Akaike Information Criterion) with a parsimony rule: Open Gauge picks the lowest degree whose AIC isn't beaten by more than 2 points by a higher degree, so it won't fit unnecessary higher-order wiggle to noise. You can also pin a specific degree explicitly in the wizard.

Coefficient covariance. When there are more data points than fitted parameters, Open Gauge also computes the fitted coefficients' covariance matrix (np.polyfit(..., cov=True)), stored alongside the calibration. This matters because evaluating the curve at a point using two or more fitted coefficients together — without their covariance — understates the true uncertainty (GUM Annex H.3 shows 30%+ underestimates from dropping it). Open Gauge stores this matrix so a correct propagation is possible; there isn't yet an Open Gauge feature that evaluates a saved calibration's curve at a live reading, but the covariance is captured for when there is.

A degenerate fit — as few or fewer data points than fitted parameters (e.g. a 2-point straight-line fit) — has zero residual degrees of freedom: no covariance matrix is stored, and the fit-residual uncertainty term (see The uncertainty budget) is marked as having no finite degrees of freedom.

Residuals

For each point, the residual is:

$$\text{residual}_i = \text{reference}_i - f(\text{measured}_i)$$

R²

The coefficient of determination — how much of the variance in the reference values the fitted curve explains. 1.0 is a perfect fit.

RMSE

Root mean square error — the typical magnitude of the residuals:

$$\text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^n \text{residual}_i^2}$$

Max error

The single largest absolute residual across all data points. This is the number Open Gauge compares against a spec when deciding pass/fail — see Decision rules.

%FS error

Max error expressed as a percentage of the measurement span:

$$\%\text{FS error} = \frac{\text{max error}}{\text{range max} - \text{range min}} \times 100\%$$

Non-linearity

For a fit of degree ≥ 2: how far the fitted curve deviates from its own best-fit straight line, as a percentage of full scale. A large non-linearity is a signal that a higher-degree fit was the right call — not just noise the auto-degree-selection happened to pick up.

Repeatability

The standard deviation of measured values at repeated reference points (three or more readings at the same reference value):

$$\text{repeatability} = \sqrt{\frac{1}{m-1}\sum_{j=1}^m (x_j - \bar{x})^2}$$

where $x_1, \ldots, x_m$ are the measured values recorded at the same reference value. Requires at least 3 points at that reference to compute — Open Gauge only reports it when the data supports it.

Hysteresis

The largest spread of measured values recorded at the same reference value across an ascending and descending sweep — i.e. how far apart the "coming up to X" reading and the "coming down to X" reading are, for whichever reference value shows the biggest gap. Requires an up/down sweep in the data (the same reference value visited from both directions) to compute.

See Worked examples for a load-cell example that computes both repeatability and hysteresis from a real 12-point ascending/descending dataset by hand.