What the Sensor Calibration Calculator does
This calculator turns a set of calibration points into the numbers a calibration record needs: zero and span error from two points, a least-squares correction from more, and accuracy, linearity, hysteresis and repeatability expressed as percentages of span. It also applies the correction to any raw reading so you can check the result before writing it into the transmitter.
Every percentage here is of span, not of reading. A 0.5% figure on a 0-200 C range means 1 C everywhere, whereas 0.5% of reading would mean 0.1 C at 20 C and 1 C at 200 C. Calibration certificates use span, and mixing the two is how a sensor passes on paper and fails in the field.
It runs entirely in your browser. Calibration data is a site record and never leaves the tab.
How to use it
- Enter the calibration points, one per line, as reference then measured: the reference is what the standard says, the measured value is what the instrument showed.
- Add a third field, up or down, when you have taken the point on the way up the range and again on the way down. That is what makes a hysteresis figure possible; without both directions the page says so rather than inventing one.
- Repeat a point in the same direction to get a repeatability figure - three readings at mid-scale is the usual check.
- Set the configured span low and high. These default to the extremes of your reference points, but a transmitter's span is its configured range, and using the tested points instead flatters every percentage.
- Enter an acceptance tolerance as a percentage of span to get a pass or fail, and type a raw reading in the correction box to see what the fitted correction does to it.
Reading the results
Accuracy is the largest deviation from the reference, as a percentage of span. It is the number that decides pass or fail.
Linearity is the largest departure from the best-fit straight line. A sensor that reads exactly one unit high everywhere has poor accuracy and perfect linearity - and can be fixed entirely by a zero adjustment, which is why the two figures are kept apart.
Hysteresis is the gap between the up-scale and down-scale readings at the same point. It usually comes from mechanical friction or a diaphragm that does not return, and no amount of zero and span adjustment will remove it.
Repeatability is the spread of repeated readings at one point in one direction. It sets the floor: no correction can make an instrument more consistent than its own repeatability.
r-squared below about 0.999 means a straight line is the wrong model. Zero and span will then correct the ends and leave error in the middle, which is exactly where most processes run.
Worked example: a five-point check on a 0-100 C transmitter, up and down
Up-scale readings at 0, 25, 50, 75 and 100 C are 0.5, 25.4, 50.2, 75.1 and 100.1. Down-scale at 75, 50, 25 and 0 are 75.6, 50.8, 25.7 and 0.6.
The span is 100 C, so a deviation of 0.8 C is 0.8% of span. The worst deviation is the down-scale point at 50 C, reading 50.8: accuracy is 0.8% of span, which fails a 0.5% tolerance and passes a 1% one.
Hysteresis is the up-down gap at the same reference. At 50 C it is 50.8 - 50.2 = 0.6 C, or 0.6% of span, and it is the largest of the three matched points. Since hysteresis cannot be adjusted out, an instrument with 0.6% hysteresis can never be better than about 0.3% however carefully it is set up.
Zero error, from the lowest point, is +0.5 C or 0.5% of span. The span error from the two extremes is (100.1 - 0.5) - (100 - 0) = -0.4 C. The least-squares correction over all nine points comes out close to corrected = 1.0 x measured - 0.55, which is mostly an offset - confirming that this instrument is straight but shifted.
Adjust the zero first, then the span, then re-check the zero: on most transmitters a span adjustment moves the zero as well, so a single pass leaves the bottom of the range out.
Formulas and scoring rules
- Two-point gain and offset
gain = (refHigh - refLow) / (measHigh - measLow); offset = refLow - gain x measLowcorrected = gain x measured + offset.- Zero and span error
zeroError = measLow - refLow; spanError = (measHigh - measLow) - (refHigh - refLow)- Least squares
slope = Sxy / Sxx; intercept = mean(reference) - slope x mean(measured)Fitted with the measured value as x, so the result is directly usable as a correction. r2 = 1 - SSres / SStot.- Accuracy
max |measured - reference| / span- Linearity (independent)
max |measured - value the best-fit line predicts| / span- Hysteresis
max |mean(down readings) - mean(up readings)| at the same reference / span- Repeatability
max (highest - lowest) of repeated readings at the same reference and direction / span
As found and as left
A calibration record has two halves. As-found is what the instrument read before anybody touched it: that is the figure that tells you whether the product made while it drifted was in specification, and it is the only one that matters when tracing a quality problem backwards. As-left is what it read after adjustment, and it is the figure that predicts the next interval.
Run this page twice, once on each set of points, and keep both. An as-left record on its own hides exactly the information a deviation investigation needs. ISO/IEC 17025 expects the as-found condition to be reported when an instrument arrives out of tolerance, for the same reason.
Why the order of adjustment matters
On most analogue transmitters, and on many digital ones with an analogue output stage, the zero and span adjustments interact. Moving the span pivots the whole line about the bottom of the range, which shifts the zero; moving the zero translates the line, which leaves the span alone. So the field order is zero, span, then zero again - and on a badly drifted instrument, one more pass after that.
A least-squares correction applied in software has no such problem, because both coefficients are calculated at once. If the instrument accepts a gain and an offset directly, use the fitted numbers from this page rather than iterating with a screwdriver.
Limitations: what the result does not prove
- It calculates from the points you give it. It cannot know whether your reference standard is traceable, in calibration, or better than the instrument under test - and a reference no better than the unit is not a calibration at all.
- It reports no uncertainty budget. A real certificate states an expanded uncertainty with a coverage factor; the figures here are deviations, not uncertainties, and the guidance for combining them is NIST Technical Note 1297.
- Hysteresis and repeatability are only reported when the data supports them: both directions for hysteresis, repeated points for repeatability. They are never estimated from a single pass.
- A pass here is arithmetic against the tolerance you typed. It is not a calibration certificate, and it does not make anyone competent to issue one.
Privacy: where your data goes
Everything you paste, type or drop is processed in this browser tab. It is not uploaded, logged, stored or sent to analytics. Session recording and tag-manager scripts are switched off on this page.
Standards and sources
- ISO/IEC 17025 - Requirements for testing and calibration laboratories - checked 19 Sep 2026
- NIST Technical Note 1297 - Evaluating and expressing measurement uncertainty - checked 19 Sep 2026
- JCGM 100:2008 - Guide to the expression of uncertainty in measurement (GUM)
Frequently asked questions
How do I calculate zero and span error?
Zero error is the measured value minus the reference at the bottom of the range. Span error is the width of the measured range minus the width of the reference range: (measHigh - measLow) - (refHigh - refLow). Both are usually quoted as a percentage of span.
What is the difference between accuracy and linearity?
Accuracy is the worst distance from the reference; linearity is the worst distance from the best-fit straight line through the same points. An instrument reading exactly 1 unit high everywhere has 1 unit of accuracy error and zero linearity error, and a zero adjustment fixes it completely.
Why is error expressed as a percentage of span rather than of reading?
Because most instrument error sources are fixed fractions of the full range rather than of the value being measured. Percent of span gives one number that applies everywhere; percent of reading gives an error that shrinks towards zero, which is flattering and usually untrue near the bottom of the range.
How many calibration points do I need?
Two give you zero and span but cannot show curvature at all. Five points at 0, 25, 50, 75 and 100% of span, taken up and down, is the common industrial practice: it shows non-linearity and hysteresis while still fitting in a reasonable time on site.
Can hysteresis be adjusted out?
No. Zero and span move the line; hysteresis is the difference between two lines, so adjusting either one simply splits the error. Mechanical hysteresis in a diaphragm, bourdon tube or linkage is repaired or replaced, not calibrated away.
What does r-squared tell me about a sensor?
How well a straight line describes it. Above about 0.9999 a two-point correction will work everywhere. Below 0.999 the sensor is visibly curved and a linear correction leaves error in the middle of the range, so either use a multi-point correction in the instrument or accept the curvature in the uncertainty budget.
Should I use the tested points or the configured range as the span?
The configured range. If a transmitter is set 0-200 C and you tested 0-100 C, the same 1 C deviation is 0.5% of the real span but 1% of the tested one. Using the tested points halves every figure and makes a marginal instrument look comfortable.
Is this a calibration certificate?
No. It is arithmetic on the numbers you typed. A certificate needs a traceable reference standard, a stated uncertainty, environmental conditions, the as-found condition and a competent signatory - none of which a web page can supply.
Last reviewed by the A2Z.Tools team against the sources listed above.
