Calibration Bias Reference Charts

November 2025
This website contains an expanding collection of reference charts that give the calibration bias (a.k.a. reliability bias) resulting from fitting statistical models with maximum likelihood or calibrating prior prediction.
These charts are relevant for anyone using statistical models to understand the probabilities of extremes.

Calibration Bias

Introduction

Imagine you are fitting a statistical distribution to n data points. The data are assumed to be independent, and the distribution is continuous and parametric.
And imagine your goal is to make an assessment of the probability distribution of the next data point. So you want to make an out-of-sample prediction of probabilities, quantiles, densities, or random deviates.
You would hope that an x% probability from the predictive distribution you create would correspond to future events that will occur x% of the time, otherwise the probabilities are somewhat meaningless. This property is known as reliability, or calibration.
But fitting distributions in such a way that they make predictions that are well calibrated is actually not so easy to do.
The most commonly-used methods for fitting statistical distributions, which use point estimates of parameters (methods such as maximum likelihood, method of moments, L moments and probability weighted moments) do not produce calibrated probabilities.
The biggest problem for these methods is in the tail, for extremes, where they give underestimates of probabilities. They underestimate extreme probabilities and return levels because they ignore parameter uncertainty. The underestimation can be very large, and if you are designing a flood defence, pricing insurance, or figuring out what kinds of heat-waves you need to be ready for, that underestimation could be very unfortunate.

Calibration Bias Charts

The calibration bias of a method for fitting distributions can be quantified using simulations (assuming the model is correct).
This website provides the results of simulations which show the size of the calibration bias for a number of commonly-used statistical models.
The idea is that the first step towards reducing calibration bias is to understand it.
The charts cover predictions generated by maximum likelihood prediction and calibrating prior prediction.
Calibrating prior prediction is an Objective Bayesian method which was designed to be as easy to use as maximum likelihood, but to give lower calibration bias (see the references below for more information).
Bayesian predictions require a prior: in calibrating prior prediction the prior is chosen to give good calibration for any underlying parameter values, or for parameter values in a reasonable range.
In some cases (homogeneous statistical models, such as the normal, Gumbel and simple linear regression, among others), calibrating prior predictions are perfectly calibrated, with the small caveat that there may be a departure from perfect calibration if the scheme used to evaluate the Bayesian prediction equation is approximate or has not not fully converged.
In other cases (inhomogeneous statistical models, such as the GEVD and GPD, among others), calibrating prior predictions are not perfectly calibrated, but the bias is still materially lower than maximum likelihood, and new research promises to further reduce the bias.

Reasons to Use the Charts

There are two main reasons to use these charts:

There are 100s of scientific publications that have assessed the probabilities of extremes using maximum likelihood. You can use these charts to understand possible bias in the probability and return level assessments in those publications. In some cases, the bias will be small enough that it can be ignored. In other cases, it will be large enough that it cannot be ignored. When it is large, the charts can be used to estimate a correction, or to motivate recalculating the probabilities.
If you are designing a study that will involve fitting statistical distributions as a way to estimate probabilities of future events, then you can use the charts to understand whether maximum likelihood prediction or calibrating prior prediction will be good enough for your purposes.

Reading the Charts

Chart Type 1

The horizontal axis is sample size
The vertical axis is one over nominal exceedance probability (NP). Nominal probability is the probability you are asking about, that you specify, or that the model returns.
The contours show one over the true exceedance probability, also known as the predictive coverage probability (PCP). This is the probability after correction for bias.
Each chart is split into three parts, covering different ranges of inverse NP. This allows us to use axes that are linear in inverse NP, which are easy to read, give appropriate resolution in each range, and cover a wide range of probabilities, from p = 1/2 to p = 1/500.
If a study with sample size n returns a maximum likelihood assessment of the probability of an event (or quantile) as being NP then we can use the appropriate chart to derive the PCP (i.e., the true predictive probability) for that event, using the contours, as a function of n (on the horizontal axis) and NP (on the vertical axis). The PCPs are typically higher than the NPs. This is because the tail of maximum likelihood predictions is typically too thin.
The calibrating prior prediction charts are interpreted in a similar way.

Chart Type 2

These charts only apply to inhomogeneous models, where the results depend on a parameter.
The horizontal axis is now parameter value, and each chart represents one sample size.

Notes

Technical Comments

You may be wondering about subjective Bayesian predictions, and whether they are calibrated. The answer is: you should believe that they are calibrated if you believe the prior, but if you don't believe the prior, you shouldn't. The idea of calibrating prior prediction is to remove this dependence on belief, and produce Bayesian predictive distributions that are calibrated, or close to calibrated, for all observers, irrespective of their beliefs.

Acknowledgements

Many thanks to Trevor Sweeting for help with the statistical theory, Lynne Jewson for help with the group theory, and SE Jewson for help with this website.
Also, many thanks to the anonymous insurance companies that fund this research project.
And in addition, many thanks to the various people I have discussed this whole topic with over the last couple of years, including Clare Barnes, Chris Paciorek, Paul Northrop, Stephen Dupon, and the anonymous reviewers of our journal papers.

More information

This website is intentionally brief. There is much more information in the references below.

Research

There are various extensions and improvements in the pipeline. I post news about my research on Linkedin.

Contact

I can be contacted at stephen.jewson-at-gmail.com

References

1) Our initial paper on this topic, which contains a detailed technical discussion:

S. Jewson, T. Sweeting and L. Jewson (2025): Reducing Reliability Bias in Assessments of Extreme Weather Risk using Calibrating Priors: ASCMO (Advances in Statistical Climatology, Meteorology and Oceanography)

Bibtex:

@article{jewsonsweetingjewson2025,
	author = {Jewson, Stephen and Sweeting, Trevor and Jewson, Lynne},
	title = {Reducing reliability bias in assessments of extreme weather risk using calibrating priors},
	journal = {ASCMO},
	volume = {11},
	number = {1},
	pages = {1-22},
	doi = {10.5194/ascmo-11-1-2025},
	url = {https://doi.org/10.5194/ascmo-11-1-2025},
	year = {2025}
}

2) A second paper that describes the charts and gives a slightly less detailed technical discussion:

S. Jewson (2025): Maximum Likelihood and Calibration Prior Reliability Bias Reference Charts: Stats

Bibtex:

@article{jewson2025,
	author = {Jewson, Stephen},
	title = {Maximum Likelihood and Calibration Prior Reliability Bias Reference Charts},
	journal = {Stats},
	volume = {8},
	number = {4},
	doi = {10.3390/stats8040109},
	url = {https://www.mdpi.com/2571-905X/8/4/109},
	year = {2025}
}

3) A technical paper aimed at actuaries:

S. Jewson and T. Sweeting (2026): Including parameter uncertainty in loss distributions using calibrating priors: Annals of Actuarial Science

Bibtex:

@article{jewsonsweeting2026,
	author = {Jewson, Stephen and Sweeting, Trevor},
	title = {Including parameter uncertainty in loss distributions using calibrating priors},
	journal = {Annals of Actuarial Science},
	doi = {10.1017/S1748499525100237},
	url = {https://doi.org/10.1017/S1748499525100237},
	year = {2026}
}

4) Related software packages:

We have produced software packages that makes it easy to produce maximum likelihood and calibrating prior predictions: fitdistcp
On CRAN
On Pypi