Optimal decision-making based on classifiers requires that the confidence in their predictions reflect the actual error rates. When this happens, one speaks of a calibrated model. Recent work has shown that expressive neural networks are able to overfit the cross-entropy loss without losing accuracy, thus producing overconfident (i.e. miscalibrated) models.
Learning outcomes
- Understanding quantitatively how optimal decisions depend on all probabilities and not just on the predicted class.
- Learning to measure (mis)calibration of models.
- Recalibration of classifiers during training: loss functions, regularisation.
- Recalibration of classifiers, a posteriori: non-parametric, parametric, Bayesian.
Structure of the workshop
- Introduction to the nomenclature: notions of calibration, metrics, calibration functions
- Relation between calibration and accuracy
- Discussion of situations when calibration is important for performance
- Measuring miscalibration 1: histogram estimators
- Measuring miscalibration 2: statistical tests and decision process simulation
- Recalibration by post-processing
- Training calibrated classifiers
- Performance comparison of different approaches to calibration
