A calibrated classifier is one where the predicted probability of a class closely matches the rate at which that class occurs, e.g. for data points which are assigned a predicted probability of class A of .8, approximately 80 percent of such points should belong to class A if the classifier is well calibrated. This is estimated empirically by grouping data points with similar predicted probabilities for each class, and plotting the rate of each class within each bin against the predicted probability bins.
Arguments
- obj
(list of Prediction | list of ResampleResult | BenchmarkResult)
Single prediction object, list of them, single resample result, list of them, or a benchmark result. In case of a list probably produced by different learners you want to compare, then name the list with the names you want to see in the plots, probably learner shortnames or ids.- breaks
(
character(1)
| numeric)
Ifcharacter(1)
, the algorithm to use in generating probability bins. See hist for details. If numeric, the cut points for the bins. Default is “Sturges”.- groups
(
integer(1)
)
The number of bins to construct. If specified,breaks
is ignored. Default isNULL
.- task.id
(
character(1)
)
Selected task in BenchmarkResult to do plots for, ignored otherwise. Default is first task.
Value
CalibrationData. A list containing:
- proportion
data.frame with columns:
Learner
Name of learner.bin
Bins calculated according to thebreaks
orgroups
argument.Class
Class labels (for binary classification only the positive class).Proportion
Proportion of observations from classClass
among all observations with posterior probabilities of classClass
within the interval given inbin
.
- data
data.frame with columns:
Learner
Name of learner.truth
True class label.Class
Class labels (for binary classification only the positive class).Probability
Predicted posterior probability ofClass
.bin
Bin corresponding toProbability
.
- task
(TaskDesc)
Task description.
References
Vuk, Miha, and Curk, Tomaz. “ROC Curve, Lift Chart, and Calibration Plot.” Metodoloski zvezki. Vol. 3. No. 1 (2006): 89-108.
See also
Other generate_plot_data:
generateCritDifferencesData()
,
generateFeatureImportanceData()
,
generateFilterValuesData()
,
generateLearningCurveData()
,
generatePartialDependenceData()
,
generateThreshVsPerfData()
,
plotFilterValues()
Other calibration:
plotCalibration()