A calibrated classifier is one where the predicted probability of a class closely matches the rate at which that class occurs, e.g. for data points which are assigned a predicted probability of class A of .8, approximately 80 percent of such points should belong to class A if the classifier is well calibrated. This is estimated empirically by grouping data points with similar predicted probabilities for each class, and plotting the rate of each class within each bin against the predicted probability bins.
generateCalibrationData(obj, breaks = "Sturges", groups = NULL, task.id = NULL)
obj | (list of Prediction | list of ResampleResult | BenchmarkResult) |
---|---|
breaks | ( |
groups | ( |
task.id | ( |
CalibrationData. A list containing:
data.frame with columns:
Learner
Name of learner.
bin
Bins calculated according to the breaks
or groups
argument.
Class
Class labels (for binary classification only the positive class).
Proportion
Proportion of observations from class Class
among all
observations with posterior probabilities of class Class
within the
interval given in bin
.
data.frame with columns:
Learner
Name of learner.
truth
True class label.
Class
Class labels (for binary classification only the positive class).
Probability
Predicted posterior probability of Class
.
bin
Bin corresponding to Probability
.
(TaskDesc)
Task description.
Vuk, Miha, and Curk, Tomaz. “ROC Curve, Lift Chart, and Calibration Plot.” Metodoloski zvezki. Vol. 3. No. 1 (2006): 89-108.
Other generate_plot_data:
generateCritDifferencesData()
,
generateFeatureImportanceData()
,
generateFilterValuesData()
,
generateLearningCurveData()
,
generatePartialDependenceData()
,
generateThreshVsPerfData()
,
plotFilterValues()
Other calibration:
plotCalibration()