Generate data for critical-differences plot.
Source:R/plotCritDifferences.R
generateCritDifferencesData.Rd
Generates data that can be used to plot a
critical differences plot. Computes the critical differences according
to either the
"Bonferroni-Dunn"
test or the "Nemenyi"
test."Bonferroni-Dunn"
usually yields higher power as it does not
compare all algorithms to each other, but all algorithms to a
baseline
instead.
Learners are drawn on the y-axis according to their average rank.
For test = "nemenyi"
a bar is drawn, connecting all groups of not
significantly different learners.
For test = "bd"
an interval is drawn arround the algorithm selected
as a baseline. All learners within this interval are not signifcantly different
from the baseline.
Calculation:
$$CD = q_{\alpha} \sqrt{\left(\frac{k(k+1)}{6N}\right)}$$
Where \(q_\alpha\) is based on the studentized range statistic.
See references for details.
Usage
generateCritDifferencesData(
bmr,
measure = NULL,
p.value = 0.05,
baseline = NULL,
test = "bd"
)
Arguments
- bmr
(BenchmarkResult)
Benchmark result.- measure
(Measure)
Performance measure. Default is the first measure used in the benchmark experiment.- p.value
(
numeric(1)
)
P-value for the critical difference. Default: 0.05- baseline
(
character(1)
): (learner.id
)
Select alearner.id
as baseline for thetest = "bd"
("Bonferroni-Dunn") critical differences diagram. The critical difference interval will then be positioned arround this learner. Defaults to best performing algorithm.
Fortest = "nemenyi"
, no baseline is needed as it performs all pairwise comparisons.- test
(
character(1)
)
Test for which the critical differences are computed.
“bd” for the Bonferroni-Dunn Test, which is comparing all classifiers to abaseline
, thus performing a comparison of one classifier to all others.
Algorithms not connected by a single line are statistically different from the baseline.
“nemenyi” for the PMCMRplus::frdAllPairsNemenyiTest which is comparing all classifiers to each other. The null hypothesis that there is a difference between the classifiers can not be rejected for all classifiers that have a single grey bar connecting them.
Value
(critDifferencesData
). List containing:
- data
(data.frame) containing the info for the descriptive part of the plot
- friedman.nemenyi.test
(list) of class
pairwise.htest
contains the calculated PMCMRplus::frdAllPairsNemenyiTest- cd.info
(list) containing info on the critical difference and its positioning
- baseline
baseline
chosen for plotting- p.value
p.value used for the PMCMRplus::frdAllPairsNemenyiTest and for computation of the critical difference
See also
Other generate_plot_data:
generateCalibrationData()
,
generateFeatureImportanceData()
,
generateFilterValuesData()
,
generateLearningCurveData()
,
generatePartialDependenceData()
,
generateThreshVsPerfData()
,
plotFilterValues()
Other benchmark:
BenchmarkResult
,
batchmark()
,
benchmark()
,
convertBMRToRankMatrix()
,
friedmanPostHocTestBMR()
,
friedmanTestBMR()
,
getBMRAggrPerformances()
,
getBMRFeatSelResults()
,
getBMRFilteredFeatures()
,
getBMRLearnerIds()
,
getBMRLearnerShortNames()
,
getBMRLearners()
,
getBMRMeasureIds()
,
getBMRMeasures()
,
getBMRModels()
,
getBMRPerformances()
,
getBMRPredictions()
,
getBMRTaskDescs()
,
getBMRTaskIds()
,
getBMRTuneResults()
,
plotBMRBoxplots()
,
plotBMRRanksAsBarChart()
,
plotBMRSummary()
,
plotCritDifferences()
,
reduceBatchmarkResults()