Generates data that can be used to plot a
critical differences plot. Computes the critical differences according
to either the
`"Bonferroni-Dunn"`

test or the `"Nemenyi"`

test.

`"Bonferroni-Dunn"`

usually yields higher power as it does not
compare all algorithms to each other, but all algorithms to a
`baseline`

instead.

Learners are drawn on the y-axis according to their average rank.

For `test = "nemenyi"`

a bar is drawn, connecting all groups of not
significantly different learners.

For `test = "bd"`

an interval is drawn arround the algorithm selected
as a baseline. All learners within this interval are not signifcantly different
from the baseline.

Calculation:
$$CD = q_{\alpha} \sqrt{\left(\frac{k(k+1)}{6N}\right)}$$

Where \(q_\alpha\) is based on the studentized range statistic.
See references for details.

generateCritDifferencesData(bmr, measure = NULL, p.value = 0.05,
baseline = NULL, test = "bd")

## Arguments

bmr |
(BenchmarkResult)
Benchmark result. |

measure |
(Measure)
Performance measure.
Default is the first measure used in the benchmark experiment. |

p.value |
(`numeric(1)` )
P-value for the critical difference. Default: 0.05 |

baseline |
(`character(1)` ): (`learner.id` )
Select a `learner.id` as baseline for the `test = "bd"`
("Bonferroni-Dunn") critical differences
diagram. The critical difference interval will then be positioned arround this learner.
Defaults to best performing algorithm.
For `test = "nemenyi"` , no baseline is needed as it performs *all pairwise
comparisons*. |

test |
(`character(1)` )
Test for which the critical differences are computed.
“bd” for the Bonferroni-Dunn Test, which is comparing all
classifiers to a `baseline` , thus performing a comparison
of one classifier to all others.
Algorithms not connected by a single line are statistically different
from the baseline.
“nemenyi” for the PMCMR::posthoc.friedman.nemenyi.test
which is comparing all classifiers to each other. The null hypothesis that
there is a difference between the classifiers can not be rejected for all
classifiers that have a single grey bar connecting them. |

## Value

(`critDifferencesData`

). List containing:

data(data.frame) containing the info for the descriptive
part of the plot

friedman.nemenyi.test(list) of class `pairwise.htest`

contains the calculated
PMCMR::posthoc.friedman.nemenyi.test

cd.info(list) containing info on the critical difference
and its positioning

baseline`baseline`

chosen for plotting

p.valuep.value used for the PMCMR::posthoc.friedman.nemenyi.test
and for computation of the critical difference

## See also

Other generate_plot_data: `generateCalibrationData`

,
`generateFeatureImportanceData`

,
`generateFilterValuesData`

,
`generateLearningCurveData`

,
`generatePartialDependenceData`

,
`generateThreshVsPerfData`

,
`plotFilterValues`

Other benchmark: `BenchmarkResult`

,
`batchmark`

, `benchmark`

,
`convertBMRToRankMatrix`

,
`friedmanPostHocTestBMR`

,
`friedmanTestBMR`

,
`getBMRAggrPerformances`

,
`getBMRFeatSelResults`

,
`getBMRFilteredFeatures`

,
`getBMRLearnerIds`

,
`getBMRLearnerShortNames`

,
`getBMRLearners`

,
`getBMRMeasureIds`

,
`getBMRMeasures`

, `getBMRModels`

,
`getBMRPerformances`

,
`getBMRPredictions`

,
`getBMRTaskDescs`

,
`getBMRTaskIds`

,
`getBMRTuneResults`

,
`plotBMRBoxplots`

,
`plotBMRRanksAsBarChart`

,
`plotBMRSummary`

,
`plotCritDifferences`

,
`reduceBatchmarkResults`