Create a description object for a resampling strategy.

A description of a resampling algorithm contains all necessary information to create a ResampleInstance, when given the size of the data set.

Usage

makeResampleDesc(
  method,
  predict = "test",
  ...,
  stratify = FALSE,
  stratify.cols = NULL,
  fixed = FALSE,
  blocking.cv = FALSE
)

Arguments

method

(character(1))
“CV” for cross-validation, “LOO” for leave-one-out, “RepCV” for repeated cross-validation, “Bootstrap” for out-of-bag bootstrap, “Subsample” for subsampling, “Holdout” for holdout, “GrowingWindowCV” for growing window cross-validation, “FixedWindowCV” for fixed window cross validation.

predict

(character(1))
What to predict during resampling: “train”, “test” or “both” sets. Default is “test”.

...

(any)
Further parameters for strategies.

iters (integer(1)): Number of iterations, for “CV”, “Subsample” and “Bootstrap”.
split (numeric(1)): Proportion of training cases for “Holdout” and “Subsample” between 0 and 1. Default is 2 / 3.
reps (integer(1)): Repeats for “RepCV”. Here iters = folds * reps. Default is 10.
folds (integer(1)): Folds in the repeated CV for RepCV. Here iters = folds * reps. Default is 10.
horizon (numeric(1)): Number of observations in the forecast test set for “GrowingWindowCV” and “FixedWindowCV”. When horizon > 1 this will be treated as the number of observations to forecast, else it will be a fraction of the initial window. IE, for 100 observations, initial window of .5, and horizon of .2, the test set will have 10 observations. Default is 1.
initial.window (numeric(1)): Fraction of observations to start with in the training set for “GrowingWindowCV” and “FixedWindowCV”. When initial.window > 1 this will be treated as the number of observations in the initial window, else it will be treated as the fraction of observations to have in the initial window. Default is 0.5.
skip (numeric(1)): How many resamples to skip to thin the total amount for “GrowingWindowCV” and “FixedWindowCV”. This is passed through as the “by” argument in seq(). When skip > 1 this will be treated as the increment of the sequence of resampling indices, else it will be a fraction of the total training indices. IE for 100 training sets and a value of .2, the increment of the resampling indices will be 20. Default is “horizon” which gives mutually exclusive chunks of test indices.

stratify

(logical(1))
Should stratification be done for the target variable? For classification tasks, this means that the resampling strategy is applied to all classes individually and the resulting index sets are joined to make sure that the proportion of observations in each training set is as in the original data set. Useful for imbalanced class sizes. For survival tasks stratification is done on the events, resulting in training sets with comparable censoring rates.

stratify.cols

(character)
Stratify on specific columns referenced by name. All columns have to be factor or integer. Note that you have to ensure yourself that stratification is possible, i.e. that each strata contains enough observations. This argument and stratify are mutually exclusive.

fixed

(logical(1))
Whether indices supplied via argument 'blocking' in the task should be used as fully pre-defined indices. Default is FALSE which means they will be used following the 'blocking' approach. fixed only works with ResampleDesc CV and the supplied indices must match the number of observations. When fixed = TRUE, the iters argument will be ignored and is interally set to the number of supplied factor levels in blocking.

blocking.cv

(logical(1))
Should 'blocking' be used in CV? Default to FALSE. This is different to fixed = TRUE and cannot be combined. Please check the mlr online tutorial for more details.

Value

(ResampleDesc).

Details

Some notes on some special strategies:

Repeated cross-validation: Use “RepCV”. Then you have to set the aggregation function for your preferred performance measure to “testgroup.mean” via setAggregation.
B632 bootstrap: Use “Bootstrap” for bootstrap and set predict to “both”. Then you have to set the aggregation function for your preferred performance measure to “b632” via setAggregation.
B632+ bootstrap: Use “Bootstrap” for bootstrap and set predict to “both”. Then you have to set the aggregation function for your preferred performance measure to “b632plus” via setAggregation.
Fixed Holdout set: Use makeFixedHoldoutInstance.

Object slots:

id (character(1)): Name of resampling strategy.
iters (integer(1)): Number of iterations. Note that this is always the complete number of generated train/test sets, so for a 10-times repeated 5fold cross-validation it would be 50.
predict (character(1)): See argument.
stratify (logical(1)): See argument.
All parameters passed in ... under the respective argument name: See arguments.

Standard ResampleDesc objects

For common resampling strategies you can save some typing by using the following description objects:

hout: holdout a.k.a. test sample estimation (two-thirds training set, one-third testing set)
cv2: 2-fold cross-validation
cv3: 3-fold cross-validation
cv5: 5-fold cross-validation
cv10: 10-fold cross-validation

Examples

# Bootstraping
makeResampleDesc("Bootstrap", iters = 10)
#> Resample description: OOB bootstrapping with 10 iterations.
#> Predict: test
#> Stratification: FALSE
makeResampleDesc("Bootstrap", iters = 10, predict = "both")
#> Resample description: OOB bootstrapping with 10 iterations.
#> Predict: both
#> Stratification: FALSE

# Subsampling
makeResampleDesc("Subsample", iters = 10, split = 3 / 4)
#> Resample description: subsampling with 10 iterations and 0.75 split rate.
#> Predict: test
#> Stratification: FALSE
makeResampleDesc("Subsample", iters = 10)
#> Resample description: subsampling with 10 iterations and 0.67 split rate.
#> Predict: test
#> Stratification: FALSE

# Holdout a.k.a. test sample estimation
makeResampleDesc("Holdout")
#> Resample description: holdout with 0.67 split rate.
#> Predict: test
#> Stratification: FALSE