High Correlation FilterSource:
step_corr creates a specification of a recipe
step that will potentially remove variables that have large
absolute correlations with other variables.
step_corr( recipe, ..., role = NA, trained = FALSE, threshold = 0.9, use = "pairwise.complete.obs", method = "pearson", removals = NULL, skip = FALSE, id = rand_id("corr") )
A recipe object. The step will be added to the sequence of operations for this recipe.
One or more selector functions to choose variables for this step. See
selections()for more details.
Not used by this step since no new variables are created.
A logical to indicate if the quantities for preprocessing have been estimated.
A value for the threshold of absolute correlation values. The step will try to remove the minimum number of columns so that all the resulting absolute correlations are less than this value.
A character string for the
useargument to the
A character string for the
methodargument to the
A character string that contains the names of columns that should be removed. These values are not determined until
A logical. Should the step be skipped when the recipe is baked by
bake()? While all operations are baked when
prep()is run, some operations may not be able to be conducted on new data (e.g. processing the outcome variable(s)). Care should be taken when using
skip = TRUEas it may affect the computations for subsequent operations.
A character string that is unique to this step to identify it.
An updated version of
recipe with the new step added to the
sequence of any existing operations.
This step can potentially remove columns from the data set. This may cause issues for subsequent steps in your recipe if the missing columns are specifically referenced by name. To avoid this, see the advice in the Tips for saving recipes and filtering columns section of selections.
This step attempts to remove variables to keep the
largest absolute correlation between the variables less than
When a column has a single unique value, that column will be
excluded from the correlation analysis. Also, if the data set
has sporadic missing values (and an inappropriate value of
is chosen), some columns will also be excluded from the filter.
tidy() this step, a tibble with column
terms (the columns that will be removed) is returned.
Original R code for filtering algorithm by Dong Li,
modified by Max Kuhn. Contributions by Reynald Lescarbeau (for
caret package). Max Kuhn for the
library(modeldata) data(biomass) set.seed(3535) biomass$duplicate <- biomass$carbon + rnorm(nrow(biomass)) biomass_tr <- biomass[biomass$dataset == "Training",] biomass_te <- biomass[biomass$dataset == "Testing",] rec <- recipe(HHV ~ carbon + hydrogen + oxygen + nitrogen + sulfur + duplicate, data = biomass_tr) corr_filter <- rec %>% step_corr(all_numeric_predictors(), threshold = .5) filter_obj <- prep(corr_filter, training = biomass_tr) filtered_te <- bake(filter_obj, biomass_te) round(abs(cor(biomass_tr[, c(3:7, 9)])), 2) #> carbon hydrogen oxygen nitrogen sulfur duplicate #> carbon 1.00 0.32 0.63 0.15 0.09 1.00 #> hydrogen 0.32 1.00 0.54 0.07 0.19 0.31 #> oxygen 0.63 0.54 1.00 0.18 0.31 0.63 #> nitrogen 0.15 0.07 0.18 1.00 0.27 0.15 #> sulfur 0.09 0.19 0.31 0.27 1.00 0.10 #> duplicate 1.00 0.31 0.63 0.15 0.10 1.00 round(abs(cor(filtered_te)), 2) #> hydrogen nitrogen sulfur duplicate HHV #> hydrogen 1.00 0.11 0.26 0.20 0.10 #> nitrogen 0.11 1.00 0.16 0.13 0.11 #> sulfur 0.26 0.16 1.00 0.13 0.08 #> duplicate 0.20 0.13 0.13 1.00 0.94 #> HHV 0.10 0.11 0.08 0.94 1.00 tidy(corr_filter, number = 1) #> # A tibble: 1 × 2 #> terms id #> <chr> <chr> #> 1 all_numeric_predictors() corr_ubc7G tidy(filter_obj, number = 1) #> # A tibble: 2 × 2 #> terms id #> <chr> <chr> #> 1 oxygen corr_ubc7G #> 2 carbon corr_ubc7G