Collapse Some Categorical LevelsSource:
step_other creates a specification of a recipe
step that will potentially pool infrequently occurring values
into an "other" category.
step_other( recipe, ..., role = NA, trained = FALSE, threshold = 0.05, other = "other", objects = NULL, skip = FALSE, id = rand_id("other") )
A recipe object. The step will be added to the sequence of operations for this recipe.
One or more selector functions to choose variables for this step. See
selections()for more details.
Not used by this step since no new variables are created.
A logical to indicate if the quantities for preprocessing have been estimated.
A numeric value between 0 and 1, or an integer greater or equal to one. If less than one, then factor levels with a rate of occurrence in the training set below
thresholdwill be pooled to
other. If greater or equal to one, then this value is treated as a frequency and factor levels that occur less than
thresholdtimes will be pooled to
A single character value for the "other" category.
A list of objects that contain the information to pool infrequent levels that is determined by
A logical. Should the step be skipped when the recipe is baked by
bake()? While all operations are baked when
prep()is run, some operations may not be able to be conducted on new data (e.g. processing the outcome variable(s)). Care should be taken when using
skip = TRUEas it may affect the computations for subsequent operations.
A character string that is unique to this step to identify it.
An updated version of
recipe with the new step added to the
sequence of any existing operations.
The overall proportion (or total counts) of the categories are
computed. The "other" category is used in place of any categorical levels
whose individual proportion (or frequency) in the training set is less than
If no pooling is done the data are unmodified (although character data may
be changed to factors based on the value of
prep()). Otherwise, a factor is always returned with
different factor levels.
threshold is less than the largest category proportion, all levels
except for the most frequent are collapsed to the
If the retained categories include the value of
other, an error is
other is in the list of discarded levels, no error
If no pooling is done, novel factor levels are converted to missing. If pooling is needed, they will be placed into the other category.
When data to be processed contains novel levels (i.e., not contained in the training set), the other category is assigned.
tidy() this step, a tibble with columns
terms (the columns that will be affected) and
retained (the factor
levels that were not pulled into "other") is returned.
Other dummy variable and encoding steps:
library(modeldata) data(okc) set.seed(19) in_train <- sample(1:nrow(okc), size = 30000) okc_tr <- okc[ in_train,] okc_te <- okc[-in_train,] rec <- recipe(~ diet + location, data = okc_tr) rec <- rec %>% step_other(diet, location, threshold = .1, other = "other values") rec <- prep(rec, training = okc_tr) collapsed <- bake(rec, okc_te) table(okc_te$diet, collapsed$diet, useNA = "always") #> #> anything mostly anything strictly anything #> anything 3128 0 0 #> halal 0 0 0 #> kosher 0 0 0 #> mostly anything 0 8226 0 #> mostly halal 0 0 0 #> mostly kosher 0 0 0 #> mostly other 0 0 0 #> mostly vegan 0 0 0 #> mostly vegetarian 0 0 0 #> other 0 0 0 #> strictly anything 0 0 2562 #> strictly halal 0 0 0 #> strictly kosher 0 0 0 #> strictly other 0 0 0 #> strictly vegan 0 0 0 #> strictly vegetarian 0 0 0 #> vegan 0 0 0 #> vegetarian 0 0 0 #> <NA> 0 0 0 #> #> other values <NA> #> anything 0 0 #> halal 4 0 #> kosher 5 0 #> mostly anything 0 0 #> mostly halal 23 0 #> mostly kosher 36 0 #> mostly other 509 0 #> mostly vegan 164 0 #> mostly vegetarian 1715 0 #> other 172 0 #> strictly anything 0 0 #> strictly halal 7 0 #> strictly kosher 7 0 #> strictly other 209 0 #> strictly vegan 104 0 #> strictly vegetarian 442 0 #> vegan 66 0 #> vegetarian 331 0 #> <NA> 0 12145 tidy(rec, number = 1) #> # A tibble: 5 × 3 #> terms retained id #> <chr> <chr> <chr> #> 1 diet anything other_ez6XT #> 2 diet mostly anything other_ez6XT #> 3 diet strictly anything other_ez6XT #> 4 location oakland other_ez6XT #> 5 location san francisco other_ez6XT # novel levels are also "othered" tahiti <- okc[1,] tahiti$location <- "a magical place" bake(rec, tahiti) #> # A tibble: 1 × 2 #> diet location #> <fct> <fct> #> 1 strictly anything other values # threshold as a frequency rec <- recipe(~ diet + location, data = okc_tr) rec <- rec %>% step_other(diet, location, threshold = 2000, other = "other values") rec <- prep(rec, training = okc_tr) tidy(rec, number = 1) #> # A tibble: 6 × 3 #> terms retained id #> <chr> <chr> <chr> #> 1 diet anything other_p2QWY #> 2 diet mostly anything other_p2QWY #> 3 diet strictly anything other_p2QWY #> 4 location berkeley other_p2QWY #> 5 location oakland other_p2QWY #> 6 location san francisco other_p2QWY # compare it to # okc_tr %>% count(diet, sort = TRUE) %>% top_n(4) # okc_tr %>% count(location, sort = TRUE) %>% top_n(3)