step_other creates a specification of a recipe step that will potentially pool infrequently occurring values into an "other" category.

step_other(
  recipe,
  ...,
  role = NA,
  trained = FALSE,
  threshold = 0.05,
  other = "other",
  objects = NULL,
  skip = FALSE,
  id = rand_id("other")
)

Arguments

recipe

A recipe object. The step will be added to the sequence of operations for this recipe.

...

One or more selector functions to choose which variables that will potentially be reduced. See selections() for more details.

role

Not used by this step since no new variables are created.

trained

A logical to indicate if the quantities for preprocessing have been estimated.

threshold

A numeric value between 0 and 1 or an integer greater or equal to one. If it's less than one then factor levels whose rate of occurrence in the training set are below threshold will be "othered". If it's greater or equal to one then it's treated as a frequency and factor levels that occur less then threshold times will be "othered".

other

A single character value for the "other" category.

objects

A list of objects that contain the information to pool infrequent levels that is determined by prep.recipe().

skip

A logical. Should the step be skipped when the recipe is baked by bake.recipe()? While all operations are baked when prep.recipe() is run, some operations may not be able to be conducted on new data (e.g. processing the outcome variable(s)). Care should be taken when using skip = TRUE as it may affect the computations for subsequent operations

id

A character string that is unique to this step to identify it.

Value

An updated version of recipe with the new step added to the sequence of existing steps (if any).

Details

The overall proportion (or total counts) of the categories are computed. The "other" category is used in place of any categorical levels whose individual proportion (or frequency) in the training set is less than threshold.

If no pooling is done the data are unmodified (although character data may be changed to factors based on the value of strings_as_factors in prep.recipe()). Otherwise, a factor is always returned with different factor levels.

If threshold is less than the largest category proportion, all levels except for the most frequent are collapsed to the other level.

If the retained categories include the value of other, an error is thrown. If other is in the list of discarded levels, no error occurs.

If no pooling is done, novel factor levels are converted to missing. If pooling is needed, they will be placed into the other category.

When data to be processed contains novel levels (i.e., not contained in the training set), the other category is assigned.

When you tidy() this step, a tibble with columns terms (the columns that will be affected) and retained (the factor levels that were not pulled into "other") is returned.

See also

Examples

library(modeldata)
data(okc)

set.seed(19)
in_train <- sample(1:nrow(okc), size = 30000)

okc_tr <- okc[ in_train,]
okc_te <- okc[-in_train,]

rec <- recipe(~ diet + location, data = okc_tr)


rec <- rec %>%
  step_other(diet, location, threshold = .1, other = "other values")
rec <- prep(rec, training = okc_tr)

collapsed <- bake(rec, okc_te)
table(okc_te$diet, collapsed$diet, useNA = "always")
#>                      
#>                       anything mostly anything strictly anything other values
#>   anything                3128               0                 0            0
#>   halal                      0               0                 0            4
#>   kosher                     0               0                 0            5
#>   mostly anything            0            8226                 0            0
#>   mostly halal               0               0                 0           23
#>   mostly kosher              0               0                 0           36
#>   mostly other               0               0                 0          509
#>   mostly vegan               0               0                 0          164
#>   mostly vegetarian          0               0                 0         1715
#>   other                      0               0                 0          172
#>   strictly anything          0               0              2562            0
#>   strictly halal             0               0                 0            7
#>   strictly kosher            0               0                 0            7
#>   strictly other             0               0                 0          209
#>   strictly vegan             0               0                 0          104
#>   strictly vegetarian        0               0                 0          442
#>   vegan                      0               0                 0           66
#>   vegetarian                 0               0                 0          331
#>   <NA>                       0               0                 0            0
#>                      
#>                        <NA>
#>   anything                0
#>   halal                   0
#>   kosher                  0
#>   mostly anything         0
#>   mostly halal            0
#>   mostly kosher           0
#>   mostly other            0
#>   mostly vegan            0
#>   mostly vegetarian       0
#>   other                   0
#>   strictly anything       0
#>   strictly halal          0
#>   strictly kosher         0
#>   strictly other          0
#>   strictly vegan          0
#>   strictly vegetarian     0
#>   vegan                   0
#>   vegetarian              0
#>   <NA>                12145

tidy(rec, number = 1)
#> # A tibble: 5 x 3
#>   terms    retained          id         
#>   <chr>    <chr>             <chr>      
#> 1 diet     anything          other_ez6XT
#> 2 diet     mostly anything   other_ez6XT
#> 3 diet     strictly anything other_ez6XT
#> 4 location oakland           other_ez6XT
#> 5 location san francisco     other_ez6XT

# novel levels are also "othered"
tahiti <- okc[1,]
tahiti$location <- "a magical place"
bake(rec, tahiti)
#> # A tibble: 1 x 2
#>   diet              location    
#>   <fct>             <fct>       
#> 1 strictly anything other values

# threshold as a frequency
rec <- recipe(~ diet + location, data = okc_tr)

rec <- rec %>%
  step_other(diet, location, threshold = 2000, other = "other values")
rec <- prep(rec, training = okc_tr)

tidy(rec, number = 1)
#> # A tibble: 6 x 3
#>   terms    retained          id         
#>   <chr>    <chr>             <chr>      
#> 1 diet     anything          other_p2QWY
#> 2 diet     mostly anything   other_p2QWY
#> 3 diet     strictly anything other_p2QWY
#> 4 location berkeley          other_p2QWY
#> 5 location oakland           other_p2QWY
#> 6 location san francisco     other_p2QWY
# compare it to
# okc_tr %>% count(diet, sort = TRUE) %>% top_n(4)
# okc_tr %>% count(location, sort = TRUE) %>% top_n(3)