What is the most efficient kmeans clustering package in R?

Question

Sorry if this seems vague, but I have a data set with over 100 columns with characteristics I want to cluster with, and ~10^6 rows. Using

kmeans(dataframe, centers = 100,
             nstart = 20,
             iter.max = 30)

Takes over an hour on an i7-6700K. It does not use multiple cores, so is that something which can be done?

Thanks!

Answer 1

You could try using ClusterR, especially the function MiniBatchKmeans

Here is an example of usage:

some data (smaller than yours - 300k rows and 30 columns)

z <- rbind(replicate(30, rnorm(1e5, 2)),
           replicate(30, rnorm(1e5, -1)),
           replicate(30, rnorm(1e5, 5)))

library(ClusterR)
km_model <- MiniBatchKmeans(z, clusters = 3, batch_size = 20, num_init = 5, max_iters = 100, 
                         init_fraction = 0.2, initializer = 'kmeans++', early_stop_iter = 10,
                         verbose = F)

pred <- predict_MBatchKMeans(z, km_model$centroids)

object pred contains the associated clusters:

table(pred)
pred
     1      2      3 
100000 100000 100000 

I'd say that was a perfect separation. Increasing the batch size and number of initiations is advisable if the function is fast for you.

Speed:

library(microbenchmark)
microbenchmark(km_model <- MiniBatchKmeans(z, clusters = 3, batch_size = 20, num_init = 5, max_iters = 100, 
                                           init_fraction = 0.2, initializer = 'kmeans++', early_stop_iter = 10,
                                           verbose = F))

Unit: seconds
                                                                                                                                                                                     expr
 km_model <- MiniBatchKmeans(z, clusters = 3, batch_size = 20, num_init = 5, max_iters = 100, init_fraction = 0.2, initializer = "kmeans++",      early_stop_iter = 10, verbose = F)
      min       lq     mean   median       uq      max neval
 3.338328 3.366573 3.473403 3.444095 3.518813 4.176116   100