Retrieving the optimal number of clusters in R

Question

I have data for which I want to evaluate the optimal number of clusters according to the Gap statistic.

I read the page on gap statistic in r which gives the following example:

gs.pam.RU <- clusGap(ruspini, FUN = pam1, K.max = 8, B = 500)
gs.pam.RU

When I call gs.pam.RU.Tab, I get

Clustering Gap statistic ["clusGap"].
B=500 simulated reference sets, k = 1..8
 --> Number of clusters (method 'firstSEmax', SE.factor=1): 4
         logW   E.logW         gap     SE.sim
[1,] 7.187997 7.135307 -0.05268985 0.03729363
[2,] 6.628498 6.782815  0.15431689 0.04060489
[3,] 6.261660 6.569910  0.30825062 0.04296625
[4,] 5.692736 6.384584  0.69184777 0.04346588
[5,] 5.580999 6.238587  0.65758835 0.04245465
[6,] 5.500583 6.119701  0.61911779 0.04336084
[7,] 5.394195 6.016255  0.62205988 0.04243363
[8,] 5.320052 5.921086  0.60103416 0.04233645

From which I want to retrieve the number of clusters. But, contrary to the pamk function which enables to get this number easily, I couldn't find a way to get this number using clusGap.

I then tried using the maxSE function, but I have no clue to what the arguments f and SE.f represent or how I can get them from the data matrix.

Any easy way to retrieve this optimal number of clusters?

Answer 1

The answer is in the output:

...
 --> Number of clusters (method 'firstSEmax', SE.factor=1): 4
...

This is the number of clusters producing the maximum value of gap (which is in row 4 of the table).

The arguments to maxSE(...) are the gap and SE.sim, respectively:

with(gs.pam.RU,maxSE(Tab[,"gap"],Tab[,"SE.sim"]))
# [1] 4

It is sometimes useful to plot gap, to see how well differentiated the clustering options are:

plot(gs.pam.RU)
gap.range <- range(gs.pam.RU$Tab[,"gap"])
lines(rep(which.max(gs.pam.RU$Tab[,"gap"]),2),gap.range, col="blue", lty=2)