Estimating the Number of Clusters via Normalized Cluster Instability

Abstract

We improve current instability-based methods for the selection of the number of clusters k in cluster analysis by developing a normalized cluster instability measure that corrects for the distribution of cluster sizes, a previously unaccounted driver of cluster instability. We show that our normalized instability measure outperforms current instability-based measures across the whole sequence of possible k and especially overcomes limitations in the context of large k. We also compare, for the first time, model-based and model-free approaches to determine cluster-instability and find their performance to be comparable. We make our method available in the R-package cstab.

Publication
In arXiv
Date