Visualize documents embeddings and clustering

Question

I have the following dataframe:

print(df)
document                embeddings
   1        [-1.1132643 ,  0.793635  ,  0.8664889]
   2        [-1.1132643 ,  0.793635  ,  0.8664889]
   3        [-0.19276126,  -0.48233205,  0.17549737]
   4        [0.2080252 ,  0.01567003, 0.0717131]

I want to cluster and visualize them to see the similarities between the documents. What is the best method/steps to do this?

This is just a small dataframe, the original dataframe has more than 20k documents.

Answer 1

Document vectors in your case reside in a 768-dimensional euclidean space. Meaning in a 768-dimensional coordinate space, each point represents a document. Assuming these have been trained correctly, it's safe to imagine that contextually similar documents should be closer to each other in this space as compared to different ones. This may allow you to apply a clustering method to club similar documents together.

For clustering, you can use multiple clustering techniques such as -

1. Kmeans (clusters based on euclidean distances)
2. Dbscan (clusters with the notion of density)
3. Gaussian mixtures (clusters based on a mixture of k gaussians)

You can use Silhouette score to find the optimal number of clusters for the clustering algorithm to best create separations in clusters.

For visualization, you can ONLY visualize in 3D or 2D space. This means you will have to use some dimensionality reduction methods to reduce the 768 dimensions to 3 dimensions or 2 dimensions.

This can be achieved with the following algorithms set to 2 or 3 components -

1. PCA
2. T-SNE
3. LDA (requires labels)

Once you have clustered the data AND reduced the dimensionality of the data separately, you can use matplotlib to plot each of the points in a 2D/3D space and color each point based on its cluster (0-7) to visualize documents and clusters.

#process flow
(20k,768) -> K-clusters (20k,1) ------|
                                      |--- Visualize (3 axis, k colors)
(20k,768) -> Dim reduction (20k,3)----|

Here is an example of the goal you are trying to achieve -

Here, you see the first 2 components of data from T-SNE, and each color represents the clusters you have created from your clustering method of choice (deciding the number of clusters using silhouette score)

EDIT: You can apply dimensionality reduction to project your 768-dimensional data into a 3D or 2D space and THEN cluster using a clustering method. This would reduce the amount of computation you have to handle since now you are clustering only on 3 dimensions instead of 768, but at cost of information that might help you discriminate clusters better.

#process flow                                 
                                    |------------------------|  
(20k,768) -> Dim reduction (20k,3)--|                        |-- Visualize
                                    |--- K-Clusters (20k,1)--|