Tensorflow: Hierarchical Softmax Implementation

Question

I'm currently having text inputs represented by vector, and I want to classify their categories. Because they are multi-level categories, I meant to use Hierarchical Softmax.

Example:

 - Computer Science
     - Machine Learning
     - NLP
 - Economics
 - Maths
     - Algebra
     - Geometry

I don't know how to implement it in Tensorflow. All examples I've met is using other frameworks.

Thanks

Answer 1

Practically if your total number of categories is in the range of hundreds to thousands (less than 50K), you don't need to consider using hierarchical softmax, which is designed to run training faster for classifying into millions of categories (for example, the number of words in a vocabulary).

In my experience (with Naive Bayesian and neural networks), utilizing the hierarchical structure at training time does not necessarily improve your classification quality.

However, if you are interested to implement Hierarchical Softmax anyway, that's another story.

Answer 2

Finally, I have changed to use Pytorch. It's easier and more straight-forward than Tensorflow.

• For anyone further interested in implement HS, you can have a look at my sample instructions: https://gist.github.com/paduvi/588bc95c13e73c1e5110d4308e6291ab

• For anyone still want a Tensorflow implementation, this one is for you: https://github.com/tansey/sdp/blob/87e701c9b0ff3eacab29713cb2c9e7181d5c26aa/tfsdp/models.py#L205. But it's a little messy, and the author recommended using Pytorch or other dynamic graph framework