tensorflowjs - Is there an equivalent method for tokenizer in javascript?

Question

I'm building an NLP classifier in python and would like to build a hosting HTML page for a demo. I want to test on a sample text to see the prediction and this is implemented in python through tokenizing the text and then padding it before predicting. Like this:

tf.tokenizer.texts_to_sequences(text)
token_list = tf.tokenizer.texts_to_sequences([text])[0]
token_list_padded = pad_sequences([token_list], maxlen=max_length, padding=padding_type)

The problem is that I'm new to javascript, so is there tokenization and padding methods in javascript like in python?

Answer 1

There is not yet a tf.tokenizer in js as there is in python.

A simple js.tokenizer has been described here. A more robust approach would be to use the tokenizer that comes with universal sentence encoder