Using custom beam scorer in TensorFlow CTC (language model)

Question

Is it possible to customize beam scorer in TensorFlow CTC implementation from Python side? I see this possibility in comment for CTCBeamSearchDecoder C++ class constructor but wonder how to provide this functionality for Python users?

Specific issue that we have is the plugging of language model into CTC based speech decoder. Language model can possibly be a pre-trained TensorFlow sub-graph, capable of outputting probabilities for beam score adjustment. But we need a way to inject this into beam scorer.

Answer 1

There's currently no API for Python to use language model with a custom scorer. Contributions are welcome, but there's some difficulty in making this possible in the Python API as it would require running the TF LM sub-graph in an independent session inside the decoder op, and those wouldn't blend nicely together.

The easiest way of doing this is in C++ and requires extending the BaseBeamScorer class along with a BeamState (similar to what can be seen in tests) and further run CTCBeamSearchDecoder::Decode on top of the outputs from the tensorflow graph that would normally go in the ctc_beam_search_decoder op.

By doing this, your BeamScorer implementation could make use of any language model you have at hand and simply needs to return the appropriate scores when expanding the beam from one state to another.