Real time speech recognition using WebRTC, Node.js and speech recognition engine

Question

A. What I am trying to implement.

A web application allowing real-time speech recognition inside web browser (like this).

B. Technologies I am currently thinking of using to achieve A.

• JavaScript
• Node.js
• WebRTC
• Microsoft Speech API or Pocketsphinx.js or something else (cannot use Web Speech API)

C. Very basic workflow

1. Web browser establishes connection to Node server (server acts as a signaling server and also serves static files)
2. Web browser acquires audio stream using getUserMedia() and sends user's voice to Node server
3. Node server passes audio stream being received to speech recognition engine for analysis
4. Speech recognition engine returns result to Node server
5. Node server sends text result back to initiating web browser
6. (Node server performs step 1 to 5 to process requests from other browsers)

D. Questions

1. Would Node.js be suitable to achieve C?
2. How could I pass received audio streams from my Node server to a speech recognition engine running separately from the server?
3. Could my speech recognition engine be running as another Node application (if I use Pocketsphinx)? So my Node server communicates to my Node speech recognition server.

Answer 1

Would Node.js be suitable to achieve C?

Yes, though there are no hard requirements for that. Some people are running servers with gstreamer, for example check

http://kaljurand.github.io/dictate.js/

node should be fine too.

How could I pass received audio streams from my Node server to a speech recognition engine running separately from the server?

There are many ways for node-to-node communication. One of them is http://socket.io. There are also plain sockets. The particular framework depends on your requirements for fault-tolerance and scalability.

Could my speech recognition engine be running as another Node application (if I use Pocketsphinx)? So my Node server communicates to my Node speech recognition server.

Yes, sure. You can create a node module to warp pocketsphinx API.

UPDATE: check this, it should be similar to what you need:

http://github.com/cmusphinx/node-pocketsphinx