Pitch detection

I'm currently developing an electron app, which I hope will be able to measure the pitch of guitar input on desktop.

My initial idea is one tone at a time so please let me know if FTT is appropriate.

Edit: per comments it seems that FTT is not great so I'm considering using Harmonic Product Spectrum for example

I don't have too much experience with node.js, but so far I've managed to fork the broken microphone package and tweak it a bit to be able to fetch a wav format data from sox.

This is the actual code that spawns the process and fetches the data (simplified, it actually has a startCapture method which spawns the recording process):

const spawn = require('child_process').spawn;
const PassThrough = require('stream').PassThrough;

const audio = new PassThrough;
const info = new PassThrough;

const recordingProcess = spawn('sox', ['-d', '-t', 'wav', '-p'])
recordingProcess.stdout.pipe(audio);
recordingProcess.stderr.pipe(info);

And in another js file, I listen for the data event:

mic.startCapture({format: 'wav'});
mic.audioStream.on('data', function(data) {
    /* data is Uint8Array[8192] */
});

Ok so I'm getting an array of data which seems to be a good start. I know I should be applying somehow a pitch detection algorithm to start the pitch analysis

Am I going in the right direction? What format should this data be in? How can I use this data for pitch detection?

What is the best pitch detection algorithm for JavaScript?

A compilation of pitch detection algorithms for Javascript. Supports both the browser and node. YIN - The best balance of accuracy and speed, in my experience. Occasionally provides values that are wildly incorrect. AMDF - Slow and only accurate to around +/- 2%, but finds a frequency more consistenly than others.

How to create a pitch volume meter using audio analyzer?

In startUserMedia () function create an analyzer and connect the stream source. Assign createAudioMeter (ctx) to meter and connect with streamNode. Call drawLoop () function to create pitch volume meter.

How to get the pitch volume of audio from a stream?

In startUserMedia () function create an analyzer and connect the stream source. Assign createAudioMeter (ctx) to meter and connect with streamNode. Call drawLoop () function to create pitch volume meter. Get the voice pitch volume and set minimum_volume = 130. You can change the value if you want.

How do I find the pitch of a WAV file?

To find the pitch of a wav file, we can use the wav-decoder library to extract the data into such an array. This assumes you are using an npm-compatible build system, like Webpack or Browserify, and that your target browser supports WebAudio. Ample documentation on WebAudio is available online, especially on Mozilla's MDN.

Since you're getting a buffer with WAV data, you can use the wav-decoder library to parse it, and then feed it to the pitchfinder library to obtain the frequency of the audio.

const Pitchfinder = require('pitchfinder')
const WavDecoder = require('wav-decoder')
const detectPitch = new Pitchfinder.YIN()

const frequency = detectPitch(WavDecoder.decode(data).channelData[0])

However, since you're using Electron, you can also just use the MediaStream Recording API in Chromium.

First of all, this will only work with Electron 1.7+, because it uses Chromium 58, the first version of Chromium to include a fix for a bug which prevented the AudioContext from decoding audio data from the MediaRecorder.

Also, for the purposes of this code, I'll be using ES7 async and await syntax, which should run just fine on Node.js 7.6+ and Electron 1.7+.

So let's assume your index.html for Electron looks like this:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>Frequency Finder</title>
  </head>
  <body>
    <h1>Tuner</h1>

    <div><label for="devices">Device:</label> <select id="devices"></select></div>

    <div>Pitch: <span id="pitch"></span></div>
    <div>Frequency: <span id="frequency"></span></div>

    <div><button id="record" disabled>Record</button></div>
  </body>

  <script>
    require('./renderer.js')
  </script>
</html>

Now let's get to work on the renderer script. First, let's set a few variables we'll be using:

const audioContext = new AudioContext()
const devicesSelect = document.querySelector('#devices')
const pitchText = document.querySelector('#pitch')
const frequencyText = document.querySelector('#frequency')
const recordButton = document.querySelector('#record')
let audioProcessor, mediaRecorder, sourceStream, recording

Alright, now onto the rest of the code. First, let's populate that <select> drop down in the Electron window with all the available audio input devices.

navigator.mediaDevices.enumerateDevices().then(devices => {
  const fragment = document.createDocumentFragment()
  devices.forEach(device => {
    if (device.kind === 'audioinput') {
      const option = document.createElement('option')
      option.textContent = device.label
      option.value = device.deviceId
      fragment.appendChild(option)
    }
  })
  devicesSelect.appendChild(fragment)

  // Run the event listener on the `<select>` element after the input devices
  // have been populated. This way the record button won't remain disabled at
  // start.
  devicesSelect.dispatchEvent(new Event('change'))
})

You'll notice at the end, we call an event that we've set on the <select> element in the Electron window. But, hold on, we never wrote that event handler! Let's add some code above the code we just wrote:

// Runs whenever a different audio input device is selected by the user.
devicesSelect.addEventListener('change', async e => {
  if (e.target.value) {
    if (recording) {
      stop()
    }

    // Retrieve the MediaStream for the selected audio input device.
    sourceStream = await navigator.mediaDevices.getUserMedia({
      audio: {
        deviceId: {
          exact: e.target.value
        }
      }
    })

    // Enable the record button if we have obtained a MediaStream.
    recordButton.disabled = !sourceStream
  }
})

Let's also actually write a handler for the record button, because at this moment it does nothing:

// Runs when the user clicks the record button.
recordButton.addEventListener('click', () => {
  if (recording) {
    stop()
  } else {
    record()
  }
})

Now we display audio devices, let the user select them, and have a record button... but we still have unimplemented functions - record() and stop().

Let's stop right here to make an architectural decision.

We can record the audio, grab the audio data, and analyse it to obtain its pitch, all in renderer.js. However, analysing the data for pitch is an expensive operation. Therefore, it would be good to have the ability to run that operation out-of-process.

Luckily, Electron 1.7 brings in support for web workers with a Node context. Creating a web worker will allow us to run the expensive operation in a different process, so it doesn't block the main process (and UI) while it is running.

So keeping this in mind, let's assume that we will create a web worker in audio-processor.js. We'll get to the implementation later, but we'll assume it accepts a message with an object, {sampleRate, audioData}, where sampleRate is the sample rate, and audioData is a Float32Array which we'll pass to pitchfinder.

Let's also assume that:

If processing of the recording succeeded, the worker returns a message with an object {frequency, key, octave} - an example would be {frequency: 440.0, key: 'A', octave: 4}.
If processing of the recording failed, the worker returns a message with null.

Let's write our record function:

function record () {
  recording = true
  recordButton.textContent = 'Stop recording'

  if (!audioProcessor) {
    audioProcessor = new Worker('audio-processor.js')

    audioProcessor.onmessage = e => {
      if (recording) {
        if (e.data) {
          pitchText.textContent = e.data.key + e.data.octave.toString()
          frequencyText.textContent = e.data.frequency.toFixed(2) + 'Hz'
        } else {
          pitchText.textContent = 'Unknown'
          frequencyText.textContent = ''
        }
      }
    }
  }

  mediaRecorder = new MediaRecorder(sourceStream)

  mediaRecorder.ondataavailable = async e => {
    if (e.data.size !== 0) {
      // Load the blob.
      const response = await fetch(URL.createObjectURL(data))
      const arrayBuffer = await response.arrayBuffer()
      // Decode the audio.
      const audioBuffer = await audioContext.decodeAudioData(arrayBuffer)
      const audioData = audioBuffer.getChannelData(0)
      // Send the audio data to the audio processing worker.
      audioProcessor.postMessage({
        sampleRate: audioBuffer.sampleRate,
        audioData
      })
    }
  }

  mediaRecorder.start()
}

Once we start recording with the MediaRecorder, we won't get our ondataavailable handler called until recording is stopped. This is a good time to write our stop function.

function stop () {
  recording = false
  mediaRecorder.stop()
  recordButton.textContent = 'Record'
}

Now all that's left is to create our worker in audio-processor.js. Let's go ahead and create it.

const Pitchfinder = require('pitchfinder')

// Conversion to pitch from frequency based on technique used at
// https://www.johndcook.com/music_hertz_bark.html

// Lookup array for note names.
const keys = ['C', 'C#', 'D', 'D#', 'E', 'F', 'F#', 'G', 'G#', 'A', 'A#', 'B']

function analyseAudioData ({sampleRate, audioData}) {
  const detectPitch = Pitchfinder.YIN({sampleRate})

  const frequency = detectPitch(audioData)
  if (frequency === null) {
    return null
  }

  // Convert the frequency to a musical pitch.

  // c = 440.0(2^-4.75)
  const c0 = 440.0 * Math.pow(2.0, -4.75)
  // h = round(12log2(f / c))
  const halfStepsBelowMiddleC = Math.round(12.0 * Math.log2(frequency / c0))
  // o = floor(h / 12)
  const octave = Math.floor(halfStepsBelowMiddleC / 12.0)
  const key = keys[Math.floor(halfStepsBelowMiddleC % 12)]

  return {frequency, key, octave}
}

// Analyse data sent to the worker.
onmessage = e => {
  postMessage(analyseAudioData(e.data))
}

Now, if you run this all together... it won't work! Why?

We need to update main.js (or whatever the name of your main script is) so that when the main Electron window gets created, Electron is told to provide Node support in the context of the web worker. Otherwise, that require('pitchfinder') doesn't do what we want it to do.

This is simple, we just need to add nodeIntegrationInWorker: true in the window's webPreferences object. For example:

mainWindow = new BrowserWindow({
  width: 800,
  height: 600,
  webPreferences: {
    nodeIntegrationInWorker: true
  }
})

Now, if you run what you've put together, you'll get a simple Electron app that lets you record a small section of audio, test its pitch, and then display that pitch to screen.

This will work best with small snippets of audio, as the longer the audio, the longer it takes to process.

If you want a more complete example that goes more in depth, such as the ability to listen and return pitch live instead of making the user click record and stop all the time, take a look at the electron-tuner app that I've made. Feel free to look through the source to see how things are done - I've done my best to make sure it is well commented.

Here's a screenshot of it:

Screenshot of electron-tuner

Hopefully all this helps you in your efforts.

Pitch detection - Node.js

Tags:

javascript

node.js

audio

fft

Alvaro

People also ask

1 Answers

Adaline Simonian

Recent Activity

Donate For Us