Load audiodata into AudioBufferSourceNode from <audio/> element via createMediaElementSource?

Question

Is it possible to have an audiofile loaded from <audio/>-element via createMediaElementSource and then load the audio data into a AudioBufferSourceNode?

Using the audio-element as a source (MediaElementSource) seems not to be an option, as I want to use Buffer methods like noteOn and noteGrain.

Loading the audiofile directly to the buffer via XHR unfortunately isn't an option neither ( see Open stream_url of a Soundcloud Track via Client-Side XHR?)

Loading the buffer contents from the audio elements seems to be possible though:

http://www.w3.org/2011/audio/wiki/Spec_Differences#Reading_Data_from_a_Media_Element

Or is it even possible to directly use the buffer of an <audio/>-element as a sourceNode?

Answer 1

Seems as if it's not possible to extract the audiobuffer from an MediaElementSourceNode.

see https://groups.google.com/a/chromium.org/forum/?fromgroups#!topic/chromium-html5/HkX1sP8ONKs

Any reply proving me wrong is very welcome!

Answer 2

This is possible. See my post at http://updates.html5rocks.com/2012/02/HTML5-audio-and-the-Web-Audio-API-are-BFFs. There is also a code snippet and example there. There are a few outstanding bugs, but loading an <audio> into the Web Audio API should work as you want.

// Create an <audio> element dynamically.
var audio = new Audio();
audio.src = 'myfile.mp3';
audio.controls = true;
audio.autoplay = true;
document.body.appendChild(audio);

var context = new webkitAudioContext();
var analyser = context.createAnalyser();

// Wait for window.onload to fire. See crbug.com/112368
window.addEventListener('load', function(e) {
  // Our <audio> element will be the audio source.
  var source = context.createMediaElementSource(audio);
  source.connect(analyser);
  analyser.connect(context.destination);

  // ...call requestAnimationFrame() and render the analyser's output to canvas.
}, false);

Answer 3

Today 2020+ it is possible via the audioWorklet Node

https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkletProcessor/AudioWorkletProcessor

Running in the AudioWorkletContext so only that is there you can pass the binary raw data via messages

// test-processor.js
class RecorderWorkletProcessor extends AudioWorkletProcessor {
  constructor (options) {
    super()
    console.log(options.numberOfInputs)
    console.log(options.processorOptions.someUsefulVariable)
  }
  // @ts-ignore 
  process(inputs, output, parameters) {
      /**
      * @type {Float32Array} length 128 Float32Array(128)
      * non-interleaved IEEE754 32-bit linear PCM 
      * with a nominal range between -1 and +1, 
      * with each sample between -1.0 and 1.0.
      * the sample rate depends on the audioContext and is variable
      */
      const inputChannel = inputs[0][0];  //inputChannel Float32Array(128)
      const { postMessage } = this.port;
      postMessage(inputChannel)  // float32Array sent as byte[512] 
      return true; // always do this!
  }
}

main code

const audioContext = new AudioContext()

const audioMediaElement = audioContext.createMediaElementSource(
  /** @type {HTMLAudioElement} */ audio
);

await audioContext.audioWorklet.addModule('test-processor.js')
const recorder = new AudioWorkletNode(audioContext, 'test-processor', 
  {
     processorOptions: {
     someUsefulVariable: new Map([[1, 'one'], [2, 'two']])
  }
});

/**
 *   
 * Objects of these types are designed to hold small audio snippets, 
 * typically less than 45 s. For longer sounds, objects implementing 
 * the MediaElementAudioSourceNode are more suitable. 
 * The buffer contains data in the following format: 
 * non-interleaved IEEE754 32-bit linear PCM (LPCM)
 * with a nominal range between -1 and +1, that is, a 32-bit floating point buffer, 
 * with each sample between -1.0 and 1.0.  
 * @param {ArrayBufferLike|Float32Array} data 
 */
 const convertFloatToAudioBuffer = (data) => {
    const sampleRate = 8000 | audioContext.sampleRate
    const channels = 1;
    const sampleLength = 128 | data.length; // 1sec = sampleRate * 1
    const audioBuffer = audioContext.createBuffer(channels, sampleLength, sampleRate); // Empty Audio
    audioBuffer.copyToChannel(new Float32Array(data), 0); // depending on your processing this could be already a float32array
    return audioBuffer;
}
let startAt = 0
const streamDestination = audioContext.createMediaStreamDestination();
/**
 * Note this is a minimum example it plays only the first sound 
 * it uses the main audio context if it would use a
 * streamDestination = context.createMediaStreamDestination();
 * 
 * @param {ArrayBufferLike|Float32Array} data 
 */
const play = (data) => {
    const audioBufferSourceNoce = audioContext.createBufferSource();  
    audioBufferSourceNoce.buffer = convertFloatToAudioBuffer(data);

    const context = audioContext; // streamDestination; // creates a MediaStream on streamDestination.stream property
    audioBufferSourceNoce.connect(context);    

    // here you will need a hugh enqueue algo that is out of scope for this answer
    startAt = Math.max(context.currentTime, startAt);
    source.start(startAt);
    startAt += buffer.duration;
    audioBufferSourceNoce.start(startAt);
}

// Here is your raw arrayBuffer ev.data
recorder.port.onmessage = (ev) => play(ev.data);

// connect the processor with the source
audioMediaElement.connect(recorder);

Note

This is only the bare minimum to console.log the data that comes from the recorder processor and we take only 1 channel when you really want to process that data you should consider doing that in a worker again register a handler that post the data directly to that worker as else your mainprocess could get unresponsive if you do a lot of processing.