Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there a way to stop Web Audio API decodeAudioData method memory leak?

The Problem

When creating audio buffers using the Web Audio API, there are buffers created by the decodeAudioData method, which reside in memory and are apparently not accessible through JavaScript. They seem to hang around for the entire life of a browser tab, and never get garbage collected.

Possible Reason For the Problem

I know that these buffers are separated from the main thread and set on another thread for asynchronous decoding. I also know that the API spec says that decodeAudioData should not be allowed to decode the same input buffer twice, which I assume is why a copy of the decoded buffer and/or the encoded input buffer are kept around. However, on memory limited devices like Chromecast, this causes huge amounts of memory to accumulate and Chromecast crashes.

Reproducibility

In my example code, I fetch an mp3 using Ajax and then pass the arraybuffer into the decodeAudioData function. Normally within that function there is a onsuccess callback which can take the decoded AudioBuffer as a parameter. But here in my code, I don't even pass that in. Therefore I also don't do anything with the decoded buffer after decoding it. It is not referenced anywhere within my code. It is entirely left in the native code. However, every call to this function increases the memory allocation and it is never released. For example, in Firefox about:memory shows the audiobuffers there for the life of the Tab. Non-reference should be sufficient for the garbage collector to get rid of these buffers.

My main question then is, is there any reference to these decoded audio buffers, say within the audiocontext object, or somewhere else that I can try to remove them from memory? Or is there any other way that I can cause these stored and unreachable buffers to disappear?

My question differs from all the others currently on SO regarding decodeAudioData because I show that the memory leak happens even without the user storing any reference or even using the returned decoded audio buffer.

Code To Reproduce

function loadBuffer() {
    // create an audio context
    var context = new (window.AudioContext || window.webkitAudioContext)();

    // fetch mp3 as an arraybuffer async
    var url = "beep.mp3";
    var request = new XMLHttpRequest();
    request.open("GET", url, true);
    request.responseType = "arraybuffer";

    request.onload = function () {

        context.decodeAudioData(
                request.response,
                function () {// not even passing buffer into this function as a parameter
                    console.log("just got tiny beep file and did nothing with it, and yet there are audio buffers in memory that never seem to be released or gc'd");
                },
                function (error) {
                    console.error('decodeAudioData error', error);
                }
        );
    };

    request.onerror = function () {
        console.log('error loading mp3');
    }
    request.send();
}

To anticipate some possible responses.

  1. I must use Web Audio API because I am playing four part harmony from four audio files on Chromecast and the html audio element does not support multiple simultaneous playback on Chromecast.
  2. Probably any JS library you may reference [e.g. Howler.js, Tone.js, Amplitude.js etc.] is built upon the Web Audio API, and so they will all share this memory leak problem.
  3. I know that the WAA is implementation dependent on a per browser basis. My primary concern at the moment is Chromecast, but the problem exists for every browser I've tried.
  4. Therefore, I think it is a spec related issue where the spec requires the non-dupe encoding rule, and so implementers keep copies of the buffer around on a browser level thread so they can check them against new xhr inputs. If the spec writer's happen to read my question, is there not a way that the user can have the option for this behavior, and opt out of it if they wish in order to prevent the internal buffer storage on mobile and thin memory platforms?
  5. I have not been able to find any reference to these buffers in any JS object.
  6. I know that I can audio_context.close() and then hope for garbage collection of all the resources held by the audio_context, and then hope that I can reinstantiate the audio_context with a new one, but that has not empirically been timely enough for my application. Chromecast crashes before GC takes out the trash.
like image 784
Scott Doherty Avatar asked Jan 31 '19 16:01

Scott Doherty


Video Answer


1 Answers

Pragmatic Workaround

I have found a method to solve the problem of the Web Audio API audiobuffers handing around indefinitely and crashing Chromecast and other mobile platforms. [[ I have not tested this on all browsers - your mileage may vary. ]]

LOADING STAGE

  1. Load the document using Web Audio API inside an iFrame.
  2. Load your audio buffers and do whatever you do to play them.

CLEARING STAGE

  1. Call sourceNode.stop on all of the playing nodes you have reference to.
  2. Call source.disconnect(); on all source nodes.
  3. Call gainNode.disconnect(); on all gain nodes those source nodes are associated with (and whatever other kind of WAA nodes you might be using that have a disconnect method)
  4. Set all referenced gainNodes and sourceNodes to null;
  5. Null out any buffers you have referenced both decoded and your xhr fetched encoded audiobuffers;
  6. KEY: Within the WAA page call audio_context.close(); then set audio_context=null; (this can be done from the parent of the iFrame using contentWindow).
  7. Note: Some of these nulling steps may not be absolutely necessary, however this approach has worked for me.

RE-LOADING STAGE

  1. Reload the iframe from the parent page. This will cause all of the audiobuffers to be garbage collected ON THE NEXT GC ROUND, including the ones in the hidden (non JS) areas of memory.
  2. Your iframe will have to reinstantiate the web audio context and load its buffers and create nodes etc. just as you did when you first loaded it.

Notes: You must decide when you are going to use this clearing method (e.g. after so many buffers have been loaded and played). You can do it without an iframe, but you may have to reload the page once or twice to get garbage collection to fire. This is a pragmatic workaround for those who need to load lots of Web Audio API audio buffers on memory thin platforms like Chromecast or other mobile devices.

FROM PARENT

  function hack_memory_management() {
                var frame_player = document.getElementById("castFrame");
                //sample is the object which holds an audio_context
               frame_player.contentWindow.sample.clearBuffers();
                 setTimeout(function () {
                    frame_player.contentWindow.location.reload();
                }, 1000);
            }

INSIDE WAA IFRAME

CrossfadeSample.prototype.clearBuffers = function () {
    console.log("CLEARING ALL BUFFERS -IT'S UP TO GC NOW'");
    // I have four of each thing because I am doing four part harmony

    // these are the decoded audiobuffers used to be passed to the source nodes
    this.soprano = null;
    this.alto = null;
    this.tenor = null;
    this.bass = null;
    if (this.ctl1) {

        //these are the control handles which hold a source node and gain node 
        var offName = 'stop';
        this.ctl1.source[offName](0);
        this.ctl2.source[offName](0);
        this.ctl3.source[offName](0);
        this.ctl4.source[offName](0);

        // MAX GARGABE COLLECTION PARANOIA

        //disconnect all source nodes
        this.ctl1.source.disconnect();
        this.ctl2.source.disconnect();
        this.ctl3.source.disconnect();
        this.ctl4.source.disconnect();

        //disconnect all gain nodes
        this.ctl1.gainNode.disconnect();
        this.ctl2.gainNode.disconnect();
        this.ctl3.gainNode.disconnect();
        this.ctl4.gainNode.disconnect();

        // null out all source and gain nodes
        this.ctl1.source = null;
        this.ctl2.source = null;
        this.ctl3.source = null;
        this.ctl4.source = null;

        this.ctl1.gainNode = null;
        this.ctl2.gainNode = null;
        this.ctl3.gainNode = null;
        this.ctl4.gainNode = null;
    }

    // null out the controls
    this.ctl1 = null;
    this.ctl2 = null;
    this.ctl3 = null;
    this.ctl4 = null;

    // close the audio context
    if (this.audio_context) {
        this.audio_context.close();
    }
    // null the audio context
    this.audio_context = null;

};

Update:

Sadly, even this does not reliably work and Chromecast can still crash given a few clear and loads of new mp3s. See "My present solution" elsewhere on this page.

like image 92
Scott Doherty Avatar answered Oct 23 '22 06:10

Scott Doherty