Record and processing audio concurrently with Thread in Android

Question

I am writing a small app that capture the audio from the android MIC, performs a FFT on the input and then graphs the chart to the user. I am trying to do the recording and graphing concurrently (obviously with a small delay from being recorded to being graphed). I am attempting to launch two threads, one to read and one to process. However, I am having synchronization issues when I process it seems to only be receiving (or not) zeros. Any advise would be greatly appreciated. :)

public class Plotter extends Activity {

/* plotting objects */
private static GraphicalView mView;
private LineGraph line = new LineGraph();

private boolean recordAudio = true; // record?
private AudioRecord mRecorder = null; // audio object
private Menu mMenu; // app menu

private static final String LOG_TAG = "Frequency Plotter"; // debug tag

private Mfft mfft = null; // FFT class
private static final int BUF_SIZE = 8192; // amount to read in

private Thread listener = null;
private Thread processor = null;

Stack<Float> items = new Stack<Float>();

/* colors for line */
private int[] colors = {Color.BLUE,Color.CYAN,Color.DKGRAY,Color.GRAY,
        Color.GREEN,Color.LTGRAY,Color.MAGENTA,Color.RED,Color.WHITE,Color.YELLOW};

private void processAudio(){

    ArrayList<Double> real = new ArrayList<Double>();

    try{

        Random randomGenerator = new Random();
        float[] in = new float[2048];
        Arrays.fill(in,1);

        while(true){

            synchronized(items){

                while(items.size() < 2048)
                    items.wait();

                items.notifyAll();

                for(int i=0; i < 2048; i++){

                    in[i] = items.pop();    
                }
            }

            double[] ret = mfft.fft(2048,44100,in); // get FFT of data
            TimeSeries dataset = new TimeSeries( (real.size()+1)/2048 + "" ); 
            XYSeriesRenderer renderer = new XYSeriesRenderer(); // customized renderer

            // Customization time
            renderer.setColor(colors[randomGenerator.nextInt(10)]);
            renderer.setPointStyle(PointStyle.SQUARE);
            renderer.setFillPoints(true);
            line.addRenderer(renderer); // add custom renderer

            for(int i = 0; i < 2048; i++){
                real.add(ret[i]);
                dataset.add(real.size()-1,ret[i]); // Add it to our graph
            }

            line.addDataset(dataset); // add data to line
            mView.repaint(); // render lines
        }

    }catch(Exception e){
        Log.e(LOG_TAG, e + " ");
    }
}

private void writeToBuffer(short[] in) {

    synchronized(items){

        for(int i = 0; i < BUF_SIZE; i++){ // copy to create float
            items.push((float)in[i]);   
        }
        items.notifyAll();
    }
}

private void listen(){

    final short[] in = new short[BUF_SIZE];
    mRecorder = new AudioRecord(
            MediaRecorder.AudioSource.MIC, // source
            44100, // frequency (HERTZ)
            AudioFormat.CHANNEL_IN_MONO, // channel
            AudioFormat.ENCODING_PCM_16BIT, // format
            BUF_SIZE // size data packet
            );

    mRecorder.startRecording();

    while(recordAudio){
        try{    
            /* read next part */
            mRecorder.read(in,0,BUF_SIZE); // read from device
            writeToBuffer(in);

        }catch(Exception t){
            /* something went horribly wrong!!!*/
            recordAudio = false;
            Log.e(LOG_TAG, "Failure reading" + t.getMessage());
        }   
    }
}

private void startRecording(){

    /* create a new thread that will run the recording in the background */

    listener = new Thread(
        new Runnable(){

            public void run(){
                listen();
            }
    });
    listener.start();

    /* small delay to produce */
    try {
         Thread.sleep(100);
    } catch (InterruptedException e1) {
         e1.printStackTrace();
    }

    /* create a thread to process the audio */
    processor = new Thread(
        new Runnable(){
            public void run(){
                processAudio();
            }
    });
    processor.start();
}

private void stopRecording(){

    recordAudio = false;
    mRecorder.stop();
    mRecorder.release();
    mRecorder = null;
}

/** clear the current chart */
private void clearChart(){

    line = new LineGraph();
    this.onStart();
}

/** Called when the activity is first created. */
@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.main);
}

@Override
protected void onStart() {

    super.onStart();

    /* instantiate */
    mfft = new Mfft(); // instance of the FFT class
    mView = line.getView(this); // get the chart view

    /* new horizontal layout */
    LinearLayout ll = new LinearLayout(this);
    ll.setOrientation(LinearLayout.HORIZONTAL);

    ll.addView(mView); // add chart to layout

    setContentView(ll); // set layout
}

@Override
public boolean onOptionsItemSelected(MenuItem item) {

    // Handle item selection

    switch (item.getItemId()) {
        case R.id.record:

            startRecording();

            item.setEnabled(false); // disable start
            mMenu.findItem(R.id.stop).setEnabled(true); // enable stop

            return true;
        case R.id.stop:

            stopRecording();

            item.setEnabled(false);  // disable stop
            mMenu.findItem(R.id.clear).setEnabled(true); // enable stop

            return true;
        case R.id.clear:

            clearChart(); // clear chart

            item.setEnabled(false);  // disable clear
            mMenu.findItem(R.id.record).setEnabled(true); // enable stop

            return true;
        default:
            return super.onOptionsItemSelected(item);
    }

}

@Override
public boolean onCreateOptionsMenu(Menu menu) {

    mMenu = menu;
    MenuInflater inflater = getMenuInflater();
    inflater.inflate(R.menu.my_menu, menu);
    return true;
}
}

Edit: Added full definitions.

Answer 1

• several thoughts...

similar example code, Audalyzer

Unfortunately the author has stopped development on this project, but the source tarball is still available online. In particular note: org.hermit.android.io.AudioReader.java . You read the audio and pass it via a Stack object, this author uses short [] arrays. (still that does not seem like it should be your problem source...) http://code.google.com/p/moonblink/downloads/detail?name=SourceTarball.zip

BUF_SIZE thoughts

Your audio buffer (BUF_SIZE = 8192) feels a bit small. How does that relate to AudioRecord.getMinBufferSize()? I used 2x minBufferSize, and that's without doing any calculations on it (only read/write).

Handler thoughts

I'm still reviewing your code, unclear how your threads communicate. But, your problem sounds like it needs a way for the threads to communicate a Handler.

Below are the links I've been reviewing to grasp how to use Handlers and communicate between threads effectively:

• threads - nice overview of handlers (withOUT looper). w/ code example: com.indy.testing.TestMain.java.MyThread.java http://indyvision.net/2010/02/android-threads-tutorial-part-3/

• threads - ok overview of handlers and loopers http://techtej.blogspot.com/2011/02/android-passing-data-between-main.html

• threads w/ 2way comm. w/ code example: sample.thread.messaging.ThreadMessaging.java http://codinghard.wordpress.com/2009/05/16/android-thread-messaging/