Java audio visualizer- How to capture real-time sound output to speaker?

Question

tl;dr For future readers, recording real-time audio is not possible (for now) with Java or C#. Use C++, as it provides a plethora of audio api's.

---

My goal is to get the current sound played on a Windows machine, and analyze the sound much like a graphic audio visualizer does (get the volume property and Hz(base and treble)). When I say current sound, I mean if one was to play a Youtube video or Spotify song, and this program would read that audio output. I have NO intention to play sound, but capture it in real-time and visualize it.

In attempting to do so, I read on how to build an audio waveform display and it touches on how to convert an audio file to an array of bytes (a Line). This doesn't help because It wont get the current sound. I also read on how to capture audio as well, and this java accessing sound tutorial, neither of those answer my question because they both require a song file to be loaded.

I'm just not understanding this at all. I'm totally clueless, and any help would be appreciated.

Edit: I did a little more looking around, and the second answer from this source lead me to the conclusion that: I could find all of the audio devices, see which one is producing sound. I don't know what to do after that.

Edit 2 (edited again): From experimenting and looking around, I wrote this code below. I think this gets me in the direction I'm wanting, but I don't know how to finish it.

    Mixer.Info[] mixers = AudioSystem.getMixerInfo();
    for (Mixer.Info mixerInfo : mixers) {
        Mixer mixer = AudioSystem.getMixer(mixerInfo);
        try {
            mixer.open();
            Line.Info[] lines = mixer.getTargetLineInfo();
            for (Line.Info linfo : lines) {

                Line line = AudioSystem.getLine(linfo);

                //here I'm opening the line, but I don't know how to grab data
                line.open();

            }
        } catch (LineUnavailableException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

I used this source: Checking The Level of Audio-Playback in a mixers line, but I'm not looking to check for all the lines that are playing volume, I just need the users default Mixer, get that Line, and be able to analyze the data.

Edit 3: I have tried:

    //creating a format for getting sound
    float sampleRate = 8000;
    int sampleSizeInBits = 16;
    int channels = 2;
    boolean signed = true;
    boolean bigEndian = true;
    AudioFormat format = new AudioFormat(sampleRate, sampleSizeInBits, channels, 
        signed, bigEndian);

    //creating a line based off of the format
    DataLine.Info info = new DataLine.Info( TargetDataLine.class, format);
    TargetDataLine line = (TargetDataLine) AudioSystem.getLine(info);

    //opening and starting that line
    line.open(format);
    line.start();

    while (conditionIsTrue){
        //here, I don't know what to put as the parameters.
        //Had I known, I don't know how I would get to analyze the data
        line.read();
    }

I think I'm on the right path using the code above, but I don't know how to extract the sound and find the bpm, base, treble, etc.

Edit 4: This was an interesting read : Real-time low latency audio processing in Java. This doesn't touch on what classes and how to actually implement this, but it provides some insight.

Edit 5: @AndrewThompson Using this piece of code based off of your link I was able to iterate over the available source and target lines.

Mixer.Info[] mixers = AudioSystem.getMixerInfo();
    for (Mixer.Info mixerInfo : mixers) {
        Mixer mixer = AudioSystem.getMixer(mixerInfo);
        try {
            mixer.open();
            Line.Info[] sourceLines = mixer.getSourceLineInfo();
            Line.Info[] targetLine = mixer.getTargetLineInfo();
            for (Line.Info sourceLinfo : sourceLines) {
                System.out.println(sourceLinfo );
            }
            for (Line.Info targetLinefo : targetLine) {
                System.out.println(targetLinefo);
            }

        } catch (LineUnavailableException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }

The output looks like this:

interface SourceDataLine supporting 8 audio formats, and buffers of at least 32 bytes
interface Clip supporting 8 audio formats, and buffers of at least 32 bytes
interface SourceDataLine supporting 8 audio formats, and buffers of at least 32 bytes
interface Clip supporting 8 audio formats, and buffers of at least 32 bytes
interface SourceDataLine supporting 8 audio formats, and buffers of at least 32 bytes
interface Clip supporting 8 audio formats, and buffers of at least 32 bytes
HEADPHONE target port
SPEAKER target port

I have then created a method that gets the sound levels of all the lines which looks like this:

private static void getVolumeOfAllLines() {
    Mixer.Info[] mixers = AudioSystem.getMixerInfo();
    for (Mixer.Info mixerInfo : mixers) {
        Mixer mixer = AudioSystem.getMixer(mixerInfo);
        try {
            mixer.open();
            Line.Info[] lines = mixer.getSourceLineInfo();
            for (Line.Info linfo : lines) {
                DataLine line = (DataLine)AudioSystem.getLine(linfo);
                if(line != null)
                    System.out.println(line.getLevel());
            }
        } catch (LineUnavailableException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
}

-in attemps to find the current line playing sound, indicating a higher volume. This returns:

-1.0
-1.0
-1.0
-1.0
-1.0
-1.0

No progress.

---

New Code:

    private static void debug(){
    Mixer.Info[] mixers = AudioSystem.getMixerInfo();
    for (Mixer.Info mixerInfo : mixers) {
        Mixer mixer = AudioSystem.getMixer(mixerInfo);
        try {
            mixer.open();
            Line.Info[] lines = mixer.getTargetLineInfo();


            AudioFormat format = new AudioFormat(
                    AudioFormat.Encoding.PCM_SIGNED,
                    44100,
                    16, 2, 4,
                    44100, false);

            AudioFormat[] tdl = AudioSystem.getTargetFormats(AudioFormat.Encoding.PCM_SIGNED, format);

            for (Line.Info linfo : lines) {

                //Line line = AudioSystem.getLine(linfo);


                TargetDataLine line = null;
                DataLine.Info info = new DataLine.Info(TargetDataLine.class,
                        format); // format is an AudioFormat object
                if (!AudioSystem.isLineSupported(info))
                {
                    System.out.println("line not supported:" + line );
                }

                try
                {
                    line = (TargetDataLine) AudioSystem.getLine(info); //error
                    line.open(format);
                    System.out.println("line opened:" + line);

                    line.start();

                    byte[] buffer = new byte[1024];
                    int ii = 0;
                    int numBytesRead = 0;
                    while (ii++ < 100) {
                        // Read the next chunk of data from the TargetDataLine.
                        numBytesRead =  line.read(buffer, 0, buffer.length);

                        System.out.println("\nnumBytesRead:" + numBytesRead);
                        if (numBytesRead == 0) continue;
                        // following is a quickie test to see if content is only 0 vals
                        // present in the data that was read.

                        for (int i = 0; i < 16; i++)
                        {
                            if (buffer[i] != 0)
                                System.out.print(".");
                            else
                                System.out.print("0");
                        }
                    }

                } catch (LineUnavailableException ex) {
                    ex.printStackTrace();
                    //...
                }
            }
        } catch (LineUnavailableException e) {
            e.printStackTrace();
        }
    }
}

Answer 1

There is a good example from the Java tutorials that will help you to extract the PCM data from a line. In the tutorial titled Using Files and Format Converters there is a code example under the section heading "Reading Sound Files". The relevant portion is the "snippet" example, and is marked by the code:

  // Here, do something useful with the audio data that's 
  // now in the audioBytes array...

At this point, you hve access to the individual bytes of the line, and can take and assemble them into PCM according to the format of the sound file. There are several other stackoverflow questions which deal with the specifics of going to and from bytes to PCM.

---

Am adding some code in response to comments.

Per not being able to cast to TargetDataLine, the following, extracted from the tutorial, allowed me to cast a line to TargetDataLine.

    AudioFormat format = new AudioFormat(
        AudioFormat.Encoding.PCM_SIGNED, 
        44100,
        16, 2, 4, 
        44100, false);

    TargetDataLine line = null;
    DataLine.Info info = new DataLine.Info(TargetDataLine.class, 
        format); // format is an AudioFormat object
    if (!AudioSystem.isLineSupported(info)) 
    {
        System.out.println("line not supported:" + line );
    }

    try 
    {
        line = (TargetDataLine) AudioSystem.getLine(info);
        line.open(format);
        System.out.println("line opened:" + line);

        line.start();

        byte[] buffer = new byte[1024];
        int ii = 0;
        int numBytesRead = 0;
        while (ii++ < 100) {
       // Read the next chunk of data from the TargetDataLine.
            numBytesRead =  line.read(buffer, 0, buffer.length);

            System.out.println("\nnumBytesRead:" + numBytesRead);
               if (numBytesRead == 0) continue;
     // following is a quickie test to see if content is only 0 vals
    // present in the data that was read.

               for (int i = 0; i < 16; i++)
               {
                   if (buffer[i] != 0)
                       System.out.print(".");
                   else
                       System.out.print("0");
               }
            }

        } catch (LineUnavailableException ex) {
            ex.printStackTrace();
        //... 
    }
}

But I'm just grabbing a line using a CD quality format scheme, I haven't tried to figure out which line has the sound from the YouTube channel that is playing.

---

OP and I went to chat and continued to hack on this, but were unable to work through to a solution. Seems many others have looked at this and given up, too. I'm hoping the bounty proves tempting -- this is an interesting issue.

Answer 2

There is no good solution using java. It's better to use jni to access different os hardware.

In windows, NAudio is a good choice. I follow its demo-Record Soundcard Output with WasapiLoopbackCapture ,compile a console exe and used it in Runtime.getRuntime().exec

Answer 3

The thing that worked for me was to use Virtual Audio Cable. It creates a virtual input output duo where the output feeds the audio data to input and then you can easily get the data using Target data Line. Nothing else worked since it's a hardware limitation not software so we have to emulate virtual hardware in order to capture the output in realtime.

https://vb-audio.com/Cable/