Run script based on sound VU level in vb.net

Question

I'm designing an application to farm one of my games for me. However, The approach I'm going with is to listen for a sound or spike in sound levels before running a desired macro combination. The audio I'm looking to monitor is the current PC's audio output, not a microphone or external device.

This subject seem's to be vary vague on tutorials or information, however from my knowledge I found a post here that explains to use a BuckSoft.DirectSound project?

So based on the information I found, I assume you do something like the following?

If AnalogSignalMeter1.LeftLevel > 0 Or AnalogSignalMeter1.RightLevel > 0 Then

    ' Do Something

End If

From personal prospective I would love to help clarify a solution to the public as this topic is not well documented and avoided. I'm open to all solutions or suggestions, however, my focus is on vb.net and will consider C# if needed.

---

Another option that I have seen is the CoreAudio API. I have seen this API used on multiple posts for the ability to extract the current sound levels, however I have not seen examples for reading the current Master VU meter and fader data/Levels.

Private Function GetVol() As Integer        'Function to read current volume setting
    Dim MasterMinimum As Integer = 0
    Dim DevEnum As New MMDeviceEnumerator()
    Dim device As MMDevice = DevEnum.GetDefaultAudioEndpoint(EDataFlow.eRender, ERole.eMultimedia)
    Dim Vol As Integer = 0

    With device.AudioEndpointVolume
        Vol = CInt(.MasterVolumeLevelScalar * 100)
        If Vol < MasterMinimum Then
            Vol = MasterMinimum / 100.0F
        End If
    End With
    Return Vol
End Function

Data I'm looking to gather:

As showed bellow, I'm NOT looking to gather the physical sound level % but rather looking to run actions if the VU levels spike from 0. This means if I play a video or sound file, the application will hear a sound coming from the current work station and perform a desired action.

Bellow will be my rough example of how I plan to use or collect data from my prospective. Using an timer within vb.net I can have an statement consistently looking for an change in "VUSoundLevels" (Not a real statement) and run a script when an change/input happens.

Private Sub Timer1_Tick()

    If VUSoundLevels > 0 Then

        ' Run Code & Exit Loop

    End IF

End Sub

Answer 1

Here is a C# solution that uses Windows Core Audio Library

This API has the notion of Session (which kinda corresponds to what you see in the Volume Mixer). So, I've provided an AudioSession wrapper class that can give you various information on all current sessions in Windows, with details like the session process id and possibly it's name, display name, icon, etc.

This class also has a GetChannelsPeakValues() methods which uses the IAudioMeterInformation interface to get peak value for each audio channel.

Here is a C# console app example (but the AudioSession class supports any UI technology) that, when ran, will display the peak values for each channel of the chrome browser instance (run some video or sound from the browser and the numbers should start moving). If you don't have chrome, use another process of your choice.

class Program
{
    static void Main(string[] args)
    {
        // Here, I'm just monitoring chrome, one of the possible sessions.
        // If you want the whole speakers peak values, use the AudioSession.GetSpeakersChannelsPeakValues() method
        foreach (var session in AudioSession.EnumerateAll())
        {
            if (session.Process?.ProcessName == "chrome")
            {
                do
                {
                    var values = session.GetChannelsPeakValues();
                    if (values.Length == 0)
                        continue;

                    Console.WriteLine(string.Join(" ", values.Select(v => v.ToString("00%"))));
                }
                while (true);
            }
            session.Dispose();
        }
    }
}

And here is the support C# code:

public class AudioSession : IDisposable
{
    private readonly Lazy<Icon> _icon;
    private readonly Lazy<Process> _process;
    private readonly IAudioSessionControl2 _control;

    public AudioSession(IAudioSessionControl2 control)
    {
        _control = control;
        control.GetState(out var state);
        State = state;
        control.GetGroupingParam(out var guid);
        GroupingParam = guid;
        IconPath = GetString(control.GetIconPath);
        DisplayName = GetString(control.GetDisplayName);
        _icon = new Lazy<Icon>(GetIcon, true);
        _process = new Lazy<Process>(() => Process.GetProcessById(ProcessId), true);
        Id = GetString(control.GetSessionIdentifier);
        InstanceId = GetString(control.GetSessionInstanceIdentifier);
        control.GetProcessId(out var pid);
        ProcessId = pid;
        IsSystemSounds = control.IsSystemSoundsSession() == 0;
    }

    public AudioSessionState State { get; }
    public string IconPath { get; }
    public string DisplayName { get; }
    public Guid GroupingParam { get; }
    public Icon Icon => _icon.Value;
    public string Id { get; }
    public string InstanceId { get; }
    public int ProcessId { get; }
    public Process Process => _process.Value;
    public bool IsSystemSounds { get; }

    public float[] GetChannelsPeakValues()
    {
        var meter = (IAudioMeterInformation)_control;
        meter.GetMeteringChannelCount(out var channelCount);
        var values = new float[channelCount];
        meter.GetChannelsPeakValues(channelCount, values);
        return values;
    }

    private delegate int GetStringFn(out IntPtr ptr);
    private static string GetString(GetStringFn fn)
    {
        fn(out var ptr);
        if (ptr == IntPtr.Zero)
            return null;

        try
        {
            var s = Marshal.PtrToStringUni(ptr);
            if (!string.IsNullOrWhiteSpace(s) && s.StartsWith("@"))
            {
                var sb = new StringBuilder(256);
                if (SHLoadIndirectString(s, sb, sb.Capacity, IntPtr.Zero) == 0)
                {
                    s = sb.ToString();
                }
            }
            return s;
        }
        finally
        {
            Marshal.FreeCoTaskMem(ptr);
        }
    }

    private Icon GetIcon()
    {
        if (string.IsNullOrWhiteSpace(IconPath))
            return null;

        var index = ParseIconLocationPath(IconPath, out var path);
        // note this may only work if the OS bitness is the same as this process bitness
        var hIcon = ExtractIcon(IntPtr.Zero, path, index);
        return hIcon == IntPtr.Zero ? null : Icon.FromHandle(hIcon);
    }

    public override string ToString() => DisplayName;
    public void Dispose() => _icon.Value?.Dispose();

    public static float[] GetSpeakersChannelsPeakValues()
    {
        // get the speakers (1st render + multimedia) device
        var deviceEnumerator = (IMMDeviceEnumerator)(new MMDeviceEnumerator());
        deviceEnumerator.GetDefaultAudioEndpoint(EDataFlow.eRender, ERole.eMultimedia, out IMMDevice speakers);
        if (speakers == null)
            return new float[0];

        // get meter information
        speakers.Activate(typeof(IAudioMeterInformation).GUID, 0, IntPtr.Zero, out object o);
        var meter = (IAudioMeterInformation)o;
        if (meter == null)
            return new float[0];

        meter.GetMeteringChannelCount(out var count);
        if (count == 0)
            return new float[0];

        var values = new float[count];
        meter.GetChannelsPeakValues(count, values);
        return values;
    }

    public static IEnumerable<AudioSession> EnumerateAll()
    {
        // get the speakers (1st render + multimedia) device
        var deviceEnumerator = (IMMDeviceEnumerator)(new MMDeviceEnumerator());
        deviceEnumerator.GetDefaultAudioEndpoint(EDataFlow.eRender, ERole.eMultimedia, out IMMDevice speakers);
        if (speakers == null)
            yield break;

        // activate the session manager, we need the enumerator
        speakers.Activate(typeof(IAudioSessionManager2).GUID, 0, IntPtr.Zero, out object o);
        var sessionManager = (IAudioSessionManager2)o;
        if (sessionManager == null)
            yield break;

        // enumerate sessions for on this device
        sessionManager.GetSessionEnumerator(out IAudioSessionEnumerator sessionEnumerator);
        sessionEnumerator.GetCount(out int count);

        for (int i = 0; i < count; i++)
        {
            sessionEnumerator.GetSession(i, out var sessionControl);
            if (sessionControl != null)
            {
                var meter = sessionControl as IAudioMeterInformation;
                yield return new AudioSession(sessionControl);
            }
        }
    }

    [DllImport("shlwapi", CharSet = CharSet.Unicode)]
    private extern static int SHLoadIndirectString(string pszSource, StringBuilder pszOutBuf, int cchOutBuf, IntPtr ppvReserved);

    [DllImport("shlwapi", CharSet = CharSet.Unicode)]
    private static extern int PathParseIconLocation(string pszIconFile);

    [DllImport("shell32", CharSet = CharSet.Unicode)]
    private static extern IntPtr ExtractIcon(IntPtr ptr, string pszExeFileName, int nIconIndex);

    private static int ParseIconLocationPath(string location, out string path)
    {
        if (location == null)
            throw new ArgumentNullException(nameof(location));

        path = string.Copy(location);
        int index = PathParseIconLocation(path);
        int pos = path.LastIndexOf('\0');
        if (pos >= 0)
        {
            path = path.Substring(0, pos);
        }

        if (path.StartsWith("@"))
        {
            path = path.Substring(1);
        }
        return index;
    }
}

[ComImport]
[Guid("BCDE0395-E52F-467C-8E3D-C4579291692E")]
public class MMDeviceEnumerator
{
}

public enum EDataFlow
{
    eRender,
    eCapture,
    eAll,
    EDataFlow_enum_count
}

public enum ERole
{
    eConsole,
    eMultimedia,
    eCommunications,
    ERole_enum_count
}

[Guid("a95664d2-9614-4f35-a746-de8db63617e6"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public partial interface IMMDeviceEnumerator
{
    [PreserveSig]
    int EnumAudioEndpoints(EDataFlow dataFlow, uint dwStateMask, out IntPtr ppDevices);

    [PreserveSig]
    int GetDefaultAudioEndpoint(EDataFlow dataFlow, ERole role, out IMMDevice ppEndpoint);

    [PreserveSig]
    int GetDevice([MarshalAs(UnmanagedType.LPWStr)] string pwstrId, out IMMDevice ppDevice);

    [PreserveSig]
    int RegisterEndpointNotificationCallback(IntPtr pClient);

    [PreserveSig]
    int UnregisterEndpointNotificationCallback(IntPtr pClient);
}

[Guid("d666063f-1587-4e43-81f1-b948e807363f"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public partial interface IMMDevice
{
    [PreserveSig]
    int Activate([MarshalAs(UnmanagedType.LPStruct)] Guid iid, uint dwClsCtx, [In, Out] IntPtr pActivationParams, [MarshalAs(UnmanagedType.IUnknown)] out object ppInterface);

    [PreserveSig]
    int OpenPropertyStore(uint stgmAccess, out IntPtr ppProperties);

    [PreserveSig]
    int GetId(out IntPtr ppstrId);

    [PreserveSig]
    int GetState(out uint pdwState);
}

[Guid("C02216F6-8C67-4B5B-9D00-D008E73E0064"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IAudioMeterInformation
{
    [PreserveSig]
    int GetPeakValue(out float pfPeak);

    [PreserveSig]
    int GetMeteringChannelCount(out int pnChannelCount);

    [PreserveSig]
    int GetChannelsPeakValues(int u32ChannelCount, [Out, MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 0)] float[] afPeakValues);

    [PreserveSig]
    int QueryHardwareSupport(out int pdwHardwareSupportMask);
}

[Guid("77aa99a0-1bd6-484f-8bc7-2c654c9a9b6f"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public partial interface IAudioSessionManager2
{
    // IAudioSessionManager
    [PreserveSig]
    int GetAudioSessionControl(IntPtr AudioSessionGuid, uint StreamFlags, out IAudioSessionControl2 SessionControl);

    [PreserveSig]
    int GetSimpleAudioVolume(IntPtr AudioSessionGuid, uint StreamFlags, out IntPtr AudioVolume);

    // IAudioSessionManager2
    [PreserveSig]
    int GetSessionEnumerator(out IAudioSessionEnumerator SessionEnum);

    [PreserveSig]
    int RegisterSessionNotification(IntPtr SessionNotification);

    [PreserveSig]
    int UnregisterSessionNotification(IntPtr SessionNotification);

    [PreserveSig]
    int RegisterDuckNotification([MarshalAs(UnmanagedType.LPWStr)] string sessionID, IntPtr duckNotification);

    [PreserveSig]
    int UnregisterDuckNotification(IntPtr duckNotification);
}

[Guid("E2F5BB11-0570-40CA-ACDD-3AA01277DEE8"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IAudioSessionEnumerator
{
    [PreserveSig]
    int GetCount(out int SessionCount);

    [PreserveSig]
    int GetSession(int SessionCount, out IAudioSessionControl2 Session);
}

public enum AudioSessionState
{
    AudioSessionStateInactive = 0,
    AudioSessionStateActive = 1,
    AudioSessionStateExpired = 2,
}

[Guid("bfb7ff88-7239-4fc9-8fa2-07c950be9c6d"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public partial interface IAudioSessionControl2
{
    // IAudioSessionControl
    [PreserveSig]
    int GetState(out AudioSessionState pRetVal);

    [PreserveSig]
    int GetDisplayName(out IntPtr pRetVal);

    [PreserveSig]
    int SetDisplayName([MarshalAs(UnmanagedType.LPWStr)] string Value, [MarshalAs(UnmanagedType.LPStruct)] Guid EventContext);

    [PreserveSig]
    int GetIconPath(out IntPtr pRetVal);

    [PreserveSig]
    int SetIconPath([MarshalAs(UnmanagedType.LPWStr)] string Value, [MarshalAs(UnmanagedType.LPStruct)] Guid EventContext);

    [PreserveSig]
    int GetGroupingParam(out Guid pRetVal);

    [PreserveSig]
    int SetGroupingParam([MarshalAs(UnmanagedType.LPStruct)] Guid Override, [MarshalAs(UnmanagedType.LPStruct)] Guid EventContext);

    [PreserveSig]
    int RegisterAudioSessionNotification(IntPtr NewNotifications);

    [PreserveSig]
    int UnregisterAudioSessionNotification(IntPtr NewNotifications);

    // IAudioSessionControl2
    [PreserveSig]
    int GetSessionIdentifier(out IntPtr pRetVal);

    [PreserveSig]
    int GetSessionInstanceIdentifier(out IntPtr pRetVal);

    [PreserveSig]
    int GetProcessId(out int pRetVal);

    [PreserveSig]
    int IsSystemSoundsSession();

    [PreserveSig]
    int SetDuckingPreference(bool optOut);
}

Answer 2

The KEY to this (my solution) working is that you have your VOLUME MIXER showing while this form is running.

This may seem a little out there, but it may work and is A LOT easier than trying to tap into the sound card information. What I propose is to use color identification and some good ol win32 api's. First, you will need to show your Volume mixer (or sound settings) like I did here (I am going to use the Volume Mixer for this example and focus on the Amazon Music control):

Then, using the following code (Module and Form):

Imports System.Runtime.InteropServices

Module Module1

    <DllImport("user32.dll")>
    Private Function GetDC(ByVal hwnd As IntPtr) As IntPtr
    End Function

    <DllImport("user32.dll")>
    Private Function ReleaseDC(ByVal hwnd As IntPtr, ByVal hdc As IntPtr) As Int32
    End Function

    <DllImport("gdi32.dll")>
    Private Function GetPixel(ByVal hdc As IntPtr, ByVal nXPos As Integer, ByVal nYPos As Integer) As UInteger
    End Function

    Public Function GetPixelColor(ByVal x As Integer, ByVal y As Integer) As System.Drawing.Color
        Dim hdc As IntPtr = GetDC(IntPtr.Zero)
        Dim pixel As UInteger = GetPixel(hdc, x, y)
        Dim clr As Color

        ReleaseDC(IntPtr.Zero, hdc)

        clr = Color.FromArgb(255, (pixel And &HFF), (pixel And &HFF00) >> 8,
                                (pixel And &HFF0000) >> 16)


        Return clr
    End Function

End Module  

And Here is the form code (Simple):

Public Class Form1   

    Const c_blnDebug as Boolean = True  
    'Make false to run your program with settings  
    'Make true to get location and colors  

    Const c_intRedThresh As Integer = 90  
    'Threshold color must be less than or equal to this  

    Const c_intGreenThresh As Integer = 170  
    'Threshold color must be greater than or equal to this  

    Const c_intBlueThresh As Integer = 90  
    'Threshold color must be less than or equal to this  


    Private Sub Timer1_Tick(sender As Object, e As EventArgs) Handles Timer1.Tick  
        Dim ptArr() As Point  
        Dim intI As Integer  
        Dim clrTemp As Color  
        Dim intRed As Integer  
        Dim intGreen As Integer  
        Dim intBlue As Integer  

        'Set the pixel locations to watch if NOT DEBUGGING  
        ReDim ptArr(0 To 2)  
        'at source level  
        ptArr(0).X = 1762  
        ptArr(0).Y = 870  
        '-1 pixel  
        ptArr(1).X = 1762  
        ptArr(1).Y = 869  
        '+1 pixel  
        ptArr(2).X = 1762  
        ptArr(2).Y = 871  

        If c_blnDebug Then
            Debug.Print(GetPixelColor(MousePosition.X, MousePosition.Y).ToString & vbCrLf &
                    "X: " & MousePosition.X & vbCrLf & "Y: " & MousePosition.Y)
        Else
            For intI = 0 To 2
                clrTemp = GetPixelColor(ptArr(intI).X, ptArr(intI).Y)
                intRed = clrTemp.R
                intGreen = clrTemp.G
                intBlue = clrTemp.B

                If ((intRed < c_intRedThresh) And
                    (intGreen >= c_intGreenThresh) And
                    (intBlue <= c_intBlueThresh)) Then
                    'Sound Spike do your business
                    Debug.Print("Found Spike @ " & Now)
                    Exit For
                End If
            Next intI
        End If
    End Sub

    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
        If c_blnDebug Then
            Timer1.Interval = 1000  '1 second
        Else
            Timer1.Interval = 250   '0.25 seconds
        End If
        Timer1.Enabled = True
    End Sub
End Class

Can then get you the mouse position that you need to watch as well as the colors (roughly) as shown in my debug.print results:

Color [A=255, R=51, G=190, B=51]
X: 1762
Y: 870
Color [A=255, R=51, G=190, B=51]
X: 1762
Y: 870
Color [A=255, R=51, G=191, B=51]
X: 1762
Y: 870
Color [A=255, R=51, G=188, B=51]
X: 1762
Y: 870
Color [A=255, R=51, G=195, B=51]
X: 1762
Y: 870
Color [A=255, R=232, G=17, B=35]
X: 1491
Y: 646

Therefore, what I have chosen is to watch the mouse position coordinates +/- (1) pixel. For (Red < 90), (Green > 170), and (Blue < 90) as a threshold.

Once this is done I can simply set the timer to every 250 milliseconds and check for the colors. This works like a charm and is super simple!

I hope this helps, I get the following results!

Found Spike @ 2/5/2019 10:16:14 AM
Found Spike @ 2/5/2019 10:16:17 AM
Found Spike @ 2/5/2019 10:16:17 AM
Found Spike @ 2/5/2019 10:16:18 AM
Found Spike @ 2/5/2019 10:16:19 AM
Found Spike @ 2/5/2019 10:16:19 AM
Found Spike @ 2/5/2019 10:16:21 AM
Found Spike @ 2/5/2019 10:16:21 AM
Found Spike @ 2/5/2019 10:16:21 AM
Found Spike @ 2/5/2019 10:16:21 AM
Found Spike @ 2/5/2019 10:16:23 AM

And, here is a screen shot (above code does not reflect this, only bare bones approach) of the Final Product (or at least until I decide to add more):