Kinect V2 - Simple Sound Detection C++

This is a very simple post showing an implementation of the detection of a sound, any sound above a threshold, using the Kinect V2 SDK. The sample is written as a C++ XAML Windows Store app. The code uses the usual pattern for retrieving data from the Kinect via the SDK; that is,

Get Sensor

Choose Datasource

Open a Reader on the DataSource

Subscribe to the FrameArrived event on the Reader

Open the Sensor

This looks like this:

1. void MainPage::OnLoaded(Platform::Object ^sender, Windows::UI::Xaml::RoutedEventArgs ^e)
2. {
3.     _kinect = KinectSensor::GetDefault();
4.  
5.     auto audioSource = _kinect->AudioSource;
6.  
7.     // Allocate byte array to hold the audio data
8.     _audioBuffer = ref new Array<byte>(audioSource->SubFrameLengthInBytes);
9.  
10.     _reader = audioSource->OpenReader();
11.  
12.     _reader->FrameArrived += ref new Windows::Foundation::TypedEventHandler<WindowsPreview::Kinect::AudioBeamFrameReader ^, WindowsPreview::Kinect::AudioBeamFrameArrivedEventArgs ^>(this, &MainPage::OnFrameArrived);
13.     _kinect->Open();
14. }

 

The the FrameArrived handler goes something like this:

Get BeamFrame Collection

GetBeamFrame Collection first member and iterate over the SubFrames within

For each valid SubFrame copy out the Audio data

Process the audio data

1. void MainPage::OnFrameArrived(WindowsPreview::Kinect::AudioBeamFrameReader ^sender, WindowsPreview::Kinect::AudioBeamFrameArrivedEventArgs ^args)
2. {
3.     auto beamCollection = args->FrameReference->AcquireBeamFrames();
4.     if (beamCollection == nullptr)
5.         return;
6.     if (beamCollection->Size < 1)
7.         return;
8.  
9.     for (auto subFrame : beamCollection->GetAt(0)->SubFrames)
10.     {
11.         if (subFrame == nullptr)
12.             continue;
13.  
14.         // Continue until timeout exceeded
15.         if (_lastDetected.Duration != 0.0 && (subFrame->RelativeTime.Duration < _lastDetected.Duration + TIMEOUT))
16.             continue;
17.  
18.         subFrame->CopyFrameDataToArray(_audioBuffer);
19.  
20.         double sum = 0;
21.         int stride = sizeof(float);
22.         float sq = 0.0f;
23.         int sampleCounter = 0;
24.  
25.         // Loop over audio data bytes...
26.         for (int i = 0; i < _audioBuffer->Length; i += stride)
27.         {
28.             memcpy(&sq, &_audioBuffer[i], sizeof(float));
29.             sum += sq * sq;
30.             sampleCounter++;
31.             if (sampleCounter < 40)
32.                 continue;
33.             float energy = (float) (10.0 * log10(sum / sampleCounter));
34.             sampleCounter = sum = 0;
35.  
36.             if (energy > ENERGY_THRESHOLD)
37.             {
38.                 _lastDetected = subFrame->RelativeTime;
39.  
40.                 // Sound detected!! Do Stuff....
41.                 break;
42.             }
43.         }
44.     }
45. }

The sample is here https://github.com/peted70/kinectv2-detectsound.git