Visual Gesture Builder – Kinect 4 Windows v2 (code)

My previous post looked into my experimentation with using Visual Gesture Builder to create a gesture builder database (gbd) with a view to incorporating code into an application for detecting those trained gestures. I used the example of a military salute and will now describe the code for using this in a c#/xaml windows store app. I started by using some boilerplate code from a previous post which registers events to receive skeleton data (required for gesture detection) and colour data (to display the colour image from the RGB camera).

 

1. _gestureDatabase = new VisualGestureBuilderDatabase(@"Gestures/salute.gbd");
2. _gestureFrameSource = new VisualGestureBuilderFrameSource(_kinect, 0);
3.  
4. // Add all gestures in the database to the framesource..
5. _gestureFrameSource.AddGestures(_gestureDatabase.AvailableGestures);
6.  
7. foreach (var gesture in _gestureDatabase.AvailableGestures)
8. {
9.     if (gesture.Name == "salute")
10.     {
11.         _salute = gesture;
12.     }
13.     if (gesture.Name == "saluteProgress")
14.     {
15.         _saluteProgress = gesture;
16.     }
17. }
18.  
19. _gestureFrameReader = _gestureFrameSource.OpenReader();
20. _gestureFrameReader.IsPaused = true;
21. _gestureFrameReader.FrameArrived += reader_FrameArrived;

 

This snippet shows the initialisation code for importing the gesture database and registering to receive events when new data arrives.Note that I hold references to Gesture objects so that I can identify these when the data arrives.The two gestures I am interested in are the “salute” gesture which is a discrete gesture and “saluteProgress” which is a continuous gesture and will report progress for the salute. (Note. details of what, how and why for those different gesture types are in my previous post). This code follows the familiar pattern followed by the Kinect SDK of creating a frame source and opening a reader on that source which can be used to register for data events.Since we need a tracking id for gesture detection I have paused the reader until we have a valid one (this will be retrieved from the skeleton data, see code snippet below).

 

1. if (body != null && body.IsTracked)
2. {
3.     _gestureFrameSource.TrackingId = body.TrackingId;
4.     _gestureFrameReader.IsPaused = false;
5.     break;
6. }

 

When the gesture data arrives, info about the current status of each gesture can be pulled out if the frame and in my sample I just display these on simple UI elements.

 

1. void reader_FrameArrived(VisualGestureBuilderFrameReader sender, VisualGestureBuilderFrameArrivedEventArgs args)
2. {
3.     using (var frame = args.FrameReference.AcquireFrame())
4.     {
5.         if (frame != null && frame.DiscreteGestureResults != null)
6.         {
7.             var result = frame.DiscreteGestureResults[_salute];
8.             
9.             if (result.Detected == true)
10.             {
11.                 var progressResult = frame.ContinuousGestureResults[_saluteProgress];
12.                 Progress.Value = progressResult.Progress;
13.             }
14.             else
15.             {
16.                 Progress.Value = 0.0;
17.             }
18.  
19.             GestureText.Text = result.Detected ? "TRUE" : "FALSE";
20.             ConfidenceText.Text = result.Confidence.ToString();
21.         }
22.     }
23. }

 

Here’s a screenshot of the sample in action:

 

The sample project is located here http://1drv.ms/1C4RQGU