Kinect 4 Windows V2 – Unity 3D

After installing the Kinect v2 SDK from here http://www.microsoft.com/en-us/download/details.aspx?id=44561 you can also download the supporting Unity 3D plugins here http://go.microsoft.com/fwlink/?LinkID=513177. Note that the plugins require Unity 3D Pro and they expose APIs for Kinect for Windows core functionality, visual gesture builder and face to Unity apps. The zip file containing the Unity packages also contains two sample scenes Green Screen and Kinect View. Lets take a look at KinectView first:

 

So, I stumbled around a bit on the next step as initially I tried opening the KinectView scene from it’s existing location and this doesn’t seem to work very well. The result I got was that when I examined the game objects there was an error on each of the scripts. After some head-scratching I watched the video here http://channel9.msdn.com/Series/Programming-Kinect-for-Windows-v2/04 and at about 10:54 the presenter shows copying the scene locally to the current project. That fixed the issue for me so I was back up and running. You can plug in your Kinect sensor and run the game and then explore the scripts to see how the data is retrieved from the sensor and applied to GameObjects in your scene.

 

 

My idea was to create a very simple Kinect sample which will move particle systems around following the positions of the users hands. So lets step through what I did.

First, I chose File > New Project to bring up the Unity project wizard:

I added the Visual Studio Tools as I’m more comfortable editing and debugging in Visual Studio. You can download the tools from here https://visualstudiogallery.msdn.microsoft.com/20b80b8c-659b-45ef-96c1-437828fe7cf2.

Once the project is created I imported the Kinect Unity package; Assets > Import Package > Custom Package… I navigated to where I installed the Kinect package and chose Kinect.2.0.1410.19000.unitypackage.

I left everything selected and imported the various libraries and scripts. The editor shows that I have an empty scene just containing a Camera.

The first thing to do was to try to get the Kinect sensor data into the scene, so to do this I created a new c# script in the project and copied the code from the KinectView sample in the file BodySourceManager.cs. The code retrieves the sensor and opens a reader on it for the body data and also reads the current frame’s data in the scripts update() method. The update() method is called as part of the game loop so we are ‘pulling’ the Kinect skeleton data each time we are called. The class also has a method GetData() to allow the data to be accessed by other scripts.

I created an empty GameObject called BodySourceManager and associated the BodySourceManager.cs script with it so the code to set up the Kinect would be executed. Next, I created another empty GameObject and associated a new script which would get the body data and position the GameObject depending on the position of one of the joints in the skeleton. I made the joint type a variable so I could set up the positions of various GameObjects via the Unity editor. The intention was that if I then add a particle system as a child of this new GameObject then it would follow the tracked position of say, the left hand and I could add others for each joint that I wanted to track.

The code for the update loop showing how the game object position is set from the joint. 

1. void Update ()
2. {
3.     if (_bodySourceManager == null)
4.     {
5.         return;
6.     }
7.  
8.     _bodyManager = _bodySourceManager.GetComponent<BodyManager>();
9.     if (_bodyManager == null)
10.     {
11.         return;
12.     }
13.  
14.     Body[] data = _bodyManager.GetData();
15.     if (data == null)
16.     {
17.         return;
18.     }
19.  
20.     // get the first tracked body...
21.     foreach (var body in data)
22.     {
23.         if (body == null)
24.         {
25.             continue;
26.         }
27.  
28.         if (body.IsTracked)
29.         {
30.             var pos = body.Joints[_jointType].Position;
31.            this.gameObject.transform.position = new Vector3(pos.X, pos.Y, pos.Z);
32.             break;
33.         }
34.     }
35. }

The next step was to add two new GameObjects, one for each hand and set their joint types to HandLeft and HandRight and then add particle system’s to each as a child. Then the particle systems had their parameters tweaked until they looked how I wanted. It seems to me that the combination of Unity and Kinect is a pretty powerful one and I’m looking forward to seeing the results in the Windows Store. Please find the sample project here http://1drv.ms/1tyKlIT.