Openframeworks + Kinect 4 Windows V2

OpenFrameworks

I am in the process of reviewing various frameworks for visualising my Kinect windows store apps because prototyping ideas Direct3D is a bit labour intensive. My attention has turned towards Openframeworks. This is the official description:

openFrameworks is an open source C++ toolkit designed to assist the creative process by providing a simple and intuitive framework for experimentation.

The project is designed to be a wrapper around other creative open frameworks; here are some it pulls in:

 

• OpenGL, GLEW, GLUT, libtess2 and cairo for graphics
• rtAudio, PortAudio, OpenAL and Kiss FFT or FMOD for audio input, output and analysis
• FreeType for fonts
• FreeImage for image saving and loading
• Quicktime, GStreamer and videoInput for video playback and grabbing
• Poco for a variety of utilities
• OpenCV for computer vision
• Assimp for 3D model loading
</ul>

 

This sounds ideal for my requirements

 

WINDOWS STORE APP

Retrieving the source from here https://github.com/MSOpenTech/openFrameworks instead of usual OpenFrameworks source tree helps as MS Open Tech have done the work to create an OF Project generator which supports Windows Store Applications. This includes the conversion from OpenGL to DirectX and retaining the structure of an OF app so existing apps could be just dropped into place. There is a re-built project generator exe located here \msopentechopenframeworks\openFrameworks\projectGenerator\projectGeneratorSimple_winrt.exe

 



which will generate a Windows Store c++ project with OF set up and ready to go.

 



Let’s go through the items in this project to gain a better understanding of what happens when the app runs.

 

The windows app project is a c++/DirectX windows store app project which makes use of a SwapChainBackgroundPanel which effectively allows you to compose XAML UI elements over a full-screen DirectX rendering context. The source code for OpenFrameworks is included in the solution as a separate project. Project Angle is utilised to carry out the translation from OpenGL to DirectX (see http://msopentech.com/blog/2013/12/20/msopentechcontributestoangle/ for further info about this) as can be seen in the source files AngleBase.h and AngleBase.cpp.

 

Running the app out of the box gives (note I am running Windows 10 preview hence the window adornments):



One piece of the project structure I didn’t previously mention is the ‘src’ folder which contains the OF app itself. The code contains a class which has methods for a drawing loop and also stubs for responding to various events mouse, keyboard, etc. So the idea is that you can just fill the stubs with your OF code.

1. void ofApp::setup(){
2.  
3. }
4.  
5. //--------------------------------------------------------------
6. void ofApp::update(){
7.  
8. }
9.  
10. //--------------------------------------------------------------
11. void ofApp::draw(){
12.  
13. }
14.  
15. //--------------------------------------------------------------
16. void ofApp::keyPressed(int key){
17.  
18. }
19.  
20. //--------------------------------------------------------------
21. void ofApp::keyReleased(int key){
22.  
23. }
24.  
25. //--------------------------------------------------------------
26. void ofApp::mouseMoved(int x, int y ){
27.  
28. }
29.  
30. //--------------------------------------------------------------
31. void ofApp::mouseDragged(int x, int y, int button){
32.  
33. }
34.  
35. //--------------------------------------------------------------
36. void ofApp::mousePressed(int x, int y, int button){
37.  
38. }
39.  
40. //--------------------------------------------------------------
41. void ofApp::mouseReleased(int x, int y, int button){
42.  
43. }
44.  
45. //--------------------------------------------------------------
46. void ofApp::windowResized(int w, int h){
47.  
48. }
49.  
50. //--------------------------------------------------------------
51. void ofApp::gotMessage(ofMessage msg){
52.  
53. }
54.  
55. //--------------------------------------------------------------
56. void ofApp::dragEvent(ofDragInfo dragInfo){
57.  
58. }

I’m not yet familiar with the OF API so for illustrative purposes I’ll have a crack at rendering the colour stream from a Kinect v2 device.

 

KINECT

To get started I will add a reference to the Kinect SDK to the Windows Store project – you can do this by right-clicking the project (in the solution explorer) and choosing > Add > References



Choose to ‘Add New Reference’ and select Extensions and as long as you have the Kinect SDK installed correctly you will see



WindowsPreview.Kinect in there. Choose that and you should be good to go.

I copied some code from a previous post Face Tracking Native – Kinect 4 Windows v2 which displayed the colour stream from the Kinect in a WriteableBitmap. This time the code will pull the colour frames from within the OF update() method and copy the image pixels to an ofImage object which will be used to draw the images with a call to ofImage::draw() from within the apps draw() method. The code for the update and draw methods is shown here:

1. void ofApp::update()
2. {
3.     // Pull the frame...
4.     {
5.         auto multiFrame = _multiFrameReader->AcquireLatestFrame();
6.         if (multiFrame != nullptr)
7.         {
8.             if (multiFrame->ColorFrameReference != nullptr)
9.             {
10.                 auto colorFrame = multiFrame->ColorFrameReference->AcquireFrame();
11.  
12.                 if (colorFrame != nullptr)
13.                 {
14.                     colorFrame->CopyConvertedFrameDataToArray(_colorPixels, ColorImageFormat::Rgba);
15.                     
16.                     _img.setFromPixels(_colorPixels->Data, colorFrame->FrameDescription->Width, colorFrame->FrameDescription->Height, ofImageType::OF_IMAGE_COLOR_ALPHA);
17.                     _colorFrameProcessed = true;
18.                 }
19.             }
20.         }
21.     }
22. }
23.  
24. //--------------------------------------------------------------
25. void ofApp::draw()
26. {
27.     ofClear(255, 0, 0);
28.  
29.     if (!_kinect->IsAvailable)
30.     {
31.         ofDrawBitmapString("No Kinect", 50, 10);
32.         return;
33.     }
34.     _img.draw(0, 0);
35. }

The project can be downloaded here http://1drv.ms/1vqiSd0. I’m hoping this will come in handy the next time I need to quickly create a creative prototype.