Connect with the Kinect

The last couple of blog posts, I've concentrated on advanced research topic of tracking the hands in some way. I feel I should return to the basis of Kinect, and the original intent for the device: putting you in the game. The paper I read discusses a framework for mapping a person's body motions to that of an avatar. But it's not just one to one. It's an enhanced mapping. For example, if you jump, your avatar could jump over an entire building. So then the challenge in building the framework becomes evident. How do you give these enhanced action to the player, while still making the player feel like they are "in the game," or are still in perfect control?

To even begin to tackle this problem, you must make predefined gestures to try and match input to. Kind of the same process as, does the player press the "X" button. After you have your pre-defined gestures, you take the input gestures of the player, apply machine-learning (classification) algorithms to decide which gesture the input is closest to. There is one snag. Taking too much time between getting input and displaying the result can make the player feel out of synche with their avatar. To combat this, the paper suggests determining the gesture before the player has completed it. For example, if the player jumps, the algorthm decides the player has done the "jump" gesture before the player reaches the peek of his jump.

Another step in implementing exaggerated action is how you display and work the output on the screen. In order to show exaggerated output, you must first have predefined animations for the specific action. So when the player jumps, the avatar will jump, but the angle of the legs may be completely different since the jump is scripted. Taking this into account there must also be a seemless transition between the non-scripted one-to-one motion before the action, and the scripted animation activated by the gesture. While the paper doesn't go into exactly how this seemless transition occures, I theorize it can be eazily done by interpolating (using math to estimate in-between limb positions) frames between the last non-scripted frame, and the first scripted frame.

- Kao Pyro of the Azure Flame

Source:  
Bleiweiss, A., Eshar, D., Kutliroff, G., Lerner, A., Oshrat, Y., & Yanai, Y. (2010). Enhanced interactive gaming by blending full-body tracking and gesture animation. SA '10 ACM SIGGRAPH ASIA 2010 Sketches .
http://dl.acm.org/citation.cfm?id=1899950.1899984&coll=DL&dl=GUIDE