Quantifying Experience and Task Performance in 3D Serious Games

Abstract

Mixed reality systems allow the development of different 3D immersive games, by immersing a live captured 3D model of a person in a virtual environment and enabling interactions and collaborations among geographically distributed people. Serious gaming is one application domain for a mixed reality system in which games are developed for primary purpose of education or training rather than just entertainment. Different aspects of a serious game such as visual, interaction, immersion, etc. influence the user’s perceived experience. Task performance in a serious game reflects how efficiently and accurately users carry out the assigned tasks. For instance, in a serious game for a virtual STEM (Science, Technology, Engineering and Mathematics) experiment, the user’s task performance can be characterized by how accurately the user follows the given procedure and how efficiently the goals are accomplished. User’s task performance in such serious games is typically influenced by his/her experience of the provided virtual environment for carrying out the assigned task. In this dissertation, we focus on the problem of quantifying experience and task performance in 3D serious games by addressing the following questions - (i) Can we, and if so how do we, quantify the user’s experience and potentially improve it in different serious games? (ii) Can we, and if so how do we, map the user’s task performance in a real world scenario to the corresponding virtual world serious game? Also, is there a correlation that exists between the user’s task performance and experience in serious games? Since serious games is a wide domain covering countless applications, it would be difficult if not impossible to generalize and answer the above questions for all of them. Hence, we focus on solving the research questions for 3 different domains of serious games developed using a mixed reality framework, namely Exergames, Multi-Modal Collaborative Virtual Laboratory (MMCVL) and Penalty training game. In order to quantify the visual experience, a learning-based objective measure is developed that emulates human perception of the 3D human open mesh quality. For a mixed reality application, a large amount of 3D data is generated and transmitted across the network, even for a single RGB-D Kinect camera. To reduce this data, based on the available bandwidth, a visual quality based vertex selection technique and a sweep-line based meshing technique is used. The user experience of a mixed reality game is improved, if accurate interactions are provided for a wide range of motion. Multiple cameras are needed to provide a complete representation of the person from all directions. A fast skeleton-based re-calibration method is developed that performs continuous and simultaneous extrinsic calibration of multiple Kinect cameras. Skeletal poses from multiple Kinect cameras are combined in order to generate a high quality combined 3D point cloud model. User studies are performed to evaluate the effect of 3 aspects - visual, interaction and immersion, on the overall quality of experience. For automatically assessing the task performance of a single user or a group of users in a 3D serious game, we formally express the assessment logic using an Augmented Hierarchical Task Network (A-HTN). Game authors are provided with an authoring script to help them incorporate the assessment logic in the game. Recording mechanism is used to store user’s task performance for assessment as well as for future reference. User studies are conducted on the 3 serious games domains and correlation analysis is performed to show that the user’s task performance improves if high quality of experience is provided in 3D serious games.