Assessment
Figure 1: A researcher interacting with students during gameplay

Augmented reality (AR) can utilize the following strategies for in-game assessment:

Observations– Due to the outdoor and physical nature of AR learning, observations present some unique opportunities and challenges. The research team found that shadowing a group of students while holding a clipboard to fill out an observation protocol was not as effective as documenting the entire experience via video camera. Written observations do not capture the complexity and nuance of all the various interactions the students experience with virtual characters, digital items, and each other. Furthermore, small mobile High Definition cameras, such as the Flip, capture the AR experience from multiple angles.

Player Reports– Periodically throughout the game-play, researchers asked students to explain what they are doing and why they are doing it. This player-report data collection mirrors the think-aloud and talk-aloud protocol used in qualitative research where the objective is to “illuminate what’s going on in a person’s head during the performance of a task,” such as the team-based problem solving within Outbreak (Patton, 2002, p. 385; Ericsson & Simon, 1993).

Log Files– Capturing player movement, such as the data collection path chosen and the length of time teams spent at each collection site (e.g., habitat), can provide researchers with a good picture of a team’s data collection strategy when combined with the talk-aloud data. In addition to player movement, these log files have the capacity to capture handheld communication, research activity, access to hint screens or tutorials, responses to embedded assessment, and other data points, providing the researchers with a ‘cognitive audit trail’ for deep analysis of game play and the projected learning within (Montola, Stenros, & Waern, 2009; Dede, 2009).

Figure 2: A multiple choice screen assessing student learning

Embedded Assessment– This can be inserted in multiple formats, including alphanumeric keypads for fill in the blank and sentence completion, multiple choice, and interactive drag and drop. The use of handheld-based embedded assessments allows researchers to more closely align the game content to standardized testing items while still maintaining the immersive nature of the AR experience. It also minimizes the “racing” mentality among student players, since questions must be answered correctly before continuing the game (Dunleavy, Dede, & Mitchell, 2008).

While the inevitable advancement of technology will reduce technical and logistical challenges, two larger contextual issues pose greater challenges to the implementation of AR games and the required assessment therein: 1) standardized testing accountability pressure and the associated fact that we do not test for 21st century skills (Dunleavy, 2006; Dunleavy, Heinecke, & Dexter, 2007); and 2) difficulty of assessing collaborative problem solving and scientific inquiry, resulting in a paucity of valid and reliable quantitative instruments aligned with these domains (Dede, 2009). Both of these challenges need to be addressed not only within academic institutions, but also by policy makers at the federal, state, and district level before serious games such as AR are going to become another tool in the ecology of learning environments.

 

For a more in-depth analysis assessing Augmented Reality learning experiences, please read the complete chapter from which the information above was drawn:

Dunleavy, M., & Simmons, B. (2011). Assessing learning and identity in augmented reality science games. In L. Annetta & S. Bronack (Eds.),
Serious educational game assessment (pp. 221-240). Rotterdam, The Netherlands: Sense Publishers.

References

Dede, C. (2009). Learning Context: Gaming, Simulations, and Science Learning in the Classroom. National Research Council. Retrieved from
Dunleavy, M. (2006). 1-to-1 laptop use among nationally board certified teachers. Unpublished doctoral dissertation. University of Virginia,
Charlottesville.
Dunleavy, M., Dede, C., & Mitchell, R. (2009). Affordances and limitations of immersive participatory augmented reality simulations for
teaching and learning. Journal of Science Education and Technology: Volume 18, Issue1, Page 7
Dunleavy, M., Heinecke, W. & Dexter, S. (2007). What added value does a 1:1 student to laptop ratio bring to technology-supported teaching
and learning? Journal of Computer Assisted Learning.

Ericsson, K.A. & Simon, H.A. (1993). Protocol analysis: Verbal reports as data. Revised edition. Cambridge, MA: The MIT Press.

Montola, M., Sternos, J., & Waern, A. (2009). Pervasive games theory and design. Burlington, MA: Elsevier, Inc.

Patton, M.Q. (2002). Qualitative evaluation and research methods (3rd ed.) Thousand Oaks, CA, US: Sage Publications, Inc.