Summer Science Research Project

Examining Cognitive Control and Brain Function in Video Games

Department of Neuroscience/Psychology
Mentor: Kira Bailey PhD
Mentees: Joy Buraima, Isabelle Rodriguez, Navami Shenoy

Abstract

A growing body of evidence suggests that action video game (AVG) experience is associated with improvements in visual/spatial attention and cognitive control (Green & Bavelier, 2003, 2006, 2007; Green, Pouget, & Bavelier, 2010) and changes in brain function (Knols et al., 2017). The significance of this finding lies in the implication that the skills acquired in an AVG might be transferred to other contexts (Boot, Blakely, & Simons, 2011; Green & Bavelier, 2003). This has led some researchers (Bavelier et al., 2012; Green & Bavelier, 2008) to recommend the use of AVGs in training protocols among populations that would benefit from enhanced visual attention and cognition (e.g., older adults, pilots, military personnel). These recommendations may be premature, however; there are several methodological criticisms of the past research (Boot, Blakely, & Simons, 2011; Bisoglio et al., 2014), one of which is that the use of readily available commercial video games does not allow for strong experimental control over the numerous variables that could influence cognitive skills. Unreal Engine 4 was used to create three video games that simulate standard laboratory assessments of cognitive control; two simulate the Flanker task (Ericksen & Ericksen 1974) while the third video game simulates the N-back task. Two benefits of this approach are 1) greater ecological validity in measuring cognitive control, and 2) the ability to control numerous variables in the video game in order to directly test hypotheses. The realistic video games should provide behavioral data comparable to behavioral data obtained from the standard psychological tasks, while also providing insight into the mechanisms for video game effects on cognition.

Simplified Abstract

Video games impact cognitive control, the processes that allow us to plan and execute our goal-directed behavior. But how? We are not sure. 

Some studies involving video games have found positive changes in cognitive control, while others show negative changes. However, video games in the market are not designed to train cognitive abilities, so it is difficult to know which aspects of the games are causing these changes. We are designing video games that replicate the tasks used by researchers to study cognitive control. This not only gives us greater experimental control but it also allows us to better ascertain the effects of specific game features on brain activity and behavior. 

The significance of the research lies in the implication that the skills acquired in action video games (AVGs) may help train other populations to enhance visual attention and cognition. These populations could include older adults, military personnel, and pilots. Using Unreal Engine 4, Buraima, Rodriguez, and Shenoy created 3 video games from scratch to imitate the Flanker task and N-back task.

Cognitive Tasks Explained

N-Back Task

The N-back task measures working memory and working memory capacity. Participants are presented with a sequence of stimuli one-by one. Most commonly, the stimuli are letters. For each stimulus, they need to decide if the current stimulus is the same as the stimulus present N trials ago. The N can be 1 trial, 2 trials, 3 trials, etc. The higher the trial number, the more difficult the task. The factors that influence performance include the N trial, speed of the stimuli presentation, and the size of the set of stimuli.

One-Back task

Stimuli/Target are back-to-back

Two-Back Task

Stimuli/Target have one non-target in between

Three-Back Task

Stimuli/Target have two non-stimuli in between

Flanker Test

The Eriksen Flanker test is a cognitive task which assesses a participant's ability to suppress inappropriate responses given a particular context. A target stimulus is flanked by non-target stimuli which can correspond to either of the following directional responses:

1.) an identical directional response (termed a congruent flanker)

2.) an opposite directional response (termed an incongruent flanker) 

or 

3.) neither of the aforementioned directional responses (termed a neutral flanker).

Letters, colors, numbers and even arrows have been used as stimuli on variants of the Flanker test.

A sample congruent trial for a Flanker task.

A sample incongruent trial for a Flanker task.

An example of a neutral flanker for a given trial.

Rodriguez's Game

N-back Video Game- Rodriguez

The player controls the direction in which the mannequin in the game turns. The stimuli are represented by different fast food items: lollipop, donut, hot dog, pizza, and ice cream. The stimuli are presented on a billboard on either the left or right corner of the sidewalk. A predetermined path was created by Rodriguez, that path must be followed for the player to win the game. The game will measure the reaction time— the time it took the player to press a button, as well as accuracy— did the player turn at the correct stimulus.
Three levels exist: 1-back, 2-back, and 3-back. It is expected that errors will increase as the N-back increases.

Screenshots from the Game

N-Back

N-Back
N-back

N-back
N-back

N-back

Buraima's Game

Flanker Task Video Game

Key presses corresponding to respective congruency trials are defined are the start of the game. The participant runs indefinitely on one of various tracks until they are presented with a stimuli set. The stimuli come in the form of Y bots of differing colors. Participants have a given amount of time to respond, after which their character keeps running until presentation of another stimuli set. 4 levels (each representing different themes) were made in this fashion. Reaction time (how long it takes participant to respond) and accuracy (how appropriate the participant's responses are) will be measured throughout the game. Note that only congruent and incongruent trials were represented in this game (i.e., neutral flankers are omitted).

Screenshots from the Game

Shenoy's Game

Flanker Task Video Game

The player walks through the game universe until they encounter a stimulus. Each stimulus can be either congruent or incongruent and consists of zombies and/or cyborgs. The participant fights the stimulus with specific key presses that correspond with the correct attack to “defeat” the target in the middle. The player controls these responses, while the path followed by them is predetermined. The game involves four such levels and measures the accuracy and reaction time of the responses.

Screenshots from the Game

Future Research

Each video game will be available online so that behavioral data can be collected even with social distancing. Participants will be able to play the game without having to come to the lab, and we will receive accuracy and reaction time data. This data will allow us to ensure that the video games are valid and are having the effect they were designed to have.
When it is safe again, participants will be brought to the lab to play the video games whilst EEG/ERP data is being collected to later be analyzed.