On-road and simulated driving: Concurrent and discriminant validation

Abstract

Introduction

A converging pair of studies investigated the validity of a simulator for measuring driving performance/skill.

Study 1

A concurrent validity study compared novice driver performance during an on-road driving test with their performance on a comparable simulated driving test.

Results

Results showed a reasonable degree of concordance in terms of the distribution of driving errors on-road and errors on the simulator. Moreover, there was a significant relationship between the two when driver performance was rank ordered according to errors, further establishing the relative validity of the simulator. However, specific driving errors on the two tasks were not closely related suggesting that absolute validity could not be established and that overall performance is needed to establish the level of skill.

Study 2

A discriminant validity study compared driving performance on the simulator across three groups of drivers who differ in their level of experience - a group of true beginners who had no driving experience, a group of novice drivers who had completed driver education and had a learner's permit, and a group of fully licensed, experienced drivers.

Results

The findings showed significant differences among the groups in the expected direction -- the various measures of driving errors showed that beginners performed worse than novice drivers and that experienced drivers had the fewest errors. Collectively, the results of the concurrent and discriminant validity studies support the use of the simulator as a valid measure of driving performance for research purposes.

Impact on industry

These findings support the use of a driving simulator as a valid measure of driving performance for research purposes. Future research should continue to examine validity between on-road driving performance and performance on a driving simulator and the use of simulated driving tests in the evaluation of driver education/training programs.

Introduction

Young, novice drivers are greatly overrepresented in crashes (Evans, 1987, Gebers et al., 1993, Mayhew and Simpson, 1990, Mayhew and Simpson, 1995, Mayhew and Simpson, 1999, Mayhew et al., 2004, National Highway Traffic Safety Administration [NHTSA], 1994, Williams, 1996, Williams, 2003). Research has consistently shown that the elevated crash risk among young drivers arises from two primary sources: inexperience in driving or skill deficiencies, and age-related factors associated with lifestyle (Jessor, 1987, Mayhew and Simpson, 1995, Simpson, 1996, Williams and Ferguson, 2002). Driver education has, for decades, been a popular and widespread response to this problem. It is endorsed by insurance companies as well as parents and teens, who believe it is effective in making safer drivers (Fuller and Bonney, 2004a, Fuller and Bonney, 2004b, Mayhew, 2007, Plato and Rasp, 1983, Williams and Ferguson, 2004). Program evaluations have, however, failed to produce compelling evidence that young people who complete driver education programs have fewer crashes than those who receive less formal driver instruction (Christie, 2001, Engrstrom et al., 2003, Lonero and Mayhew, 2010, Mayhew and Simpson, 2002, Roberts et al., 2002, Senserrick and Haworth, 2005, Vernick et al., 1999).

On the other hand, such evaluations have often not assessed other outcomes such as changes in knowledge, attitudes, skills, beliefs, and motivations. Moreover, some evaluations have used groups that were not comparable, or used sample sizes too small to reliably detect an effect (Clinton and Lonero, 2006, Engrstrom et al., 2003, Haworth et al., 2000, Lonero and Mayhew, 2010). As a consequence, these evaluations have offered little insight into why driver education courses have failed to reduce collisions, or of equal importance, how these courses can be improved to produce significant safety benefits.

To help address the shortcomings of previous evaluations, a large scale evaluation of driver education is currently being conducted. One segment of the evaluation involves an examination of changes in student outcomes, including knowledge, attitudes, opinions, driving practices, and skills. Most of these outcomes are being measured using a questionnaire, the New Driver Survey, which has been rigorously and extensively developed and tested (Mayhew, Simpson, Marcoux, & Lonero, 2009). Among these outcomes are driving skills, which are critical to the overall evaluation since one of the major purposes of driver education is to teach students how to drive and prepare them to pass the road test.

Accordingly, one segment of the large scale evaluation will seek to determine if exposure to driver education enhances performance skills, particularly because skills appear to be closely related to the risk of collision. To illustrate, a recent study that compared collision-involved teens with those who were collision-free found that the former were more likely to have important skill deficiencies (Mayhew, Simpson, Singhal, & Desmond, 2006). They also rated themselves as significantly less skilled, overall, particularly in anticipating hazards, preventing a skid, controlling the vehicle in a skid, predicting traffic situations ahead, reacting quickly, and driving in the dark. Collision-involved teens were also more likely to commit unintentional driving errors associated with an increased risk of collision.

Therefore, measuring driving skills is of central importance to an evaluation of driver education. Although self-reports provide valuable insights into driving skills, a more objective measure such as that obtained from an on-road test or from a simulator, would greatly enhance the validity of inferences. Driving simulators provide a more controlled and safer environment for measuring performance than an on-road test. However, as noted by Shechtman, Classen, Awadzi, and Mann (2009, p. 380), “driving simulators must be validated before they are used for driving assessment and/or training.” Fortunately there is evidence of validity from studies comparing on-road driving performance with specific driving tasks such as braking (Hoffman, Brown, Lee, & McGehee, 2002) and lateral control (Blaauw, 1982; see Shechtman et al., 2009 for a review). Moreover, a few studies (e.g., Harms, 1996, Törnros, 1998) have successfully compared overall driving performance on-the-road with performance on the simulator, a methodology that is considered to be the “gold standard” for validation (e.g., Bédard et al., 2010, Shechtman et al., 2009).

Establishing the validity of a driving simulator to measure driving skills was the purpose of the two studies reported here. A concurrent validity study was designed to compare driving performance on-the-road with driving performance on a simulator (i.e., how well does a drive test on road correlate with a drive test on a simulator). A discriminant validity study compared driving performance on the simulator among drivers with different levels of experience -- a group of unlicensed beginners compared to a group of novice drivers compared to a group of experienced drivers. This study design determined the degree to which a simulated drive test discriminates between “unskilled” and “skilled” drivers defined in terms of their driving experience.

These two validation studies were conducted in Winnipeg, Manitoba, Canada. Manitoba has a Graduated Driver Licensing Program that requires someone to be at least 16 years of age, or a minimum of 15 years, 6 months and enrolled in a high school driver education course, to obtain a learner's permit. The learner's permit is held for a minimum of nine months and the novice is required to pass a road test as a condition of obtaining their intermediate license.