To relate the standardized road test to video recordings of naturalistic driving in older adults with a range of cognitive impairment.
To relate the standardized road test to video recordings of naturalistic driving in older adults with a range of cognitive impairment.
Cross-sectional observational study.
Academic medical center memory disorders clinic.
One hundred three older drivers (44 healthy, 59 with cognitive impairment) who passed a road test.
Error rate and global ratings of safety (pass with and without recommendations, marginal with restrictions or training, or fail) made by a professional driving instructor.
There was fair agreement between global ratings on the road test and naturalistic driving. More errors were detected in the naturalistic environment, but this did not affect global ratings. Error scores between settings were significantly correlated, and the types of errors made were similar. History of crashes corrected for miles driven per week was related to road test error scores but not naturalistic driving error scores. Global cognition (Mini-Mental State Examination) was correlated with road test and naturalistic driving errors. In healthy older adults, younger age was correlated with fewer errors on the road test and more errors in naturalistic driving.
Road test performance is a reasonable proxy for estimating fitness to drive in older individuals' typical driving environments, but differences between performance assessed using these two methods remain poorly understood and deserve further study.
The road test is widely accepted as the criterion standard for licensing new drivers. Given its face validity, it has also been adapted as a tool to monitor older adults for potential declines in driving ability related to physical illness or cognitive impairment. Test characteristics for on-road tests in cognitively healthy older adults show good interrater and test–retest reliability, internal consistency, and correlation between global ratings and performance scores in research settings.[2, 3] Road test performance is also related to office-based cognitive assessment[4-10] and history of motor vehicle crashes.
Despite this, little is known about the relationship between road test and naturalistic driving behavior. Although the road test occurs in a real-world environment, there are environmental differences between the two settings that may affect its ecological validity. For example, the road test may occur in an unfamiliar or more-complex environment. It is well documented that some older adults with and without cognitive impairment restrict their driving space and may reduce the complexity of their driving environments (e.g., avoid nighttime and highway driving).[12-14] As such, placing participants on a new and possibly more-difficult course may put additional cognitive demands on them. Formal test-taking with potential consequences may also lead to anxiety. In a large study of road test performance in healthy and cognitively impaired older adults, road testing was associated with test-taking fear and anxiety in some participants. In a retrospective review of road test performance of older adults, road test failures substantially increased when individuals were no longer allowed to use their own vehicles during the road test. Older adults may therefore be more susceptible to the negative influences of anxiety and course and vehicle unfamiliarity during the road test. These factors may in turn elicit driving difficulties that are not apparent in their typical environments, resulting in inappropriate failures of individuals who are safe to drive. Conversely, some drivers may be on their “best behavior” while being monitored by an in-car instructor. Moreover, older adults may benefit from being accompanied by a driving instructor who is providing navigation instructions and cueing regarding traffic situations or driving rules. Prior research suggests that cueing enhances performance on the road test.
There are no accepted standards for road tests for older drivers[18, 19] in part because knowledge about older adults' driving behavior is largely derived from self- or caregiver report and not direct observation. Based on survey data, older adults limit their driving to well-known streets, reduce their weekly mileage, and reduce the complexity of their driving by avoiding highways and night driving.[14, 21-23] Recent naturalistic data using in-car recording devices confirm these self-report findings and demonstrate that driving behavior declines when older adults are faced with challenging driving situations and are driving in environments further away from home. When driving was monitored over 1 week, older drivers had fewer demerit points for unsafe driving but made different types of errors, including not stopping at stop signs and turning errors, than younger participants.
The validity of the road test in predicting the types of errors and problems detected using naturalistic methods is largely unknown given the paucity of studies directly comparing road tests with naturalistic driving performance. There are many environmental differences between these two settings, and factor analysis comparing a standardized road test with naturalistic driving suggests that the road test requires a more-limited set of skills than naturalistic driving, raising concern that road tests are not optimally designed to estimate driving competence in older adults. Furthermore, it is unclear whether the structured and cued format of the road test enhances driving performance or reveals problems that would not otherwise have been identified. The objective of this study was to compare a road test with 4 hours of video-recorded naturalistic driving in older adults with a range of cognitive impairment.
Study participants were aged 60–90 with a valid driver's license and no at-fault crashes within the past year. Physician or family member concern about driving competence was not required for study entry. Healthy older adults had no history of dementia and a Mini-Mental State Examination (MMSE) score greater than 26. Cognitively impaired participants were recruited primarily from a hospital-based memory disorders clinic during routine follow-up visits for their memory disorder. Healthy participants were spouses of cognitively impaired participants or were recruited through community advertisements. Dementia severity was measured using the Clinical Dementia Rating scale (CDR); only cognitively impaired participants with CDR scores of 0.5 or 1 were included. Cognitively impaired participants met diagnostic criteria for possible or probable Alzheimer's disease (AD) based on the National Institute of Neurological and Communicative Disorders and Stroke/Alzheimer's Disease and Related Disorders Association criteria. Cognitively impaired participants had been on a stable dose of a cholinesterase inhibitor for 6 weeks, if prescribed. Study informants were individuals who had accompanied the participant while driving at least once monthly during the preceding year and could report on driving history. Study informants provided information on crash history and miles driven for the memory-disordered participants and were available to drive the participant home if they failed the on-road test. Exclusion criteria for both groups included reversible causes of dementia, physical or ophthalmological disorders that might impair driving ability, mental retardation, schizophrenia, bipolar disorder, and alcohol or substance abuse within the previous year. Anxiolytic and antipsychotic medications were permitted, but dosages were required to be stable for 6 weeks before study entry.
Written informed consent was obtained from all participants. The Rhode Island Hospital institutional review board approved all procedures. Participants were screened for inclusion criteria at an office visit that included the neurological examination, vision screen, MMSE, and CDR. Participants were scheduled for a road test if inclusion criteria were met. If they failed the road test, they could retest within 2 weeks, but they were encouraged to refrain from driving until the retest was completed. Of the initial 122 participants enrolled, 10 failed the road test (CDR = 0, n = 1; CDR = 0.5, n = 5; CDR = 1, n = 4). One participant retested and passed. Cameras were then installed in their primary vehicle to record naturalistic driving. Participants who failed the road test or elected not to retest were terminated from the study because of ethical considerations related to video recording drivers who could not pass a road test. A letter recommending driving cessation was sent to the primary care physician, and the driving instructor counseled participants and family members.
Miles driven per week and crash history were obtained at the baseline visit from the informant for the memory disordered participants and according to self-report for the healthy controls. The research assistant obtained this information using a semistructured interview format.
A professional driving instructor (who was blind to diagnosis) administered the Rhode Island Road Test (RIRT) to participants. The same driving instructor rated all participants for the duration of the study. The RIRT was administered within 1 month of the baseline office assessment during daylight hours under good road conditions. The test covered 6.5 miles of urban terrain without highway driving and required 45 minutes to complete. The driving instructor accompanied the participant in a specially fitted vehicle that had a brake on the passenger side for emergency use. A pretest of basic vehicle operation was performed in a parking lot before the test. The driving instructor provided only oral instructions to complete the course.
The RIRT is an adaptation of the Washington University Road Test, a standardized driving measure with previously established reliability. The RIRT showed adequate interrater reliability for 20 participants (rated by a second professional driving instructor in the backseat), with almost perfect agreement for the global rating (kappa = 0.83 for linear weighted ratings and 0.92 for quadratic weighted ratings)[31, 32] and a strong correlation coefficient for the error score Spearman correlation coefficient (r) = 0.87.
Twenty-eight driving maneuvers and behaviors were rated on a 3-point scale (0 = unimpaired, 1 = mildly impaired, 2 = moderately to severely impaired). (See for complete description.) The majority of participants had the same number of opportunities to engage in a driving maneuver during the test. If an event was not encountered (e.g., response to pedestrian), it was not included in the overall proportional score. Total scores ranged from 0 to 960, with higher scores reflecting poorer performance. Two dependent variables were generated. The first variable was average error severity score, reflecting the sum of ratings (range 0–2) for each event divided by the number of observed maneuvers. The second variable was a categorical rating of overall driving ability (pass with or without recommendations, marginal with restrictions, marginal with training, fail). Pass indicated that the participant's driving performance would be unlikely to result in crashes or violations. Recommendations included behaviors that could improve safety. Marginal indicated that the driver could continue to drive but should restrict driving to particular locations, times, or traffic density or enroll in driving lessons. Fail indicated that the driver exhibited behavior that had a high probability of leading to crashes that could not be easily remediated.
Low-profile cameras (3.5 inches wide by 2.5 inches deep by three inches tall) were installed on the same day that the road test was passed on the dashboard and in the back of the vehicle. One camera faced the participant, another faced forward, and the back two cameras faced diagonally forward to capture the environment to the sides of the vehicle. Participants were instructed to drive in their typical environment and to follow their daily routine. Video was downloaded to a digital video recording device under the passenger seat. Cameras were installed for 2 weeks. The research assistant reviewed all driving segments acquired and provided approximately 4 hours of consecutive daytime driving to the driving instructor for review. Only daytime driving was rated because nighttime video quality was suboptimal for coding.
The same driving instructor completed the video ratings 1 month after the road test to maximize comparability of ratings across environments. The instructor was not allowed to review his ratings from the road test. Naturalistic driving was rated using the Composite Driving Assessment Scale (CDAS), a measure developed for this study. Scale content was based on input from the study driving instructor's standard assessment and from the RIRT rating scale. Behaviors were divided into discrete events (maneuvers) and global events (attention, attitude, reaction time). Because the number and types of driving situations in naturalistic driving assessment cannot be controlled in the same fashion as on a road test, each item was given a global rating of unimpaired, mildly impaired, or moderately to severely impaired using the same 3-point Likert scale as the RIRT (range 0–2). Total scores ranged from 0 to 60, with higher scores reflecting poorer performance. The error severity score was an average score reflecting the sum of the ratings for each behavior divided by the maximum number of observed behaviors. The driving instructor also made a global rating of safety identical to the RIRT. Interrater agreement for the global ratings between two separate driving instructors rating 25 videos was in the moderate range for the global ratings (kappa = 0.45) and error scores (r = 0.62).
Course difficulty for naturalistic driving was rated on a Likert scale ranging from 1 (simple) to 10 (extremely challenging). One rating was made for the 4 hours of driving to reflect the modal driving environment.
For the participant who retested on the road test and remained in the study, the result of the initial road test was included in analyses. Group comparisons for baseline characteristics were made using independent-sample t-tests for continuous variables and chi-square or Fisher exact tests for categorical variables. A linear regression model was applied to examine the association between driving errors and history of crashes. Analyses were performed using S-Plus 8.2 (Insightful Corporation, Seattle, WA).
Demographic characteristics are presented in Table 1 for participants who passed the road test and had cameras installed in their cars (N = 103). The final group consisted of 44 healthy participants, 41 cognitively impaired participants with a CDR rating of 0.5 (questionable to very mild dementia) and 18 with a CDR rating of 1 (mild dementia). Given the small sample size of cognitively impaired participants with a CDR of 1, cognitively impaired participants were combined into one group for all comparisons. Cognitively impaired participants were older (t(101) = −3.44, P < .001) and less educated (t(101) = 3.58, P < .001) and drove fewer miles per week (t(101) = 5.04, P < .001) than the healthy participants. In terms of driving history, both groups reported few crashes during the 3 years before enrollment (n = 14 overall) or violations in the year before enrollment. There were no differences in crash rate per year (t(101) = −1.08, P = .28) or percentage of violations between the two groups (Fisher test, P = .17).
|Characteristic||No Cognitive Impairment, n = 44||Cognitive Impairment, n = 59||Total, N = 103|
|Age, mean ± SD||71.2 ± 7.6||76.0 ± 6.0a||73.9 ± 7.2|
|Race (Caucasian, %)||100.0||91.5%||95.2|
|Education, years, mean ± SD||16.3 ± 3.8||13.8 ± 3.4a||14.9 ± 3.8|
|Mini-Mental State Examination, mean ± SD||29.5 ± 0.7||25.2 ± 2.8a||27.0 ± 3.0|
|Years driving, mean ± SD||53.1 ± 6.8||55.5 ± 9.9||54.5 ± 8.7|
|Miles driven per week, mean ± SD||200.8 ± 114.9||98.8 ± 90.5a||142.4 ± 133.1|
|Crash in past year (%)||13.6||8.5||10.7|
|Crashes per year/10,000 miles driven, mean ± SD||0.2 ± 0.4||1.4 ± 7.5||0.8 ± 5.7|
|History of violations in past year (%)||13.6||5.1||8.7|
To address any potential biases in the cognitively impaired group, refusers (n = 64) were compared with enrolled cognitively impaired participants. Refusers were similar in age (76.5 ± 6.2; t(121) = 0.47, P = .64) and sex (37% male; χ2 = 2.44, P = .12) to enrolled cognitively impaired participants, although refusers were more cognitively impaired as measured according to the MMSE (23.6 ± 3.2; t(121) = −2.95, P < .001).
Table 2 presents group comparisons of driving outcomes. Cognitively impaired participants made more errors and were rated more poorly on global ratings of safety on the road test and naturalistic driving than cognitively healthy participants. The complexity of driving environments was comparable between the groups despite fewer miles driven per week in the cognitively impaired participant group.
|Driving Assessment||No Cognitive Impairment, n = 44||Cognitive Impairment, n = 59||t||P-value|
|Error score, mean ± SD||0.04 ± 0.03||0.08 ± 0.06||−4.20||<.001|
|Global rating (%)|
|Pass, no recommendations||56.8||20.3||<.001|
|Pass with recommendations||29.5||32.2|
|Marginal with restrictions||11.4||30.5|
|Marginal with training||2.3||15.2|
|Error score, mean ± SD||0.10 ± 0.08||0.19 ± 0.13||−4.08||<.001|
|Global rating (%)|
|Pass, no recommendations||61.4||35.6||.01|
|Pass with recommendations||22.7||20.3|
|Marginal with restrictions||13.6||20.3|
|Marginal with training||2.3||16.9|
|Driving course difficulty, mean ± SD||5.80 ± 1.15||5.68 ± 1.32||0.47||.64|
In the overall group, ratings on the road test and naturalistic driving showed fair agreement, (kappa = .33), reflecting an effect size in the moderate range. Fifty-three percent of participants passed the road test and naturalistic driving. Only four participants (4%) failed naturalistic driving after receiving a pass with recommendations (n = 2), marginal with restrictions (n = 1), or marginal with training (n = 1) rating on the road test. Error scores in the two environments were correlated (r = 0.41, P < .001), but error score severity was higher in the naturalistic environment (0.15 ± 0.12) than on the road test (0.06 ± 0.05) (t(102) = −8.31, P < .001).
Tables 3 and 4 show specific types of driving errors ranked according to severity in both driving environments; only the 10 maneuvers rated most poorly of the entire CDAS and RIRT measures are presented, for brevity. On average, specific maneuvers were rated as mildly impaired in both environments. Although the CDAS and RIRT do not rate identical maneuvers, inspection of the most-severe error types across environments shows overlap of half of the maneuvers, including checking blind spots, making complete stops, lane keeping, traffic awareness, and speed control. Checking blind spots was the most notable error made in both environments. Between-group comparisons for each of these maneuvers showed that the cognitively impaired participants were more impaired than healthy participants on more maneuvers on the road test (6/10 maneuvers) than with naturalistic driving (2/10 maneuvers).
|Item||Overall, N = 103||No Cognitive Impairment, n = 44||Cognitive Impairment, n = 59|
|Checks blind spotsa||0.87||0.62||1.06b|
|Uses mirrors for lane change||0.25||0.13||0.34b|
|Approaches intersection at appropriate speed||0.17||0.19||0.16|
|Makes complete stopsa||0.17||0.12||0.21|
|Brakes smoothly and accurately||0.14||0.13||0.15|
|Item||Overall, N = 103||No Cognitive Impairment, n = 44||Cognitive Impairment, n = 59|
|Checks blind spotsa||0.77||0.41||1.03b|
|Makes complete stopsa||0.64||0.60||0.67|
|Awareness of driving of others||0.51||0.32||0.66|
|Responds to signage||0.48||0.36||0.56|
|Appropriate response to emergency vehicles||0.38||0.25||0.47|
To determine whether road test or naturalistic driving performance is a more-sensitive measure of driving risk, history of crashes, corrected for miles driven per week during the 3 years before enrollment, was entered as a covariate in a linear regression model predicting error scores on the RIRT and CDAS. Two separate models were run using the RIRT and the CDAS as outcomes. Crash history was associated with error scores on the RIRT (F(1,101) = 7.38, P = .01, coefficient of variation (R2) = 0.07, but not on the CDAS (F(1,101) = 0.60, P = .44, R2 = 0.01).
Disease severity and age were then examined to determine whether there were any demographic or disease characteristics that might affect the relationship between road test and naturalistic driving performance. As expected, global cognitive status as measured using the MMSE was correlated with the road test (r = −0.32, P = .001) and naturalistic driving (r = −0.22, P = .02), with poorer performance on the MMSE associated with higher driving error scores in both settings. Age was correlated with the road test (r = 0.26, P = .01) but not naturalistic driving (r = 0.10, P = .30). When the groups were separated, for cognitively impaired participants, there was no relationship between age and performance on the road test (r = 0.06, P = .64) or naturalistic driving (r = 0.13, P = .32). For the healthy older adults, younger age was correlated with lower error severity on the road test (r = 0.40, P = .01) and greater error severity for naturalistic driving (r = −0.32, P = .04).
Individuals with cognitive impairment had higher error scores and poorer global ratings of driving competence on the road test and naturalistic driving than cognitively healthy older adults. Cognitively impaired participants had a greater number of more-impaired maneuvers than cognitively intact participants on the road test than with naturalistic driving, suggesting that the road test may be a more cognitively demanding task than naturalistic driving. These results are consistent with several studies demonstrating that mild cognitive impairment or early Alzheimer's disease affects driving skills even if the person is able to pass a road test.[6, 23] Furthermore, global cognitive status (MMSE) was associated with driving errors in both settings. Taken together, the findings suggest that both methods of driving assessment are sensitive to cognitive impairment beyond the effects of normal aging.
Given the limitations of the road test, it was of primary interest to determine whether performance on a standardized road test was related to individuals' driving behavior in their own environment. Results showed fair agreement between global ratings of safety and error scores on the road test and naturalistic environment, suggesting that the road test is a reasonable proxy for driving behavior in the naturalistic setting for older adults with and without cognitive impairment. This is encouraging and suggests that the road test may not necessarily enhance driving performance through structure and cueing or, conversely, cause problems because of anxiety or course or vehicle unfamiliarity that would not be problems in the participants' typical driving environment.
For both groups, more errors were committed in the naturalistic setting than on the road test, but this did not affect global ratings of safety. It is possible that the driving instructor judged the types of errors detected in the naturalistic setting to be less severe than the errors made on the road test. Alternatively, ratings may differ because the driving instructor had more opportunities to make judgments about errors while watching the videos of the naturalistic driving than during the much shorter driving test. Errors may have also been interpreted differently during the in-car experience than the purely observational ratings made using video recordings for naturalistic driving. Qualitatively, the types of errors made on the road test and naturalistic setting were similar, with half of the error types overlapping between the two environments. Failure to check blind spots was the most egregious error in both settings, and awareness of traffic, making complete stops, speed control, and lane keeping were other error types frequently identified in both settings. These types of errors are generally consistent with other studies showing that older adults with cognitive impairment have significant difficulties with lane checking, lane changing, and merging, all of which require checking of blind spots.[6, 7]
It was of interest to determine whether higher error scores in the naturalistic setting were related to a real-world safety measure. To accomplish this, history of crashes 3 years before enrollment was recorded and corrected for miles driven. This correction was made because past studies have shown that individuals with cognitive impairment have higher crash rates when correcting for miles driven.[23, 35] In the current study, crashes per mile driven were positively associated with road test but not naturalistic driving errors, suggesting that the road test may be a more-sensitive measure of crash risk, but it is important to emphasize that, in this group of participants, history of crashes was rare, and the majority had no crashes. Report of crash history was also self-reported, so the reporting could have been biased. Participants with an at-fault crash within 1 year of enrollment were also excluded, so the highest-risk drivers may have been excluded. It is also possible that the few participants who had crashes biased this relationship. These findings should be replicated in a sample with a greater range of crash history.
As expected, disease severity was related to road test and naturalistic driving ratings. Age had a unique effect on driving performance in each of these environments in the healthy older adults. More specifically, younger age was associated with fewer driving errors on the road test and more driving errors in the naturalistic setting, suggesting that the younger participants were able to modify their behavior on the road test to “best behavior” or that older participants modified their driving environment to reduce errors, but individuals with cognitive impairment did not.
This study has limitations. The evaluation of naturalistic driving was restricted by using video technology. It was not possible to observe directional signal or pedal use, and more overlap in error types might have been seen between the two environments if these behaviors had been measured in the naturalistic setting. The composition of the sample also limits the generalizability of these findings. Individuals who refused to participate in the study were more cognitively impaired. As such, the sample largely comprised participants with cognitive impairment in the mild range, making it difficult to apply the findings to individuals with more-advanced dementia. In addition, cognitively impaired participants were slightly older and less educated than the healthy participants. Both groups were driving in comparably difficult settings during their naturalistic driving, despite cognitively impaired participants driving fewer miles per week, suggesting that any differences in the complexity of naturalistic driving environments did not confound the findings. Finally, crash history and miles driven per week were self-reported and could be biased.
Because of safety limitations, only drivers who passed the road test were allowed to be evaluated naturalistically. Consequently, how drivers deemed to be unsafe on the road test might have performed in naturalistic driving is unknown. The cognitively impaired participant who initially failed the road test was judged to be safe based on their video-recorded naturalistic driving. Conversely, four participants who passed the road test ultimately failed naturalistic driving. Overall, the positive predictive value of safe driving for the road test appears to be acceptable, but the negative predictive value cannot be assessed because the worst drivers who failed the road test did not have an assessment of naturalistic driving. The instructor was allowed to provide advice on driving restrictions (e.g., restrict driving to local areas, avoid driving at night or on highways, drive with a copilot), which could have changed their behavior in the naturalistic setting, although this factor would have biased against finding a significant relationship between naturalistic and road test driving, suggesting that the relationship between the two settings may be more robust than the data indicate.
In conclusion, the road test appears to provide a reasonable estimate of driving in the naturalistic setting in older adults with and without cognitive impairment. Although previous work suggests that different skills may be emphasized in the road test than in the naturalistic environment, those differences do not seem to affect the concordance between overall safety ratings and errors in the two environments. These results do not adequately address whether performance in the naturalistic setting or during a road test is a better predictor of driving safety in this population. Future longitudinal naturalistic studies that include safety outcomes are needed to make this determination.
This work was supported by the National Institute on Aging at the National Institutes of Health (R01 AG016335 to BRO).
We would like to acknowledge the participation of the following individuals in this research program: Mr. Timothy Souza from the ABC/ACE Driving School, who performed the road test and ratings of video recorded naturalistic driving assessments; Stephanie K. Knott for assistance in data collection; and the participants who graciously offered their time to the study.
Conflict of Interest: None of the authors have any financial or personal conflicts of interest that would affect the presentation or interpretation of the findings presented in this paper.
Author Contributions: All authors contributed significantly to the work presented in this manuscript.
Sponsor's Role: The sponsor had no direct role in design, methods, subject recruitment, data collection, analysis, or preparation of the manuscript.