A generally robust approach to hypothesis testing in independent and correlated groups designs
Address reprint requests to: Harvey J. Keselman, Department of Psychology, University of Manitoba, Winnepeg, MB R3T 2N2, Canada. E‐mail: kesel@ms.umanitoba.ca.
Abstract
Standard least squares analysis of variance methods suffer from poor power under arbitrarily small departures from normality and fail to control the probability of a Type I error when standard assumptions are violated. These problems are vastly reduced when using a robust measure of location; incorporating bootstrap methods can result in additional benefits. This paper illustrates the use of trimmed means with an approximate degrees of freedom heteroskedastic statistic for independent and correlated groups designs in order to achieve robustness to the biasing effects of nonnormality and variance heterogeneity. As well, we indicate when a boostrap methodology can be effectively employed to provide improved Type I error control. We also illustrate, with examples from the psychophysiological literature, the use of a new computer program to obtain numerical results for these solutions.
Citing Literature
Number of times cited according to CrossRef: 53
- Sandra Baez, Michel Patiño-Sáenz, Jorge Martínez-Cotrina, Diego Mauricio Aponte, Juan Carlos Caicedo, Hernando Santamaría-García, Daniel Pastor, María Luz González-Gadea, Martín Haissiner, Adolfo M. García, Agustín Ibáñez, The impact of legal expertise on moral decision-making biases, Humanities and Social Sciences Communications, 10.1057/s41599-020-00595-8, 7, 1, (2020).
- Samantha Pegg, Lindsay Dickey, Emma Mumper, Ellen Kessel, Daniel N. Klein, Autumn Kujawa, Stability and change in emotional processing across development: A 6‐year longitudinal investigation using event‐related potentials, Psychophysiology, 10.1111/psyp.13438, 56, 11, (2019).
- Marie Aguirre, Auriane Couderc, Justine Epinat-Duclos, Olivier Mascaro, Infants discriminate the source of social touch at stroking speeds eliciting maximal firing rates in CT-fibers, Developmental Cognitive Neuroscience, 10.1016/j.dcn.2019.100639, 36, (100639), (2019).
- Daniela M. Pfabigan, Shihui Han, Converging electrophysiological evidence for a processing advantage of social over nonsocial feedback, Cognitive, Affective, & Behavioral Neuroscience, 10.3758/s13415-019-00737-9, (2019).
- Fu Guo, Xue-shuang Wang, Wei-lin Liu, Yi Ding, Affective preference measurement of product appearance based on event-related potentials, Cognition, Technology & Work, 10.1007/s10111-018-0463-5, 20, 2, (299-308), (2018).
- Daniela M. Pfabigan, Anna M. Wucherer, Claus Lamm, Internal control beliefs and reference frame concurrently impact early performance monitoring ERPs, Cognitive, Affective, & Behavioral Neuroscience, 10.3758/s13415-018-0604-6, 18, 4, (778-795), (2018).
- Birgit Rauchbauer, Daniela M. Pfabigan, Claus Lamm, Event-related potentials of automatic imitation are modulated by ethnicity during stimulus processing, but not during motor execution, Scientific Reports, 10.1038/s41598-018-30926-4, 8, 1, (2018).
- Joseph Dien, Best practices for repeated measures ANOVAs of ERP data: Reference, regional channels, and robust ANOVAs, International Journal of Psychophysiology, 10.1016/j.ijpsycho.2016.09.006, 111, (42-56), (2017).
- Autumn Kujawa, Ellen M. Kessel, Ashley Carroll, Kodi B. Arfer, Daniel N. Klein, Social processing in early adolescence: Associations between neurophysiological, self-report, and behavioral measures, Biological Psychology, 10.1016/j.biopsycho.2017.07.001, 128, (55-62), (2017).
- Ann Clawson, Peter E. Clayson, Cierra M. Keith, Christina Catron, Michael J. Larson, Conflict and performance monitoring throughout the lifespan: An event-related potential (ERP) and temporospatial component analysis, Biological Psychology, 10.1016/j.biopsycho.2017.01.012, 124, (87-99), (2017).
- Autumn Kujawa, Ashley Carroll, Emma Mumper, Dahlia Mukherjee, Ellen M. Kessel, Thomas Olino, Greg Hajcak, Daniel N. Klein, A longitudinal examination of event-related potentials sensitive to monetary reward and loss feedback from late childhood to middle adolescence, International Journal of Psychophysiology, 10.1016/j.ijpsycho.2017.11.001, (2017).
- Douglas J. Angus, Andrew J. Latham, Eddie Harmon‐Jones, Matthias Deliano, Bernard Balleine, David Braddon‐Mitchell, Electrocortical components of anticipation and consumption in a monetary incentive delay task, Psychophysiology, 10.1111/psyp.12913, 54, 11, (1686-1705), (2017).
- Paige Ethridge, Autumn Kujawa, Melanie A. Dirks, Kodi B. Arfer, Ellen M. Kessel, Daniel N. Klein, Anna Weinberg, Neural responses to social and monetary reward in early adolescence and emerging adulthood, Psychophysiology, 10.1111/psyp.12957, 54, 12, (1786-1799), (2017).
- Yiwen Wang, Zhen Zhang, Yiming Jing, Emilio A. Valadez, Robert F. Simons, How do we trust strangers? The neural correlates of decision making and outcome evaluation of generalized trust, Social Cognitive and Affective Neuroscience, 10.1093/scan/nsw079, 11, 10, (1666-1676), (2016).
- Michael J. Larson, Peter E. Clayson, Cierra M. Keith, Isaac J. Hunt, Dawson W. Hedges, Brent L. Nielsen, Vaughn R.A. Call, Cognitive control adjustments in healthy older and younger adults: Conflict adaptation, the error-related negativity (ERN), and evidence of generalized decline with age, Biological Psychology, 10.1016/j.biopsycho.2016.01.008, 115, (50-63), (2016).
- Rehab Alhasani, Akshata Nayak, Tony Szturm, Mayur Nankar, Sue Boreskie, Geri Brousseau, The Feasibility of a Novel Dual-Task Exercise Program Which Integrates Balance, Gaze, Mobility and Cognition in Community Dwelling Older Adults: Protocol for a Randomized Clinical Pilot Trial, Advances in Aging Research, 10.4236/aar.2015.43012, 04, 03, (96-111), (2015).
- Tyson V. Barker, Sonya Troller-Renfree, Daniel S. Pine, Nathan A. Fox, Individual differences in social anxiety affect the salience of errors in social contexts, Cognitive, Affective, & Behavioral Neuroscience, 10.3758/s13415-015-0360-9, 15, 4, (723-735), (2015).
- Michael J. Larson, Peter E. Clayson, Mark Primosch, Marco Leyton, Scott C. Steffensen, The Effects of Acute Dopamine Precursor Depletion on the Cognitive Control Functions of Performance Monitoring and Conflict Processing: An Event-Related Potential (ERP) Study, PLOS ONE, 10.1371/journal.pone.0140770, 10, 10, (e0140770), (2015).
- Tony Szturm, Jordan Hochman, Christine Wu, Lix Lisa, Karen Reimer, Beth Wonneck, Andrea Giacobbo, Games and Telerehabilitation for Balance Impairments and Gaze Dysfunction: Protocol of a Randomized Controlled Trial, JMIR Research Protocols, 10.2196/resprot.4743, 4, 4, (e118), (2015).
- Haykaz Mangardich, Robert A. Cribbie, Assessing clinical significance using robust normative comparisons, Psychotherapy Research, 10.1080/10503307.2014.889329, 25, 2, (239-248), (2014).
- Ann Clawson, Peter E. Clayson, Whitney Worsham, Oliver Johnston, Mikle South, Michael J. Larson, How about watching others? Observation of error-related feedback by others in autism spectrum disorders, International Journal of Psychophysiology, 10.1016/j.ijpsycho.2014.01.009, 92, 1, (26-34), (2014).
- H. J. Keselman, Sphericity Test, Wiley StatsRef: Statistics Reference Online, 10.1002/9781118445112, (2014).
- H. J. Keselman, Burt Holland, Robert A. Cribbie, Multiple Comparison Procedures, Wiley StatsRef: Statistics Reference Online, 10.1002/9781118445112, (2014).
- H. J. Keselman, Rand R. Wilcox, Multiple Comparison Tests: Nonparametric and Resampling Approaches, Wiley StatsRef: Statistics Reference Online, 10.1002/9781118445112, (2014).
- Ann Clawson, Peter E. Clayson, Mikle South, Erin D. Bigler, Michael J. Larson, An Electrophysiological Investigation of Interhemispheric Transfer Time in Children and Adolescents with High-Functioning Autism Spectrum Disorders, Journal of Autism and Developmental Disorders, 10.1007/s10803-013-1895-7, 45, 2, (363-375), (2013).
- Joseph Dien, Eric S. Brian, Dennis L. Molfese, Brian T. Gold, Combined ERP/fMRI evidence for early word recognition effects in the posterior inferior temporal gyrus, Cortex, 10.1016/j.cortex.2013.03.008, 49, 9, (2307-2321), (2013).
- Michael J. Larson, Ann Clawson, Peter E. Clayson, Scott A. Baldwin, Cognitive conflict adaptation in generalized anxiety disorder, Biological Psychology, 10.1016/j.biopsycho.2013.08.006, 94, 2, (408-418), (2013).
- Ann Clawson, Peter E. Clayson, Michael J. Larson, Cognitive control adjustments and conflict adaptation in major depressive disorder, Psychophysiology, 10.1111/psyp.12066, 50, 8, (711-721), (2013).
- Julia W. Y. Kam, Elizabeth Dao, Maria Stanciulescu, Hamish Tildesley, Todd C. Handy, Mind Wandering and the Adaptive Control of Attentional Resources, Journal of Cognitive Neuroscience, 10.1162/jocn_a_00375, 25, 6, (952-960), (2013).
- Peter E. Clayson, Michael J. Larson, Adaptation to Emotional Conflict: Evidence from a Novel Face Emotion Paradigm, PLoS ONE, 10.1371/journal.pone.0075776, 8, 9, (e75776), (2013).
- Michael J. Larson, Ann Clawson, Peter E. Clayson, Mikle South, Cognitive Control and Conflict Adaptation Similarities in Children and Adults, Developmental Neuropsychology, 10.1080/87565641.2011.650337, 37, 4, (343-357), (2012).
- Justin M. Aronoff, Daniel J. Freed, Laurel M. Fisher, Ivan Pal, Sigfrid D. Soli, The Effect of Different Cochlear Implant Microphones on Acoustic Hearing Individualsʼ Binaural Benefits for Speech Perception in Noise, Ear and Hearing, 10.1097/AUD.0b013e31820dd3f0, 32, 4, (468-484), (2011).
- Lene Symes, Judith McFarlane, Lorraine Frazier, Maria C. Henderson-Everhardus, Gayle McGlory, Kathleen Bachtel Watson, Yan Liu, Charles E. Rhodes, Ron Cornelis Hoogeveen, Exploring Violence Against Women and Adverse Health Outcomes in Middle Age to Promote Womenʼs Health, Critical Care Nursing Quarterly, 10.1097/CNQ.0b013e3181e6d7c4, 33, 3, (233-243), (2010).
- Joseph Dien, Charles A. Michelson, Michael S. Franklin, Separating the visual sentence N400 effect from the P400 sequential expectancy effect: Cognitive and neuroanatomical implications, Brain Research, 10.1016/j.brainres.2010.07.099, 1355, (126-140), (2010).
- Guillermo Vallejo, Manuel Ato, M. Paula Fernández, A robust approach for analyzing unbalanced factorial designs with fixed levels, Behavior Research Methods, 10.3758/BRM.42.2.607, 42, 2, (607-617), (2010).
- G. Vallejo, M. P. Fernández, P. E. Livacic-Rojas, Analysis of unbalanced factorial designs with heteroscedastic data, Journal of Statistical Computation and Simulation, 10.1080/00949650802482386, 80, 1, (75-88), (2010).
- Joseph Dien, The ERP PCA Toolkit: An open source program for advanced statistical analysis of event-related potential data, Journal of Neuroscience Methods, 10.1016/j.jneumeth.2009.12.009, 187, 1, (138-145), (2010).
- Todd C. Handy, Daniel Smilek, Lena Geiger, Cindy Liu, Jonathan W. Schooler, ERP Evidence for Rapid Hedonic Evaluation of Logos, Journal of Cognitive Neuroscience, 10.1162/jocn.2008.21180, 22, 1, (124-138), (2010).
- H. J. Keselman, Rand R. Wilcox, Lisa M. Lix, James Algina, Katherine Fradette, Adaptive robust estimation and testing, British Journal of Mathematical and Statistical Psychology, 10.1348/000711005X63755, 60, 2, (267-293), (2010).
- Lisa M. Lix, Rachel T. Fouladi, Robust step‐down tests for multivariate independent group designs, British Journal of Mathematical and Statistical Psychology, 10.1348/000711006X117853, 60, 2, (245-265), (2010).
- Jenifer Larson-Hall, Richard Herrington, Improving Data Analysis in Second Language Acquisition by Utilizing Modern Developments in Applied Statistics, Applied Linguistics, 10.1093/applin/amp038, 31, 3, (368-390), (2009).
- Joseph Dien, Evaluating two‐step PCA of ERP data with Geomin, Infomax, Oblimin, Promax, and Varimax rotations, Psychophysiology, 10.1111/j.1469-8986.2009.00885.x, 47, 1, (170-183), (2009).
- Joseph Dien, Aminda J. O'Hare, Evidence for automatic sentence priming in the fusiform semantic area: Convergent ERP and fMRI findings, Brain Research, 10.1016/j.brainres.2008.09.045, 1243, (134-145), (2008).
- Aminda J. O'Hare, Joseph Dien, Lauren D. Waterson, Cary R. Savage, Activation of the posterior cingulate by semantic priming: A co-registered ERP/fMRI study, Brain Research, 10.1016/j.brainres.2007.10.095, 1189, (97-114), (2008).
- Joseph Dien, Michael S. Franklin, Charles A. Michelson, Lisa C. Lemen, Christy L. Adams, Kent A. Kiehl, fMRI characterization of the language formulation area, Brain Research, 10.1016/j.brainres.2008.06.107, 1229, (179-192), (2008).
- Michael S. Franklin, Joseph Dien, James H. Neely, Elizabeth Huber, Lauren D. Waterson, Semantic priming modulates the N400, N300, and N400RP, Clinical Neurophysiology, 10.1016/j.clinph.2007.01.012, 118, 5, (1053-1068), (2007).
- Joseph Dien, Michael S. Franklin, Christopher J. May, Is “Blank” a suitable neutral prime for event-related potential experiments?, Brain and Language, 10.1016/j.bandl.2005.08.002, 97, 1, (91-101), (2006).
- Jennifer L. Beaumont, Lisa M. Lix, Kathleen J. Yost, Elizabeth A. Hahn, Application of Robust Statistical Methods for Sensitivity Analysis of Health-Related Quality of Life Outcomes, Quality of Life Research, 10.1007/s11136-005-2293-1, 15, 3, (349-356), (2006).
- H. J. Keselman, Burt Holland, Robert A. Cribbie, Multiple Comparison Procedures, Encyclopedia of Statistics in Behavioral Science, 10.1002/0470013192, (2005).
- H. J. Keselman, Rand R. Wilcox, Multiple Comparison Tests: Nonparametric and Resampling Approaches, Encyclopedia of Statistics in Behavioral Science, 10.1002/0470013192, (2005).
- H. J. Keselman, Sphericity Test, Encyclopedia of Statistics in Behavioral Science, 10.1002/0470013192, (2005).
- H. J. Keselman, Robert A. Cribbie, Burt Holland, Pairwise Multiple Comparison Test Procedures: An Update for Clinical Child and Adolescent Psychologists, Journal of Clinical Child & Adolescent Psychology, 10.1207/s15374424jccp3303_19, 33, 3, (623-645), (2004).
- Rhonda K. Kowalchuk, H. J. Keselman, James Algina, Repeated Measures Interaction Test with Aligned Ranks, Multivariate Behavioral Research, 10.1207/s15327906mbr3804_2, 38, 4, (433-461), (2003).
