Original Article
Meta‐regression approximations to reduce publication selection bias
Abstract
Publication selection bias is a serious challenge to the integrity of all empirical sciences. We derive meta‐regression approximations to reduce this bias. Our approach employs Taylor polynomial approximations to the conditional mean of a truncated distribution. A quadratic approximation without a linear term, precision‐effect estimate with standard error (PEESE), is shown to have the smallest bias and mean squared error in most cases and to outperform conventional meta‐analysis estimators, often by a great deal. Monte Carlo simulations also demonstrate how a new hybrid estimator that conditionally combines PEESE and the Egger regression intercept can provide a practical solution to publication selection bias. PEESE is easily expanded to accommodate systematic heterogeneity along with complex and differential publication selection bias that is related to moderator variables. By providing an intuitive reason for these approximations, we can also explain why the Egger regression works so well and when it does not. These meta‐regression methods are applied to several policy‐relevant areas of research including antidepressant effectiveness, the value of a statistical life, the minimum wage, and nicotine replacement therapy. Copyright © 2013 John Wiley & Sons, Ltd.
Citing Literature
Number of times cited: 72
- Leif D. Nelson, Joseph Simmons and Uri Simonsohn, Psychology's Renaissance, Annual Review of Psychology, 69, 1, (511), (2018).
- Timo Gnambs and Markus Appel, Narcissism and Social Networking Behavior: A Meta‐Analysis, Journal of Personality, 86, 2, (200-212), (2017).
- Dolores Albarracín, Kristina Wilson, Man-pui Sally Chan, Marta Durantini and Flor Sanchez, Action and inaction in multi-behaviour recommendations: a meta-analysis of lifestyle interventions, Health Psychology Review, 12, 1, (1), (2018).
- Sven Kepes and Manoj A. Thomas, Assessing the robustness of meta-analytic results in information systems: publication bias and outliers, European Journal of Information Systems, (1), (2018).
- Mehmet Ugur, Sefa Awaworyi Churchill and Edna Solomon, TECHNOLOGICAL INNOVATION AND EMPLOYMENT IN DERIVED LABOUR DEMAND MODELS: A HIERARCHICAL META-REGRESSION ANALYSIS, Journal of Economic Surveys, 32, 1, (50), (2018).
- Ole Rogeberg, Rune Elvik and Michael White, Correction to: ‘The effects of cannabis intoxication on motor vehicle collision revisited and revised’ (2016), Addiction, 113, 5, (967-969), (2018).
- Jordan P. Davis, Tara M. Dumas, Daniel A. Briley and Steve Sussman, A meta‐analysis of the association between substance use and emerging adult development using the IDEA scale, The American Journal on Addictions, 27, 3, (166-176), (2018).
- T. D. Stanley and Hristos Doucouliagos, Neither fixed nor random: weighted least squares meta‐regression, Research Synthesis Methods, 8, 1, (19-42), (2016).
- T. D. Stanley, Hristos Doucouliagos and John P. A. Ioannidis, Finding the power to reduce publication bias, Statistics in Medicine, 36, 10, (1580-1598), (2017).
- John P. A. Ioannidis, T. D. Stanley and Hristos Doucouliagos, The Power of Bias in Economics Research, The Economic Journal, 127, 605, (F236-F265), (2017).
- John Protzko, Raising IQ among school-aged children: Five meta-analyses and a review of randomized controlled trials, Developmental Review, 10.1016/j.dr.2017.05.001, 46, (81-101), (2017).
- Zahid Latif, Yang mengke, Danish, Shahid Latif, Liu Ximei, Zulfiqar Hussain Pathan, Shafaq Salam and Zeng Jianqiu, The dynamics of ICT, foreign direct investment, globalization and economic growth: Panel estimation robust to heterogeneity and cross-sectional dependence, Telematics and Informatics, (2017).
- Betsy Jane Becker, Ariel M. Aloe, Maren Duvendack, T.D. Stanley, Jeffrey C. Valentine, Atle Fretheim and Peter Tugwell, Quasi-experimental study designs series—paper 10: synthesizing evidence for effects collected from quasi-experimental studies presents surmountable challenges, Journal of Clinical Epidemiology, 89, (84), (2017).
- Sefa Awaworyi Churchill and Siew Ling Yew, The effect of military expenditure on growth: an empirical synthesis, Empirical Economics, (2017).
- Nicholas Grebe, Rachael Falcon and Steven Gangestad, p-Curve and Selection Methods as Meta-Analytic Supplements for Biologists: A Demonstration of Effect Size Estimation in Studies of Human Fluctuating Asymmetry, Symmetry, 9, 7, (98), (2017).
- Bob Uttl, Carmela A. White and Daniela Wong Gonzalez, Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related, Studies in Educational Evaluation, 54, (22), (2017).
- Desirée Blázquez, Juan Botella and Manuel Suero, The Debate on the Ego-Depletion Effect: Evidence from Meta-Analysis with the p-Uniform Method, Frontiers in Psychology, 8, (2017).
- Malte Friese, Julius Frankenbach, Veronika Job and David D. Loschelder, Does Self-Control Training Improve Self-Control? A Meta-Analysis, Perspectives on Psychological Science, 12, 6, (1077), (2017).
- F. Knapp, W. Viechtbauer, R. Leonhart, K. Nitschke and C. P. Kaller, Planning performance in schizophrenia patients: a meta-analysis of the influence of task difficulty and clinical and sociodemographic variables, Psychological Medicine, 10.1017/S0033291717000459, 47, 11, (2002-2016), (2017).
- Jarle Møen and Helge Sandvig Thorsen, Publication Bias in the Returns to R&D Literature, Journal of the Knowledge Economy, 8, 3, (987), (2017).
- Petr Polák, The productivity paradox: A meta-analysis, Information Economics and Policy, 38, (38), (2017).
- Winnie W.S. Mak, Phoenix K.H. Mo, Gloria Y.K. Ma and Maggie Y.Y. Lam, Meta-analysis and systematic review of studies on the effectiveness of HIV stigma reduction programs, Social Science & Medicine, 188, (30), (2017).
- Mehmet Ugur and Arup Mitra, Technology Adoption and Employment in Less Developed Countries: A Mixed-Method Systematic Review, World Development, 96, (1), (2017).
- Arlin J. Benjamin, Sven Kepes and Brad J. Bushman, Effects of Weapons on Aggressive Thoughts, Angry Feelings, Hostile Appraisals, and Aggressive Behavior: A Meta-Analytic Review of the Weapons Effect Literature, Personality and Social Psychology Review, (108886831772541), (2017).
- Anna Soveri, Jan Antfolk, Linda Karlsson, Benny Salo and Matti Laine, Working memory training revisited: A multi-level meta-analysis of n-back training studies, Psychonomic Bulletin & Review, 24, 4, (1077), (2017).
- Elizabeth A. Stevens, Melissa A. Rodgers and Sarah R. Powell, Mathematics Interventions for Upper Elementary and Secondary Students: A Meta-Analysis of Research, Remedial and Special Education, (074193251773188), (2017).
- Sefa Awaworyi Churchill, Mehmet Ugur and Siew Ling Yew, Does Government Size Affect Per‐Capita Income Growth? A Hierarchical Meta‐Regression Analysis, Economic Record, 93, 300, (142-171), (2016).
- Stuart J. Ritchie, Publication bias in a recent meta‐analysis on breastfeeding and IQ, Acta Paediatrica, 106, 2, (345-345), (2016).
- T. D. Stanley, Limitations of PET-PEESE and Other Meta-Analysis Methods, Social Psychological and Personality Science, 8, 5, (581), (2017).
- Junhua Dang, An updated meta-analysis of the ego depletion effect, Psychological Research, (2017).
- Andreas Schneck, Examining publication bias—a simulation-based evaluation of statistical tests on publication bias, PeerJ, 5, (e4115), (2017).
- Jasper H B de Groot and Monique A M Smeets, Human Fear Chemosignaling: Evidence from a Meta-Analysis, Chemical Senses, 42, 8, (663), (2017).
- Joseph P. Simmons and Uri Simonsohn, Power Posing: P-Curving the Evidence, Psychological Science, 28, 5, (687), (2017).
- T.D. Stanley, Yes, nicotine replacement therapy's effectiveness is much lower than often reported, Journal of Clinical Epidemiology, 81, (144), (2017).
- Giovanni Sala, Alexander P. Burgoyne, Brooke N. Macnamara, David Z. Hambrick, Guillermo Campitelli and Fernand Gobet, Checking the “Academic Selection” argument. Chess players outperform non-chess players in cognitive skills related to intelligence: A meta-analysis, Intelligence, 61, (130), (2017).
- Jennifer M. Murray, Sarah F. Brennan, David P. French, Christopher C. Patterson, Frank Kee and Ruth F. Hunter, Effectiveness of physical activity interventions in achieving behaviour change maintenance in young and middle aged adults: A systematic review and meta-analysis, Social Science & Medicine, 192, (125), (2017).
- Reid Ewing and Robert Cervero, “Does Compact Development Make People Drive Less?” The Answer Is Yes, Journal of the American Planning Association, 83, 1, (19), (2017).
- W. Kip Viscusi, Best Estimate Selection Bias in the Value of a Statistical Life, Journal of Benefit-Cost Analysis, (1), (2017).
- S. Awaworyi Churchill and S.L. Yew, Are government transfers harmful to economic growth? A meta-analysis, Economic Modelling, 64, (270), (2017).
- John R. Hughes, Thomas R. Fanshawe and Lindsay F. Stead, Is nicotine replacement really ineffective? A reply to Stanley and Massey, Journal of Clinical Epidemiology, 81, (143), (2017).
- Ole Rogeberg and Rune Elvik, The effects of cannabis intoxication on motor vehicle collision revisited and revised, Addiction, 111, 8, (1348-1359), (2016).
- Etan Orgel, Jeanine M Genkinger, Divya Aggarwal, Lillian Sung, Michael Nieder and Elena J Ladas, Association of body mass index and survival in pediatric leukemia: a meta-analysis, The American Journal of Clinical Nutrition, 103, 3, (808), (2016).
- Alexander Etz, Joachim Vandekerckhove and Daniele Marinazzo, A Bayesian Perspective on the Reproducibility Project: Psychology, PLOS ONE, 11, 2, (e0149794), (2016).
- Blakeley B. McShane, Ulf Böckenholt and Karsten T. Hansen, Adjusting for Publication Bias in Meta-Analysis, Perspectives on Psychological Science, 10.1177/1745691616662243, 11, 5, (730-749), (2016).
- Justin McCrary, Garret Christensen, Daniele Fanelli and Daniele Marinazzo, Conservative Tests under Satisficing Models of Publication Bias, PLOS ONE, 11, 2, (e0149590), (2016).
- Saiying Steenbergen-Hu, Matthew C. Makel and Paula Olszewski-Kubilius, What One Hundred Years of Research Says About the Effects of Ability Grouping and Acceleration on K–12 Students’ Academic Achievement, Review of Educational Research, 86, 4, (849), (2016).
- John Kitchener Sakaluk, Exploring Small, Confirming Big: An alternative system to The New Statistics for advancing cumulative and replicable psychological research, Journal of Experimental Social Psychology, 66, (47), (2016).
- Robbie C. M. van Aert, Jelte M. Wicherts and Marcel A. L. M. van Assen, Conducting Meta-Analyses Based on p Values, Perspectives on Psychological Science, 11, 5, (713), (2016).
- Germà Bel and Raymond Gradus, Effects of unit-based pricing on household waste collection demand: A meta-regression analysis, Resource and Energy Economics, 44, (169), (2016).
- Martin Paldam, Simulating an empirical paper by the rational economist, Empirical Economics, 50, 4, (1383), (2016).
- Gregory Francis, Implications of “Too Good to Be True” for Replication, Theoretical Claims, and Experimental Design: An Example Using Prominent Studies of Racial Bias, Frontiers in Psychology, 7, (2016).
- Yanbo Wang, Qingfei Min and Shengnan Han, Understanding the effects of trust and risk on individual behavior toward social media platforms: A meta-analysis of the empirical evidence, Computers in Human Behavior, 56, (34), (2016).
- Michael R. Cunningham and Roy F. Baumeister, How to Make Nothing Out of Something: Analyses of the Impact of Study Sampling and Statistical Interpretation in Misleading Meta-Analytic Conclusions, Frontiers in Psychology, 7, (2016).
- Daniël Lakens, Joe Hilgard and Janneke Staaks, On the reproducibility of meta-analyses: six practical recommendations, BMC Psychology, 4, 1, (2016).
- T.D. Stanley and Shelby Massey, Evidence of nicotine replacement's effectiveness dissolves when meta-regression accommodates multiple sources of bias, Journal of Clinical Epidemiology, 79, (41), (2016).
- Maime Guan and Joachim Vandekerckhove, A Bayesian approach to mitigation of publication bias, Psychonomic Bulletin & Review, 23, 1, (74), (2016).
- Sven Kepes, Michael A. McDaniel and Martin Voracek, The Validity of Conscientiousness Is Overestimated in the Prediction of Job Performance, PLOS ONE, 10, 10, (e0141468), (2015).
- Michiel van Elk, Dora Matzke, Quentin F. Gronau, Maime Guan, Joachim Vandekerckhove and Eric-Jan Wagenmakers, Meta-analyses are no substitute for registered replications: a skeptical perspective on religious priming, Frontiers in Psychology, 6, (2015).
- Megan de Linde Leonard and T.D. Stanley, Married with children: What remains when observable biases are removed from the reported male marriage wage premium, Labour Economics, 33, (72), (2015).
- P. Crosetto, A. Filippin and J. Heider, A Study of Outcome Reporting Bias Using Gender Differences in Risk Attitudes, CESifo Economic Studies, 61, 1, (239), (2015).
- Tim Bogg and Leanne Lasecki, Reliable gains? Evidence for substantially underpowered designs in studies of working memory training transfer to fluid intelligence, Frontiers in Psychology, 5, (2015).
- Timo Gnambs, What makes a computer wiz? Linking personality traits and programming aptitude, Journal of Research in Personality, 58, (31), (2015).
- T. D. Stanley and Hristos Doucouliagos, Neither fixed nor random: weighted least squares meta‐analysis, Statistics in Medicine, 34, 13, (2116-2127), (2015).
- , Contemporary Global Perspectives on Gender Economics, (39)
- Emma Onraet, Alain Van Hiel, Kristof Dhont, Gordon Hodson, Mark Schittekatte and Sarah De Pauw, The Association of Cognitive Ability with Right-wing Ideological Attitudes and Prejudice: A Meta-analytic Review, European Journal of Personality, 29, 6, (599), (2015).
- Hristos Doucouliagos, T.D. Stanley and W. Kip Viscusi, Publication selection and the income elasticity of the value of a statistical life, Journal of Health Economics, 33, (67), (2014).
- Sara Lazzaroni and Peter A.G. van Bergeijk, Natural disasters' impact, factors of resilience and development: A meta-analysis of the macroeconomic literature, Ecological Economics, 107, (333), (2014).
- David I. Miller, Kyle M. Nolla, Alice H. Eagly and David H. Uttal, The Development of Children's Gender‐Science Stereotypes: A Meta‐analysis of 5 Decades of U.S. Draw‐A‐Scientist Studies, Child Development, , (2018).
- T. D. Stanley, Hristos Doucouliagos and Piers Steel, DOES ICT GENERATE ECONOMIC GROWTH? A META‐REGRESSION ANALYSIS, Journal of Economic Surveys, , (2018).
- John E. Scofield, Erin M. Buchanan and Bogdan Kostic, A meta-analysis of the survival-processing advantage in memory, Psychonomic Bulletin & Review, 10.3758/s13423-017-1346-0, (2017).
- Randolph Luca Bruno and Maria Cipollina, A meta‐analysis of the indirect impact of foreign direct investment in old and new EU member states: Understanding productivity spillovers, The World Economy, , (2017).
- Sebastian Gechert and Ansgar Rannenberg, WHICH FISCAL MULTIPLIERS ARE REGIME‐DEPENDENT? A META‐REGRESSION ANALYSIS, Journal of Economic Surveys, , (2018).
