Academic Emergency Medicine

Cover image for Vol. 22 Issue 6

Edited By: David C. Cone, MD

Online ISSN: 1553-2712

Statistics and Research Methodology

Welcome to the Statistics and Research Methodology Virtual Issue

Compiled by Craig D. Newgard, MD, MPH

These articles were compiled based on recommendations from Academic Emergency Medicine and Annals of Emergency Medicine statistical/methods reviewers, recognition as “landmark” statistical publications and frequently cited methodology articles. Papers from emergency medicine journals (Academic Emergency Medicine, Annals of Emergency Medicine and Prehospital Emergency Care) are featured. However, additional references are provided for each topic from non-emergency medicine journals (many of which are written by emergency physicians) as potentially helpful references.



Research fundamentals: statistical considerations in research design: a simple person's approach.

Research fundamentals: IV. Choosing a research design.

Emergency medicine animal research: does use of randomization and blinding affect the results?

Study designs and evaluation models for emergency department public health research.

Statistical methodology: II. Reliability and variability assessment in study design, Part A.

Statistical methodology: II. Reliability and validity assessment in study design, Part B.


Additional references:

  • Schulz KF, Altman DG, Moher D and the CONSORT group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. Ann Intern Med. 2010;152:726-32. (Also co-published in PLoS Med, BMJ, J Clin Epidemiol, BMC Med, Trials, Obstet Gynecol, and Int J Surg)
  • Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG; Consolidated Standards of Reporting Trials Group. CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol. 2010;63:e1-37.
  • Gilda Piaggio, PhD, Diana R. Elbourne, PhD, Stuart J. Pocock, PhD, Stephen J. W. Evans, MSc, Douglas G. Altman, DSc, for the CONSORT Group. Reporting of Noninferiority and equivalence Randomized Trials:  Extension of the CONSORT 2010 Statement . JAMA. 2012;308:2594–2604.
  • Calvert M, Blazeby J, Altman DG, Revicki DA, Moher D, Brundage D. Reporting of patient–reported outcomes in randomized trials – The CONSORT PRO extension. JAMA 2013;309:814–22.
  • von Elm, E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, for the STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies. Ann Intern Med. 2007;147:573–577.
  • Vandenbroucke JP, von Elm E, Altman DG,. Gøtzsche PC, et al for the STROBE initiative. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration. Ann Intern Med. 2007;147:W–163–W–194.
  • Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta–analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med. 2009 Aug 18;151(4):W65–94.
  • Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta–analyses: the PRISMA statement. Ann Intern Med. 2009 Aug 18;151(4):264–9, W64. Epub 2009 Jul 20.
  • Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF, for the QUOROM group. Improving the quality of reports of meta–analyses of randomized controlled trials: the QUOROM statement. Lancet 1999;354:1896–1900.
  • Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, Moher D, et al for the Meta–analysis Of Observational Studies in Epidemiology (MOOSE) Group. Meta–analysis of observational studies in epidemiology. A proposal for reporting. JAMA 2000;283:2008–2012.


Advanced statistics: how to determine whether your intervention is different, at least as effective as, or equivalent: a basic introduction.

Are we looking for superiority, equivalence, or noninferiority? Asking the right question and answering it correctly.

An overview of the adaptive designs accelerating promising trials into treatments (ADAPT–IT) project.

Additional references:

  • Meurer WJ, Lewis RJ, Berry DA. Adaptive clinical trials: a partial remedy for the therapeutic misconception? JAMA. 2012 Jun 13;307(22):2377–8.
  • Moye LA. End–point interpretation in clinical trials: the case for discipline. Controlled Clinical Trials 1999;20:40–49.
  • Altman DG, Dore CJ. Randomisation and baseline comparisons in clinical trials. Lancet 1990;335:149–153.


Statistical methods in prehospital research.

Statistical methodology: IV. Analysis of variance, analysis of covariance, and multivariate analysis of variance.

Parametric statistical tests: unnecessary assumptions, computers, and the search for the trustworthy p–value.

Advanced statistics: linear regression, part I: simple linear regression.

Advanced statistics: linear regression, part II: multiple linear regression.

Advanced statistics: up with odds ratios! A case for odds ratios when outcomes are common.

Logistic regression: a brief primer.

Statistical models and Occam's razor.

Advanced statistics: statistical methods for analyzing cluster and cluster–randomized data.

An introduction to the Bayesian analysis of clinical trials.

Bayesian modeling and real–world problems.

Statistical methodology: IX. survival analysis.

Statistical methodology: V. Time series analysis using autoregressive integrated moving average (ARIMA) models.

Advanced statistics: bootstrapping confidence intervals for statistics with "difficult" distributions.

Statistical methodology: III. Receiver operating characteristic (ROC) curves.

Statistical methodology: VIII. Using confirmatory factor analysis (CFA) in emergency medicine research.

Statistical methodology: VII. Q–methodology, a structural analytic approach to medical subjectivity.

Additional references:

  • Kwok H, Lewis RJ. Bayesian hierarchical modeling and the integration of heterogeneous information on the effectiveness of cardiovascular therapies. Circ Cardiovasc Qual Outcomes. 2011 Nov 1;4(6):657–66.
  • Goodman SN. Toward evidence–based medical statistics. 2: The Bayes factor. Ann Intern Med 1999;130:1005–1013.
  • Spiegelhalter DJ, Myles JP, Jones DR, Abrams KR. An introduction to bayesian methods in health technology assessment. BMJ 1999;319:508–512.
  • D. J. Spiegelhalter, L. S. Freedman, and M. K. B. Parmar. Bayesian approaches to randomized trials. Journal of the Royal Statistical Society Series A, 157:357–416, 1994.
  • M. D. Hughes. Reporting Bayesian analyses of clinical trials. Statis Medicine, 12:1651–1663, 1993.
  • Greenland S. Prinicples of multilevel modeling. Int J Epidemiol 2000;29:158–167.
  • Clark TG, Bradburn MJ, Love SB, Altman DG. Survival analysis part I: basic concepts and first analysis. Br J Cancer 2003;89:232–238.
  • Bradburn MJ, Clark TG, Love SB, Altman DG. Survival analysis part II: multivariate data analysis – an introduction to concepts and methods. Br J Cancer 2003;89:431–436.
  • Bradburn MJ, Clark TG, Love SB, Altman DG. Survival analysis part III: multivariate data analysis – choosing a model and assessing its adequacy of fit. Br J Cancer 2003;89:605–611.
  • Clark TG, Bradburn MJ, Love SB, Altman DG. Survival analysis part IV: further concepts and methods in survival analysis. Br J Cancer 2003;89:781–786.
  • Whiting–O’Keefe QE, Henke C, Simborg DW. Choosing the correct unit of analysis in medical care experiments. Medical Care 1984;22:1101–1114.
  • Altman DG, Bland JM. Statistic notes: unit of analysis. BMJ 1997;314:1874
  • Royston P, Altman DG. Regression using fractional polynomials of continuous covariates: parsimonious parametric modeling. Appl Statist. 1994;43:429–467.


The importance of "shrinkage" in subgroup analyses.

Subgroups, Reanalyses, and Other Dangerous Things.

Additional references:

  • Brookes ST, Whitely E, Egger M, et al. Subgroup analyses in randomized trials: risks of subgroup–specific analyses; power and sample size for the interaction test. J Clin Epidemiol 2004;57(3):229–36.
  • Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet 2000;355(9209):1064–9.
  • Pocock SJ, Assmann SE, Enos LE, Kasten LE. Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practice and problems. Stat Med. 2002 Oct 15;21(19):2917–30.
  • Schulz KF, Grimes DA. Multiplicity in randomised trials II: subgroup and interim analyses. Lancet. 2005 May;365(9471):1657–61.
  • Lagakos SW. The challenge of subgroup analyses – reporting without distorting. N Engl J Med. 2006 Apr 20;354(16):1667–9.
  • Yusuf S, Wittes J, Probstfield J, Tyroler HA. Analysis and interpretation of treatment effects in subgroups of patients in randomized clinical trials. JAMA 1991;266:93–98.
  • Oxman AD, Guyatt GH. A consumer's guide to subgroup analyses. Ann Intern Med 1992;116:78–84.
  • Rothwell PM. Treating Individuals 2. Subgroup analysis in randomised controlled trials: importance, indications, and interpretation. Lancet 2005;365:176–86.
  • Moye LA, Deswal A. Trials within trials: confirmatory subgroup analyses in controlled clinical experiments. Control Clin Trials 2001;22:605–619.
  • Brookes ST, Whitley E, Peters TJ, et al. Subgroup analyses in randomised controlled trials: quantifying the risks of false positives and false negatives. Health Technology Assessment 2001;5(33):1 – 64.


Advanced statistics: missing data in clinical research––part 1: an introduction and conceptual framework.

Advanced statistics: missing data in clinical research––part 2: multiple imputation.

The validity of using multiple imputation for missing prehospital data in a state trauma registry.

Evaluating the Use of Existing Data Sources, Probabilistic Linkage and Multiple Imputation to Build Population–Based Injury Databases Across Phases of Trauma Care.

Additional references:

  • Rubin DB. Multiple Imputation for Nonresponse in Surveys. New York: John Wiley & Sons, Inc., 1987.
  • Little RJA and Rubin DB. Statistical analysis with missing data. 2nd edition. New Jersey: John Wiley and Sons, Inc. 2002.
  • Raghunathan T, Lepkowski, Van Hoewyk J, Solenberger P. A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology 2001;27:85–95.
  • Van Der Heijden GJMG, Donders ART, Stijnen T, Moons KGM. Imputation of missing values is superior to complete case analysis and the missing–indicator method in multivariable diagnostic research: A clinical example. J Clin Epid. 2006;59:1102–9.
  • Crawford SL, Tennstedt SL, McKinlay JB. A comparison of analytic methods for non–random missingness of outcome data. J Clin Epidemiol. 1995;48:209–219.
  • Joseph L, Belisle P, Tamim H, Sampalis JS. Selection bias found in interpreting analyses with missing data for the prehospital index for trauma. Journal of Clinical Epidemiology. 2004;57:147–153.
  • Greenland S, Finkle WD. A critical look at methods for handling missing covariates in epidemiologic regression analyses. American Journal of Epidemiology. 1995;142:1255–1264


Probabilistic linkage of computerized ambulance and inpatient hospital discharge records: a potential tool for evaluation of emergency medical services.

Validation of probabilistic linkage to match de–identified ambulance records to a state trauma registry.

(see also above article – Newgard et al. AEM 2012.)


Chart reviews in emergency medicine research: where are the methods?

Advanced statistics: understanding medical record review (MRR) studies.


Advanced statistics: the propensity score–a method for estimating treatment effect in observational research.

Additional references:

  • Stukel TA, Fisher ES, Wennberg DE, Alter DA, Gottlieb DJ, Vermeulen MJ. Analysis of observational studies in the presence of treatment selection bias: effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. JAMA. 2007 Jan 17;297(3):278–85.
  • McClellan M, McNeil BJ, Newhouse JP. Does more intensive treatment of acute myocardial infarction in the elderly reduce mortality? Analysis using instrumental variables. JAMA. 1994 Sep 21;272(11):859–66.


Qualitative research methodologies in emergency medicine.

Additional references:

  • Kuper A, Reeves S, Levinson W. An introduction to reading and appraising qualitative research. BMJ 2008;337:a288. doi:10.1136/bmj.a288
  • Lingard L, Albert M, Levinson W. Qualitative research. Grounded theory, mixed methods, and action research. BMJ 2008;337:a567. doi: 10.1136/bmj.39602.690162.47
  • Hodges BD, Kuper A, Reeves S. Qualitative research. Discourse analysis. BMJ 2008;337:a879. doi:10.1136/bmj.a879
  • Reeves S, Kuper A, Hodges BD. Qualitative research methodologies: Ethnography. BMJ 2008;337:a1020. doi:10.1136/bmj.a1020
  • Green J, Britten N. Qualitative research and evidence based medicine. BMJ 1998;316:1230–1232.
  • Mays N, Pope C. Qualitative research in health care. Assessing quality in qualitative research. BMJ 2000;320:50–52.
  • Pope C, Ziebland S, Mays N. Qualitative research in health care. Analysing qualitative data. BMJ 2000;320:114–116.
  • Giacomini MK, Cook DJ for the Evidence–Based Medicine Working Group. Users’ guides to the medical literature XXIII. Qualitative research in health care. A. Are the results of the study valid? JAMA 2000;284:357–362.
  • Giacomini MK, Cook DJ for the Evidence–Based Medicine Working Group. Users’ guides to the medical literature XXIII. Qualitative research in health care. B. What are the results and how do they help me care for my patients? JAMA 2000;284:478–482.


Methodologic standards for the development of clinical decision rules in emergency medicine.

Some thoughts on the stability of decision rules.

Medical decisionmaking: let's not forget the physician.

Additional references:

  • Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA. 1997 Feb 12;277(6):488–94.


Achieving graphical excellence: suggestions and methods for creating high–quality visual displays of experimental data.

Additional references:

  • Schriger DL, Altman DG, Vetter JA, Heafner T, Moher D: Forest plots in reports of systematic reviews: a cross–sectional study reviewing current practice. Int J Epid Int J Epid 2010;39:421–9.
  • Schriger DL, Savage DF, Altman DG. The Presentation of Continuous Outcomes in Randomized Trials: An Observational Study of a Missed Opportunity [In revision at BMJ]


Additional references:

  • Detsky AS, Naglie IG. A clinician's guide to cost–effectiveness analysis. Ann Intern Med 1990;113:147–154.
  • Russell LB, Gold MR, Siegel JE, Daniels N, Weinstein MC, for the Panel on Cost–Effectiveness in Health and Medicine. The role of cost–effectiveness analysis in health and medicine. JAMA 1996;276:1172–1177.
  • Weinstein MC, Siegel JE, Gold MR, Kamlet MS, Russell LB for the Panel on Cost–Effectiveness in Health and Medicine. Recommendations of the panel on cost–effectiveness in health and medicine. JAMA 1996;276:1253–1258.
  • Siegel JE, Weinstein MC, Russell LB, Gold MR for the Panel on Cost–Effectiveness in Health and Medicine. Recommendations for reporting cost–effectiveness analyses. JAMA 1996;276:1339–1341.


Evaluating bias and variability in diagnostic test results.

Statistical methodology: I. Incorporating the prevalence of disease into the sample size calculation for sensitivity and specificity.

Additional references:

  • Irwig L, Bossuyt P, Glasziou P, Gatsonis C, Lijmer J. Evidence base of clinical diagnosis. Designing studies to ensure that estimates of test accuracy are transferable. BMJ 2002;324:669–671
  • Whiting P, Rutjes AWS, Reitsma JB, Glas AS, Bossuyt PMM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy. A systematic review. Ann Intern Med 2004;140:189–202.
  • Lijmer JG, Mol BW, Heisterkamp A, Bonsel GJ, Prins MH, van der Meulen JHP, Bossuyt PMM. Empirical evidence of design–related bias in studies of diagnostic tests. JAMA 1999;282:1061–1066.
  • Bossuyt PM, Reitsma JB, Bruns DE, et al for the STARD group. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Ann Intern Med. 2003 Jan 7;138(1):40–4.
  • Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MM, Sterne JA, Bossuyt PM; the QUADAS–2 Group. QUADAS–2: A Revised Tool for the Quality Assessment of Diagnostic Accuracy Studies. Ann Intern Med. 2011 Oct 18;155(8):529–536.


What is confidence? Part 1: The use and interpretation of confidence intervals.

What is confidence? Part 2: Detailed definition and determination of confidence intervals.

p<0.05:Threshold for decerebrate genuflection.

Additional references:

  • Gardner MJ, Altman DG. Confidence intervals rather than P values: estimation rather than hypothesis testing. BMJ. 1986;292:746–750.
  • Poole C. Low P–values or narrow confidence intervals: which are more durable? Epidemiology 2001;12:291–294.
  • Goodman SN. Toward evidence–based medical statistics. 1: the p–value fallacy. Ann Intern Med 1999;130:995–1004.
  • Braitman LE. Confidence intervals assess both clinical significance and statistical significance. Ann Intern Med 1991;114:515–517.


Research fundamentals VI: misconduct in biomedical research.

Advanced statistics: applying statistical process control techniques to emergency medicine: a primer for providers.

Additional references:

  • Pereira TV, Horwitz RI, Ioannidis JP. Empirical evaluation of very large treatment effects of medical interventions. JAMA. 2012 Oct 24;308(16):1676–84.
  • Maclure M, Schneeweis S. Causation of bias: the episcope. Epidemiology 2001;12:114–122.
  • Bland JM, and Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet 1986;8476:307–310.
  • Schriger DL: Finding Truths in Clinical Medicine: Through the looking glass – cracked. Ann Emerg Med. 2001;38:566–9.
  • Schriger DL: How do we draw inference from "negative" studies? Ann Emerg Med. 2003;41:69–71.
  • Schriger DL. Getting the right message: avoiding overly optimistic interpretations of the scientific literature. Ann Emerg Med. 2006;48:75–6.
  • Schriger DL. Dynamic research, static literature. Ann Emerg Med. 2010;56:339–40. PMID: 20868905.