M > 1

Analysis of Treatment x Replication Designs

Authors

  • SALLY JACKSON,

    1. Sally Jackson is Associate Professor of Communication at the University of Arizona.Dale E. Brashers is now an instructor in the Department of Communication at Ohio State University.
    Search for more papers by this author
  • DALE E. BRASHERS

    1. Sally Jackson is Associate Professor of Communication at the University of Arizona.Dale E. Brashers is now an instructor in the Department of Communication at Ohio State University.
    Search for more papers by this author

  • The authors thank Bert Meuffels for useful comments and suggestions on an earlier version of this article. Special thanks go to Daniel J. O'Keefe for supplying effect size data and other meta-analysis statistics and for helpful comments on the manuscript. The support of the Netherlands Institute for Advanced Study in the Humanities and Social Sciences and of the University of Oklahoma Provost's Office is gratefully acknowledged.

Abstract

Increasingly, communication experiments are incorporating replication/actors for the purpose of controlling confounds and increasing generalizability. If replications are considered to be samples of possible treatment implementations, treating the replication factor as random is more appropriate than treating it as fixed. Study 1 shows that treating sampled replications as a fixed effect leads to potentially serious alpha inflation in the test of the treatment effect while treating sampled replications as random controls alpha at its nominal level. Study 2 addresses a common objection to treating replications as random: the argument that to do so will lead to unacceptably low power in statistical testing. Although experiments with very few replications are likely to be deficient in power, the results of Study 2 establish that power can be improved to an unexpected degree by a relatively modest increase in the number of replications.

Ancillary