A simulation study has been carried out to compare the results from using different randomization methods to assess the significance of the F-statistics for factor effects with analysis of variance. Two-way and three-way designs with and without replication were considered, with the randomization of observations, the restricted randomization of observations, and the randomization of different types of residuals. Data from normal, uniform, exponential, and an empirical distribution were considered. It was found that usually all methods of randomization gave similar results, as did the use of the usual F-distribution tables, and that no method of analysis was clearly superior to the others under all conditions. © 1998 John Wiley & Sons, Ltd.