Reduced sociability is a core feature of autism spectrum disorders (ASD) and is highly disabling, poorly understood, and treatment refractory. To elucidate the biological basis of reduced sociability, multiple laboratories are developing ASD-relevant mouse models in which sociability is commonly assessed using the Social Choice Test. However, various measurements included in that test sometimes support different conclusions. Specifically, measurements of time the “test” mouse spends near a confined “stimulus” mouse (chamber scores) sometimes support different conclusions from measurements of time the test mouse sniffs the cylinder containing the stimulus mouse (cylinder scores). This raises the question of which type of measurements are best for assessing sociability. We assessed the test-retest reliability and ecological validity of chamber and cylinder scores. Compared with chamber scores, cylinder scores showed higher correlations between test and retest measurements, and cylinder scores showed higher correlations with time spent in social interaction in a more naturalistic phase of the test. This suggests that cylinder scores are more reliable and valid measures of sociability in mouse models. Cylinder scores are reported less commonly than chamber scores, perhaps because little work has been done to establish automated software systems for measuring the former. In this study, we found that a particular automated software system performed at least as well as human raters at measuring cylinder scores. Our data indicate that cylinder scores are more reliable and valid than chamber scores, and that the former can be measured very accurately using an automated video analysis system in ASD-relevant models. Anat Rec, 2011. © 2011 Wiley-Liss, Inc.