The performance of Radio-Isotope IDentification (RIID) algorithms using gamma spectroscopy is increasingly becoming important. For example, sensors at locations that screen for illicit nuclear material rely on isotope identification to resolve innocent nuisance alarms arising from naturally occurring radioactive material. Recent data collections for RIID testing consist of repeat measurements for each of several scenarios to test RIID algorithms. Efficient allocation of measurement resources requires an appropriate number of repeats for each scenario. To help allocate measurement resources in such data collections for RIID algorithm testing, we consider using only a few real repeats per scenario. In order to reduce uncertainty in the estimated RIID algorithm performance for each scenario, the potential merit of augmenting these real repeats with realistic synthetic repeats is also considered. Our results suggest that for the scenarios and algorithms considered, approximately 10 real repeats augmented with simulated repeats will result in an estimate having comparable uncertainty to the estimate based on using 60 real repeats. Published in 2009 by John Wiley & Sons, Ltd.