Statistical principal components analysis for retrieval experiments

Abstract

In this article, the statistical principal components analysis (PCA) is proposed as a method for performance comparisons of different retrieval strategies. It is shown that the PCA method can reveal implicit performance relations among retrieval systems across information needs (i.e., queries, topics). For illustration, the TREC 12 robust track data have been reevaluated by the PCA method and have been shown to reveal easily the performance relations that are hard to see with traditional techniques. Therefore, PCA promises a uniform evaluation framework that can be used for large-scale evaluation of retrieval experiments. In addition to the mean average precision (MAP) measure, relative analytic distance (RAD) is proposed as a new performance summary measure based on the same notion introduced by PCA.