Assessment of predictions in the model quality assessment category

Authors

  • Domenico Cozzetto,

    1. Department of Biochemical Sciences, University of Rome “La Sapienza”, P. le A. Moro, 00185 Rome, Italy
    Search for more papers by this author
    • Domenico Cozzetto and Andriy Kryshtafovych contributed equally to this work.

  • Andriy Kryshtafovych,

    1. Protein Structure Prediction Center, Genome Center, University of California, Davis, California
    Search for more papers by this author
    • Domenico Cozzetto and Andriy Kryshtafovych contributed equally to this work.

  • Michele Ceriani,

    1. Department of Biochemical Sciences, University of Rome “La Sapienza”, P. le A. Moro, 00185 Rome, Italy
    Search for more papers by this author
  • Anna Tramontano

    Corresponding author
    1. Department of Biochemical Sciences, University of Rome “La Sapienza”, P. le A. Moro, 00185 Rome, Italy
    2. Istituto Pasteur–Fondazione Cenci Bolognetti, University of Rome “La Sapienza”, P. le A. Moro, 00185 Rome, Italy
    • Department of Biochemical Sciences “Rossi Fanelli,” University of Rome “La Sapienza”, P.le Aldo Moro, 5, 00185 Rome, Italy
    Search for more papers by this author

Abstract

The article presents our evaluation of the predictions submitted to the model quality assessment (QA) category in CASP7. In this newly introduced category, predictors were asked to provide quality estimates for protein structure models. The QA category uses the automatically produced models that are traditionally distributed to CASP participants as input for predictions. Predictors were asked to provide an index of the quality of these individual models (QM1) as well as an index for the expected correctness of each of their residues (QM2). We computed the correlation between the observed and predicted quality of the models and of the individual residues achieved by the participating groups and evaluated the statistical significance of the differences. We also compared the results with those obtained by a “naïve predictor” that assigns a quality score related to how close the model is to the structure of the most similar protein of known structure. The aims of a method for assessing the overall quality of a model can be twofold: selecting the best (or one of the best) model(s) among a set of plausible choices, or assigning a nonrelative quality value to an individual model. The applications of the two strategies are different, albeit equally important. Our assessment of the QA category demonstrates that methods for addressing the first task effectively do exist, while there is room for improvement as far as the second aspect is concerned. Notwithstanding the limited number of groups submitting predictions for residue-level accuracy, our data demonstrate that a respectable accuracy in this task can be achieved by methods relying on the comparison of different models for the same target. Proteins 2007. © 2007 Wiley-Liss, Inc.

Ancillary