SEARCH

SEARCH BY CITATION

Keywords:

  • CASP;
  • residue-residue contact prediction;
  • RR

ABSTRACT

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. CONCLUSIONS
  7. ACKNOWLEDGMENTS
  8. REFERENCES
  9. Supporting Information

We present the results of the assessment of the intramolecular residue-residue contact predictions from 26 prediction groups participating in the 10th round of the CASP experiment. The most recently developed direct coupling analysis methods did not take part in the experiment likely because they require a very deep sequence alignment not available for any of the 114 CASP10 targets. The performance of contact prediction methods was evaluated with the measures used in previous CASPs (i.e., prediction accuracy and the difference between the distribution of the predicted contacts and that of all pairs of residues in the target protein), as well as new measures, such as the Matthews correlation coefficient, the area under the precision-recall curve and the ranks of the first correctly and incorrectly predicted contact. We also evaluated the ability to detect interdomain contacts and tested whether the difficulty of predicting contacts depends upon the protein length and the depth of the family sequence alignment. The analyses were carried out on the target domains for which structural homologs did not exist or were difficult to identify. The evaluation was performed for all types of contacts (short, medium, and long-range), with emphasis placed on long-range contacts, i.e. those involving residues separated by at least 24 residues along the sequence. The assessment suggests that the best CASP10 contact prediction methods perform at approximately the same level, and comparably to those participating in CASP9. Proteins 2014; 82(Suppl 2):138–153. © 2013 Wiley Periodicals, Inc.


Abbreviations
FM

free modeling

MCC

the Matthews correlation coefficient

RR

residue-residue (contacts)

TBM

template-based modeling.

INTRODUCTION

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. CONCLUSIONS
  7. ACKNOWLEDGMENTS
  8. REFERENCES
  9. Supporting Information

Inter-residue contacts have been shown instrumental in reconstructing protein backbones by means of distance geometry or restrained molecular dynamics.[1-3] This finding suggested that the prediction of intramolecular contacts in proteins can serve as an intermediate step toward accurate prediction of the three-dimensional structure, and triggered extensive research to connect protein sequence and structure with a “two-span bridge”: from sequence to contacts and from contacts to structure. To build such a bridge, the researchers focused on predicting contacts with accuracy sufficiently high to be useful for structure modeling on one side, and on building a structure from incomplete/inaccurate contact data, on the other.

As far as the area of structure rebuilding is concerned, a series of papers published in the 1990s demonstrated that protein contact maps can indeed serve as scaffolds for building protein structures even when the maps are sparse or contain just a fraction of correct contacts.[4-8] A few features related to the tolerance of these methods to data uncertainty and incompleteness were discovered. In particular, in a pioneering work,[1] Havel et al. speculated that it is better to know many distances imprecisely rather than a few distances accurately. Saitoh et al.[5] noticed that the only factor largely influencing the quality of the reconstructed structures is the long-range geometrical constraint. Skolnick et al. suggested[7] that knowing contacts for one in every seven residues would be sufficient to recover the structure of short proteins. Later, Vassura et al.[9] claimed that knowing one in four actual contacts might be enough to facilitate rebuilding tertiary structure with 5 Å accuracy. Although in general it is still unclear what accuracy, coverage, and distribution of contacts along the sequence are needed to be useful in practice, it has become common knowledge that information on just a few correct contacts can be valuable for improving structure prediction. This is especially true for the long-range contacts, which impose strong constraints on the three-dimensional structure and effectively narrow the search space of possible conformations. The usefulness of the contact approach was illustrated in the current edition of CASP, where predictors in the newly introduced contact-assisted structure prediction category (see the contact-assisted assessment article, this issue) were able to build substantially better models using information provided by the organizers on some of the long-range contacts in the target structures. Other studies also report that incorporating contact information into protein folding programs such as Rosetta and I-TASSER leads to improvement of the 3D models.[10, 11]

Returning to the first bridge span in the “two-span bridge” analogy, substantial attention was dedicated to the prediction of intramolecular contacts. Much of the research in this area stemmed from the hypothesis of correlated mutations, suggesting that pairs of residues that mutate in a coordinated fashion during evolution are likely to be in contact. In the 1990s, the first articles demonstrating the applicability of this idea to contact prediction were published.[12-14] After these promising results, a series of contact prediction methods developing this concept further appeared in the literature.[15] Quite recently, the 20-year-old idea received a new twist as several articles claimed improved accuracy of contact prediction through disentangling the direct pairwise couplings from the background network of coordinately mutating positions.[15-22] Besides the coordinated mutations approaches, many other contact prediction methods were developed based on different or hybrid methodological concepts. In general, they are based on machine-learning techniques incorporating sequence-related features such as the sequence evolutionary profile of the target, secondary structure, and solvent accessibility—to name just a few. These methods use neural networks,[23-29] support vector machines,[30-32] hidden Markov models,[33-35] genetic algorithms,[36] random forest models,[37] and learning classifier systems.[38] Many of the methods mentioned above were tested in CASP experiments achieving different levels of success.

The prediction of residue-residue contacts has been a part of the CASP experiment since CASP2[39] (1996), however, the prediction format and the assessment procedures have been standardized only in CASP6–CASP9.[40-43] For CASP10, we developed an infrastructure for an automatic evaluation of the RR predictions and visual analysis of the results.[44] Here we analyze the results obtained by groups participating in CASP10 and quantify progress in the area compared with the previous CASPs.

MATERIALS AND METHODS

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. CONCLUSIONS
  7. ACKNOWLEDGMENTS
  8. REFERENCES
  9. Supporting Information

RR prediction format and definition of a contact

The RR prediction format and definition of intramolecular contacts in CASP10 have not changed since previous rounds of CASP. A pair of residues is defined to be in contact when the distance between their Cβ atoms (Cα in case of GLY) is less than 8.0 Å. Depending on the separation along the sequence, short-, medium- and long-range contacts are between residues separated by 6 to 11, 12 to 23, and at least 24 residues, respectively. The contacts with a separation of less than six residues are not considered as they typically correspond to contacts within secondary structure elements. The participating groups were asked to submit a list of pairs of residues predicted to be in contact. Each reported contact had to be annotated with a probability score in the [0;1] range, reflecting the predictor confidence in assigning the contact. Unlike the previous rounds of CASP, only one set of contact predictions per target was allowed in CASP10 for each participating group.

Sets of domains evaluated

The evaluation of predictions was carried out on a per-domain basis. The domains with detectable homology to proteins of known structures were not included in the evaluation as in these cases contacts could easily be derived from the template structures. Thus, we used only the domains for which structural templates did not exist or were very difficult to identify, that is, the domains classified in the FM, TBM/FM, or TBM_hard categories.[45] The complete list of CASP10 domains with their classifications is available at http:/predictioncenter.org/casp10/domains_summary.cgi.

We assessed the performance of contact prediction methods on two sets of domains.

Set 1 (denoted as “FM”) comprises 15 FM and 1 FM/TBM domains. For these domains templates did not exist or could not be reliably identified based on the target sequence. Set 1 is our main evaluation set and is consistent with the sets used in previous rounds of CASP.

Set 2 (hereinafter referred to as “FM + TBM_hard”) is an extension of the previous set obtained by adding the domains from the TBM_hard category (13 entries). These are the hardest TBM targets, for which templates exist but are hard to identify or to properly align with the target. As a consequence, the scores of all submitted three-dimensional models for these targets were rather poor, not exceeding 50 GDT_TS units.[45]

We also performed the assessment on two sets of targets generated from the original two sets by eliminating non-globular proteins consisting of repeated structural blocks: Set 1R = Set 1 {T0653-D1, T0695-D1}, and Set 2R = Set 2 {T0653-D1, T0671-D2, T0690-D1, T0695-D1}.

The first three targets removed from Set 2 are the well-known leucine-rich repeats,[46] while the last one is a three-helical spectrin bundle repeated five times.[62] All four structures are built with repeated structural blocks for which good templates exist. Since the majority of contacts for these domains could be derived from the templates, their inclusion could introduce a bias in the evaluation. In practice, differences in the results on the original and the reduced sets were minor for the majority of analyses, and therefore we present here the results only for the original datasets, except for the domain-length dependence analysis, where using the reduced sets is more appropriate.

An estimate of the difficulty of individual domains for contact prediction is shown in Supporting Information Figure S1.

Sets of evaluated contacts

To compare the performance of contact prediction methods we used two different approaches. In the first approach, we trimmed the predicted lists of contacts to the same number of contacts per target (see the Reduced contact lists subsection below); in the second, we “padded” the lists by assigning a probability value of 0 to all non-listed contacts. The both procedures ensure that the participating groups are compared on the same number of contacts.

Preprocessing of predictions

For multidomain targets, we extracted the lists of inter-residue contacts for each individual domain. This step was necessary as predictions were submitted for the entire targets, but evaluated on a per-domain basis (see above). We also considered contacts between residues from different domains as their correct prediction can be useful in predicting the orientation of the interacting domains.

For each prediction, we separated short-, medium-, and long-range contacts and assessed them independently. The medium and long-range contacts were also assessed together.

Reduced contact lists

For every domain, the lists described above were trimmed to the L/5 and L/10 contacts predicted with higher probability (L is the length of the domain). The number L/5 (or L/10) is rounded to the closest integer, and if there are multiple entries corresponding to the same probability they are considered in the order provided by the predictor. To be included in the evaluation, the filtered list of contacts had to comprise at least L/5 or L/10 contacts. In order to assess also the groups that submitted only very small numbers of contacts, we also evaluated predictions on the five contacts with the highest assigned probability values, regardless of the domain length.

Thus, for every group we generated 12 reduced lists of contacts per predicted domain, whenever possible. The results for all lists of contacts and all contact range categories are available at http:/predictioncenter.org/casp10/rr_results.cgi. In this paper we focus on the results for the L/5 lists of long-range contacts. The numbers of domains predicted on these datasets for each of the participating groups are summarized in Figure 1. Two groups (G334 and G077) submitted just a few predictions for the evaluated domains and one (G246) did none, so we excluded them from the analysis and present the results on the reduced lists for the remaining 23 groups. For every group, the final scores on the reduced datasets are averages of the per-domain scores.

image

Figure 1. Number of domains per group for which the L/5 list of long-range contacts were evaluated. Two groups RBO-CON (G334) and FLOUDAS (G077) submitted too few predictions and are not included in the subsequent analyses.

Download figure to PowerPoint

Padded contact maps

As contact probability maps generated from submitted predictions are sparse, they are usually unsuitable for many analyses that require complete predictions (i.e. we need each pair of residues to be predicted either in contact or not). We remediate the “sparseness” problem here by setting the values of the empty cells of contact probability maps to zero (“padded” lists). In other words, pairs of residues that are missing in predictions are considered as non-contacts. Under such assumption, each prediction list classifies every pair of residues within the selected range to one of the four cases: TP, correctly predicted contact; FP, non-contact predicted as contact; TN, correctly “predicted” non-contact (i.e., the non-contact not included in the predicted contact list); and FN, contact “predicted” as non-contact (i.e., the contact missing in the submitted list).

We only assessed the groups that submitted predictions for at least 10 domains on the “padded” datasets—these are the same 23 groups as above, plus group G334. As in the case of the reduced contact lists, in this article we concentrate on the analysis of the performance of the participating groups for the long-range contacts only. Differently from the assessment on the reduced contact lists, the final group scores on the padded datasets are calculated from the data on all domains pooled together.

Evaluation procedure

In CASP10 we have substantially expanded the set of evaluation tools to assess residue-residue contact predictions. Besides the methods used in the previous CASPs, we introduced several new evaluations providing an alternative point of view on methods' performance. While in previous CASPs the assessors analyzed the results exclusively on the “reduced” datasets, implicitly concentrating on two aspects of contact prediction: (1) how good are methods in identifying the most reliable predicted contacts and (2) how accurate are the methods in predicting contacts with the highest reliability, in this CASP we complemented the assessment with analyses on the full sets of contacts addressing the issue of how accurate are all submitted contact predictions, including those predicted with lower reliability. Below, we briefly outline all evaluation procedures, focusing in more detail on the new evaluation measures.

Basic scoring functions and group performance on the reduced datasets

Since CASP6, predictions in the RR category have been evaluated on the reduced contact lists using two main scores: precision = TP/(TP + FP), and Xd. The detailed description of these scores can be found in the previous CASP contact assessment articles.[40-43] Note, that in those papers the measure defined by the formula TP/(TP + FP) was called “accuracy” (Acc); here we have changed its name to “precision” to be consistent with the classic descriptive statistics definition. The precision-based results are discussed in the main text of this article, while the Xd-based results are shown in the Supporting Information.

Based on these two scores, the performance of groups was further compared with two strategies: cumulative z-score ranking (sum of precision-based and Xd-based z-scores) and “head-to-head” comparisons.[43]

Evaluation measures for the padded datasets

Matthews' correlation coefficient and other binary descriptive statistics measures

For the assessment of the effectiveness of the predictive methods as binary classifiers we used four evaluation measures.

The first two are precision and recall, a.k.a. sensitivity:

  • display math

They were already used in previous CASPs, but were shown to be equivalent on the reduced prediction sets.[41] On the complete datasets, precision and recall are not inter-dependent any more as the number of predicted contacts is different for different predictions. Based on the formulae, one can notice that each of these measures takes into account only two out of the four parameters of prediction quality (TP, FP, TN, and FN) and therefore focuses on the specific aspects of predicting contacts only (ignoring non-contacts).

The F-score is a more comprehensive measure as it combines precision and recall

  • display math

and inherits useful features typical to both measures. However, the F-measure still does not take the true negative rate into account.

Even though employing measures that take all parameters of contact prediction into account may seem beneficial, it should be approached with caution, as in our case two binary classes of prediction (contacts and non-contacts) are disproportionally distributed in the structure (contacts constitute just a small fraction of all pairs of residues). As it was discussed in the CASP9 disorder assessment article,[47] the Matthews correlation coefficient (MCC)

  • display math

is a well-suited measure for handling cases with imbalanced class frequencies. The MCC was shown to provide a more appropriate account of the skewed data than many other methods, and not to favor over-prediction of any classes. Therefore, in this article we consider this measure as the main estimator of binary classifiers on the expanded datasets.

Precision-recall curve analysis

In previous rounds of CASP, the probability score assigned to every predicted contact was used in assessment only to select the most reliable contacts (according to the predictors' estimates) for the reduced evaluation datasets. However one can argue that the probability score holds valuable information that can be used both in modeling of the structure and in assessment. For example, it can be used to test the ability of predictors to correctly rank the predicted contacts and select the proper cut-off separating contacts (positive cases) from non-contacts (negative cases).

To address these issues we carried out the analysis based on the precision-recall (PR) curves, which are widely used in statistical evaluations of disproportional datasets.[48-51] The PR-curve analysis is conceptually similar to the well-known ROC-curve analysis,[52] but differs in that the parametric curves are plotted in the (recall, precision) coordinates. Davis and Goadrich[53] proved that the dominant curve in ROC space corresponds to the dominant curve in PR space and vice versa, and showed that the curves in PR space may be more informative for skewed data, as ROC curves tend to provide overly optimistic results in such cases.

In essence, a PR-curve illustrates the relationship between the precision and recall of a predictor for a set of probability thresholds. For each threshold, a record (pair of residues in our case) is considered as a positive example (contact) if its predicted probability is equal to or greater than the threshold value. The area under the PR-curve, AUC_PR, is indicative of the classifier's accuracy, with a value of 1 corresponding to a perfect predictor. The AUC_PR values were calculated using the software developed by Davis and Goadrich[53] and freely available from their website.[54]

The Jaccard distance for clustering methods

The dissimilarity between two groups for each target is defined in terms of the Jaccard distance:[55]

  • display math

where M11 is the number of common contacts predicted by groups i and j, M10 and M01 are the contacts only predicted by group i and j, respectively. The J-score has values in the range of [0;1], with the value of 0 corresponding to identical predictors and 1 - to completely dissimilar ones.

The tie-breaking procedure for defining the first correct/incorrect contact

If prediction contains several contacts with the same probability value, the position of the first correct/incorrect prediction is assigned regardless of whether there are incorrect/correct predictions with the same probability. In other words, if the correct prediction with the highest probability has the same probability, and therefore the same rank R, as one or more incorrect predictions, the correct prediction is assigned rank R. Analogously, the position of the first incorrect prediction is assigned regardless of whether there are correct predictions with the same probability, i.e. if the first incorrect prediction has the same rank R as a correct prediction, the first incorrect prediction is assigned rank R.

RESULTS

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. CONCLUSIONS
  7. ACKNOWLEDGMENTS
  8. REFERENCES
  9. Supporting Information

Participating methods: Brief description and similarity

In CASP10 26 groups submitted predictions of intra-molecular contacts, including 22 automated servers and four expert groups. Three groups used new methods, while others used modified techniques developed earlier and tested in previous rounds of CASP. Table 1 presents a short description of the participating publicly available contact prediction servers. A more detailed overview of all the methods participating in CASP10 can be found in the CASP10 Abstract Book.[56]

Table 1. The Publicly Available Contact Prediction Servers Participating in CASP10
Server name and URL addressCASP10 groupBrief description of the method
  1. a

    New methods according to the CASP10 Abstract Book.

CMAPproa. Available at: http:/scratch.proteomics.ics.uci.edu/G305Deep neural networks architecture allowing progressive refinement of contact prediction.
Distill, Distill-roll. Available at: http://distill.ucd.ie/distill/G072 and G087Two-dimensional-recursive neutral networks.
ICOS. Available at: http:/icos.cs.nott.ac.uk/servers/psp.htmlG184Inhouse machine-learning technique taking into account nine-residue window profiles, secondary structure, and other features.
MULTICOM-CLUSTER. Available at: http:/casp.rnet.missouri.edu/svmcon.htmlG081An SVM tool. The input data include secondary structure, solvent accessibility, and sequence profile.
MULTICOM-CONSTRUCTa. Available at: http:/iris.rnet.missouri.edu/dncon/G222Ensembles of deep networks.
MULTICOM-NOVEL,MULTICOM-REFINE. Available at: http:/casp.rnet.missouri.edu/nncon.htmlG424 and G125Recursive neural networks. MULTICOM-REFINE has a separate module to predict contacts in beta-sheets.
PROC_S3. Available at: http:/www.abl.ku.edu/proc/proc_s3.htmlG257Random Forest models incorporating more than 1000 sequence-related features.
SAM-T06, SAM-T08. Available at: http:/compbio.soe.ucsc.edu/SAM06/ and http:/compbio.soe.ucsc.edu/SAM08/G381 and G113Recursive neural networks using the correlated mutations in MSA.
Samcha-servera. Available at: http:/binfolab12.kaist.ac.kr/conti/G112SVM incorporating more than 800 sequential features.

Not all methods are conceptually different as oftentimes they rely on similar prediction techniques using similar mathematical apparatus and predictive features. To illustrate this, we clustered the methods participating in CASP10 based on the pair-wise Jaccard distance (see Materials). Figure 2 shows the results of the method clustering. As one can notice, four lowest level clusters encompass two prediction groups each from the same research centers, i.e. two Proc-S, Distill, Multicom, and confuzz methods. It is apparent that the clustered groups use similar methodologies with slight modifications in the implementation of the method.

image

Figure 2. Dendrogram illustrating the similarity among different methods as judged by the number of common predictions for all targets.

Download figure to PowerPoint

Group performance on the reduced datasets: Precision and Xd

The results of the analysis of the group performance for long-range contacts in the L/5 contact lists are presented in Figure 3. For each group we show the values of precision and cumulative z-score (sum of precision-based and Xd-based z-scores) averaged over all predicted domains from the “FM” and “FM + TBM_hard” datasets (see Materials for a detailed description of the datasets and evaluation measures).

image

Figure 3. Precision (A) and cumulative z-score (B) for the participating groups on the two sets of the evaluated domains (FM and FM + TBM_hard). The data are shown for the top L/5 long-range contacts. Groups in both panels are ordered according to their cumulative z-score on FM targets.

Download figure to PowerPoint

Panel A of Figure 3 demonstrates that the precision of the current prediction methods on FM targets does not exceed 20%. The three best performing groups on the FM targets (G125, G222, and G424) attain precision of 19% and belong to the same family of methods (Multicom, group leader J. Cheng, University of Missouri). Multicom-construct method (G222) was also shown to reach the highest score according to the Xd measure (see Fig. S2 in Supporting Information), and is ranked first according to the cumulative z-score (Fig. 3, panel B). It should be mentioned, though, that the difference in performance of this method and the others is marginal, as Student's t-tests did not reveal statistically significant difference in the performance of the top ten methods (see Table 2 for precision and Table S1 in Supporting Information for Xd). This statement is supported by the results of the “head-to-head” comparison (Table 3 and Table S2 in Supporting Information) where no method was shown to consistently over-score any other method on more than half of the domains.

Table 2. Results of the Paired Student's t-Test on the Precision Score for (A) FM and (B) FM, TBM-Hard Domains for Top 10 Groups According to the Cumulative z-Score Ranking
AG222G358G305G413G424G125G113G087G489G314
  1. The tables show the P values (cells below the diagonal) of the Student's t-tests performed for each pair of the groups on the common set of domains (the numbers above the diagonal). Shaded cells indicate statistically indistinguishable results at the significance level of 0.05.

G222x91511141412151415
G3580.49x96889989
G3050.180.3x12151512161515
G4130.110.190.19x11118121111
G4240.070.090.440.13X1511151414
G1250.160.10.390.130.26x11151414
G1130.50.340.390.240.080.08x121112
G0870.260.460.40.460.360.320.44x1515
G4890.330.10.370.40.340.320.310.49x14
G3140.190.210.410.090.210.340.370.480.48x
           
BG489G087G222G413G358G072G305G113G424G125
G489x272618162727212526
G0870.03x2820182929232728
G2220.020.34x19182828232627
G4130.050.340.08x102020141819
G358<0.010.030.260.12x1818181717
G072<0.010.080.360.430.11x29232728
G3050.010.070.010.180.250.21x232728
G113<0.010.120.180.380.470.240.14x2122
G4240.010.03<0.010.36<0.010.110.170x27
G1250.010.03<0.010.33<0.010.110.1900.39x
Table 3. The “Head-to-Head” Comparison of the Performance of the Groups Based on the precision Score for (A) FM and (B) FM, TBM-Hard Domains for the Top 10 Groups According to the Cumulative z-Score Ranking
A Group 2
G222G358G305G413G424G125G113G087G489G314
  1. The rows show the fraction of common domains for which the precision score of the group in the row is higher than that of the group in the column. Cases of equal scores are not counted.

Group 1G222X44.4%46.7%63.6%50.0%50.0%41.7%66.7%64.3%46.7%
 G35844.4%x44.4%16.7%75.0%75.0%33.3%33.3%37.5%55.6%
 G30533.3%33.3%x58.3%40.0%40.0%33.3%50.0%60.0%40.0%
 G41327.3%66.7%25.0%x36.4%36.4%25.0%58.3%45.5%27.3%
 G42421.4%12.5%53.3%45.5%x13.3%18.2%46.7%35.7%28.6%
 G12528.6%12.5%46.7%45.5%13.3%x18.2%53.3%35.7%21.4%
 G11333.3%44.4%50.0%50.0%63.6%63.6%x66.7%45.5%50.0%
 G08726.7%55.6%37.5%41.7%33.3%26.7%33.3%x26.7%40.0%
 G48928.6%50.0%40.0%45.5%50.0%50.0%45.5%60.0%x35.7%
 G31433.3%33.3%46.7%54.5%64.3%57.1%33.3%60.0%57.1%x
            
B Group 2
G489G087G222G413G358G072G305G113G424G125
Group 1G489x63.0%53.8%55.6%75.0%77.8%66.7%71.4%72.0%73.1%
 G08725.9%x35.7%45.0%61.1%55.2%51.7%43.5%55.6%50.0%
 G22242.3%60.7%x63.2%55.6%57.1%60.7%56.5%69.2%74.1%
 G41338.9%55.0%31.6%x70.0%50.0%50.0%35.7%61.1%63.2%
 G35818.8%27.8%38.9%20.0%x50.0%50.0%33.3%76.5%70.6%
 G07214.8%24.1%35.7%45.0%44.4%x44.8%39.1%59.3%57.1%
 G30533.3%31.0%25.0%40.0%33.3%41.4%x30.4%51.9%50.0%
 G11323.8%47.8%26.1%35.7%44.4%52.2%47.8%x71.4%72.7%
 G42420.0%29.6%15.4%27.8%17.6%33.3%40.7%14.3%x29.6%
 G12519.2%35.7%14.8%26.3%17.6%35.7%39.3%13.6%14.8%x

For the set of FM and TBM_hard domains, there is a group clearly outperforming the others, Multicom (G489), the results of which (Fig. 3) definitely look better than those of other groups (precision over 35% with the next best value of 24% for the Distill_roll group). The Multicom group is shown to be statistically better than all other predictors on the FM + TBM_hard set of targets (see Table 2 in the main text and Table S1 in Supporting Information) and consistently better than other methods in head-to-head comparisons (Table 3 and Table S2 in Supporting Information). However, it should be mentioned that the method used by group G489 is not conceptually an ab initio contact prediction method, as it relies on the three-dimensional models submitted by CASP10 servers. The better performance of this group on the FM + TBM_hard dataset can be explained by the method's consensus strategy, which works well on the TBM targets that constitute a substantial fraction of the FM + TBM_hard dataset.

Dependence of group performance on the domain length and the depth of alignment

Figure S1 (Supporting Information) shows that the contacts are harder to predict for some domains. The predictive difficulty of a domain is not always directly connected with the availability of templates, and from Figure S1 it can be seen that in CASP10 the third easiest target (T0739-D2) is in fact an FM domain, while the second hardest (T0668-D1) is a template-based target. This raises the question of which other features, besides template availability, may influence the accuracy of contact prediction. In particular, we investigated the influence of domain length and depth of alignment.

Figure 4(A) shows the precision of the best 10 performing groups as a function of domain length. The CASP10 FM dataset covers a wide range of domain length spanning from 58 to 535 residues. Two domains are short (under 60 residues), two rather long (over 390 residues) and the remaining 12 are of medium length (80–220 residues). On four of the domains (the shortest two and one from each of the medium and long sub-ranges), the best groups reach a very high precision (over 50%). It should be noticed, though, that the two longest domains in this graph (T0653-D1 and T0695-D1) represent non-globular targets with a repeated topology (see the description of Set 2R in Materials), and this may introduce bias in the analysis. Therefore, we analyzed per-group trends in the results excluding these two domains. Inspection of the graph reveals that the vast majority of groups reach better precision on shorter targets.

image

Figure 4. Precision of the prediction methods as a function of domain length (A) and depth of the alignment (B). The data are shown for the top L/5 long-range contacts.

Download figure to PowerPoint

To analyze the dependence of group performance on the depth of the target alignments, we searched for sequence homologs for each target with PSI-BLAST[57] running five iterations against the non-redundant database with parameters “-h 0.05 -v 1000 -b 1000.” The number of hits covering at least 75% of target's sequence was used as a measure of the alignment depth. The depth of the alignment for CASP10 FM targets varied from just a few hits (for T0726-D3, T0741-D1, T0740-D1) to more than a thousand for two repeat-topology domains (T0653-D1 and T0695-D1). Figure 4(B) shows that CASP10 methods are in general insensitive to the alignment depth, as no trend in the data can be detected. As precision of group performance depends on target length, we also tested a hypothesis that length can be a contributing factor in how precision depends on depth of alignment. Our additional analysis showed that this is not the case.

Group performance on the untrimmed contact lists: PR-curve and MCC analyses

Figure 5 and Table 4 present a different perspective on the methods' performance based on the PR-curve analysis, MCC and other descriptive statistics measures (see Materials).

image

Figure 5. PR-curves for all predicted long-range contacts on FM domains.

Download figure to PowerPoint

Table 4. Descriptive Statistics Scores Calculated for the Predictions Treated in the Context of the Complete Contact Maps for Long-Range Contacts for FM Domains
GroupNo domTPFPTNFNMCCPrecision (%)Recall (%)F1AUC_PR
  1. The results are sorted according to the MCC score.

G4891584111,33127,579818380.1316.931.40.1130.095
G08716117522,200266,87615100.1275.043.80.0900.065
G2221690515,025274,05117800.1205.733.70.0970.043
G0721681413,674275,40218710.1125.630.30.0950.049
G39616100621,195267,88116790.1094.537.50.0810.038
G25716133143,378245,69813540.0923.049.60.0560.038
G1131355911,025266,17617850.0914.823.80.0800.025
G31416125542,987246,08914300.0852.846.70.0530.032
G1251659712,454276,62220880.0834.622.20.0760.034
G4131284134,071152,5437420.0822.453.10.0460.038
G4241654010,788278,28821450.0814.820.10.0770.035
G081163124287284,78923730.0786.811.60.0860.018
G11213154289,198107,6282690.0761.785.10.0330.027
G1841657015,934273,14221150.0653.521.20.0590.021
G13910129482,600167,4566250.0631.567.40.0300.015
G3321673825,302263,77419470.0632.827.50.0510.026
G3811344813,232263,96918960.0613.319.10.0560.019
G305161952123,658165,4187330.0581.672.70.0300.038
G358161541969287,10725310.0577.35.70.0640.026
G1801249030,968155,64610930.0351.631.00.0300.012
G3341218241154,08121660.0196.90.80.0140.015
G4751219857261,34921690.0092.20.90.0120.009
G098121398270,4571345−0.0051.31.00.0110.018
G462121198170,4581347−0.0071.10.80.0090.018

The PR-curve analysis clearly identifies the top performing group, G489 (Multicom), which reaches an AUC_PR score of 9.5%. Again, we remind here that this group does not predict contacts directly from the sequence but relies on the submitted three-dimensional models. The two other groups that stand out in the PR-curve analysis are G087 and G072, both from the Distill family of methods (group leader G. Pollastri, University College Dublin).

The results of the PR-analysis (AUC_PR scores) are shown to be well correlated with the MCC and F1 scores presented in Table 4. The Pearson correlation coefficients for these two pairs of scores are 0.76 and 0.71, respectively. Also there is a high correlation (0.90) between the MCC and F1 scores. At the same time, the correlation between other measures presented in Table 4 is substantially lower (except for the F1 – precision correlation) confirming that these (low-correlated) measures highlight different aspects of contact prediction.

Position of the first correct and incorrect contact

The prediction of contacts in protein structures can be used as input for computational methods aimed at structure prediction and, in this case, the correct ranking of the contacts in terms of their probability might not be necessarily relevant. On the other hand, prediction of specific contacts in a protein might shed light on its functional or structural properties and in this case, their correctness should be experimentally tested before drawing conclusions. This is usually done by designing appropriate mutations of the residues predicted to be in contact, expressing the mutated protein(s) and testing their function (see for example Refs. [58-61]). Clearly, one would like to perform as few experiments as possible. Since contact predictions are provided together with estimates of their reliability, it is reasonable to expect that the contacts would be tested in the order they appear in the list of predictions. This raises the question of how much down the ordered list of contacts is the first correct prediction for a given method.

We computed the position of the first correct prediction as well as the position of the first error for each target and each group considering short, medium, and long-range contacts. The results of this analysis are available from the CASP10 web site (http://predictioncenter.org/casp10/rr_additional.cgi). As in other sections, here we concentrate on the results for long-range contacts on FM targets.

Figure 6(A) shows, for each group, the percentage of times in which the first correct prediction is found in a given position; Figure 6(B) shows the percentage of times in which the first incorrect prediction is found in a given position. Group G489 that performs better than the other groups has a correct prediction in the first position on the L/5 contact lists 56% of the times and in 13% of the cases the first correct prediction is in position 2. Other groups also often have the first correct prediction ranking high in the list. It is instructive to compare the two parts of the figure. For example, group G184 has a correct prediction in one of the top positions about 40% of the time, but also often it has an incorrect prediction in the first positions. This is due to the fact that this group often assigns the same probability values to a set of contacts, some correct and some incorrect.

image

Figure 6. Percent of cases where the first correct (A) and first incorrect (B) prediction is in the reported position for each group. Rows are ordered according to the percentage in the first column of A. The data are shown for the top L/5 long-range contacts in FM domains.

Download figure to PowerPoint

Interdomain contact predictions

The prediction of contacts between different domains can be extremely useful in cases where multidomain proteins are modeled using different templates for the different domains, since the step of packing together the partial models can, and often does, introduce errors.

We analyzed the number of cases in which different participating groups correctly predicted contacts between residues belonging to two different domains. The results for interdomain long-range contacts in FM targets are summarized in Table 5, and the example for target T0658 is shown in Figure 7. Table 5 shows that in this analysis the best results are achieved by group G489, followed by groups G112 and G072.

Table 5. Results of the Prediction of Long-Range Contacts in Which the Contacting Residues Belong to Two Different Domains
GroupFPTPPrecision (%)
  1. The data are for the L/5 contacts with higher predicted probability.

G489265186.4
G08725972.6
G07226172.6
G4758422.3
G11224652.0
G38121331.4
G3347411.3
G08121720.9
G33226120.8
G13918210.5
G18021710.5
G42423110.4
G0773500.0
G09822100.0
G11323100.0
G12525900.0
G18423200.0
G22224900.0
G25730500.0
G30530500.0
G31430500.0
G35821200.0
G39630500.0
G41321800.0
G46221500.0
image

Figure 7. Example of the prediction of inter-domain contacts for target T0658. This is a two domain protein with the first domain (residues 20–185) being an FM target and the second (residues 186–540)—a template based target. The top panel shows L/5 contacts correctly predicted by at least one group as arcs connecting the corresponding residues indicated by circles. We show all the residues involved in correctly predicted contacts in the first (FM) domain, both intra- and inter-domain, and only the residues involved in correctly predicted inter-domain contacts for the second (TBM) domain. The size of the circle is proportional to the number of contacts the residue makes in the experimental structure. Blue and yellow circles are residues belonging to the first and second domain, respectively. The color of the connecting arcs indicates the frequency with which the corresponding contact was predicted by the groups. Red, green, and gray lines indicate contacts predicted with a frequency below the median, between the median and the third quartile and above the third quartile, respectively. The bottom figure shows the three-dimensional structure of the protein with the first domain in blue and the second in yellow. The correctly predicted contacts are indicated by sticks with the same color scheme as the corresponding arcs in the top panel.

Download figure to PowerPoint

Also in this case, one can ask the question of how often the contacts predicted with the highest probabilities are correct. The results, shown in Figure S3 (Supporting Information) again highlight that group G489 is particularly effective in ranking the predicted contacts.

Comparison of CASP10 with previous experiments

Establishing progress in contact prediction is not a trivial task as targets, methods, and databases change in time. Unfortunately, no methods are available to adequately take all these relevant factors into account. We report here a comparison of the results without attempting to make any claim about the presence of real and measurable progress.

Figure 8 shows the results of the top 10 groups in the latest three CASPs on FM domains for the L/5 lists of long-range contacts (CASP10 results for the FM + TBM_hard domains are also included for comparison). On average, the CASP8 predictions (12 domains) have the highest precision—24.6%, followed by CASP9 (29 domains)—21.4%, CASP10 (FM + TBM_hard, 28 domains)—21.4%, and CASP10: (FM, 16 domains)—17.4%. These results may indicate lack of substantial progress or, alternatively, be a consequence of the growing difficulty of targets in subsequent CASPs.[63]

image

Figure 8. Precision of prediction for the top 10 groups in latest three CASPs.

Download figure to PowerPoint

CONCLUSIONS

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. CONCLUSIONS
  7. ACKNOWLEDGMENTS
  8. REFERENCES
  9. Supporting Information

The assessment of the state-of-the-art in contact prediction shows that the current precision of the best contact prediction methods on long-range contacts averages around 20%—the same limit observed in several previous CASPs. We look forward to seeing the results of the new methods that have recently appeared. Their published results in tests other than CASP have certainly stirred a lot of attention and it is therefore likely that we will see a renewed interest in the development of novel methods in contact prediction that will lead to improved results. We believe that progress in the field is objectively offset by the increased difficulty of the targets in CASP10 and that the depth of the alignments available for these targets made them less attractive for these new methods. At the same time, it should be mentioned that the list of CASP targets does mirror the proteins that the biological community considers interesting and worth an effort.

The predictions submitted by the best performing groups are statistically indistinguishable on the set of free-modeling domains. When hard template-based targets are added to the dataset, the results of the Multicom group, which uses consensus strategy to extract the contacts from predicted three-dimensional structures, are better than the others. Among the remaining groups, two implementations of the Distill method and ab initio predictors from the Multicom series of methods quite consistently perform better.

Based on the CASP10 data, we show that shorter domains are in general easier targets for contact prediction, and that the difficulty of predicting contacts in domains is not correlated with the depth of target sequence alignment.

ACKNOWLEDGMENTS

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. CONCLUSIONS
  7. ACKNOWLEDGMENTS
  8. REFERENCES
  9. Supporting Information

The authors thank John Moult and David Jones for useful suggestions.

REFERENCES

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. CONCLUSIONS
  7. ACKNOWLEDGMENTS
  8. REFERENCES
  9. Supporting Information

Supporting Information

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. CONCLUSIONS
  7. ACKNOWLEDGMENTS
  8. REFERENCES
  9. Supporting Information

Additional Supporting Information may be found in the online version of this article.

FilenameFormatSizeDescription
prot24340-sup-0001-suppinfo.pdf293KSupplementary Information

Please note: Wiley Blackwell is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.