The field of transmission electron microscopy, and more specifically single particle cryo-EM, has undergone dramatic growth over the last decade, emerging as a powerful technique in structural biology that can often be used to determine structures that are not easily studied by other methods.1, 2 Near-atomic resolution has now been achieved in several cases and it is clear that this will be more common in the years ahead. However, structures of difficult macromolecular targets at lower resolutions are still the mainstay of the field. As such, fully interpreting structures of fragile macromolecular assemblies, membrane proteins, and other challenging targets at 6–30 Å resolution requires the use of data from other methods combined with sophisticated computational modeling. In fact, cryo-EM presents both a challenge and a blessing for methods development in the modeling community, as it offers additional restraints that can resolve ambiguities often encountered in modeling studies.

Cryo-EM maps span a wide range of resolutions, ranging from ∼3 to ∼30 Å.2, 3 For protein structures, this range can be broken down into three broad categories: First, in the 10–30 Å resolution range, quaternary structure can be observed and rigid-body fitting of X-ray structures or homology models is the primary method for more detailed map interpretation. Second, in the 6–10 Å resolution range, secondary structure elements (SSEs) appear. At 8–10 Å resolution, α-helices become resolvable, followed by β-sheets as 6 Å resolution is approached. These secondary structural elements permit some direct interpretation of the map in many cases, even without domain crystal structures. When atomistic models of molecular components are available, the presence of secondary structure means docking can be performed with much greater reliability and accuracy. It also becomes possible to apply flexible fitting techniques, which adjust individual SSEs to compensate for the conformational differences that are often observed between cryo-EM structures and the models being fitted. Finally, at 3–6 Å resolution, de novo backbone tracing of density maps becomes possible, with full atomistic model building becoming possible towards 3 Å resolution. Due to the fact that the best existing cryo-EM resolutions are still considered low by X-ray crystallography standards, existing crystallographic model-building techniques are often of limited use for interpreting cryo-EM maps. Therefore, many efforts are under way to develop new model building and de novo backbone-tracing tools specifically for these moderately lower resolution maps.

In parallel with the rapid improvement in cryo-EM methods and techniques for map interpretation, the number of depositions of EM maps and map-derived models in the EM Data Bank (EMDB) and Protein Data Bank (PDB) has been increasing rapidly in recent years, as shown in Figure 1.1 However, there are as yet no standardized and widely accepted methods for validating the density maps and models over different resolution ranges. EMDataBank has therefore convened a task force of experts to discuss the need to establish such standards.4

thumbnail image

Figure 1. EM entries in EMDB and PDB, cumulative by year (*2012 data is through March).

Download figure to PowerPoint

The idea for the cryo-EM modeling challenge arose from a round-table discussion at the January 2010 “Modeling of Cryo-EM Maps” workshop ( organized by Wah Chiu (National Center for Macromolecular Imaging) and Klaus Schulten (U. Illinois Urbana-Champaign), and hosted by the Texas Learning and Computation Center at the University of Houston. The ∼100 participants at that meeting included representatives from several major modeling groups as well as cryo-EM practitioners. At the meeting, the modeling-methods developers voiced a request to the cryo-EM community to identify a set of maps that would be suitable for them to use as standards for methods development and testing. The cryo-EM researchers, in turn, asked the methods developers to provide more detailed and practical information as to how and when their methods could be applied. These discussions led to the organization of the modeling challenge, which was held later the same year, and led in turn to a workshop at the Pacific Symposium for Biocomputing (PSB) the following year to discuss the results.5

The exercise was called a challenge rather than a competition because, unlike CASP6, 7 or CAPRI,8 there was no “true” answer to be revealed and used for quantitative comparison of results. The goal was not to identify a winner in each category of the challenge, but simply to explore as many available methods as possible and to extend lines of communication between modeling-methods developers and cryo-EM-data producers. Thirteen cryo-EM density maps were selected and modeling-community members were challenged to apply their tools to extract as much information as possible from each map and to submit their results to the challenge web server ( In the end, we had 130 submissions in all categories, and the server remains open as a public repository for results and sample data for those in the modeling community.

The major constraints applied in the map-selection process for the challenge were:

  • 1
    To sufficiently explore all three resolution ranges described above, so that any given method would be applicable to a minimum of three maps in the set.
  • 2
    To include examples of large, highly symmetric viruses; smaller, intermediate symmetry assemblies; and completely asymmetric structures.
  • 3
    To include representative structures from different research groups that use different cryo-EM software packages and different experimental equipment, to explore the qualitative differences that may exist among published cryo-EM maps ostensibly at the same resolution.

The modeling methods were divided into six categories:

  • Volume Analysis: techniques for annotating density maps without producing a full atomic model

    • Map segmentation

    • Secondary structure element annotation

    • Protein-backbone tracing

  • Structural Modeling: techniques for producing or altering atomic models based on density maps

    • Rigid-body fitting

    • Flexible modeling

    • De novo modeling

It is important to reiterate that, while comparisons of the results obtained with different methods would be interesting, the primary purpose of the challenge was to provide the cryo-EM and modeling communities with a sorely needed resource and a unique venue for the exchange of ideas. For the cryo-EM community, the challenge has yielded a repository of solved examples in which available software has been applied to specific real-world structures. For the modeling community, the challenge has created a set of reliable, published cryo-EM maps on which to test its methods and the opportunity to compare results obtained using different modeling methods. A summary of the challenge submissions is shown in Table I. The submissions included 35 method/map combinations, covering 96 of the 130 submitted results, in which multiple software packages were applied to a single map, and comparisons could potentially be made. Indeed, several such comparison studies are now underway.

Table I. Breakdown of the Submissions for the Cryo-EM Modeling Challenge
  1. a Denotes a software method covered in this special issue of Biopolymers. The outlined boxes represent map/software combinations where a comparison among different software would be possible based on the submitted results.

Thumbnail image of

This special issue was designed to be a compilation of papers by the participants in the challenge to describe their results in detail. However, some groups studied only a few structures using only a single software package, whereas other groups used multiple tools across most or all of the maps in the challenge. Accordingly, the level of detail in the manuscripts contained herein also varies considerably. Nonetheless, all eight offer additional details not available directly from the submissions in the challenge.

Importantly, the challenge server ( remains open, not only for reviewing the results, but also for new submissions, of which a few have been received already. We hope that this site will serve as a repository of results as groups in the modeling community improve their methods and new methods emerge. The results of the challenge may also be useful for developers who are creating new validation methods as recommended by the Electron Microscopy Validation Task Force.4 Although there may be further challenges in the future, it is likely that these would be much more tightly focused on some specific map-based modeling problem.


  1. Top of page
  2. Acknowledgements

This workshop and associated challenge was supported by NIH grants P41RR002250, P41GM103832 and R01GM079429. The authors thank all the challenge participants for their contributions to this project.


  1. Top of page
  2. Acknowledgements