I have had the pleasure of teaching a course in research design to graduate students in ecology, evolution, and conservation biology at the University of Nevada, Reno, since 1991. We discuss classic (Chamberlin 1890, Platt 1964) and unusual (Benjamin and Mangel 1999) papers in philosophy of science, provocative papers about statistical issues in designing research, many published in ESA journals (Newman et al. 1997, Cottingham et al. 2005), and case studies of disputes about design and analysis of ecological experiments (e.g., Cahill et al. 2001, 2004, Bradley et al. 2003, Louda et al. 2004). Students do a lot of writing, with the goal of learning how to prepare persuasive research proposals. Each student's writing is critiqued by the entire class, so they get suggestions for improvement not only from me but also from their peers, who may have different research interests, simulating a proposal review panel.
A fundamental part of learning how to do research that receives scant attention in most textbooks on research design is the very first step: choosing a good problem. (See Karban and Huntzinger  for an exception.) Although many graduate students may not have to face the challenge of asking a question that leads to productive research because they are working as research assistants on funded projects of their advisors, I think it's worth discussing how to find interesting and important problems in my research design course, because these students may have an opportunity to choose their own problems in their future careers. Even if not (e.g., an M.S. student who gets an agency job in which a team leader assigns research projects), there may be opportunities to redefine problems in ways that make them more promising.
There are scattered sources that are helpful in thinking about choosing a good research problem. For inspiration, I like Aldo Leopold's advice:
First and foremost, the field of inquiry must promise to yield no gainful knowledge. ... Second, and of like importance, the quest must be so difficult, and promise such long and devious paths, as to hold out no assurance of ultimate success. It must demand the keenest research nose, set in the stoutest and most highly trained hounds of science. Thirdly, the quest must lie in no single field of science.
From a letter to Judge Botts, quoted in Meine 1988:347–348
For practical advice, The Craft of Research (Booth et al. 2003) is valuable, although not targeted specifically to graduate students or scientists, but to students in all fields with the task of writing a research paper for class. Loehle (1990) provides an idiosyncratic but entertaining perspective (“Don't be an expert,” “Don't read the literature,” “The four hour work day”), which he elaborates in “Thinking strategically: power tools for personal and professional advancement” (Loehle 1996). Root-Bernstein (1982) analyzes “the problem of problems” and Wolff (2000) provokes sometimes heated discussion in classes with students interested in conservation and management.
In thinking about how scientists learn to choose good research problems, I wondered what criteria beginning students of ecology might use and how these might compare to criteria used by established researchers. Therefore I surveyed students enrolled in my research design class at the University of Nevada, Reno in 1997, 1999, 2003, and 2007. I also surveyed faculty and postdoctoral fellows in our program in Ecology, Evolution, and Conservation Biology, either by sending them the survey form directly or asking students in the class to ask their advisors or other senior researchers in their lab groups to fill out the form. I report here some interesting results of this survey, although avoid most statistical analysis because some faculty may have completed the anonymous survey more than once, and in any case it would be difficult to argue that answers of students and their advisors are independent. However, I did avoid soliciting answers from “old-timers” among the faculty in the latter years.
The survey asked respondents to rate the importance of six criteria for choosing a research problem, based on past experience and/or future plans. The six criteria were novelty, generality, tractability, complexity, money, and practical importance, and the rating descriptors were unimportant, of minor importance, somewhat important, quite important, and very important. The survey form included definitions of the criteria (Table 1). The cumulative numbers of respondents over four years were 95 students and 38 faculty and postdoctoral fellows.
Graduate students were relatively consistent in their rankings of the six criteria between 1997 and 2007 (Fig. 1). Practical importance ranked highest in all four years in which surveys were done, with most students considering practical importance of a problem as quite or very important in all years. Novelty and money were also ranked highly, while the other criteria were considered somewhat important or of minor importance. Few students or faculty rated any criterion as unimportant.
I didn't compare ratings of faculty and postdocs across time because of small sample sizes in some years, but there were interesting differences between these senior researchers and students when data were aggregated across years (Fig. 2). Practical importance and money were more important criteria for students, with the biggest difference for practical importance where the mean rating corresponded to quite important for students but only somewhat important for senior researchers. The other criteria were more important for senior researchers, with the biggest differences for generality and tractability. Fifty percent of faculty and postdocs rated generality as very or quite important for choosing a research problem, while only 24% of graduate students rated generality this highly. For tractability, the corresponding values were 47% and 26%.
Like ecology students elsewhere, many of ours are strongly motivated by love of nature and a desire to help conserve it, so it's not surprising that practical importance helps define good research problems for them. Our faculty includes both basic scientists and conservation biologists who might rate practical importance differently, although most of those who would call themselves conservation biologists do both basic and applied research and so have a broader perspective than new graduate students. Similarly, faculty have had enough experience getting grants and contracts to support research that money may not seem as much of a challenge as it is for graduate students.
It's probably also not surprising that senior researchers give more weight to generality in choosing research problems than do graduate students. Senior researchers know more, so may have more insight about the broader implications of their research than graduate students. They have records of completing successful research, so may focus less on logistical problems and more on the larger ramifications of projects than graduate students. However, for graduate students with ambitions for future careers in academia or as research leaders in government or industry, it will be essential to plan research that will have broad significance. Their ability to do so will likely develop with experience, but the ones who can most successfully relate their projects to the “big picture” will get the best postdoctoral and subsequent employment opportunities.
The most interesting difference to me between graduate students and senior researchers in responses to this survey was the greater importance attached by faculty and postdocs to tractability as a criterion for choosing a research problem. Perhaps graduate students are idealistic not only in wanting to help save the world, but also in thinking that means can be found to answer the research questions necessary to do this, no matter how challenging those questions may be. This too is not a surprising result: idealism is often associated with youth and relative inexperience, realism with the greater age and experience of senior researchers. Nonetheless, it's a result that makes me wonder how best to leaven that idealism with a dose of pragmatism without altogether suppressing it.
Today's graduate students in ecology, together with the rest of their generation, inherit a global environmental crisis of unprecedented scope and complexity. If tractability becomes too important an issue in choosing how to tackle the multitude of problems that are part of this crisis, these students risk becoming fatalists or nihilists, which will have grave consequences for future generations. In my few remaining years of teaching research design, I hope not only to help students learn to write effective research proposals, but also encourage them to remain optimistic that they can successfully tackle large problems. There is no other choice.