Monitoring and promoting the impact of pedagogically related scholarship

Authors


Academia is undergoing a revolution in the evaluation of the impact of research. The traditional measures of publication numbers and citations, along with somewhat newer metrics of impact factors for journals and h-indices for individuals seem obsolete to many in the community and skewed toward established publication venues and large labs [1-3].

The concept of “altmetrics” has arisen in an attempt to capture impact more holistically than citations alone [4]. Altmetrics include downloads and online views, bookmarking, mentions in blog posts, news stories, and social media, as well as more usual measures like peer review and patents. One advantage of altmetrics, in addition to their wider reach, is the ability to capture impact quickly rather than accumulating citations over many years.

Some well-known scientists [5], professional societies [6], and companies [7] have embraced new approaches to measuring impact, and have called for less use of impact factors, normalizing for laboratory size in evaluating numbers of publications, and inclusion of elements such as training outcomes in rating a research group. The Sloan Foundation recently awarded a grant to the National Information Standards Organization in order to study and recommend best practices in alternative metrics [8]. However, the altmetrics model has not been uniformly embraced, with some scientists and other academics worried that there is no expert assessment in social media and that some of the measures can be too easily manipulated [9].

Whatever the outcome for research publications, the impact of pedagogy research and innovative teaching is even more difficult to measure. Even when this work is published in traditional venues, it is less likely to be cited than other kinds of research, as practitioners pick up ideas and use them in classrooms rather than writing new publications. This phenomenon is reflected in the much lower impact factors for even highly-ranked education journals compared to highly-ranked research journals [10]. Beyond publication, ideas are spread through the education community as much through meetings and other means as through publication. Several authors have suggested that journal papers are the worst way to disseminate findings about student learning [11, 12]. Perhaps more importantly, the impact of educational research might be better measured by increases in student learning rather than by new scholarship.

There have been some attempts to capture impact of science education publications, although the altmetrics community has not yet been much engaged in these efforts. Towns and Kraft [10] documented the rankings of chemical education journals by chemical education researchers as an alternative to impact factors. Sally Fincher, via the “Disciplinary Commons,” has examined faculty portfolios as a way to measure dissemination and impact, and educational portfolios have also been suggested for assessment of scholarship in academic medicine [13, 14], although these authors do not address impact explicitly other than to note that adoption of methods at other institutions could be included in a portfolio.

Our dissatisfaction with traditional measures for this non-traditional scholarship, along with the possibilities opened up by the altmetrics movement, has encouraged us to begin listing measures of impact other than publication that might be tracked by altmetrics tools, and to compile ways for educators and pedagogy researchers to gather information about their own publications for the purpose of wider recognition and publicity.

We suggest a multi-pronged approach to raising the profile of the “scholarship of teaching” [15]: (1) encouraging companies in the burgeoning altmetrics market to include metrics that apply primarily to scholarship of teaching and learning (see Table 1); (2) documenting traditional citations with the full array of tools now available (see Table 2); (3) making research and materials about teaching more available via social media sites (see Table 3).

Table 1. Alternative metrics for teaching/pedagogy work
Textbooks (both hardcopy and ebooks): chapters, books, problems
Mention in grant proposals other than your own
Use by other instructors in courses: web pages, learning management systems
Images and other visualizations
Lecture capture
Expert testimony or consultancy
Table 2. Gathering data about your publication: Article level metrics
Web of Knowledge; http://webofknowledge.com/Determine the number of citations within Web of Knowledge indexed journalsInstitutional subscription required
Google Scholar; http://scholar.google.com/Determine the number of citations within Google Scholar indexed journalsFree
Google Link Search; from http://www.google.com; Search: link: URL of your articleDetermine if an article is linked from a web pageFree
Mendeley; http://www.mendeley.com/Determine the number of times an article has been saved by Mendeley users, and provides some information on the affiliations of those users.Free Registration required
Altmetric Bookmarklet; http://altmetric.com/bookmarklet.phpDetermine if an article has been mentioned on Facebook, Google Plus, Twitter or in a blog; reported on by news media, etc.Free web browser install (Chrome, Firefox, Safari)
Impact Story; http://impactstory.org/Determine the impact of an author's work using a variety of sources.Free Registration required
Table 3. Social media sites for scientists and other scholars
SiteDescription
ORCID: orcid.orgYour ORCID (Open Researcher & Contributor ID) is a unique identifier that you can use to manage a record of research activities. (It is particularly useful if you have a common name or have changed names.) It can help altmetrics tools find your work.
Google Scholar My Citations: scholar.google.com/citationsTrack citations to your publications. Check who is citing your publications, graph citations over time, compute citation metrics. View publications by colleagues and track their metrics. Create a public profile that can appear in Google Scholar when someone searches for your name.
Research Gate: researchgate.netA site for scientists to share their work and connect with others. Tracks views and downloads.
Academia.edu: academia.eduCreate a profile and connect with other scholars. You can share information about and upload publications, post teaching documents, and link to your online publication venues.
ArXiv: arxiv.orgOpen-access, moderated repository for scholarly articles (electronic pre-prints) in a variety of scientific fields and pedagogy in those fields. Automatically updates pre-prints with DOI and journal references when published. ArXiv document IDs are used by Altmetric It.

Table 1 lists a few metrics other than publications and patents that are not yet captured by any traditional or alternative metrics. Some of these are suggested in credentials review by institutions that have a career track which recognizes the scholarship of teaching, such as the University of Queensland [16].

Until alternative metrics are more widely accepted, it will continue to be important for us as education researchers to track our own impact via as many methods as possible. We should find out who is citing our work, saving it, using it in presentations, or sharing it online. One key piece of information is the Digital Object Identifier, or DOI, for our articles. It is the digital equivalent of a work's fingerprint, but not all articles have DOIs. An article's DOI can often be found on the article's web page, or by using CrossRef's DOI Lookup (http://www.crossref.org/guestquery/). Once we have the DOI we can use it to find data on the number of times it has been downloaded and cited, among other metrics. The following tables include information that is relevant to both scientific or other scholarship as well as pedgagogy/teaching publications, but which may not be as often publicized to those in education fields.

Table 2 lists some of the sites where the DOI or other identifiers can be helpful and what sorts of information can be obtained. Also note that many online journals now include article-level metrics (ALMs) on individual article pages.

There are a variety of places on the web to share our work, create research- and teaching-oriented profiles, and connect with others. Different sites will suit different people; we suggest choosing one or two that work best for you. Table 3 lists a few of these sites.

Finally, if we want others to cite our teaching innovations, we should do the same for them; pay it forward. Here are some ways to ensure that scholars and teachers whose work you are using in your teaching receive acknowledgment for their contributions, even outside of using traditional citation metrics:

  • Blog: If you already blog, document the sources of or inspiration for your teaching methods in your posts, and be sure to link to them, preferably with DOI. Such a citation once a semester can be sufficient.
  • Use social bookmarking and networking tools: Twitter, Facebook, Google+, CiteULike, and/or delicious.com are some of the places you can share links to the articles you have used. Again, best practice is to use a DOI link when available. If your setting allow for public sharing, ImpactStory and Altmetric, it will identify the links.
  • Include citations on syllabi: If you post your syllabus on the open web or in your institution's repository, consider added a “works cited” section to the end, where you list articles and DOIs or other publications and URLs that have influenced your teaching methods.
  • Email the author: If all else fails, email the authors and let them know you are using their article, blogpost, or other publication.

As should be apparent from even this brief set of links and references, the field of citation metrics is exploding and rapidly changing. If education researchers want to be recognized, we will need to use the current tools and enter the discussion with new players. Until these new tools are widely recognized and used by others, we will need to be assertively self-promoting. We would welcome comments and suggestions on any aspect of this article.

Acknowledgements

The authors are grateful to: Mel Chua at Purdue and his blog (www.blog.melchua.com) for thought-provoking posts about open access and references about STEM education publications; Mike Buschman of Plum Analytics (www.plumanalytics.com) for a virtual primer on altmetrics and an introduction to the altmetrics marketplace; Ravi Ravishanker, Wellesley CIO, for his enthusiasm and support.