Team calls for creative approaches to accountability
By Eddy Ball
Librarian Cathy Sarli and bioinformaticist Kristi Holmes, Ph.D., urged scientists and administrators to think outside the citations analysis box to quantify research outcomes and translational impact, during an April 9 presentation at NIEHS.
The Washington University in St. Louis (WUSL) researchers are among several people now thinking divergently about a tracking system framework for evaluating biomedical research — one that includes indicators of impact on public and community health, economic benefit, medical practice, patient outcomes, regulation and legislation, new research directions, and even product development and insurance reimbursement.
Hosted by NIEHS Health Scientist Administrator Kristi Pettibone, Ph.D., who is also involved in high impact tracking for the Institute’s Partnerships in Environmental Public Health (PEPH) programs (see related story), Holmes and Sarli presented the results of ongoing efforts using the Becker Medical Library Model for Assessment of Research Impact. Their model challenges the bibliometric analytical paradigm for accountability long enshrined in tenure/promotion guidelines and grant/program review, with a novel multipronged guide with more than 80 indicators for metric analysis of biomedical research.
Beyond the assumption that a high citation count equates with high impact
Sarli, who delivered the first part of the team presentation, referred early in her talk to a statement in the conclusion of a 2007 commentary by Robert Wells and Judith Whitworth, M.D., on assessing research outcomes. “It is no longer enough to measure what we can,” they wrote. “We need to measure what matters.”
For people involved in today’s biomedical research, Sarli explained, what can be measured easily — frequency of citation — is only a small part of what really matters. And what really matters — the meaningful research outcomes — she added, may be some of the most difficult information to find and quantify.
Spurred by a request in May 2007 by WUSL researcher Mae Gordon, Ph.D., for assessment of the impact of the Ocular Hypertension Treatment Study (OHTS), the Becker team began looking beyond citations for impact in other areas. Gordon had anecdotal evidence of the OHTS study impact, but she wondered if quantitative, independent indicators were available.
Sarli and Holmes, who joined the effort later on, began the search for novel indicators. Using resources from databases, personal anecdote, organizational guidelines, trade publications, and other so-called grey sources, the team began accumulating evidence that OHTS research had impacted something beyond the world of the PubMed-indexed literary canon, an odyssey they discuss in a 2010 article.
Challenges for measuring what really matters
In her part of the presentation, Holmes reinforced Sarli’s assessment of the challenges of using the new evaluation paradigm. She conceded it’s a messy, time-consuming, and labor-intensive process that is non-linear and non-standardized.
“This is certainly not an exercise for the faint of heart,” Holmes told the audience. The supporting documentation may not be publicly available or easily accessible, requiring efforts beyond the standard search. And, it can take creative thinking to establish a direct correlation between the research and the outcomes.
“Researchers need to learn to tell their stories,” Holmes said, “be more proactive, and use strategies that enhance their research impact.” As budgets tighten and legislators look more critically at funding for scientific research, Holmes predicted, scientists need to demonstrate, clearly, the returns on investment of taxpayer dollars.
Holmes suggested that biomedical researchers begin thinking, at the outset, about potential effects and outcomes in terms of value and benefit beyond publications. She suggested strategies as simple as using the declarative voice in titles, using names consistently, and writing clearer abstracts with a general, as well as scientific audience, in mind.
Improved dissemination strategies, she added, can include such activities as setting up websites for research output, publishing in open access journals, creating public access repositories for data, taking advantage of social media and outreach opportunities, and tracking the impact of studies according to the kinds of global measures that will resonate with the public and show tangible outcomes in terms of improved health.
Sarli CC, Dubinsky EK, Holmes KL. 2010. Beyond citation analysis: a model for assessment of research impact. J Med Libr Assoc 98(1):17-23.
Wells R, Whitworth JA. 2007. Assessing outcomes of health and medical research: do we measure what counts or count what we can measure? Aust New Zealand Health Policy 4:14.