The Importance of the “Approach” Criterion On NIH Grant Scores

Sally Rockey, Deputy Director of Extramural Research at NIH, posted data on her blog concerning reviewer behavior on proposals reviewed in 2010. Recall that with the new NIH grant format, NIH implemented scores of one to five (lower is better) in each of five categories: Significance, Innovation, Approach, Investigators, and Environment, as well as the overall impact score. Of 54,727 applications reviewed in FY10, 32,546 were discussed and received overall impact scores. The NIH generated some data on these scores. Among their findings:

*While reviewers used the full scoring range (one through five) for each of the five review categories, their scores were distributed more widely for the Approach category.

*Criterion scores are moderately correlated with each other and with the overall impact score.

*Of the five scoring categories, the one best able to predict the overall impact score was the Approach section (followed by significance, innovation, investigators, environment.)

The language surrounding the changes to the NIH grant format in Zerhouni’s efforts to enhance peer review stressed the importance of Impact, Significance, and Innovation. And in the new format, the length of the Research Strategies was halved, forcing grantees to compress “Approach” sections such as the exhaustive literature review and the detailed methodologies. However, from the data in Rockey’s blog post, we might surmise that reviewers still heavily weigh the Approach category. And based on my own experience with pink sheets in the new format, reviewers’ nearly-insatiable desire for preliminary data appears to continue unabated, despite the reduced page limits on proposals.

Multiple Regression To Predict Impact Scores Using Criterion Scores