Factor 1: Importance of the Research
OVERVIEW OF NIH SCORING
NIH research project grants (RPGs) submitted on or after January 25, 2025 will be evaluated using the new NIH Simplified Peer Review Framework. If you want the history and rationale for NIH’s implementation, read our previous blog post. As part of this change, NIH is providing resources such as an FAQ and other resources.
The two important terms are: Impact/Priority Scores and Criterion Scores.
Reviewers assign criterion scores to each NIH review criterion. Scores range from 1 to 9 (1 is best). Currently, NIH review criteria are: significance, investigator, innovation, approach, and environment.
Impact scores are assigned to applications. Assuming your application is discussed, all study section members (not just assigned reviewers) privately vote on a numerical impact/priority score from 1 to 9 (1 is best). Final overall impact score is determined by calculating the average of all members’ impact scores and multiplying by 10. Impact scores run from 10 to 90, where 10 is best. A good impact/priority score is a major basis for award decisions. Criterion scores are not used to calculate the final impact/priority score.
Upcoming changes to peer review
The new Simplified Review Framework takes the five existing regulatory criteria — significance, innovation, approach, investigator, and environment — and reorganizes them into three factors:
- Factor 1: Importance of the research
- Factor 2: Rigor and feasibility
- Factor 3: Expertise and resources
Factors 1 and 2 will be scored, but Factor 3 will not. However, all three factors will go into the overall impact assessment.
The goal is that this type of scoring will be much more focused on the actual impact and feasibility of the science.During a recent All About Grants podcast, Drs. Stephanie Constant and Lisa Steele explained that, while this has always been important, it’s being emphasized now to refocus reviewers to these high-level and important criteria that are felt to be the driving force for what makes a very impactful project.Below is a side-by-side, color-coded comparison of the current review criteria and where they will fall under the new paradigm. As a reminder, the five items under the current review criteria are all scored. Under the simplified review framework, investigators and environment will be considered but not scored.
| Current Review Criteria | Simplified Review Framework |
|---|---|
| – Significance – scored – Innovation – scored | Factor 1: Importance of the Research – Significance, Innovation – Scored 1 – 9 |
| – Approach – scored | Factor 2: Rigor and Feasibility – Approach (includes Inclusion and Clinical Trial (CT) Study Timeline) – Scored 1 – 9 |
| – Investigator(s) – scored – Environment – scored | Factor 3: Expertise and Resources – Investigators, Environment – Evaluated as appropriate or gaps identified; gaps require explanation – Considered in overall impact; no individual score |
NEW FRAMEWORK: FACTOR 1
Meg’s Thoughts
The table below compares the review criteria of the current review framework and the simplified framework. There aren’t big surprises in the new framework for Significance. In terms of creating subheaders in your application, we see some of our old friends there: Importance of the Research, Knowledge Gap / Critical Problem / Technical Advance, Rigor of Prior Research, Project Rationale (avoid using the term Scientific Premise, which actually refers to an outdated scoring criterion), and Background.
For Innovation under the new framework, we see a big (and welcome!) change, though we will see how reviewers score based on it. In the old framework, there is a heavy emphasis on innovation. There has always been reviewer guidance stating that a project does not need to be using innovative strategies to be fundable; it was the one scoring criterion where in theory, you could get a poor score and still be funded. But I think we all agree that in practice, if an applicant was proposing research using non-innovative approaches, they risked a poor innovation score, even if those approaches were the most scientifically appropriate. It created a dilemma for the applicant. However under the new framework, the very first item under the scoring criterion is “Evaluate the extent to which innovation influences the importance of undertaking the proposed research. Note that while technical or conceptual innovation can influence the importance of the proposed research, a project that is not applying novel concepts or approaches may be of critical importance for the field.” In my opinion, this change is LONG overdue!
Many applicants struggled to clearly delineate between Significance and Innovation, and by combining them, you avoid this issue. You could always include all the material under the header “Importance of the Research” rather than attempting to divide the two. In addition, by combining the two, it seems to de-emphasize the need to use innovative techniques and strategies, which again is not always the most scientifically sound way to conduct a given project. For Cycle I 2025 applications, I suspect I will play around with different strategies for writing to these new scoring criteria. If I have a client who is using tried-and-true strategies, I will likely create a single “Importance of the Research” header. For others where the split seems more obvious, I might divide them into the traditional Significance and Innovation sections. Under Significance I am likely to lift subheaders directly from the text of the scoring criteria.
| Current | Under the Simplified Review Framework Factor 1. Importance of the Research |
|---|---|
| Significance – Does the project address an important problem or a critical barrier to progress in the field? Is the prior research that serves as the key support for the proposed project rigorous? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field? | Significance – Evaluate the importance of the proposed research in the context of current scientific challenges and opportunities, either for advancing knowledge within the field, or more broadly. Assess whether the application addresses an important gap in knowledge in the field, would solve a critical problem, or create a valuable conceptual or technical advance. – Evaluate the rationale for undertaking the study, the rigor of the scientific background for the work (e.g., prior literature and/or preliminary data) and whether the scientific background justifies the proposed study. Innovation – Evaluate the extent to which innovation influences the importance of undertaking the proposed research. Note that while technical or conceptual innovation can influence the importance of the proposed research, a project that is not applying novel concepts or approaches may be of critical importance for the field. – Evaluate whether the proposed work applies novel concepts, methods, or technologies or uses existing concepts, methods, technologies in novel ways, to enhance the overall impact of the project. |
| Innovation – Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed? | Innovation – Evaluate the extent to which innovation influences the importance of undertaking the proposed research. Note that while technical or conceptual innovation can influence the importance of the proposed research, a project that is not applying novel concepts or approaches may be of critical importance for the field. – Evaluate whether the proposed work applies novel concepts, methods, or technologies or uses existing concepts, methods, technologies in novel ways, to enhance the overall impact of the project. |
