I was about to start talking about the specific review criteria for K-awards (and F’s), and realized that I needed to do the general criteria first. Training/Career development will come next.
There are two easily accessible sources of information about reviewing. The first is to people submitting, and the next is to people reviewing. Also the Review Guidelines page has lots of links to things about the entire process. It should not be surprising, but perhaps is a bit, that the two are almost the same. This is a level of transparency that did not always exist at NIH. NIH also has a site for “Peer Review News and Advice” which publishes a newsletter called Peer Review Notes, which is a pdf that you can find at the home page. The link takes you to the most recent one. This is info for everyone.
Here, btw, is the link for the summary table called “review criteria at a glance”.
There are five main criteria for NIH grants:
Except that in order of importance for reviews, in my view, and not always explicitly stated I would say. (some people would say that 1 & 2 are about equally important, others would say 4 is more important than 3, depending on who you are). For young investigators 3 matters much more.
Each reviewer gives you a score between 1 & 9 for each of this criteria. Your final score is not a simple arithmetic average of the five. In fact, each reviewer assigns their own final score ( called Overall Impact). I’ve had reviews where I’ve gotten 1’s on 2-5, and a 5 on Approach, and had an overall score of 4.5 (or 45, as it is reported). Info on arriving at numbers can be found here.
Reviewers are given specific questions as guidelines for evaluating each of this criteria. They differ among mechanisms, but what is below is standard. Other mechanisms have additional guidance, but these still hold.
R01, R03, R21, R34. Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?
This is basically: are you doing something that NIH perceives as important. What will be the impact of what you do? See the advice at NIAID. You can make it easy for the reviewer by putting sentences in the SA’s or Significance section that are along the lines of:
“The significance of this project is …” or “If the aims of this project are achieved, we will…”
R01, R03, R21, R34. Are the PD/PIs, collaborators, and other researchers well suited to the project? If Early Stage Investigators or New Investigators, or in the early stages of independent careers, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative or multi-PD/PI, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project?
The issues in the investigators section are answered (mostly) in a well crafted biosketch. In short, the reviewers want to know Can You Do This Project. The other place to answer these criteria is with preliminary data, where you show that you can do what you say you will do.
Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?
Innovation means “is it new, creative, etc”. Innovation can be in applying an old method to a new problem, addressing a different disease, or building a new instrument, either solid & physical or verbal & survey. Advice from NIAID, which can be taken with a grain of salt. Whether you are trying to be too innovative is something that your local mentors should help with. No matter how wonderful your research is, or you think it is, never submit a grant that you know won’t get funded.
Your project should move the frontier of knowledge forward. Striving for a paradigm shift is not advisable.
The difference between innovation & significance is subtle. Significance is the impact on the field, which by definition needs to be something new. Innovation is what is new, and how it is new. Again, it is good to have a sentence at the end of your SA’s (and in your abstract) that says:
“The innovation of this work lies in….”
R01, R03, R21, R34. Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed?
Approach is what are you doing. The significance is the importance if it is done, the approach is how are you going to get there. This is not just the methods, it is also the hypotheses and the links between the hypotheses, methods, data collected. One question to ask yourself: have I shown how these experiments will produce data that will test the specific hypotheses and address the aims I proposed?
R01, R03, R21, R34. Will the scientific environment in which the work will be done contribute to the probability of success? Are the institutional support, equipment and other physical resources available to the investigators adequate for the project proposed? Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?
Do you have the setting in which to do your work? Do you have the equipment you need? Do you have the support of additional personnel to do the stuff you can’t? If you are at a MRU this is not so hard. Just make sure you have letters from everyone. If you are not at a MRU, then it is a bit more work, but you can always get help from people elsewhere, who are willing to lend the expertise in what you don’t know or can’t do. Don’t be afraid to ask, and anyone who says they have to have money to help you is a psycho that you don’t want to work with in any case.
There are more sections in the review – animals, human subjects, etc, but they do not factor into your final score.
While all of the advice says things like “you don’t have to be perfect in every category to get a proposal funded”, that is increasingly becoming nonsense. All of these must be excellent and solid to get a fundable “overall impact score”. One continuing issue is what is the difference between significance score (one of five) and the overall impact which is the final score. Here’s what NIH says:
Significance: Does the project address an important problem or critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?
Overall Impact: Reviewers will provide an overall impact score to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following five core review criteria, and additional review criteria (as applicable for the project proposed).
This is worthy of yet another post. later… time to get back to grant writing.