|
Epi Wit & Wisdom Resources
Why Research
Proposals Are Marked Down—Reflections of One NIH Reviewer
By Robert
Wallace, M.D., M.Sc.
The National Institutes of
Health is the major funding source in the US for epidemiologic
research proposals. Grant applications are funneled through the
Epidemiology Study Section, a peer review group consisting of two
sub-committees of 20 members each. This body reviews and rates
approximately 350 individual research proposals each year. Dr.
Robert Wallace from the Department of Preventive Medicine and
Environmental Health at the University of Iowa has been a member of
this Study Section since 1979. The following article is taken from
remarks prepared by Dr. Wallace for a round table discussion held at
the recent APHA meeting in Los Angeles.
The opinions expressed are those
of one reviewer offered to new investigators and others in the hope of
improving the quality and success of submitted proposals. Examples
used are fictitious and similarities to actual grants are coincidental
and unintended.
1) Inadequate Attention to the
Basic and Clinical Science Aspects of the Epidemiological Proposal
Many grants are marked down
because the author has not reconciled the proposed work with existing
scientific knowledge. The scientific rationale must occur in a context
of existing knowledge from relevant disciplines impinging on the
problem. While it is true that an empirical association between a
putative causal agent and a disease can have preventive implications
regardless of biologic explanation, the reviewer should show that
pertinent basic knowledge has been considered. Many epidemiologic
proposals are reviewed by basic scientists in pertinent substantive
areas.
2) Insufficient Persuasion That
the Proposed Research Will Add to Knowledge of Disease Etiology,
Mechanism or Control
This takes many forms. Some
investigators imply (in my view mistakenly) that a study is justified
merely because of the enormity or public health significance of the
problem at hand or because the problem has not been adequately studied
epidemiologically. In many instances, studies are proposed in areas
that require resolution, but in which a number of conflicting studies
already exist. Approaching these areas with similar methods is not
especially attractive unless the reviewers can be persuaded that new
insights or resolution might be anticipated. A particularly difficult
problem arises in evaluation of a data set already collected when
funds are requested only for analysis. In this circumstance,
persuasiveness of scientific value is just as important as for
proposals undertaking new data collection.
3) Misuse of Personal
Reputations
In general, the reputation and
productivity record of senior investigators adds to the attractiveness
of a proposal. However, this is not enough to carry a mediocre or
inadequately prepared document. In cases where junior scientists are
principal investigators, demonstrating that senior guidance or
consultation is available can help, but this should be reflected in
the text of the proposal, not merely on the personnel page. Reviewers
know that many senior scientists are busy (e.g., as departmental
executives) and may be skeptical of the level of input actually
possible.
4) Inadequate Characterization
of Exposures
Most investigators characterize
and define diseases and cases with considerable clarity. However,
sometimes unavoidably, the agents or exposures are less well-defined
and quantified, making the proposal less attractive. Can study
subjects (and even more cogently, surrogate respondents) remember
prior exposures? Types or names of drugs, agents at the job site, and
diet are examples of exposures where some demonstration of validity is
helpful. Similarly, quantification of prior exposures to physico-chemical
agents such as dusts, low level x-rays, pesticides or allergens must
be credible. Assistance from or collaboration with those having
appropriate dosimetric skills is helpful. The sometimes necessary
substitution of proxy measures for exposures (e.g. duration at a
particular job for an actual chemical exposure) may require additional
discussion.
5) Misconceptions About the
Research Budget.
Clarity in justifying the budget
is crucial, because it is often difficult for reviewers to evaluate
local costs and budgetary situations. Salaries and ancillary expenses
vary greatly. Proportions of time committed by investigators and staff
is somewhat arbitrary. Due to shrinking research funds, high budgets
are always looked at somewhat askance, though if appropriate they are
no deterrence. Budgets that seem too high or too low may bespeak
investigator inexperience. Artificially inflating the budget so that
it will be cut to the “right” level in my view detracts from reviewer
enthusiasm. Proposals that are for finishing a project with an expired
budget or for analyzing existing data should explain clearly why
analyses were not done in the original time frame. Finally, budgets
that imply or overtly state that investigator salaries are needed all
or in part to maintain employment do not carry much weight.
6) Writing For the Appropriate
Reviewer.
The triage process for submitted
proposals seems to work reasonably well. However, even with outside
specialty opinions, the reviewers may not be fully schooled in the
specific area of your proposal. For example, if you wish to study the
epidemiology of adrenal-cortical carcinoma, your proposal will be
evaluated by at least two experienced cancer epidemiologists, but
neither may have ever worked with that particular site. Therefore, the
proposal should contain appropriate biologic and methodologic
background for someone not immediately familiar with the details of
that site. Omission of important information which might be known only
to a reviewer skilled in a subspecialty area unfortunately can detract
from a research plan.
7) Inadequate Sample Sizes.
Many investigators quite
appropriately give power estimates of the likelihood of obtaining a
significant finding (e.g., relative risk), based on various sample
sizes. These estimates are often based on the major hypothesis to be
tested. Frequently there are also sub-hypotheses, to be tested on some
fraction of the study population, which may have a lower power because
of limited subject numbers. Indeed, sometimes sub-hypotheses or
interactions may be a substantial part of the proposal’s promise, yet
sample sizes for these latter analyses may be inadequate. One example
is the distribution of occupational categories among cases with
disease X and controls. The proportion of subjects with a given
occupation is likely to be small and the chances of significant
results uncertain. Be sure sample sizes are adequate for each
hypothesis discussed. In general, hypothesis-generating proposals are
not as well received as those hypothesis-testing.
8) Over-dependence on An
Existing Cohort.
Many investigators have
established cohorts for long-term study. Many have had important
results in the first three to five years of observation. However, the
investigators should not assume that prior findings and vaguely stated
plans for further analysis alone will be sufficient. Specific goals
and potential findings should be clearly described, and a case should
be made that such findings are not merely secondary refinements of
previous analyses. One deficit of long-term studies is that
independent variables collected in the past may with time lose their
specificity or cogency. Reviewers often ask how long such a study
might be expected to continue. Anticipate this question in your
discussion.
9) Proposed Activities are Being
Conducted Elsewhere.
Sometimes otherwise highly
qualified proposals are marked down because the reviewers know that
the same or similar studies are already in progress elsewhere. Of
course, no two studies are ever totally alike and replication of
important studies in diverse populations is useful. Yet, since
research funds are limited, reviewers may be averse to supporting
another proposal in a given area. Unfortunately, there is usually no
formal mechanism, within the National Institutes of Health or
elsewhere, to determine if there is work in progress in a specific
area, and this issue depends capriciously on the personal knowledge of
reviewers.
10) The “Will it Work?”
Syndrome.
It seems to me that after
evaluating the details and components of a proposal, the reviewer
steps back and asks, despite the methodologic flaws that all
real-world studies have, if the study will work and be productive.
This is intangible and giving advice to the investigator is difficult.
Proposal organization, legibility and clarity can help, and are
especially important with the recent emphasis on shorter proposals.
Placing potential findings in the context of scientific advancement
lends credibility. Consulting NIH program directors in advance of
proposal submission may yield important and helpful counsel. The
investigator should remember to demonstrate awareness of potential
pitfalls, document collaborative arrangements, and show competence in
the use and limitations of new technologies or advanced biostatistical
methods. Finally, the investigator should step back and look at
his/her own work as a total “package.” Correctable defects may be
found.
Published December
1981
|
