The Voice of Epidemiology


    Web EpiMonitor

► Home ► About ► News ► Jobs ► Events ► Resources ► Contact


Humor Quotes Wit & Wisdom EpiSource Miscellany Editor's Tips Triumphs Links Archives
Society for Epidemiologic Research (SER) Presidential Addresses

SER President Highlights Problems With NIH Peer Review and Offers Suggestions for Change

How many epidemiologists does it take to change a light bulb? The answer is five. One to install the bulb and four to critique the methods.

This was the joke told by SER President Leslie Bernstein in her opening remarks to the audience gathered in Edmonton, Alberta for the 30th annual Society for Epidemiologic Research meeting in mid-June. She may have been poking fun at epidemiologists but she had a more serious point in mind—namely that too much of a good thing (i.e. criticism), or at least too much of a good thing done poorly, can cause very serious consequences for epidemiologists.

In wondering what topic to select for her SER address, Bernstein was influenced by the comments of National Breast Cancer Coalition members who asked her, “why are epidemiologists so incredibly critical of each other?” According to these members, many of whom are laypersons, epidemiologists are much more rigid and likely to be unsupportive of their colleagues’ ideas than are scientists in other fields.          

The epidemiology profession’s proclivity for fault finding is a cause for concern, according to Bernstein, because not all the reviews epidemiologists receive are sound. “We can all cite examples of reviews we consider to be unscientific, that have misunderstood our intent, and contain criticisms that have little bearing on our ability to carry out our proposed study successfully,” said Bernstein. Based on these observations and experiences, and because of its importance to professional success, Bernstein chose to focus her presidential remarks on the status of peer review at NIH.

Problems with Peer Review

Bernstein identified several problems with peer review in NIH study sections as it is currently practiced. The first is the high volume of work imposed on reviewers that causes fatigue over time and makes reviewing a “wearisome chore.” Also, many section reviewers do not read all of the numerous grant proposals submitted to them and rely on the opinions of the primary and secondary reviewers to score their proposals. Thus, not all the scores from a review panel are independent measures of the quality of the proposal.

Second, applicants cannot anticipate who the reviewer will be and there is “the luck of the draw,” according to Bernstein. The interests of the reviewer play a critical role in the reviewer’s level of interest and enthusiasm for a proposal. Also, personal feelings can come into play and whether or not a reviewer likes or dislikes the researcher can “affect the tone” of the review, says Bernstein. Some reviewers may be competitors of the researchers and that creates a conflict of interest situation. How much an individual proposal is affected by these vagaries is the “luck of the draw” problem.

Third, there is a lack of accountability for peer reviewers. “As reviewers of epidemiology proposals, we often focus excess weight on issues of statistical power, minor logistical issues, minor problems that may be encountered in achieving secondary aims,” says Bernstein. “How do we hold reviewers accountable for such actions?”


To help correct these failings in the peer review system, Bernstein urged her colleagues to serve on peer review panels even if tired. It could be seen as jury duty or made a precondition for receipt of a grant, she said. Also, giving feedback to NIH is potentially useful, and she got the ball rolling by offering the following suggestions for improvement:

1) Lessen the reviewer workload by composing “Study Section Pools” of regular members

2) Require that reviews be submitted in advance and “review the reviews” to assure high quality

3) Facilitate interaction between the reviewer and the applicant, possibly by submitting the review in advance to the applicant and allowing for the opportunity to rebut the review

4) Develop an oversight mechanism to review study section membership, appropriateness of reviewers, and reviewer behavior within the review process.

In addition to suggestions for NIH, Bernstein offered these suggestions to other epidemiologists:

1) Insist that we only review grants for which we are appropriate reviewers

2) Be aware that every comment we make at a review can affect the priority score

3) Prioritize our comments so that other members of a study section will understand that the minor issues are not fatal ones

4) Use the review process to make helpful suggestions for improving the study while being careful not to redesign the study

5) Train our graduate students and post-docs to ensure that they understand the process, can compete, and that they know how to review

Dr. Bernstein is professor of preventive medicine, senior associate dean for faculty affairs, and scientific director of the Cancer Surveillance Program at the University of Southern California School of Medicine in Los Angeles.

Published July 1997  v

Postscript  2000

            In my SER presidential remarks made in 1997, I addressed the crises epidemiologists were facing as subjects of the peer review process and as peer reviewers implementing the process. Many of the problems I described still exist, including our extremely critical stance relative to our colleagues’ proposals. In addition, we now have to determine whether an epidemiologic study is “innovative.” In our field this could be considered an oxymoron. Further, we now face a growing problem with “conflict of interest,” a result of our need to conduct collaborative studies, particularly in molecular and genetic epidemiology, to ensure sufficient statistical power. Those of us who collaborate widely and who serve on numerous advisory committees are unable to serve as reviewers when it might be perceived that a conflict of interest exists. Perhaps we need a less stringent definition of such conflicts. In addition, I believe that we still need a workable oversight mechanism that reviews study section membership, the appropriateness of reviewers and the reviews themselves. On the plus side, over the past three years, we have seen more experienced researchers providing service on peer review panels and renewed interest in the etiology and prevention of diseases and health conditions.

Leslie Bernstein


      ©  2011 The Epidemiology Monitor

Privacy  Terms of Use  |  Sitemap

Digital Smart Tools, LLC