🚀 go-pugleaf

[Written by individuals at the American Institute of Physics and
merely posted by me, so respond to <fyi@aip.org> or other
references below.  Always posted here on sci.research; sometimes
crossposted to an eclectic and hopefully appropriate selection
of other newsgroups, with followups directed here. Back issues
may be ftp'd from NIC.HEP.NET, along with PHYSICS NEWS UPDATE
and the American Physical Society news/opinion column WHAT'S NEW;
or from pinet.aip.org. Enjoy! -jc]

GAO Finds No "Old-Boys' Network" in Peer Review System

FYI No. 133, September 2, 1994

Over the years the peer review system for awarding federal research
funds has come under fire for being an "old-boys' network," made up
of prominent academics from elite institutions channeling funding
back to their institutions and colleagues, and biased against
lesser-known schools and regions.  Members of Congress often use
this criticism to justify earmarking funds for their
constituencies.  Chairman of the Senate Governmental Affairs
Committee, John Glenn (D-Ohio) asked the General Accounting Office
(GAO) to study the issue of equity in the peer review process.
While GAO did find some weaknesses in the system at the three
agencies it examined (NSF, NIH and the National Endowment for the
Humanities), it concluded that those criticisms most commonly made
by foes lack credibility.

The 133-page report, dated June 24, 1994 and entitled, "Peer
Review: Reforms Needed to Ensure Fairness in Federal Agency Grant
Selection," looked at fairness in three areas:  1.) the selection
of reviewers and how well they represent the pool of grant
applicants; 2.) the scoring of proposals by reviewers; and 3.) the
final funding decisions of the agencies.  This FYI will focus on
findings relevant to NSF.

GAO sampled 50 successful and 50 unsuccessful proposals from five
programs within NSF: biochemistry; economics; math, algebra and
number theory; electrical engineering; and theoretical physics.
The report concludes that "Overall, peer review processes appear to
be working reasonably well."  It notes that "several factors
alleged to affect reviews were not, in fact, related to scores at
any of the agencies, [including] the applicant's region, academic
rank, or even employing department's prestige."

Contrary to allegations about an elitist system, GAO found "little
evidence to support the notion that peer review panels are staffed
disproportionately from among researchers at a handful of elite
institutions.  On the contrary, among our sample cases those from
elite institutions tended to be somewhat underrepresented among
reviewers compared to applicants." NSF program officers reported
that researchers from elite institutions were more likely to
decline requests to serve as reviewers.

The study found that while women are overrepresented on peer review
panels at NSF, they are underrepresented on external reviews.
Among regions, only the South Atlantic was found to be
underrepresented among reviewers, and contrary to allegations, New
England was not among the three highest of the nine regions for
reviewers.  Professional age of reviewers (years since PhD)
generally followed the distribution among applicants, except for
the most recent graduates, which the report found were "greatly
underrepresented" as reviewers.

GAO cites several weaknesses in the proposal rating system:
"Agencies do little to ensure that reviewers have an accurate and
similar understanding of the agency's criteria and rating scales,"
and "reviewers often relied on unwritten or informal criteria when
evaluating proposals," such as inclusion of preliminary results.

Reviewers whose own work was least proximate to the proposals were
likely to give better ratings.  This could variously be interpreted
to mean that researchers with more relevant expertise were better
able to detect flaws, or were downgrading competitors.  On average,
the more prestigious the applicant's department was perceived to
be, the better the rating received.  Proposals from applicants
known directly to reviewers got the best scores on average, while
unknown applicants "fared worst."

GAO also finds an apparent relation between race and score: on
average, whites received better scores than non-whites.  While NSF
does not provide reviewers with information on the applicant's race
or ethnicity, the report admits that some information might be
discernable based on knowledge of the applicant or institution.  It
cannot be determined, the report states, whether the relationship
is the result of bias or the quality of proposals.  NSF responds,
in comments on the report, that it "has long-standing policies,
programs and outreach efforts that are expressly designed to
encourage women, minorities and persons from other than elite
universities to apply and, other things being equal, to decide in
their favor."  The Foundation also reports that if all NSF-funded
research programs are considered, "NSF's award rates for women and
minority applicants are roughly comparable to those for men."

While the report finds certain apparent trends, it cautions that
the data is insufficient to determine whether they represent actual
bias or not.  "Much of the variation in peer review scores of
proposals," GAO concludes, "cannot be explained by the factors we
could measure....  We have interpreted this to mean that scores are
driven by agencies' criteria, including the importance of the issue
and the quality of the research design."

Although NSF gives its program officers considerable discretion in
choosing which proposals to fund, the report found proposal scores
"the dominant independent variable" in funding decisions.
Proposals with weaker scores were more likely to be funded if less
money was requested, and the larger the request, the less likely
the proposal was to be funded if the researcher was not well-known.
As GAO points out, "this result can be interpreted as showing
prudence on the part of NSF program officers."

Proposals perceived as being from top schools were twice as likely
to be funded as others; while applicants best known to reviewers
were three times as likely to be funded as unknowns.

The report offers the following general recommendations to improve
the peer review process:  address underrepresentation of
young/junior scholars as reviewers; increase monitoring of
discrimination, and test by comparing blind and conventional
reviews; develop a rating system where criteria are rated
separately as well as overall; and identify or formalize
commonly-used unwritten rules.

In addition, it provides several NSF-specific suggestions:
increase the use of panels over external reviews; more closely
monitor inclusion of women and minorities among external reviewers;
and increase efforts to calibrate ratings among reviewers.

One copy of the report, GAO/PEMD-94-1, can be obtained free of
charge from GAO, P.O. Box 6015, Gaithersburg, MD 20884-6015, or by
phone (202-512-6000) or fax (301-258-4066.)


###############
Public Information Division
American Institute of Physics
Contact:  Audrey T. Leath
fyi@aip.org
(301)209-3094
##END##########