- SCIENCE IN CANADA: PART II -
THE PEER REVIEW SYSTEM
Adolfo J. de Bold
University of Ottawa Heart Institute, Ottawa Civic Hospital,
Ottawa, Canada
Little
can be made of any activity without regard to the intrinsic value of its
product. In science, the perceived value of any given scientific activity,
including its originality, relevance and feasibility, can only be determined
by the opinion of peer scientists. Thus the peer review system is of utmost
importance in providing the basis, if not guarantying outcome, for the
successful conduction of science and its technological applications. In
practical administrative terms, the peer review system (made up by the
scientific community and an administrative infrastructure) provides funding
agencies (government or private) advice regarding the merit of individual
applications so that informed funding decisions can be made. This process,
in turn, regulates to a large extent the activity of the scientific community
itself because most scientific activity requires significant funding. One
important constraining parameter in this scheme is the amount of money
available for research which is never enough to fund all good research
proposals. In addition, policy decisions regarding areas of strategic interest
may weigh in fund allocations.
Scientists would like to see a peer review system that is free of
political or other interventions. Policy intervention, as in strategic
planning, though, is a more subtle problem. The focusing of research, be
it AIDS, cancer or heart disease, as examples of scientific policy targets,
has good grounding in public need. Trouble is, the precise path leading
to solutions in these or any other areas can not be predicted and, therefore,
targeting without an overall commitment to advance basic scientific knowledge
does not really make sense. From the fiscal point of view therefore, strategic
targeting should entail increasing research budgets and not merely redirecting
them. This is extremely difficult to accomplish at present because research
expenditures are generally decreasing as a result of deficit control measures
seen in all countries conducting significant amounts of research.
The history of science administration is full of examples regarding
the perils involved in politically- or even fiscally-motivated tinkering
with scientific activities. For this reason, successful research programs
are those supported by either agencies at arms-length from the governments
that provide their funding or by privately run agencies. That is to say
that, in federally supported agencies, once government allocates funds
for research, an activity necessarily based on political considerations,
including leadership, it must withdraw from the process completely. What
follows is fund allocation by the agency based on purely scientific considerations,
including the best way to accomplish goals of strategic importance.
Fund allocation by granting agencies is made up of two main processes:
1. The scientific evaluation of the research proposals resulting from the
peer review process, and, 2. The allocation of the granting agency budget
made on the basis of the recommendations arising from the peer review process
including existing or new programs. Obviously the first process is a scientific
activity and the second is a purely administrative one that follows the
mechanics of allocating funds according to scientific advise.
Seen as above, the peer review process is the cornerstone of scientific
and technological advancements and its proper functioning must be ensured
at all costs. Seen from the perspective of countries under development
such as many in Latin America which, although now politically far more
stable than in the past, still suffer from administrative discontinuity,
this concept is important to preserve or implement as the case may be.
It is perhaps not an exaggeration to say that no amount of technological
transfer, external fellowships or foreign aid will help unless the internal
competitive level is brought up by a well designed peer review system.
The peer review system is applied in different forms in different
countries. Canada and the USA have similar peer review systems. In fact,
it is not uncommon for scientists of either country to serve as reviewers
or referees of scientific projects of the other country.
As stated in the previous issue of INTERSCIENTIA,
the Medical Research Council of Canada (MRC) is a good example of a federal
agency that is responsible for funding biomedical research, including basic,
applied and clinical research and research training . The MRC supports
research carried out by scientists in universities, hospitals and research
institutes. Its Governing Council whose membership also form part of the
Budget committee and the Standing Committee on Science and Research, disburses
funds allocated by Parlament in each federal budget. The Budget Committee
develops options on allocating MRC's budget among its programs and competitions
taking into account current commitments and projected activity. MRC's budget
is NOT allocated by committee (discipline/science, see below ). The Standing
Committee on Science and Research holds discussions on the various options
for funding, in an effort to meld the overall budgetary decisions with
the actual data arising from application pressure, committee recommendations
and changes to the MRC budget. The recommendations of the Standing Committee
go to Council.
The MRC peer review system is based in a system of committees. There
are several committes with expertise on different areas of biomedical research.
Each Committee has a Chairperson and a Scientific Officer. They do not
review applications; instead they work together to select reviewers and
to see that Council's policies are followed. There are 8 to 12 members,
depending on the application pressure on the committee and the breadth
of scientific areas to be reviewed.
The Grants Committee members themselves are chosen by Council for
their expertise in the various fields of research. Suggestions for Committee
members come from many areas: from applicants, the Chairs and Scientific
Officers, Council members, Standing Committee on Science and Research,
University Offices of Research, and the top ranking grantees from the previous
competition. In the spring of each year, the Chair and Scientific Officer,
top ranking grantees and some members of the Standing Committee on Science
and Research meet with MRC staff to compile a list of potential members.
There are many factors which are taken into account in determining membership
on committees. Individual members are chosen for their scientific excellence,
as reflected by ability to obtain grant support for a research program,
and quality as a referee as demonstrated by breadth of knowledge and maturity
of judgment. For the committee as a whole, there are several considerations:
1. The need to cover the range of areas for which the committee
is responsible.
2. Representation in both basic and clinical research.
3. The necessity for reviewing capability in both english and french
so that applications in either official language can be evaluated by the
committee .
4. The need to avoid conflict of interest. There is usually only
one and rarely more than two members form any one institution on a committee.
This ensures that a reasonably full complement remains when members must
excuse themselves from the review of an application from their institution.
5. The need for turnover of committee members. Committee members
usually serve no more than 3 consecutive years
6. The need for regional representation and representation by gender
proportionate to membership in the canadian scientific community as a whole.
Guidelines for Committee Members have been prepared which are intended
to inform all members, new and experienced, of the current procedures which
Council has approved, for use in all Grants Committees. The Guidelines
are not rules or regulations, but rather an attempt to describe Council's
current policies. Among other things, they stress the importance of confidentiality,
extending it to all aspects of the information available to Committee members,
and to the recommendations made by each Committee to Council. Committees
are charged with recommendations to Council on the scientific merit of
each application, the funds needed for the research proposed to be carried
out and the duration of the grant. Their assessments take into account
the opinions of the external referees as well as those of each member of
the committee besides the two asked to review the application in depth.
Since both the quality of the research project and the ability of the investigator
to carry it out are being judged, peer reviews is fundamentally a process
of estimating potential. This demanding task is far from an exact science.
The assessment process takes many factors into account: the relative
weights attached to these factors vary according to the circumstances surrounding
each application, although scientific merit is always of primary importance.
The balance sought can be illustrated by a few examples. In assessment
of applications submitted for new grants by faculty members at the start
of their independent research careers, particular emphasis is put on training,
qualifications and relevant experience. The record of scientific productivity
during the training period is also taken into account. For applications
for new grants from researchers who have already held grants, the assessment
focuses more on the progress and productivity demonstrated previously than
on the applicant's training and qualifications. In the case of applications
for renewal of grants, the research potential continues to be an important
element but considerable emphasis is put on current and past productivity
and the significance of the contributions made in that field.
Rating procedures for Grant Committees
At the outset, each internal reviewer will be asked to state their
rating (to one decimal) of the application, within one of the following
categories:
Rating
4.5 4.9: Outstanding
4.0 4.4: Excellent
3.5 3.9: Very good
3.0 3.4: Solid/significant research
2.6 2.9: Needs revision
2.0 - 2.5: Needs major revision
1.0 - 1.9: Seriously flawed
0: Not acceptable
Following a full discussion of the application, the internal reviewers
come to an agreement on the consensus rating, to one decimal. If the two
internals are unable to agree on a rating, the average of the two ratings
is used.
Each committee member, including the committee reviewers, then cast
individual confidential votes within +0.5 of the consensus rating. For
example, if the internals agree that their assessment is 3.7, then individual
members may select a rating from the range of 3.2 to 4.2 the rating assigned
to the proposal is the average of these confidential votes.
All applications rated 2.5 or below by both the internal reviewers
and for which the external reviews are essentially in agreement, wlil not
be discussed further by the committees unless a committee member requests
it. Scientific Officer notes will not be available for these applications.
Budgets
The committees also reach decisions on the funds required for each
research project. The starting point of this discussion is the budget proposed
in the application. The application form only allows space for presentation
of one year's budget, reflecting Council's established practice of initially
awarding funds for future years at the same level as the first. Where the
applicant anticipates a much different budget in the subsequent years,
additional copies of the budget page may be added for each year requested.
In reaching a consensus on budgets. Council expects that a committee
will bring to bear its collective judgment and knowledge of research requirements
and costs and recommend sufficient funds to allow the work to proceed at
an appropriate rate.
Duration of grants
Applicants may seek support for 1, 2, 3 or 5 years. New grants will
usually be 3 years to allow the new research program sufficient startup
time before seeking a renewal.
Council decisions
The recommendations from all committees on funds for each application
are listed for Council in the order determined by the priority ratings.
Council uses these lists to reach its decisions, which are based on the
availability of funds and the need to consider commitments for future years.
Each applicant receives an anonymous copy of the comments made by
the referees on their application and a summary of the committee discussion
as prepared by the committee Scientific Officer.
In coming to its final decisions, Council has combined two approaches
to funding. First, the majority of applications to be funded are identified
on the basis that each committee should have the same fraction of successful
applications. Council then ranks the remaining applications from all committees
in a single list in order of rating, and fills the rest of the quota of
successful applications from the top of the list, with the proviso that
no committee should have more than a certain fraction of successful applications.
Finally, Council returns to the budgetary question. In order to
fund the number of grants, Council must decide how much, on average, to
decrease the budgetary recommendations from the committees. For the most
part those applications rated highest receive a higher percentage of the
budgets recommended by the committees.
My sincerest thanks to Ian Shuggart, Executive Director
of the Medical Research Council of Canada and Evie Gray, Director, Analysis
and Coordination Team, MRC Programs Branch, for providing information on
current MRC policy and procedure.
