U.S. Department of Health and Human Services
The Systematic Review: An Innovative Approach to Reviewing Research
David B. Larson, Lori E. Pastro, John S. Lyons and Edward Anthony
Office of Family, Community and Long-Term Care Policy, U.S. Department of Health and Human Services
April 14, 1992
PDF Version (10 PDF pages)
This paper was prepared by the U.S. Department of Health and Human Services, Office of Family, Community and Long-Term Care Policy (now the Office of Disability, Aging and Long-Term Care Policy). For additional information about this subject, you can visit the DALTCP home page at http://aspe.hhs.gov/_/office_specific/daltcp.cfm or contact the office at HHS/ASPE/DALTCP, Room 424E, H.H. Humphrey Building, 200 Independence Avenue, S.W., Washington, D.C. 20201. The e-mail address is: webmaster.DALTCP@hhs.gov. The Project Officer was David Larson.
STANDARD RESEARCH LITERATURE REVIEWS
Standard research literature reviews are the usual means of summing up and interpreting the accumulated findings of research studies in a given research field. Review results generally serve as the best guide for further research. Review results can also provide an up-to-date description of the strengths, weaknesses, and findings in a research field, and, therefore, can be used to inform makers of opinion, policy, and law.
In this second role, a sound literature review of a sufficient number of pertinent and well-done studies can give invaluable shape and clarity to decision-making and to public discussion of important issues. For example, reviews of the medical literature are essential tools in shaping clinical practice, as well as health policy, service delivery, and the research itself. Thus, great care should be taken to insure that the literature review accurately reflects the most up-to-date and accurate research on the issue being studied.
PROBLEMS WITH STANDARD LITERATURE REVIEWS
It is unfortunate, however, that the greater the need for sound research to clarify an understudied issue, the more likely such research will be absent. Until the last decade, far too many social policy literature reviews did not meet basic scientific standards because research methods weren't strictly followed. When a review method is not applied in a standard way across all studies, the results can often appear biased. As a result, many social policy literature reviews have frequently resulted in controversy.
For example, such controversy, resulting from failing to use sound review methodology, followed the publication of several studies on the occurrence of psychological effects of abortion. A comparison of two of these literature reviews demonstrated substantial differences between the reviews, not only in which articles were selected for review, but also in the results which were chosen for emphasis by the reviewers. In other words, the reviews resulted in reviewer bias largely due to neither review having any strict, objective criteria for the selection and analysis of the articles on the topic being studied.
THE SYSTEMATIC REVIEW SOLVES THE PROBLEMS FOUND IN STANDARD LITERATURE REVIEWS
The systematic review (SR) methodology minimizes the problems associated with less scientifically rigorous review methodologies by using strict, quantitative research methods that lead to objective results. The SR's main purpose is to objectively review a topic in order to produce an accurate, unbiased, and up-to-date summary of the research on that topic. Thus, the SR uses clearly specified research methods to avoid the all-too-easy introduction of bias in the selection and interpretation of the literature being studied. For example, SRs use clearly defined inclusion and exclusion criteria for both the journal and the article selection. SRs also use strict criteria to evaluate the quality of the measures used to assess the factor of review interest. Additionally, SRs use strict criteria to evaluate the quality of each study's research methodology.
META-ANALYTIC REVIEW VS. SYSTEMATIC REVIEW
Two different research review methodologies were developed to improve the quality of standard literature reviews--the meta-analytic review and the systematic review. Light and Pillemer of Harvard developed the initial conceptualizations of the quantitative review methodology used in both the meta-analytic and the SR method. Larson and Lyons then advanced this quantitative review method to produce their SR methodology.
Similarities between the SR and the meta-analytic review include replicability, resulting from the use of the scientific method and specification of study inclusion criteria. Standard literature reviews are usually not replicable, and it is essential that replication be possible, especially when literature reviews evaluate potentially controversial research topics.
However, there are a number of significant differences between SRs and meta-analytic reviews. For example, cost differences between the SR and the meta-analytic review can be significant. The SR costs only 10% to 20% the expense of a similarly sized meta-analytic review. Another major difference between these two reviews concerns their ability to study single factors of interest within an inadequately developed research field.
To be methodologically possible, the meta-analytic review requires a fairly well-developed research field, having a large body of experimental or quasi-experimental research. This type of review requires an adequate number of studies that address essentially the same research question using comparable study samples.
Alternatively, the SR is a comprehensive review method that does not require a well-developed research field, nor does it require studies to have experimental or quasi-experimental designs. SRs require no assumptions about the comparability of study samples and interventions. The SR technique looks at the frequency that a particular research question, variable, or measure was assessed, the quality of how it was assessed, and the quality of the studies that include the factor of interest.
Like meta-analysis, or even more traditional reviews, SRs can examine the key or central findings in studies, but they offer an advantage over other types of reviews because they also permit analysis of non-central or peripheral factors in the research. Thus, when the review question concerns an under-developed or infrequently studied research issue, the SR is the type of review that will most effectively produce an objective evaluation of both the frequency and the quality of assessment.
STEPS OF THE SYSTEMATIC REVIEW
The SR is comprised of several key steps: 1) selecting the factors to be studied (more than one factor may be studied in the review); 2) deciding which SR method to use -- either the exhaustive or field review; 3) assessing the frequency and quality of measurement of the factor of interest; 4) evaluating the studies containing the factor of interest; and 5) determining and maintaining reviewer reliability.
1. Selecting the factors to be studied and forming research questions:
Because the SR applies scientific research methodology, the first step involves forming research questions based on what subject (factor of interest) you want to study. For example, several of the original SRs focused on a question concerning whether the quantity or the quality of the research containing religious variables was substandard in certain clinical scientific literatures. A second question focused on the effects of pornography to determine whether existing research demonstrated harm (or lack of harm) in assessing each study's associations between exposure to pornographic materials and changes in attitudes concerning rape or aggression towards women. In each instance, the SR involved the development of clear research questions to be addressed by the review.
2. Deciding on the SR method:
The second step is to decide which SR method to use. That is, define the criteria to be used to select the types of studies to be included and the types to be excluded in the SR. There are two basic types of SR methods: exhaustive and field reviews. The inclusion and exclusion criteria are defined differently for these two types of reviews.
Both review methods select studies from peer-reviewed journals. Therefore, in both methodologies, the studies selected have undergone a peer-review process of critique and revision prior to their publication. The exhaustive review method involves identifying every possible peer-reviewed study from every relevant field of study that includes information on the factor of interest. This method requires at least the following three steps:
-
First, a multiple, overlapping, computerized literature search is undertaken using multiple key-word terms and indexes. From this search, an initial list of articles is made. Because such computerized searches miss many relevant studies (see attached piece by Bareta et al., 1990), additional steps must be taken.
-
Thus, when the initial round of articles are secured, their reference sections are studied to identify and locate additional, potentially relevant, peer-reviewed articles. Pertinent citations from these articles are obtained, added to the list, and the reference sections of these latter studies will be similarly reviewed for additional articles. When all the reference sections have been examined and no new articles can be identified, a draft list of all relevant articles is composed.
-
As the third and final step, the list is circulated to identified experts (i.e., the three to five researchers with the most publications on the research study list) and their input is solicited for additional relevant articles.
In contrast to the exhaustive review, the field systematic review method involves selecting only one field of study, and, then, selecting the leading peer-reviewed journals in that field and the years (usually a 5-10 year period) to be reviewed. The means of selecting journals to represent a particular field is to choose quantified studies from the one or two best, i.e., the most frequently cited, peer-reviewed journals in a particular research field. These representative journals can be found by using the Social Science Citation Index and the Science Citation Index. These indexes provide ratings of journals in a variety of research fields based on the frequency with which their articles are cited. The field SR is the best review to use if the goal is to define the most accurate and up-to-date research in a specific field.
To obtain a proper sample using the field SR method, the reviewer searches by hand through every journal issue and every article within the journal studies that include the review factor of interest. For examples, SR's have examined mental health factors in nursing home studies and AIDs research in general medical journals. In addition, SR's have been done reviewing religious factors in psychiatry, family medicine, and pastoral care journals. Running totals are kept of the number of articles scanned and the number of articles on the factor of interest which are found. Studies to be excluded from the SR include editorials, commentaries, and other non-quantifiable opinion pieces since the SR keeps totals and assesses only studies with quantified data.
3. Assessing, the factors to be studied:
After the sample of review articles has been obtained, the third step involves collecting data regarding the SR factor of interest. The factor of interest is examined across the reviewed articles to determine whether it is of major (frequently assessed) or minor (infrequently assessed) importance. In addition, tabulations can be made concerning whether the factor is being assessed using one or several questions and, if several questions, whether there was a reported or demonstrated reliability.
4. Evaluating the studies that include the factors of interest:
The fourth step is to evaluate the studies that include the factors of interest in order to assess the research quality of the studies themselves. In other words, if these factors are included and assessed in poorly designed studies, the dependability of the findings will be far more questionable than findings based on more adequately designed studies.
The assessment of the quality of the study methodology will require operationalization of each of the aspects of study methodology by clearly defining them. For example, in the assessment of response rate, one would not only operationalize whether the study included or did not include a response rate, but would also assess whether the response rate was low, medium, or high. In previous studies, high has been operationalized as a response rate of 70% or greater; medium has been operationalized as a response rate between 50 to 69%; and low has been operationalized as a response rate less than 50%. Additional, relevant methodological variables could include: the size of the study population; the use of a control or comparison population; the type of sampling method used; and whether study measures had demonstrated reliability.
5. Determining and maintaining reviewer reliability:
The fifth step is to train multiple reviewers to appropriately assess the factors of interest in order to maintain reviewer reliability. These reviewers must be statistically reliable, in that when reviewing the same article, the same assessments are made. We have consistently found that with proper training, SRs can be consistently accomplished with reliabilities above .90. To ensure high reliability, periodic reliability checks should be undertaken to guard against reliability decay, especially when the SR involves a large number of studies, and, thus, a large number of reviewers.
WHAT THE SYSTEMATIC REVIEW TELLS US
SRs provide extensive information on a specific research field or topic. Several things that this type of review can tell us, include:
- the number of studies assessing the factor of interest;
- the statistical reliability of measures assessing the factor of interest;
- the most often used approach to assess the factor;
- the frequency that the factor was assessed as a variable of major or minor study relevance; and
- the quality of the research studies including the factor of interest.
THE SR'S GROWING POPULARITY
SRs now have been published in a number of the leading peer-reviewed journals. One of the main reasons for this method's popularity is its scientific objectivity, particularly in assessing potentially controversial policy-relevant subjects. This innovative policy review method will continue to be more widely used in the evaluation and interpretation of policy research.
For further information, please write or call David B. Larson, M.D., M.S.P.H., Senior Policy Analyst at:
Department of Health and Human ServicesHubert Humphrey Building, Room 424E200 Independence AvenueWashington, D.C. 20201(202) 245-6613/David Larson or Lori Pastro
FOR FURTHER INFORMATION ON THE SR, SEE THE FOLLOWING BOOKS AND ARTICLES:
Bareta, Joseph C., David B. Larson, John S. Lyons, and Joseph J. Zorc. "A Comparison of Manual and MEDLARS Reviews of the Literature on Consultation-Liaison Psychiatry." American Journal of Psychiatry, vol. 147, no. 8, August 1990, pp. 1040-1042.
Beardsley, Robert S., David B. Larson, John S. Lyons, Gary L. Gottlieb, Peter Rabins, and Barry Rovner. "MINIREVIEW: Health Services Research in Nursing Homes: A Systematic Review of Three Clinical Geriatric Journals." Journal of Gerontology, vol. 44, no. 1, 1989, pp. M30-35.
Cooper, Harris M. The Integrative Research Review - A Systematic Approach. Applied Social Research Methods Series, Volume 2. Beverly Hills, California: Sage Publications, Inc., 1984.
Katerndahl, David A., and Peter A. Cohen. "Research Methodology Series Quantitatively Reviewing the Literature: The Application of Meta-Analysis." Family Practice Research Journal, vol. 6, no. 3, Spring 1987, pp. 123-129.
Larson, David B., Michael J. Donahue, John S. Lyons, Peter L. Benson, Mansell Pattison, Everett L. Worthington, Jr., and Dan G. Blazer. "Religious Affiliations in Mental Health Research Samples as Compared with National Samples." The Journal of Nervous and Mental Disease, vol. 177, no. 2, 1989, pp. 109-111.
Larson, David B., John S. Lyons, Ann A. Hohmann, Robert S. Beardsley, Wendy M. Huckeba, Peter V. Rabins, and Barry D. Lebowitz. "A Systematic Review of Nursing Home Research in Three Psychiatric Journals: 1966-1985." International Journal of Geriatric Psychiatry, vol. 4, 1989, pp. 129-134.
Larson, David B., E. Mansell Pattison, Dan G. Blazer, Abdul R. Oniran, and Berton H. Kaplan. "Systematic Analysis of Research on Religious Variables in Four Major Psychiatric Journals, 1978-1982." American Journal of Psychiatry, vol. 143, no. 3, March 1986, pp. 329-334.
Larson, David B., Kimberly A. Sherrill, John S. Lyons, Frederic C. Craigie, Jr., Samuel B. Thielinan, Mary A. Greenwold, and Susan S. Larson. "Associations Between Dimensions of Religious Commitment and Mental Health Reported in the 'American Journal of Psychiatry' and 'Archives of General Psychiatry': 1978-1989." American Journal of Psychiatry, vol. 149, no. 4, April 1992, pp. 557-559.
Light, Richard J., and David B. Pillemer. Summing Up - The Science of Reviewing Research. Cambridge, Massachusetts: Harvard University Press, 1984.
Lyons, John S., David B. Larson, Joseph C. Bareta, Ingrid Liu, Rachel Anderson, and Carolyn H. Sparks. "A Systematic Analysis of the Quantity of Aids Publications and the Quality of Research Methods in Three General Medical Journals." Evaluation and Program Planning, vol. 13, 1990, pp. 73-77.
Mann, Charles. "Meta-Analysis in the Breech." Science, vol. 249, August 1990, pp. 476-480.
Mulrow, Cynthia D. "The Medical Review Article: State of the Science." Annals of Internal Medicine, vol. 106, no. 3, March 1987, pp. 485-488.
Rabins, Peter V., Barry W. Rovner, David B. Larson, Barbara Bums, Carol Prescott, and Robert S. Beardsley. "The Use of Mental Health Measures in Nursing Home Research." Journal of the American Geriatrics Society, vol. 35, 1987, pp. 431-434.
Wang, Margaret C., Geneva D. Haertel, and Herbert J. Walberg. "What Influences Learning? A Content Analysis of Review Literature." Journal of Educational Research, vol. 84, no. 1, September/October 1990, pp. 30-43.