U.S. Department of Health and Human Services

Costs and Benefits of Health Information Technology

Paul G. Shekelle, M.D., Ph.D., Sally C. Morton, Ph.D., and Emmett B. Keeler, Ph.D.

Southern California Evidence-based Practice Center

April 2006

PDF Version (168 PDF pages)

This report was prepared under contract #290-02-0003 between U.S. Department of Health and Human Services (HHS), Office of Disability, Aging and Long-Term Care Policy (DALTCP) and the RAND Corporation. Additional funding was provided by HHS's Agency for Healthcare Research and Quality. For additional information about this subject, you can visit the DALTCP home page at http://aspe.hhs.gov/_/office_specific/daltcp.cfm or contact the ASPE Project Officer, Jennie Harvell, at HHS/ASPE/DALTCP, Room 424E, H.H. Humphrey Building, 200 Independence Avenue, S.W., Washington, D.C. 20201. Her e-mail address is: Jennie.Harvell@hhs.gov.

The opinions and views expressed in this report are those of the authors. They do not necessarily reflect the views of the Department of Health and Human Services, the contractor or any other funding organization.

TABLE OF CONTENTS

AHRQ PRESS RELEASE

REPORT INFORMATION

PREFACE

STRUCTURED ABSTRACT

EXECUTIVE SUMMARY

CHAPTER 1. INTRODUCTION

A Framework for Considering the Costs and Benefits of Health Information Technology

Elements of the Business Case

What Is Generalizable Knowledge Regarding Health Information Technology?

CHAPTER 2. METHODS

Original Proposed Key Questions

Technical Expert Panel

Literature Search

Additional Sources of Evidence

Article Review

Synthesis of Results

Peer Review

CHAPTER 3. RESULTS

Description of the Studies

The Costs and Benefits of Health Information Technology in Pediatrics

Electronic Health Records and Quality of Ambulatory Care

Economic Value of an Electronic Health Record Systems and Health Information Technology Applications

Health Information Technology and Patient Centeredness

Barriers to HIT Implementation

CHAPTER 4. DISCUSSION

Limitations

Conclusions

Recommendations for Future Research

REFERENCES

NOTES

APPENDICES

APPENDIX A: Technical Expert Panel

APPENDIX B: Information Technology in Health Search Strategies

APPENDIX C: Health Information Technology Round Two Screener

APPENDIX D: List of TEP Members Who Provided Comments

APPENDIX E: Reviewer Comments

APPENDIX F: Articles Included in HIT Interactive Database

APPENDIX G: List of Excluded Articles

LIST OF FIGURES

FIGURE 1: HIT Literature Flow

LIST OF TABLES

TABLE 1: Summary of Studies Assessing the Economic Impact of an EHR System

Appendixes and Evidence Tables for this report are also provided electronically at http://www.ahrq.gov/downloads/pub/evidence/pdf/hitsyscosts/hitsys.pdf.

AHRQ PRESS RELEASE

Advancing Excellence in Health Care
EMBARGOED FOR RELEASETuesday, April 11, 2006, 9:00 a.m., EDT	Contact:	AHRQ Public Affairs(301) 427-1855(301) 427-1922

NEW INFORMATION WILL HELP HEALTH CARE PROVIDERS ADOPT HEALTH IT

The Agency for Healthcare Research and Quality (AHRQ) released a report today acknowledging that while health information technology has been shown to improve quality of care for patients, most health care providers need more information about how to implement these technologies successfully. AHRQ is helping to fill this gap with findings from more than 100 projects across the country. These projects make up AHRQ's $166 million health IT initiative.

The report, Costs and Benefits of Health Information Technology, is a synthesis of studies that have examined the quality impact of health IT as well as the costs and organizational changes needed to implement health IT systems. This report reviews scientific data about the implementation of health IT to date, as documented in studies published through 2003. It does not project future health care benefits or savings, in contrast to other reports.

The authors conclude that scientific reviews have shown significant improvements in the quality of health care utilizing health IT systems. However, these successes have occurred primarily within large health care systems that created their own health IT systems and devoted substantial commitment and resources to these efforts. AHRQ's initiative is developing data needed about how to put health IT to work in more common health care settings such as physicians' offices and hospitals.

"HIT has the potential to enable a dramatic transformation in the delivery of health care, making it safer, more effective, and more efficient," the report concludes. "However, widespread implementation of HIT has been limited by a lack of generalizable knowledge about what types of HIT and implementation methods will improve care and manage costs for specific health organizations."

Large health care organizations and health plans have been leaders in health IT. The report points out that, by contrast, the smaller medical practices and hospitals that constitute the majority of the nation's health care providers have limited technological expertise and must depend on the purchase of commercial systems. Data about health IT implementation in these settings has been very limited, according to the report.

As a result, a predominant portion of health care providers in America have not had the information they need to calculate the impact of health IT implementation on their organizations.

"Health care providers need reliable information that tells them what they can expect when they implement health IT systems," said AHRQ Director Carolyn M. Clancy, M.D. "Leading institutions in health IT have shown that these systems can produce improved quality and patient safety. But smaller practices and hospitals need to know how these improvements can be achieved in settings like theirs, using the kinds of commercial systems they are likely to employ. AHRQ's health IT initiative is designed to generate and share the kind of information providers need."

Dr. Clancy said AHRQ's health IT initiative will help deliver this kind of information. The $166 million initiative includes more than 100 projects where health IT systems are being implemented, with an emphasis on systems in community-based health care settings, using commercially available systems. The AHRQ initiative was launched in September 2004, and most projects have 3-year duration.

The AHRQ-sponsored research will yield scientifically valid information that will share the experiences of typical providers in implementing health IT systems. This includes both the impact on quality and safety of care as well as the organizational impact of implementing health IT systems.

"AHRQ's initiative is a real-world laboratory, showing how health IT can be used successfully in typical health care settings," Dr. Clancy said. "The experiences of our grantees will be shared broadly to help all health care providers more successfully adopt health IT."

Findings from the AHRQ projects are being made available through the AHRQ National Resource Center for Health Information Technology, at http://www.healthit.ahrq.gov.

The report released today was prepared by the Southern California Evidence-based Practice Center-RAND Corporation, one of 13 evidence-based practice centers supported by AHRQ. Another study released last year by a separate group of RAND researchers estimated that wide adoption of electronic medical records and other health IT could save more than $81 billion annually and improve the quality of care.

The report was requested and funded by AHRQ and the Office of the Assistant Secretary for Planning and Evaluation. The Office of Disease Prevention and Health Promotion also provided financial support. Others requesting the report were HHS' Centers for Medicare & Medicaid Services and the Leapfrog Group, an organization of health care purchasers.

When the embargo lifts at 9:00 a.m. EDT on Tuesday, April 11, the report will be available at http://www.ahrq.gov/downloads/pub/evidence/pdf/hitsyscosts/hitsys.pdf. In addition, an interactive database providing access to the studies reviewed as part of the report will be available at http://www.healthit.ahrq.gov/tools/rand.

An article summarizing the report will be published in the May 16 edition of Annals of Internal Medicine. It also will be available starting April 11 at http://www.annals.org/cgi/content/full/0000605-200606200-00124v1

###

REPORT INFORMATION

This document is in the public domain and may be used and reprinted without permission except those copyrighted materials noted for which further reproduction is prohibited without the specific permission of copyright holders.

Suggested Citation: Shekelle PG, Morton SC, Keeler EB. Costs and Benefits of Health Information Technology, Evidence Report/Technology Assessment No. 132 (Prepared by the Southern California Evidence-based Practice Center under Contract No. 290-02-0003.) AHRQ Publication No. 06E006. Rockville, MD: Agency for Healthcare Research and Quality. April 2006.

PREFACE

The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-Based Practice Centers (EPCs), sponsors the development of evidence reports and technology assessments to assist public- and private-sector organizations in their efforts to improve the quality of health care in the United States. This report was requested and funded by the Office of the Assistant Secretary for Planning and Evaluation (ASPE) with additional funding from the Office of Disease Prevention and Health Promotion (ODPHP), U.S. Department of Health and Human Services. In addition, the report was requested by the Leap Frog Group and the Centers for Medicare & Medicaid Services (CMS). The reports and assessments provide organizations with comprehensive, science-based information on common, costly medical conditions and new health care technologies. The EPCs systematically review the relevant scientific literature on topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to developing their reports and assessments.

To bring the broadest range of experts into the development of evidence reports and health technology assessments, AHRQ encourages the EPCs to form partnerships and enter into collaborations with other medical and research organizations. The EPCs work with these partner organizations to ensure that the evidence reports and technology assessments they produce will become building blocks for health care quality improvement projects throughout the nation. The reports undergo peer review prior to their release.

AHRQ expects that the EPC evidence reports and technology assessments will inform individual health plans, providers, and purchasers, as well as the health care system as a whole, by providing important information to help improve health care quality.

We welcome written comments on this evidence report. They may be sent to: Director, Center for Outcomes and Evidence, Agency for Healthcare Research and Quality, 540 Gaither Road, Rockville, MD 20850.

STRUCTURED ABSTRACT

Objectives: An evidence report was prepared to assess the evidence base regarding benefits and costs of health information technology (HIT) systems, that is, the value of discrete HIT functions and systems in various healthcare settings, particularly those providing pediatric care.

Data Sources: PubMed®, the Cochrane Controlled Clinical Trials Register, and the Cochrane Database of Reviews of Effectiveness (DARE) were electronically searched for articles published since 1995. Several reports prepared by private industry were also reviewed.

Review Methods: Of 855 studies screened, 256 were included in the final analyses. These included systematic reviews, meta-analyses, studies that tested a hypothesis, and predictive analyses. Each article was reviewed independently by two reviewers; disagreement was resolved by consensus.

Results: Of the 256 studies, 156 concerned decision support, 84 assessed the electronic medical record, and 30 were about computerized physician order entry (categories are not mutually exclusive). One hundred twenty four of the studies assessed the effect of the HIT system in the outpatient or ambulatory setting; 82 assessed its use in the hospital or inpatient setting. Ninety-seven studies used a randomized design. There were 11 other controlled clinical trials, 33 studies using a pre-post design, and 20 studies using a time series. Another 17 were case studies with a concurrent control. Of the 211 hypothesis-testing studies, 82 contained at least some cost data. We identified no study or collection of studies, outside of those from a handful of HIT leaders, that would allow a reader to make a determination about the generalizable knowledge of the study's reported benefit. Beside these studies from HIT leaders, no other research assessed HIT systems that had comprehensive functionality and included data on costs, relevant information on organizational context and process change, and data on implementation.

A small body of literature supports a role for HIT in improving the quality of pediatric care. Insufficient data were available on the costs or cost-effectiveness of implementing such systems.

The ability of Electronic Health Records (EHRs) to improve the quality of care in ambulatory care settings was demonstrated in a small series of studies conducted at four sites (three U.S. medical centers and one in the Netherlands). The studies demonstrated improvements in provider performance when clinical information management and decision support tools were made available within an EHR system, particularly when the EHRs had the capacity to store data with high fidelity, to make those data readily accessible, and to help translate them into context-specific information that can empower providers in their work.

Despite the heterogeneity in the analytic methods used, all cost-benefit analyses predicted substantial savings from EHR (and health care information exchange and interoperability) implementation: The quantifiable benefits are projected to outweigh the investment costs. However, the predicted time needed to break even varied from three to as many as 13 years.

Conclusions: HIT has the potential to enable a dramatic transformation in the delivery of health care, making it safer, more effective, and more efficient. Some organizations have already realized major gains through the implementation of multifunctional, interoperable HIT systems built around an EHR. However, widespread implementation of HIT has been limited by a lack of generalizable knowledge about what types of HIT and implementation methods will improve care and manage costs for specific health organizations. The reporting of HIT development and implementation requires fuller descriptions of both the intervention and the organizational/economic environment in which it is implemented.

EXECUTIVE SUMMARY

Introduction

The United States health care system is at risk due to increasing demand, spiraling costs, inconsistent and poor quality of care, and inefficient, poorly coordinated care systems. Some evidence suggests that health information technology (HIT) can improve the efficiency, cost-effectiveness, quality, and safety of medical care delivery by making best practice guidelines and evidence databases immediately available to clinicians, and by making computerized patient records available throughout a health care network. However, much of the evidence is based on a small number of systems developed at academic medical centers, and little is known about the organizational changes, costs, and time required for community practices to successfully implement off-the-shelf systems.

An analysis of the usefulness of implementing HIT must take into consideration several factors:

• The potential of this technology to improve health care quality, safety, and patient satisfaction--and how this potential has been demonstrated.
• The cost-effectiveness of the technology--the business case for adoption of the technology--including the total costs of implementation (both financial and in terms of resources) and any cost savings that accrue. Concerns exist that those who bear the greatest share of such costs are not able to recoup those costs.
• The ability to generalize the effects of an HIT intervention on costs and benefits in existing systems (using published experience with or research on these systems) to the technology's use by other health care organizations.

The Leap Frog Group and a number of components of the U.S. Department of Health and Human Services (HHS)--the Centers for Medicare & Medicaid Services (CMS), the Office of the Assistant Secretary for Planning and Evaluation (ASPE), the Office of Disease Prevention and Health Promotion (ODPHP), and the Agency for Healthcare Research and Quality (AHRQ)--requested a review of the research on HIT to compile and evaluate the evidence regarding the value of discrete HIT functions and systems in various health care settings. This Evidence-based Practice Report on the costs and benefits of health information technology systems, along with an accompanying interactive database that catalogs and assesses the existing evidence was prepared by the Southern California Evidence-based Practice Center (EPC). This report systematically reviews the literature on the implementation of HIT systems in all care settings and assesses the evidence in four specific circumstances:

1. The costs and benefits of HIT for pediatric care.
2. The ability of one aspect of HIT, the electronic health record (EHR), to improve the quality of care in ambulatory care settings.
3. The costs and cost-effectiveness of implementing HER.
4. The effect of HIT on making care more patient-centered.

Methods

An electronic search of PubMed, the Cochrane Controlled Clinical Trials Register, and the Cochrane Database of Reviews of Effectiveness (DARE) was conducted for articles published from 1995 to January 2004. Additional references were obtained by reviewing the references in several major reports prepared by private industry and by RAND Health. Two reviewers, each trained in the critical analysis of scientific literature, independently reviewed each study and resolved disagreements by consensus. The principal investigator resolved any disagreements that remained unresolved after discussions between the reviewers.

Studies selected for review had to be either:

• A meta-analysis.
• A systematic review.
• Original research that tested a hypothesis (that is, a report that compared data between groups or across time periods, assessing a specific question and using statistical tests to assess differences).
• Original research that conducted predictive analyses (a report that used modeling techniques and simulations to predict the effects of an HIT implementation).

Of 855 articles screened, 256 were accepted for review. Descriptive studies of HIT implementations were identified and classified according to the categories listed below, but were not reviewed in more detail.

The contents of each selected article or report were abstracted using electronic data-abstraction forms prepared especially for this analysis. Abstracted data included the system's capabilities, interventions used, study design, implementation processes, evaluation methods, outcomes, costs, and barriers to implementation. A structured abstract was created for each report; these abstracts can be accessed in an online, interactive database created for this evidence report. (This database can be accessed at http://healthit.ahrq.gov/tools/rand.)

Results

Overall Results

Of the 256 studies reviewed, 156 were about decision support, 84 assessed the electronic medical record, and 30 were about computerized physician order entry (CPOE). One hundred twenty-four of the studies assessed the effect of the HIT system in the outpatient or ambulatory setting, while 82 assessed its use in the hospital or inpatient setting. Ninety-seven studies used a randomized design. There were 11 controlled clinical trials, 33 studies that used a pre/post design, 20 studies that did a time series, and another 17 that were case studies with a concurrent control. Among the 211 hypothesis-testing studies, 81 contained at least some cost data.

Many of the studies concerned HIT systems developed and evaluated by academic and institutional leaders in HIT.

• Regenstrief Institute in Indianapolis, IN (18 studies)
• Partners/Brigham and Women's Hospital in Boston, MA (19 studies)
• Intermountain Health in Salt Lake City, UT (11 studies)
• Kaiser Permanente health care system (5 studies)
• Vanderbilt University in Nashville, TN (2 studies)
• U.S. Department of Veterans Affairs (VA) health care system (15 studies)

Studies from these institutions have contributed greatly to our knowledge about the usefulness of particular HIT functionalities (such as CPOE or computerized electronic alerts), and are examples of what can be realized by the implementation of broadly functional HIT at these specific institutions. But these studies also have limitations, in terms of their usefulness to inform decisions about the adoption of HIT elsewhere. The primary limitation is that these HIT systems were developed over the course of many years by technology champions at these institutions and, in a process of co-evolution, were adapted particularly to the working environment and culture of their respective institutions. Consequently, the "intervention" at these sites consists not only of the HIT system but also the local champions, who were often also the evaluators in published studies. Furthermore, it is challenging to calculate the cost of the development of the HIT system as a whole, since this process occurred over many years at each institution. In addition, these systems are not commercially available from a vendor--and vendors supply most HIT systems in use in the U.S.

We were able to identify only 15 studies that used a randomized or controlled clinical (RCT or CCT) design, included cost data, and assessed HIT systems that were not from one of the leading academic and institutional HIT institutions or the United Kingdom (another setting that has limited generalizability to U.S. health care institutions). When these 15 studies were examined for their HIT functionality using the classification system developed by the Institute of Medicine, 4 of them concerned only decision support and 4 assessed HIT systems with decision support and administrative processes. The remaining seven studies addressed other single functionalities or combinations of up to three functionalities. We were not able to find a single study that used a randomized or controlled clinical trial design, that did not report data from one of the leading academic or institutional HIT systems or the U.K., that reported cost outcomes and that assessed an HIT system including at least four of the eight IOM categories of functionality.

For the 103 hypothesis-testing studies that used a design other than a randomized or controlled clinical trial, 45 reported cost data. Of these 45 studies, 23 assessed systems that were not one of the leading academic or institutional HIT systems or that came from the U.K. An examination of these 23 studies for their functionalities showed, as in the studies using an RCT or CCT design, that most studies did not evaluate systems with a broad level of functionality. Five studies assessed only decision support, and three studies each assessed only administrative processes or order entry management. Three studies assessed HIT systems with two functionalities, order entry management and decision support. The remaining nine studies assessed various combinations of two or three functionalities. No study evaluated an HIT system with at least four of the eight categories of functionality.

The literature is even sparser regarding information about the organizational context of an HIT implementation. Of the hypothesis-testing studies, we identified only 3 studies that provided information about the financial context of the organization, such as the degree of managed care/capitation penetration; 6 studies with information about system penetration; 2 studies about facilitators to implementation; 1 study explicitly discussing sustainability of the HIT intervention; 12 studies reporting extrinsic factors in valuing costs and benefits such as the health care market competitiveness; and 6 and 9 studies, respectively, reporting the initial costs of the HIT system and costs of implementation.

In summary, we identified no study or collection of studies--outside of those from a handful of HIT leadership institutions--that would allow a reader to make a determination whether the study's reported benefit was generalizable. Besides these studies from HIT leaders, no other research assessed HIT systems that had comprehensive functionality while including data on costs, relevant information on organizational context and process change, and data on implementation. This limitation in generalizable knowledge is not only a matter of study design and internal validity. Even if further randomized, controlled trials are performed, the generalizability of the evidence would remain low unless additional systematic, comprehensive, and relevant descriptions and measurements are made regarding how the technology is utilized, the individuals using it, and the environment it is used in.

The Costs and Benefits of HIT in Pediatric Settings

Early evidence shows that stand-alone clinical decision-support systems (CDSS) (such as drug dosing calculators) can reduce medication dosing errors, and CPOE plus CDSS can reduce the incidence of harmful medication errors in the inpatient pediatric and neonatal intensive care settings. Other HIT systems, such as electronic medication administration records, pharmacy-based robots, smart infusion pumps/devices, and medication bar-coding, are predicted to reduce medication errors, but need further study.

The use of CPOE plus CDSS has been demonstrated, in separate studies, to (1) reduce the frequency or duration of antibiotic use for common pediatric illnesses such as pharyngitis and otitis media, and (2) improve completeness and reduce variation in clinical documentation. In the ambulatory setting, a single study showed that an appointment reminder system is cost-effective and significantly reduces missed appointments, while in the neonatal intensive care unit, another study showed that CPOE can reduce medication and radiology turnaround times. Therefore, the evidence for HIT cost-savings in pediatrics is limited, but appears optimistic.

Electronic Health Records and the Quality of Ambulatory Care

Adoption of EHR systems is widely believed to be critical to the delivery of consistent, high-quality health care, although the current use of EHRs is limited. Seven studies were identified on the use of EHR in four ambulatory care settings (three in the United States and one in the Netherlands). The findings reported in all of these studies were primarily related to the implementation processes and to changes in clinical processes.

With the exception of one study that examined the effects of incorporating HIV care guidelines and alerts on quality of care for HIV-positive patients, all the studies assessed the effects of adding various types of information related to laboratory test and prescription ordering to EHR ordering screens. In general, these studies showed that providing laboratory test guidelines and related information on test-ordering screens was associated with a decrease in orders for overused tests and an increase in orders for underused tests; provision of formulary guidance was associated with increased adherence to a formulary for at least one class of medication; and addition of HIV care guidelines and alerts was associated with improved quality of care.

The Economic Value of an EHR System

While EHR systems may be essential for improving efficiency and quality of health care, implementation of an EHR system requires substantial capital investments and organizational change. Consequently, many health care organizations are seeking evidence from previously implemented systems about the costs and benefits of EHR adoption in order to better inform decisions about the optimal timing and strategy for implementation.

Not all of the costs and benefits reported when implementing new systems or making changes to existing systems were financial. EHRs were associated with improvements in service and other resource utilization, provider productivity, care efficiency, documentation quality, clinical decisionmaking, guideline compliance, and costs of care.

Despite considerable variation among the few studies that modeled financial costs and benefits, all predicted substantial cost savings from EHR implementation. However, these studies each made a number of assumptions, and the predicted break-even points ranged from as short a time as 3 years to as long as 13 years.

HIT and Patient-Centered Care

The evidence is sparse for the ability of HIT systems to make health care more patient-centered. The best evidence of such a change is the beneficial effect on preventive care of using computerized reminders to patients. Telemedicine and consumer health informatics also have limited evidence of benefit in specific contexts. The evidence is much more limited about the health effects of more general, interactive health information technologies such as the Internet or e-mail, or the effect on patient trust and satisfaction of implementing HIT systems such as the electronic health record.

Barriers to HIT Implementation

Studies identified a large number of barriers to the implementation of HIT. These barriers can be classified as situational barriers (including time and financial concerns), cognitive and/or physical barriers (including users' physical disabilities and insufficient computer skills), liability barriers (including confidentiality concerns), and knowledge and attitudinal barriers. Cutting across all of these categories, however, may be the need for a major structural and ideological reorganization of clinical medicine as it is now practiced in the majority of settings to be able to integrate itself with and enjoy the benefits of HIT.

Conclusions

Limitations of the Review

• The primary limitation of this review is the quality, quantity, and generalizability of the available (published) studies. Substantially more information regarding implementation may have been obtained by contacting leading HIT implementers and conducting structured interviews with them.
• Many of the costs and financial benefits of EHR will change over the years, because they depend on the changing price of such factors as hardware, software, labor, and pharmaceuticals and medical devices. Consequently, the costs reported in some of the older articles are of limited relevance.

General Conclusions

• Predictions based on statistical models suggest that HIT has the potential to assist in dramatically transforming the delivery of health care, making it safer, more effective, and more efficient. However, the experimental evidence supporting benefits from HIT is more limited.
• A number of large health care organizations have realized some of these major gains through the implementation of multifunctional, interoperable HIT systems built around an electronic health record.
• The impact of HIT implementation on the cost and quality of care is not going to be consistent across institutions, independent of context. However, the specific context within which HIT is implemented, including the setting, the clinical issues, and the patient populations, greatly influences its use and effects.
• More widespread implementation of HIT is limited by the lack of generalizable knowledge about what types of HIT and methods for its implementation will prove most useful for specific health organizations, especially for small practices and small hospitals.
• The reporting of HIT developments and implementations needs to be improved, with greater attention given to descriptions of both the intervention and the organizational/economic environment in which the technology is implemented.
• A high priority must be placed on establishing standards for the information that needs to be measured and reported in studies of HIT implementation, similar to the CONSORT standards developed for reporting clinical trials of therapeutics.
• Using existing published evidence, it is not possible to draw firm conclusions about which HIT functionalities are most likely to achieve certain health benefits--and the assessment of costs is even more uncertain.
• Existing evidence is not sufficient to clearly define "who pays for" and "who benefits from" HIT implementation in any health care organization--except those, such as Kaiser and the VA, that are responsible for paying for and delivering all the care for the defined population.
• Statistical models can be built to estimate the costs and benefits of interoperable HIT systems within and across health care provider settings, payers/purchasers, and cumulatively across the health care continuum, but these models are based on many untested assumptions.
• Implementation of HIT faces many barriers, including institutional barriers, cognitive and/or physical barriers, liability barriers, and knowledge and attitudinal barriers.

CHAPTER 1. INTRODUCTION

The use of health information technology (HIT) has been promoted as having tremendous promise in improving the efficiency, cost-effectiveness, quality, and safety of medical care delivery in our nation's healthcare system. The realization of these benefits is especially important in the context of reports that show five years of consecutive annual double-digit increases in healthcare costs and increases in the numbers of adverse health events.^{1, 2} At the same time, reports have suggested that 50 percent of all healthcare dollars are wasted on inefficient processes. Legislators and organizational leaders at the federal and state levels have emphasized the need for healthcare to follow the example of many non-healthcare industries, in which implementation of computer information technology has been critical in increasing the accessibility of mission-critical information, automating labor-intensive and inefficient processes, and minimizing human error.

The most important use for HIT may be to help reduce medical errors. This technology-based strategy has proven effective in reducing the effects of human error in industries such as banking and aviation. Clinical HIT systems may make a substantial impact on medical quality and safety by integrating relevant automated decisionmaking and knowledge acquisition tools into the practices of medical providers, thereby reducing errors of omission that result from gaps in provider knowledge or the failure to synthesize and apply that knowledge in clinical practice. These systems, when integrated within larger HIT systems, may improve medical decisionmaking and appropriate use of diagnostic tests and therapeutic agents.

In the ambulatory healthcare environment, the use of HIT offers a variety of benefits. First, it can improve the efficiency and financial health of the practice. For years, many offices have used computerized scheduling and financial systems to streamline office processes by tracking practice productivity and automating reimbursement processes. Second, the use of ambulatory electronic health records (EHRs) also offers an opportunity to monitor and improve clinical quality by improving information access and reducing duplicative documentation. And technology-based "e-prescribing" tools may improve the efficiency and safety of prescribing practices in the outpatient setting just as they have done in the hospital setting. Finally, the widespread adoption of HIT will allow the achievement of system connectivity and information exchange among providers of the same organization, among organizations, and ultimately regionally and nationwide.

However, the majority of medical organizations and providers have been slow to adopt HIT. Recent surveys of computerized physician order entry (CPOE) use show that only 9.6 percent of hospitals have CPOE completely available for use, and only half of these hospitals require use of CPOE.³ In the ambulatory setting, recent estimates place the use of electronic health records at 6 to15 percent of office-based physicians.^{4, 5} The potential advantages of widespread adoption of HIT in our nation's healthcare system make it vital to examine the scientific evidence that currently supports the relative costs and benefits of HIT, and the barriers to implementing various types of HIT systems across the spectrum of healthcare environments.

A Framework for Considering the Costs and Benefits of Health Information Technology

Private organizations deciding whether to invest in HIT must weigh the costs and benefits of doing so. Although the primary goal of nonprofit healthcare organizations may be to provide high-quality care, these organizations still need to watch the bottom line to survive, which includes understanding the costs of measures designed to improve quality. Such private return-on-investment (ROI) calculations can provide results that are quite different from those of societal cost-benefit analysis, which are often reported in clinical journals.

For example, one study showed that a hospital that installed a computerized reminder system to alert providers when patients were not up-to-date on their immunizations increased pneumococcal vaccine orders by 8 percent.⁶ Another study showed that, among the elderly, each $12 vaccination averts $20.27 in hospital costs and increases life expectancy an average of 1.2 days.⁷ From society's point of view, the reminder system saves money and improves health, so it is a win-win program. However, from a financial perspective, the hospital has spent money on a system that had no effect on the costs or revenues of current stays because the pneumococcal vaccine is not delivered in the hospital. To benefit from this intervention, the hospital must make a reputation for higher quality and convert it into profits. This is one example of the potential for a mismatch between who pays for and who accrues cost savings from HIT use. A more extreme example would be a hospital's implementation of a HIT intervention that averts future hospitalization. In this case, HIT implementation both costs the hospital money and decreases hospital revenues, even if the HIT implementation has a net cost-savings from a societal (or Medicare) perspective.

Elements of the Business Case

The business case for investing in HIT must consider both financial and nonmonetized¹ consequences. The financial aspect deals with the effect on the organization's bottom line. Any HIT investment has immediate costs in purchase, adaptation to the local organization, and staff training. So the business case for HIT depends on the downstream financial benefits exceeding the immediate costs. Because profits = revenue - costs = (revenue per patient - costs per patient) × (number of patients), long-term profits can come from increases in (profitable) patients, increases in revenues per patient, or decreases in cost per patient. The easiest of these to understand is costs per patient. All organizations benefit from becoming more efficient and reducing the costs of providing particular services. HIT can reduce the waste involved in collecting information and getting it to where it is needed for better decisionmaking. This increase in efficiency can streamline health care and billing processes, and avoid the costs of unnecessary services and of dealing with errors, both in patient care and in billing. Also, working in high quality organizations has some intangible benefits to staff, which may lead to better retention and productivity at equal levels of pay.

However, if the HIT is used to raise the quality of care or change the mix of services provided, the resulting financial costs and benefits depend on how the organization is paid and what expenses it bears. These factors can greatly affect what kind of return on investment is likely and when it will be realized. The next three paragraphs provide some examples.

A reputation for higher quality should increase the demand for an organization's services in a competitive market, but it is difficult to prove that you are better than your competition or better than you used to be. HIT can raise quality and can also generate the statistics to prove you have done so. Perceived higher quality allows organizations to increase market share and to negotiate higher prices from payers whose members demand access to those organizations, even if they have to pay slightly higher premiums to get it. In a competitive fee-for-service environment, greater market share increases revenues and may also permit some economies of scale.

HIT can also be used to increase reimbursable services per patient, such as covered immunizations and exams. HIT pays if it reduces waste, but it reduces profit if it reduces current or downstream services. Hospitals whose payments are set by DRGs (a fixed payment that depends on the diagnosis of the patient but does not vary with actual costs) benefit somewhat from shorter length of stay (although the last days of a hospitalization are the cheapest), but not from reduced readmissions (except those where a Medicare patient bounces back into the hospital before sufficient time lapses post-discharge to qualify the readmission for reimbursement as a "new" episode of care). A hospital also will not benefit financially from interventions that shift care to physicians' practices.

The biggest gains from quality and HIT come when providers are paid by means of a capitated fee system. Under such a system, any investment that reduces the total costs of care for these patients can be recouped, so it pays to reduce unnecessary services and to provide care in the most efficient setting. HIT may help to share the information needed to do so. Such reasoning was behind the Department of Veterans Affairs' (VA's) decision to develop its HIT system. Most published examples of cost-saving quality projects come from health maintenance organizations (HMOs)--for example, better diabetes or heart failure care that keeps patients out of the hospital. Also for HMOs, high quality can offset other undesirable features--such as poor access or amenities--or can justify higher premiums. The gains to HMOs of better care will be more certain when capitation payments are adequately risk adjusted. Without risk adjustment, providing high quality chronic illness care, an area where HIT is particularly useful, may have the unprofitable side effect of attracting more-expensive patients.

Because some of the financial gains from high quality may go to purchasers (employers) rather than providers, particularly in noncapitated, fee-for-service environments, some purchasers have started to pay directly for quality. If the case for HIT were strong enough, insurers might want to subsidize it in part (i.e., based on the insurer's share of the provider's caseload). However, unless an insurer covers most of the patients in a particular health care organization or insurers agree to collaborate, it does not pay one insurer to subsidize HIT for an entire provider or organization because a substantial portion of the cost savings accrue to other payers (the "free rider" problem).

Non-healthcare businesses that are selecting investments might consider only financial return on investment (ROI), but providing health care is a business with an unusual emphasis on nonmonetized goals. The nonmonetized part of the business case includes all nonmonetary arguments that the organization feels will influence the decision to adopt or reject the intervention. Examples include the following:

• Maintaining credentials
• Satisfying reporting requirements
• Satisfying a requirement to do a quality improvement project
• Avoiding exposure to liability
• Building goodwill or reputation
• "Because it's the right thing to do."

Many of these nonmonetized items have financial aspects. For example, the intervention may reduce the cost of meeting a preexisting reporting requirement. Also, many organizations, particularly nonprofits, have nonfinancial goals--such as providing high quality care--in addition to financial goals.²

What Is Generalizable Knowledge Regarding Health Information Technology?

In this report, we use the term generalizable knowledge to mean published evidence of the effects of a HIT intervention on costs and benefits that other health care organizations can use to implement HIT and reasonably expect benefits similar to those reported in the original study. Therefore, generalizable knowledge from a study has two components: (1) the internal validity of the study and (2) the utility of the information to others considering implementing HIT. We can illustrate differences in generalizable knowledge by considering some examples.

The simplest example is that of a particular pharmaceutical therapy for patients with a certain condition. In this case, a randomized, placebo-controlled trial of the new pharmaceutical agent would be a study with good internal validity. Because pharmaceuticals are manufactured for consistency in strength and are given according to specified dosing schedules, another health care organization examining the results of such a study could reasonably assume that administration of the new pharmaceutical in the same doses and to patients with similar characteristics would result in benefits similar to those reported in the original study.

A second example would be the assessment of a new surgical therapy. In such a case, the evidence would not come from a randomized, double blind, placebo-controlled trial, since this design is not generally feasible for tests of surgical therapy. Data may come from studies comparing patients randomly assigned to surgical therapy or to an alternate therapy or nonrandomized studies comparing surgically treated patients with historical controls or even case series. As the confidence in the equivalence of the comparison groups at baseline diminishes, the difference in benefit must become greater for the reader to conclude that beneficial effects on outcomes are due to differences in therapy and not other differences between groups at baseline.

Even after accepting that a particular study reports a real difference in outcomes between groups, the healthcare organization or practitioner contemplating offering surgery must consider more factors than when contemplating the prescription of a new pharmaceutical agent. Surgical therapies are not as standardized as pharmaceutical agents, and outcomes depend upon such factors as the skill of the surgical team and hospital. There is no reason to expect that every surgeon and hospital delivers equivalent care the way physicians and patients can expect a standard dose of a pharmaceutical to have equivalent potency. Hence, a study describing the effects of a surgical therapy needs to give more detail than a study describing the effects of a pharmaceutical drug, namely, enough description of the surgeon and hospital that other healthcare organizations or providers can determine whether the reported outcomes are likely to be achieved in their own clinical situation.

When considering HIT evaluation, the situation becomes even more complex. Both the intervention and the subjects of the intervention are qualitatively different in a study of HIT than in a study of a pharmaceutical or surgical intervention. HIT implementation consists of a complex organizational change undertaken to promote quality and efficiency. Studies of organizational change are fundamentally different from studies of medical therapies. Organizational interventions interact with a wide range of organizational system components. To be successful, they must address these components in a locally effective way. Thus, in a sense, these interventions are by nature not widely generalizable, in contrast to studies of narrow interventions such as pharmaceuticals, which aim to identify treatment effectiveness that is operator-independent, or generalizable across settings or providers. This difference has several important consequences. First, randomized controlled trials are not always feasible for assessing organizational change. The risks and benefits of reliance on controlled trials for evidence about interventions involving organizational change has been debated.^{8, 9} However, reliance only on randomized clinical trials for evidence of the effect of HIT on costs and outcomes risks restricting the focus to narrow and tightly defined elements of HIT. In many real-world applications, complex organizational change interventions are implemented as a series of steps, with each step dependent on the organizational response to the previous step. Therefore, we judge that generalizable knowledge must and can come from many types of studies. However, we also judge that these studies must report details of the intervention and the organizational characteristics of where the intervention was implemented to allow other organizations to make judgments about the applicability of the results.

We consider the intervention in HIT studies to have at least four components:

• Technical--including the system components being tested (which may consist of CPOE, clinical charting, or electronic prescribing); the preexisting technology infrastructure (e.g., clinical and financial systems, network); and the existing electronic interfaces and integration.
• Human factors (machine-person interface)--system usability (e.g., "user-friendliness," system response time, intuitive user interface, support for workflow processes), support for specialty or context-specific actions (e.g., clinical content, order-sets, and level and acceptability of clinical decision-support).
• Project management--effecting complex sociotechnical process change around HIT implementation, aligning IT and organizational resources to achieve project milestones, and controllership of IT budgets.
• Organizational and cultural change, which may include a partnership of medical staff and administrative leadership to govern, align incentives, and mobilize organizational inertia to achieve desired outcomes through process change.

Cutting across all four of these components is effective communication. Most organizational change and IT projects have a strong but unrecognized communication component, which encompasses, among other things, the sharing of vision, values, and information about the components of HIT system selection, as well as its implementation and use.

Without an adequate description of all of these components in a study of HIT costs and benefits, it is difficult for others to be able to infer how, or even whether, they can reproduce the results. Omitting such information would be analogous to omitting the strength or dosing schedule from the report of a study of a pharmaceutical intervention.³

Similarly, the analogue of the patient in a study of HIT is the organization. No consensus exists regarding what aspects of the organization are most important to report, but some aspects are clearly important. Aspects that have been proposed as important include size, staffing, the organization's prior experience with quality improvement initiatives, processes expected to be influenced by the intervention and how these work currently, and the financial context of the organization. These characteristics may well determine which types of HIT interventions work in a given setting. For this review, we assessed (a) whether studies measured some key organizational characteristics and (b) what those characteristics were. Such characteristics might be considered key organizational demographics, just as gender, age, and illness severity would be considered key demographic characteristics for an efficacy and safety study of a new pharmaceutical.

However, knowing even these characteristics may not be enough to understand why a HIT intervention did or did not work. An organization has to do more than simply buy the software to be successful. It must also invest in adapting the software to the organization, developing new policies and procedures, and training staff. The extent to which the organization is willing and prepared to perform these and other critical additional functions to embed the HIT into all relevant systems determines organizational readiness for change. There is unfortunately little scientific knowledge about which organizational characteristics are essential, and which, like the color of the patient's eyes when assessing the effect of taking a new pill, are unimportant. Thus, even if the description of a successful intervention includes many of the details described above, without information about organizational readiness, readers cannot know whether or not the same intervention is likely to work in their own organization and how long and expensive the transitional process might be.

CHAPTER 2. METHODS

Original Proposed Key Questions

An evidence report on the costs and benefits of HIT systems was requested by the Leap Frog Group, the Centers for Medicare and Medicaid Services (CMS), the Office of the Assistant Secretary for Planning and Evaluation (ASPE), the Office of Disease Prevention and Health Promotion (ODPHP), and the Agency for Healthcare Research and Quality (AHRQ). The purpose of the report was to develop an evidence base regarding the value of discrete HIT functions and systems in various healthcare settings.

Original key questions for the report were:

1. What does the evidence show with respect to the costs and benefits of inter-operable electronic HIT data exchange for providers and payers/purchasers?
2. What is a framework that could be used in this study to describe levels/bundles of EHR functionality and to estimate the costs and benefits by such levels/bundles of functionality by payer/purchaser and percentage of provider penetration?
3. What knowledge or evidence deficits exist regarding needed information to support estimates of cost, benefit and net value with regard to HIT systems? Discuss gaps in research, including specific areas that should be addressed, and suggest possible public and private organizational types to perform the research and/or analysis.
4. What critical cost/benefit information is required by decision makers (at various levels) in order to give a clear understanding of HIT Systems value proposition particular to them?
5. What analytic methods (e.g., sources of data, algorithms, etc.) could be used to produce evidence of the costs and benefits within and across health care provider settings, payers/purchasers, and cumulatively across the health care delivery continuum and payers, of deploying electronic health information technology functions examined in this study?
6. What are the barriers that health care providers and health care systems encounter that limit implementation of electronic health information systems?

Technical Expert Panel

Each AHRQ evidence report is guided by a Technical Expert Panel (TEP). We invited a distinguished group of scientists, clinicians, and information technology experts, including individuals with expertise in medical informatics, Internet health, and telecommunications to participate in the TEP for this report. A list of panel members is included as Appendix A⁴.

The TEP's participation in the preparation of the report began with a meeting that was conducted via conference call at the start of the project; the purpose of this meeting was to get TEP input on the scope of the project, especially the specific information technology applications to address. We were also seeking input on what constitutes evidence because most of the data on HIT implementation derive from interventions that are not RCTs, which are the usual backbone of EPC evidence reports. This particular meeting was held at two separate times in order to accommodate scheduling conflicts; TEP members were asked to participate on the date that was more convenient for them. The meetings were held on March 19 and March 26, 2004.

At this meeting, we also discussed the framework for how to conduct our research. Many TEP members were interested in HIT implementation issues, for example, what can be learned from others who have implemented HIT in various settings, including both community and academic settings. They also emphasized that HIT is often implemented through multicomponent interventions, of which IT is just one aspect.

Based on the comments received during the TEP conference calls and numerous discussions with AHRQ, it was determined that the report would focus on reviewing the evidence from existing published articles regarding the costs, benefits, and barriers to implementing HIT. Many other excellent suggestions were received during the conference calls, such as performing new cost-benefit analyses or collecting unpublished information on barriers, but the decision was made that a review of existing published evidence should precede any other analyses.

Literature Search

At the time this report was undertaken, another team at RAND was working on a project entitled "Leveraging Modern Information Technology to Transform Medical Care Delivery." This project, funded by private industry, aimed to suggest policy changes that are likely to increase the rate of adoption of HIT in the United States. One part of the project involved assessing the effects of information technology on costs, health outcomes, and adverse events. We were given the list of titles from the team's November 2003 search of PubMed, which sought systematic reviews published in English from 1995 to 2003. PubMed, which is maintained by the U.S. National Library of Medicine, is widely recognized as the premier source for bibliographic coverage of biomedical literature. It encompasses information from Index Medicus, the Index to Dental Literature, and the Cumulative Index to Nursing and Allied Health Literature (allied health includes occupational therapy, speech therapy, and rehabilitation), as well as other sources of coverage in the areas of health care organization, biological and physical sciences, humanities, and information science as they relate to medicine and health care.

Our own search for studies of HIT began with an electronic search of PubMed on January 6, 2004 for reports of original research as well as any additional articles about HIT published since 1995. We ordered all articles on the HIT topics, regardless of study design or language. Appendix B shows our specific search strategies. We also searched the Cochrane Controlled Clinical Trials Register Database and the Cochrane Database of Reviews of Effectiveness (DARE). The Cochrane Collaboration is an international organization that helps people make well-informed decisions about health care by preparing, maintaining, and promoting the accessibility of systematic reviews on the effects of heath care interventions. In December 2004, we also conducted a specific search of the journal Health Affairs, developing a list of all articles with "information technology" or "information systems" as keywords. Health Affairs has published special editions on this topic in recent years.

Additional Sources of Evidence

Several other sources of evidence were considered, based on the recommendations of the TEP. Advanced Technologies to Lower Health Care Costs and Improve Quality was published in fall 2003 by the Massachusetts Technology Collaborative in partnership with the New England Healthcare Institute. Research was conducted by the First Consulting Group and was sponsored by several Massachusetts companies involved in healthcare and health insurance. The report focuses on seven advanced technologies (including examples of HIT, such as computerized physician order entry and electronic prescribing in the inpatient and ambulatory care setting) that have demonstrated both financial benefits and improved quality of care. It also includes discussions of barriers to implementation.

The Value of Computerized Provider Order Entry (CPOE) in Ambulatory Settings was published in 2003 by the Center for Information Technology, also located in the Boston area. This group conducted an international search for both academic and commercial sources of literature and also contacted 35 vendors regarding their currently available health information technology packages. The report found that CPOE can significantly improve quality while lowering costs.

Meta-Analysis on Computer-Based Clinical Reminder Systems reports on a 1996 metaanalysis of 16 trials by Shea, DuMouchel, and Bahamonde published in the Journal of the American Medical Informatics Association (JAMIA). The authors found that computer reminders in the ambulatory care setting improved utilization of vaccinations, breast cancer screenings, and colorectal cancer screenings, but not pap smears or other preventive care. Personal files were contributed by project staff, consultants, and technical expert panel members in response to a request for any applicable unpublished literature on the costs and benefits of HIT.

Articles could have been identified in more than one way (for example, the PubMed search and personal files might contain some of the same articles).

Article Review

We reviewed the articles retrieved from the various sources against our exclusion criteria to determine whether to include them in the evidence synthesis and in the special interactive database tool we created to accompany this report (see below). A screening review form that contains a series of categorization questions was created to track the articles (see Appendix C⁴). Two reviewers, each trained in the critical analysis of scientific literature, independently reviewed each study, and resolved disagreements by consensus. The principal investigator resolved any disagreements that remained unresolved after discussions between the reviewers.

As previously indicated, this report includes evidence from articles with many different study designs. Our initial search was unrestricted by study design. The resulting articles were divided into four categories: reviews, descriptive reports, hypothesis testing-studies, and predictive analysis studies.

Review articles identified by the search were classified as either systematic (including metaanalyses) or nonsystematic. The determination of systematic versus nonsystematic was made by reading the methods section of the article to see whether an acceptable method was employed to identify evidence. This assessment was made by the Center directors working independently with consensus resolution. Only systematic reviews were considered for further inclusion.

Articles were classified as descriptive if they primarily described the workings or implementation of a HIT system. We further classified these as qualitative or quantitative, based on the presentation of information regarding such factors as number of tests ordered and costs of implementation.

A third category of articles was classified as hypothesis-testing studies, indicating that researchers attempted to answer a study question by comparing data between groups or across time periods and using statistical tests to assess differences. Hypothesis testing studies were further classified as (1) those containing an intervention with a concurrent comparison group, which included randomized and nonrandomized controlled trials and controlled before-after studies; and (2) studies with an intervention but without a concurrent comparison group, which included pre-post studies, time-series studies with more than two measurement points, and studies that used a historical control group. Additional classifications of hypothesis testing studies included those without an intervention, which were cross-sectional in nature, and "other" hypothesis testing studies.

The fourth category of studies was predictive analyses, which included studies that used modeling techniques to predict what might happen with a HIT implementation rather than what did happen. Predictive analyses include cost-effectiveness and cost-benefit analyses. They typically use data from multiple studies and depend upon several assumptions, some of which are not always explicitly stated.

Selection of Articles and Data Elements for Interactive Database

Articles that were classified as systematic reviews, meta-analyses, hypothesis-testing, or predictive analyses went on to more detailed review. For reasons discussed below, we created structured abstracts for these articles and placed them in an interactive database of HIT studies (http://healthit.ahrq.gov/tools/rand).

We looked for the following data in each article: a description of the HIT system; the purpose of the study; the year or years the study was performed; the study design; the outcomes reported; a description of the study settings; the intervention and control arm; the evaluation method; a description of the HIT system, including how the system was acquired, the year the system was installed, the capability and comprehensiveness of the system; the integration of guidelines or decision support; the interoperability; the HIT implementation strategy; the financial context, such as whether this is a managed care or capitation environment, pay for performance, or area of public accountability; the system penetration; facilitators and barriers; evidence of the HIT system sustainability; extrinsic factors in valuing costs and benefits; the cost of the HIT system or systems, including initial costs of the hardware and the software; the cost of implementation, including planning, hiring, training, temporary productivity loss, data entry, and other organizational resources; anything about long-term cost; and outcomes, in terms of changes in healthcare utilization, changes in quality of care and patient safety, changes in healthcare costs, changes in efficiency and productivity, changes in revenue, and time needed to accrue the benefit. These data were judged to be important--and, in some cases, vital--to an understanding of the study's results as generalizable knowledge.

Synthesis of Results

Based on considerations about a framework for considering costs and benefits of HIT and what constitutes generalizable knowledge, we determined that a synthesis of the results of the included studies could not be meaningfully accomplished using conventional EPC methods for such syntheses. In other words, because the interpretation of the results of HIT studies is quite context-specific, meta-analysis would not be appropriate. No studies were really homogeneous or similar enough to consider together.

Similarly, a narrative review needs an organizing construct, such as "studies about CPOE," or "studies of HIT in rural hospitals," or even "studies of HIT that incorporate decision support and report benefits and costs for patient safety in the capitated ambulatory environment." However, the possible combinations of key variables is so vast that any limited number of narrative syntheses we might produce for this evidence report would inevitably not meet the needs of many potential users. Therefore, we decided that the most useful synthesis of this evidence would be in the form of structured abstracts of the included studies, presented in the interactive searchable database, which can be used by interested readers of this report to identify those HIT studies that meet their own particular contextual requirements. We also present four narrative reviews of studies in particular contexts, to illustrate the uses of the interactive database and also as a mechanism to discuss the strengths and limitations of the evidence regarding HIT.

Peer Review

A draft of this report was prepared in April 2005 and sent to the TEP members and others for review. We received comments from the persons listed in Appendix D⁴. Each comment received was tracked in an electronic spreadsheet and addressed in preparing the final report. Peer review comments and our responses to them are listed in Appendix E. Service as a reviewer of this report should not in any way be construed as agreeing with or endorsing the content of the report.

CHAPTER 3. RESULTS

We screened 855 articles, of which 599 were rejected: 124 did not have HIT as the subject; 4 did note report relevant outcomes; 288 were descriptive qualitative studies; and 183 were categorized as descriptive quantitative studies. A total of 256 articles was included in the HIT interactive database. (Figure 1 presents this information pictorially.)

FIGURE 1. HIT Literature Flow

Description of the Studies

Of the 256 studies included in the database, 156 pertained to decision support, 84 assessed the electronic medical record, and 30 were about CPOE (categories are not mutually exclusive). One hundred twenty four of the studies assessed the effect of the HIT system in the outpatient or ambulatory setting, while 82 assessed its use in the hospital or inpatient setting. Ninety-seven studies used a randomized design; 11 were other controlled clinical trials, 33 used a pre-post design, 20 used a time series, and another 17 were case studies with a concurrent control. Among the 211 hypothesis-testing studies, 82 contained at least some cost data (or data on utilization or efficiency, that could be converted to costs).

Many of the studies concerned HIT systems developed and evaluated by academic and institutional leaders in HIT: the Regenstrief Institute, Partners/Brigham and Women's Hospital, Intermountain Health, Kaiser, Vanderbilt, and the VA health care system. The HIT systems at the Regenstrief Institute and Partners were each assessed in 18 and 19 separate studies, respectively; 15 assessed the VA health information system; 11 studied Intermountain Health; 5 studied Kaiser; and 2 assessed the HIT system at Vanderbilt. Studies from these institutions have contributed greatly to our knowledge about the usefulness of particular HIT functionalities (such as CPOE or computerized electronic alerts) and are examples of what can be realized by the implementation of broadly functional HIT at these specific institutions. But these studies also have limitations in terms of their usefulness to inform decisions about the adoption of HIT in other locations. The primary concern is that these HIT systems were developed over the course of many years by champions at these institutions, and, in a process of coevolution, were specially adapted to the working environment and culture of their respective institutions. Consequently, the "intervention" consists of not only the HIT system but also its local champions, who were often also the evaluators in published studies. Furthermore, it is challenging to calculate the cost of the development of the HIT system as a whole, since this process has occurred over many years. Finally, these systems are not commercially available from vendors, whereas most HIT systems in the United States are commercial systems.

We were able to identify only 15 studies that used a randomized or controlled clinical design, included cost data, and assessed HIT systems that were not located at one of the leading academic and institutional HIT institutions or in the United Kingdom (UK), another setting that has limited generalizability to U.S. health care institutions. When these 15 studies were examined for their HIT functionality using the classification system developed by the Institute of Medicine,⁵ four of them concerned only decision support; four assessed HIT systems with decision support and administrative processes; and one study each assessed HIT systems with health information and data storage; health information and data storage with decision support; order entry management alone; order entry management with reporting and population health management; decision support with patient support and administrative processes; and health information with data storage decision support and administrative processes. In other words, we were unable to find a single study that used a randomized or controlled clinical trial design, reported data from a site other than one of the leading academic or institutional HIT systems or the UK, reported cost outcomes, and assessed a HIT system that included at least four of the eight IOM categories of functionality.

Of 103 hypothesis-testing studies that used a design other than a randomized or controlled clinical trial, 45 reported cost data. Of the 45 studies that reported cost data, 23 assessed systems that were not one of the leading academic or institutional HIT systems or UK systems. An examination of these 23 studies for their functionalities showed, as in the studies using an RCT or CCT design, that most did not evaluate systems with a broad level of functionality. Five studies assessed only decision support, and three studies each assessed only administrative processes or order entry management. Three studies assessed HIT systems with two functionalities: order entry management and decision support. The remaining nine studies assessed various combinations of two or three functionalities. No study evaluated a HIT system with at least four of the eight categories of functionality.

Regarding information about the organizational context of a HIT implementation, the literature is even more sparse. Of the hypothesis-testing studies, we identified only three studies that provided information about the financial context of the organization, such as the degree of managed care/capitation penetration; six studies with information about system penetration; one study about facilitators to implementation; one studies explicitly discussing sustainability of the HIT intervention; twelve studies reporting extrinsic factors in valuing costs and benefits, such as the healthcare market competitiveness; and six studies and nine studies, respectively, reporting on the initial costs of the HIT system and costs of implementation. No studies explicitly discussed sustainability of the HIT intervention.

In summary, we identified no study or collection of studies, outside of those from a handful of HIT leaders, that would allow a reader to make a determination about the generalizable knowledge of the system's reported benefit. Besides these studies from HIT leaders, no other research assessed HIT systems with comprehensive functionality while also including data on costs, relevant information on organizational context and process change, and data on implementation. This limitation in generalizable knowledge is not simply a matter of study design and internal validity: Even if more randomized controlled trials are performed, the generalizability of evidence will remain low unless more systematic, comprehensive, and relevant descriptions and measurements are made regarding how the technology is utilized, the individuals using it, and the environment it is used in.

As is apparent from the preceding discussion, the interpretation of studies of HIT is highly context-specific and is not amenable to the techniques of meta-analysis frequently used in other evidence reports to summarize results across studies. Certain functionalities of HIT systems have been the subject of recent reviews, such as CPOE,¹⁰ computer-based clinical decision support systems,^{11, 12, 13} and the use of computer-based guideline implementation systems.¹⁴ We will not summarize these reviews here. Readers are referred to the interactive database of HIT studies to select those studies that are most relevant to their own situation in terms of functionalities, clinical settings, outcomes reported, and other factors. The remainder of this chapter presents four examples of syntheses of the literature for specific situations: the effect of HIT in the field of pediatrics; evidence regarding the effect of the electronic health record on quality of ambulatory care; studies that report and predict the potential benefits and costs of implementation of the electronic health record; and health information technology and patient-centered care.

The Costs and Benefits of Health Information Technology in Pediatrics

Introduction

A decision to implement health information technology should carefully weigh the costs and benefits of incorporating it into the clinical environment. This is especially true in settings involved in the healthcare of infants and children, where patterns of practice and the needs of clinicians are unique. A recent report issued by the medical informatics taskforce of the American Academy of Pediatrics (AAP) cited a number of special requirements for the effective use of electronic medical record (EMR) systems in pediatrics.¹⁵ The practice of primary care and subspecialty pediatrics requires specialized collection of growth data, immunization history, longitudinal developmental inventories, parent education, age- and weight-based norms and dosing of therapeutics, specialized terminologies, and unique school-based forms and reports.

In the area of pediatric patient-safety, a growing number of studies have described the frequency of medication errors and adverse drug events (ADEs) in both the inpatient and ambulatory settings.^{16, 17, 18, 19} For a number of reasons--including weight- and age-based medication dosing, medication unit-doses designed for adult patients, and the limited ability of children to communicate or self-check medications before they are administered,^{20, 21}--infants and children are at higher risk for serious medication errors and resultant ADEs than are adults. HIT is believed to be a vital component in the quest to improve medication safety in pediatrics.

These special requirements, combined with a small commercial market for pediatric HIT systems relative to the adult population, make the implementation of HIT in the pediatric setting challenging and perhaps costly. Clearly, more must be known about the relative costs and benefits of HIT implementation and use in pediatrics and evidence of its impact on the six quality aims identified in the IOM report, Crossing the Quality Chasm,²² to deliver safe, effective, efficient, patient-centered, timely, and equitable healthcare.

Literature

Of the 256 articles included in the database, 14 articles were determined to contain quantitative data on the costs and/or benefits of HIT use in the pediatric healthcare setting. Because of a paucity of evidence, we also included descriptive quantitative studies in this section.

Summary of Evidence

Medication Use and Patient Safety. Given recent insight into the prevalence of medication errors in the pediatric population, health information technology is believed by most to be an important tool in reducing the rate of medication errors that occur in the care of infants and children.

Mullett et al.²³ enhanced an existing adult antiinfective computerized decision-support system for use in an academic pediatric intensive care unit (PICU) and measured its impact on medication-related outcomes. The study reported a 59-percent decrease in pharmacist interventions for erroneous drug doses and a decreased number of patient days of subtherapeutic (p0.001) or excessive (p0.001) antiinfective doses. In addition, the surveyed physicians reported that the use of the system improved their antiinfective choices and perhaps reduced the likelihood of ADEs. The authors also reported a decreased number of orders per patientantiinfective course as well as decreased robust estimated costs of antiinfective use by 9 percent in the intervention group vs. control ($86.60 vs. 78.43).

A study by Fortescue and colleagues²⁴ examined and characterized 616 medication errors occurring in the pediatric inpatient units of two academic tertiary referral medical centers. In a hypothetical experiment, physician experts determined what percentage of these errors could potentially have been prevented by the implementation of safety systems. Specifically, this hypothetical experiment determined that basic CPOE would avert 60 percent of potentially harmful errors, while CPOE with clinical decision-support systems (CPOE +CDSS) would increase the prevention of harmful errors to 75.8 percent. Other HIT systems identified by the report as being important for averting medication errors in pediatrics settings included computerized/electronic medication administration record (e-MAR) (19.2 percent of potentially harmful errors), robots in pharmacy (2.5 percent), smart intravenous infusion devices (4.2 percent), medication and patient and staff bar-coding (4.2 percent), and an automated bedside medication dispensing device (5.8 percent).

A number of studies have directly measured the benefit of CPOE using a variety of error-capture methodologies and study designs in different pediatric clinical environments. In a prospective cohort study, the authors documented medication prescribing errors (MPEs) and potential adverse drug events (PADEs) in a pediatric intensive care unit before and after implementation of a "home-grown" CPOE system.²⁵ The data showed a significant reduction of both MPEs (30.1 to 90.2 percent, p 0.001) and PADEs (2.2 to 1.3 percent, p0.001). A study by Cordero and colleagues in the neonatal intensive care setting (NICU) showed that CPOE could eliminate gentamicin prescribing errors as well.²⁶ The sum of this early evidence indicates that CPOE +CDSS has significant potential to reduce harmful medication errors, but the relative costs and complexities of achieving these beneficial outcomes need to be examined further.

Immunizations. Although a growing body of literature suggests that the use of HIT in pediatrics may be an important ingredient in reducing medication errors, a key challenge for pediatric providers lies in the area of maximizing adherence to vaccination recommendations. Paper-based immunization records do not allow for rigorous population-based monitoring or quality control. Therefore, computerized immunization registries, as separate or integrated systems and with clinical decision-support or reporting capabilities, offer tremendous potential in tracking and improving the rates of adherence to recommended immunization guidelines.

Ornstein et al.²⁷ evaluated a computer-based preventive services alerting system integrated into an electronic medical record system in an academically affiliated family practice clinic. In addition to surveying patient and physicians regarding their perceptions of the reminder system, the researchers performed before-and-after audits of adherence to recommended preventative services including childhood immunizations. Of the five immunization services tracked, only the administration of diphtheria and tetanus booster showed a small but significant improvement (48.8 to 50.6 percent, p=0.02). Adherence to the other recommended vaccinations did not show a significant improvement.

Szilagyi, and colleagues,²⁸ in an academically affiliated pediatric urban clinic, used a computerized database system to generate reminder letter for influenza vaccination to patients identified with moderate to severe asthma. Eligible patients were randomized into an intervention group, which received the reminders, and a control group. After four months, a review of the medical chart revealed a significant difference in influenza vaccination rates (30 percent intervention vs. 7 percent control, p0.01). This study demonstrated that computerized disease registry systems could serve as an important tool in improving vaccination rates in pediatrics.

Effective Disease Management. In addition to providing a potential means to influence prescribing and immunization practices in pediatrics, HIT systems also hold tremendous promise in improving clinical decisionmaking and disease management.

Medication Dosage and Delivery. Chiarelli et al.²⁹ evaluated a microprocessor device with computerized algorithms for insulin dose adjustment for pediatric patients with insulin-dependent diabetes mellitus, based on self-monitored blood glucose (SMBG) levels. This prospective randomized on-off-on study revealed that although the mean glycosylated hemoglobin levels and pre-meal SMBG levels did not improve, control patients were more likely to experience episodes of hypoglycemia than were patients using the device, and patients using the device used less insulin than during their corresponding baseline phase (p0.0001) and less insulin than the control group.

Disease-Based Clinical Decision Support. Schriger and colleagues³⁰ implemented an electronic medical record in a university hospital emergency department that provided documentation advice and recommendations for laboratory testing and treatment. Using an on-off-on interrupted study design, the authors measured appropriateness of care for febrile children less than three years of age, when measured against an evidence-based guideline. No evidence was found for improvements (or worsening) in appropriateness of care during the intervention phase compared to the baseline phase. However, use of the system was found to increase documentation of essential elements of the history and physical examination by 13 percent (95% confidence interval, 10 to 15 percent) as well as documentation of after-care instructions by 33 percent (95% confidence interval, 28 to 38 percent).

The appropriate course of antibiotic treatment for acute otitis media (AOM) is an area of concern in pediatrics. A study by Christakis et al.³¹ measure the impact of HIT on the antibiotic prescribing behavior of pediatric providers in an academic pediatric residency training clinic and compared cohorts during the pre-intervention and post-intervention phases. During the postintervention phase, providers were randomized to receive point-of-care advice recommending a course of antibiotics of less than 10 days duration (primary outcome) or delayed initiation of antibiotics (secondary outcome) for the treatment of AOM. Measurement of adherence to this computerized alert showed that providers in the intervention group had a 34-percent increase compared to the control group in the proportion of antibiotic prescriptions that were for less than 10 days (p0.01). However, during the intervention period, both the intervention and control groups became more likely to prescribe antibiotics, with the intervention group deteriorating less than the control group (p0.095). The results demonstrate that the prescribing practices of pediatricians for treatment of a common pediatric illness can be affected by a computerized reminder system.

Using a similar study design, Margolis et al.³² developed a computerized algorithm system that mandated structured input of data by providers for common pediatric problems. In return, the system provided recommendations for disease management and correct use of antibiotics. The investigators demonstrated decreased use of antibiotics for OM (p0.001) and pharyngitis (p0.01) as well as increased adherence to protocol recommendations for these two disease processes in the intervention group compared with the control group. However, the use of antibiotics for upper respiratory infections (URIs) did not change. The authors noted that the structured algorithms in the HIT system did improve the documentation of clinical elements important to ideal clinical care of pharyngitis, otitis media, and upper respiratory infections. It must be noted however, that this system's rigid requirements for physician documentation also made the HIT system unusable, and the physicians refused to use the system after five weeks.

Improved Documentation. Because many studies have reported an impact of HIT use on the quality and completeness of medical documentation, a study by Carroll and colleagues focused on the impact of a personal digital assistant (PDA) on documentation discrepancies in a NICU.³³ In this before-and-after study, all the NICU resident physicians used a PDA-based charting system during the intervention phase, comparing their progress notes against a predefined reference standard during both phases. The authors demonstrated that after adjustment for covariates, PDA-based charting did reduce discrepancies in patient weights in the charts but did not affect the number of medication or vascular line discrepancies.

Timeliness, Efficiency and Cost-Effectiveness of Care. Quattlebaum et al.³⁴ studied a scheduling/practice management system that automatically generated reminder postcards for appointment the following week. In this randomized controlled trial, the authors demonstrated a reduction of the no-show rate in their pediatric ambulatory practice from 19 to 10 percent. A cost-benefit analysis of the HIT system and its impact on missed appointments revealed that for each $1 spent on reminders, an additional $7.50 of revenue was captured.

In the inpatient setting, the previously discussed study of CPOE in the NICU by Cordero and colleagues²⁶ measured not only CPOE's effect on gentamicin dosing errors but also the time from medication prescription to administration for initial doses of a single medication and radiology tests during the pre- and post-CPOE phases. The authors documented significant reductions in the average turnaround time for both medications (10.5 to 2.8 hours, p0.01) and radiology tests (42 to 32 minutes, p0.001).

Summary

Early evidence shows that stand-alone CDSS can reduce medication dosing errors, and CPOE + CDSS can reduce the incidence of harmful medication errors in the inpatient pediatric and neonatal intensive care settings. However, other HIT systems, such as electronic MAR, pharmacy-based robots, smart infusion pumps/devices, and medication bar coding, are predicted to reduce medication errors but need to be studied further.

HIT also has tremendous potential to improve vaccination rates and disease management in pediatric outpatients. CDSS and registries have been shown to be effective in increasing vaccination rates in targeted populations, but only a limited HIT impact on general pediatric immunization rates has been demonstrated. Similarly, a patient clinical decision-support device that assists insulin dosing in children with diabetes reduces episodes of hypoglycemia and overall insulin requirements, but does not affect traditional measurements of glycemic control. And the use of computerized documentation systems with integrated CDSS has been demonstrated, in separate studies, to 1) reduce the frequency or duration of antibiotic use for common pediatric illness such as pharyngitis and otitis media, and 2) improve completeness and somewhat reduce variation in clinical documentation.

In the ambulatory setting, a single study showed that an appointment reminder system is cost-effective and significantly reduces missed appointments. Another study showed that CPOE in the NICU can reduce medication and radiology turnaround times. Therefore, the evidence for HIT cost-savings in pediatrics is limited but deserving of optimism.

Conclusion

A small body of literature supports the assertion that HIT use in pediatrics is beneficial in the areas of medication safety, adherence to immunization and disease-based guidelines, patient decision-support in diabetes management, clinical documentation, patient appointments, and in-hospital order processing. No data on the costs or cost-effectiveness of implementing these systems were found, except in one case. In addition, because many of these HIT systems were tested and/or developed in academic settings, the ability to generalize these findings to other organizations is uncertain.

Electronic Health Records and Quality of Ambulatory Care

Introduction

Despite rapid advances in the biomedical sciences, a growing body of evidence shows serious shortfalls in the quality of care Americans receive,^{35, 36} and significant longstanding shortfalls in performance have persisted despite recent increases in attention to quality.^{37, 38, 39, 40} International comparisons have demonstrated similar problems in quality.^{41, 42, 43} If the United States is to realize the full value of biomedical knowledge and of financial investments made in healthcare, the mechanisms through which that knowledge is operationalized and care is delivered must be radically redesigned.

Although the content of healthcare continues to change dramatically, the methods of health care delivery have not. In particular, a vast majority of the healthcare industry continues to deliver care, manage information, and conduct clinical transactions through the use of paper records.

Although the use of electronic health records (EHRs) is limited in healthcare, there is a renewed conviction by the government, provider groups, and healthcare purchasers that widespread adoption is critical to the delivery of consistent, high-quality care. However, EHR implementation, without other important changes in the way healthcare services are provided, is unlikely to improve quality. Such process redesign and reengineering is difficult and resource-intensive and is also hampered by the complexity and fragmentation of our current healthcare system. Therefore, despite the potential benefits of widespread EHR use, better empirical evidence is needed to confirm that EHR use does in fact improve quality and--perhaps more fundamentally--to understand what capabilities EHRs need to have for quality to be improved. At present, the depth and breadth of the empirical evidence regarding EHR use and its attributable impact on the quality of care remains unclear.

The purpose of this review is to examine and synthesize the available research evidence for the impact of EHR on quality of care in the outpatient setting. The review will also attempt to differentiate the direct impact of EHRs as point-of-care and workflow tools from how EHRs have been used to indirectly achieve those results, by measuring clinical and process outcomes. We elected to focus on ambulatory care because of the large volume of health services delivered in this arena. In addition, because the vast majority of outpatient practices comprise fewer than ten providers--many of whom lack technical infrastructure and resources--it is unclear whether widespread implementation of EHRs will be feasible in this environment.

Research Study Inclusion Criteria

From our database of 256 articles, we selected all 84 papers that related to EHRs. We then screened these articles against the following inclusion criteria: (1) the study reported quality-ofcare data as study outcomes, (2) the EHR was documented to have the following minimal functionality--electronic documentation (viewing, entry, or both), results management, CPOE, and some form of decision support, (3) the study was conducted in the ambulatory setting. The criteria for functionality were chosen based on the IOM's "White Paper on Key Capabilities of an Electronic Health Record." Given the rapid technical advances in EHR systems, we reviewed additional functional criteria to provide decisionmakers with the most relevant and forward-looking information available.

Analytic Framework

The Donabedian "Structure--Process--Outcome" model for quality was used as a framework for this review.⁴⁴ In this model, structure is defined as the resources and factors involved in producing care and the manner in which those resources and factors are organized. Examples of structural quality include the number of beds in a hospital, the number of physicians in an emergency room per shift, the budget for a clinic, and the presence of disease management program for diabetes. Process of care is defined as the activities that constitute health care. Examples include screening for breast cancer, ordering laboratory tests, and prescribing a medication. Outcomes are the end results of healthcare delivery processes. They are the consequences of health services or can be logically attributed to the act of providing those services. Whereas structure relates to the environment in which healthcare is delivered and process relates to the provisions of care, outcomes are events that occur with patients and consumers--as individuals, groups, or populations.

Two aspects of this model are particularly relevant to EHRs. First, to fully assess quality of care, there need to be links from structure to process to outcomes. The technical and functional capabilities of EHRs form a structure for care. In order to derive value from the EHR structure, new clinical processes need to be designed to utilize the EHR functional structure. These EHR-mediated processes should in turn lead to a specific set of better outcomes. Second, the distinctions among structure, process, and outcome are somewhat arbitrary in the model. Health care delivery is viewed as an interconnected series of structure--process--outcome relationships. For example, in a primary care clinic of three physicians (structure 1), a patient may have an electrocardiogram performed (process 1), which shows an abnormality (outcome 1). This abnormal result necessitates a referral to a cardiologist (structure 1) who orders a stress test (process 2), which comes back suggestive of coronary artery disease (outcome 2) and so forth. This structure--process--outcome chain is central to the role of EHRs in quality, because an EHR is a tool that explicitly links the three. An EHR with decision support for diabetes management (structure) allows a physician to order a hemoglobin A1C (process) and check the results (outcome). Because this outcome is stored in the EHR database, it in turn becomes part of the structure of care. The EHR can allow or even remind the physicians to act on that result (process 2), e.g., modify the patient's insulin dose, which will, in turn, lead to a lowering of the patient's blood sugar (outcome 3) or a reduction in the likelihood of a long-term diabetic complication.

In addition to imposing the Donabedian Structure--Process--Outcome model, we organized deficits in quality by means of a conceptual framework that divides quality problems into three types: (1) the underuse of appropriate health services, (2) the misuse or inappropriate use of health services, and (3) the overuse of health services.³⁶

Analysis

Seven research studies were identified using the search criteria outlined earlier.^{45, 46, 47, 48, 49, 50, 51} Four of the seven were conducted at academic medical centers,^{45, 46, 47, 51} and three of those four were conducted at a single institution, Regenstrief Institute (the fourth was conducted at Beth Israel Hospital in Boston).^{45, 46, 47} Two studies were conducted at a large, integrated healthcare delivery network, Kaiser,^{48, 49} and one was conducted in the Netherlands.⁵⁰ All four studies from the academic medical centers assessed internally developed HIT systems, rather than a commercially available system. One study⁴⁸ assessed two different systems at two sites, one of which was internally developed by the organization and the other a commercially developed product.

All studies included data on structural quality. These varied highly and were largely qualitative in nature. In particular, reporting on the organizational and workflow changes needed to implement an EHR or a new EHR functionality was limited. All seven studies analyzed quality with respect to process of care. Six of the seven^{45, 46, 47, 48, 49, 51} assessed quality with respect to some type of outcome.

In terms of the types of problems the interventions were trying to address, six of the seven^{45, 46, 47, 48, 49, 50} included data on the effects of EHRs on decreasing overused or redundantly used healthcare services. Two^{48, 51} included measures of the effects of EHRs on appropriate but underused care. None used explicit methods to evaluate the impact of EHRs on inappropriate use of care.

EHR Systems in Use at Regenstrief

Structure. Three studies that met our criteria were conducted at the Regenstrief Institute, which includes a research institute and an ambulatory care practice affiliated with both a university medical school and a large public hospital.^{45, 46, 47}

The development of the EHR at the Institute began in the mid-1970s. Subsequent system enhancement and implementation broadened its functionality and scope of use. In 1984, CPOE was added to the EHR capabilities and became uniformly used in the outpatient setting. In the three studies covered in this analysis, the system included electronic documentation, results management, CPOE, and decision support.

All three studies examined the effects of incorporating new information elements into the process used by physicians to order diagnostic tests. Each involved the integration of EHR-stored data into physician decisionmaking at the point of care.

The first paper⁴⁵ reported on the effect of a structural change in care delivery: the addition of diagnostic test cost data to the EHR order function. Thus, physician workflow was altered through the inclusion of cost data in the order entry process, to be shown at the point of care. After physicians ordered tests, the charges for each test and the total charges for all tests were provided automatically in a new window. Physicians were then offered the option to cancel any or all tests.

This study used a complex randomized design to test the effect of the intervention. First, baseline utilization data were collected during a 14-week observation period. Second, physicians were randomized either to receive the cost data during order entry or to use the usual EHR functional interface where no cost data were provided. Finally, data collection continued for a 19-week post-intervention period.

The second paper reported on the effect of an intervention⁴⁷ in which the investigators created statistical models to predict the likelihood of abnormal results for commonly ordered diagnostic tests. These pretest probabilities were displayed to physicians immediately prior to test ordering. Data needed for the models were obtained through EHR mediated prompts to physicians and from patient-specific data already electronically stored. Physician workflow was altered by the need to enter data during the test-ordering process and by the incorporation of the pretest probability into their decisionmaking process. The number of data prompts given to physicians was not reported.

The study used a randomized design in which patients were the unit of randomization. The EHR sorted patients automatically by the predetermined allocation. When physicians cared for intervention patients, the pretest probability function was activated during the electronic ordering process. When physicians ordered tests for the control patients, no additional decision support was provided.

The third paper reported on an intervention⁴⁷ in which care delivery was modified through an intervention in which past diagnostic test results were automatically displayed as physicians ordered new tests. The last three results for a test, the time interval between tests, and the total number of times the test had been ordered for the patient were displayed at the point of care. No additional data entry was required of physicians. Physician workflow was altered by the need to incorporate past test results during decisionmaking.

The study used a complex multiphased, randomized design. First, during a 13-week pre-intervention period, baseline data were collected regarding physician test ordering patterns. During the 16-week intervention period, patients were the unit of randomization. Finally, test ordering was monitored during an 8-week post-intervention period.

Process. Each of these studies examined the impact of the EHR-related structural improvements described previously on physician diagnostic test ordering practices. In each study, the EHR-based intervention decreased the number of diagnostic tests ordered by physicians, suggesting that quality of care was improved through the decrease in overused health services.

The first study,⁴⁵ in which test charge data was displayed, showed an overall 14-percent decrease in the number of tests ordered by physicians per visit in the intervention group. Decreases were observed for both scheduled and for unscheduled visits. The multiphase design allowed additional conclusions to be made regarding the importance of maintaining the decision-support element as part of the structure of care. In the pre-intervention period, no differences in test ordering were noted. During the intervention period, physicians randomized to the decision support tool ordered 17 percent fewer tests. In the post-intervention period, after the decision support was removed from the EHR, physicians who had been in the intervention group ordered only 7 percent fewer tests than during baseline. This effect decrement suggests that the knowledge of costs the test physicians gained during the intervention was not sufficient to alter practice over time. Instead, it was the presence of the additional cost information within the structure of care that most affected performance.

The second study,⁴⁶ in which abnormal test result probabilities were displayed to providers, showed a 9 percent overall decrease in the number of tests ordered by physicians. Two tests, urinalysis and urine culture, which had been underused prior to the intervention, showed increases in ordering frequency (+14 percent and +27). However, for the other six, overused tests examined, decreases ranged from 4 percent (electrocardiogram) to 14 percent (chest x-ray). These findings suggest that providing point-of-care pretest probabilities via an EHR improves the quality of care processes by decreasing overused testing and by increasing the use of previously underused care.

The third study,⁴⁷ in which past abnormal test results were displayed, showed an overall 9decrease in the number of tests ordered by physicians. As in the first study, data were analyzed in the post-intervention period to assess the persistence of the effect. After the EHR intervention was discontinued, the researchers observed a non-statistically significant 11-percent increase in the number of tests ordered (the post-intervention period time frame was not long enough for this trend to reach statistical significance).

Outcomes. In the first study,⁴⁵ the primary outcome was diagnostic test-related charges. Charges were 13 percent ($6.68 in 1988 dollars) lower per visit for the intervention group physicians than for the control group. Decreases in charges were directly due to the decrease in the number of tests ordered, i.e., the improvements in quality of care processes. Given the likelihood that the intervention reduced overused care, this outcome increases the efficiency of care delivery.

In the second study,⁴⁶ which used statistical models to predict whether any of eight commonly ordered diagnostic tests were likely to be abnormal, the primary outcome was financial charges for tests. Overall, charges decreased 9 percent ($1.09 in 1986 dollars). A technical outcome of the study was the operating characteristics of the statistical models used to predict lab test abnormalities, which were based on data collected and stored in the EHR. All predictive models for the study tests performed well, with receiver operating curve areas generally over 0.75 (range 0.66 to 0.92).

In the third study,⁴⁷ efficiency outcomes were also observed due to the decrease in the number of tests ordered. Charges for tests in the intervention group were 13 percent lower per visit (approximately $1.82 per visit in 1986 dollars).

EHR Systems in Use at Kaiser Permanente

Structure. Two studies came from regional medical centers in the Kaiser Permanente network, an integrated, not-for-profit, nonacademic healthcare delivery system.

In the first of these two studies,⁴⁸ comprehensive EHRs were implemented in two regions (Northwest and Colorado) of the enterprise. One EHR was internally developed (Colorado) and the other externally developed by a commercial vendor (Northwest). Although the EHR systems were different, both were reported to have comparable functionality, including the following: documentation, clinical results management, CPOE for both diagnostics and medications, administrative data management, and decision support. Specific decision support functions varied between the two sites in both content and scope.

The second of these two studies provided a brief qualitative description of structural changes associated with and supporting EHR implementation. Implementation was carried out gradually, in phases, beginning in discrete areas of the ambulatory care clinics. The majority of system implementation was completed within one year of initiation. The authors note that because of phased implementation, it was "some time" before changes in health care delivery were noted. (Data related to time course of impact will be discussed in the Outcomes section of this analysis.) No data on the specific organizational drivers for adoption were included.

Structural reorganization and improvements in workflow were described briefly. Prior to the implementation of EHR in each region, the presence of multiple ambulatory sites required paper records to be physically delivered. Paper charts were warehoused and had to be delivered "several miles." For same-day and unscheduled ambulatory visits, availability of paper records was "unreliable." After EHR implementation, use of paper charts was "essentially eliminated" and electronic patient charts became available for emergency room visits, unscheduled visits, and same-day appointments. Charts also became available for telephone contacts, and the resulting improvement in clinical workflow led to more effective utilization of telephone-based care, with physicians reporting that they were better able to address patient health issues over the phone when provided with access to electronic records. The authors cite this outcome as a primary reason for decreased office visits, one of the primary outcomes of the study.

In terms of time frame of impacts, little difference in services was noted during the first year of implementation (the authors reported system implementation was mostly completed one year after implementation began).

Evaluating the effects of EHR adoption was itself a form of structural change in this study. In order to determine appropriate utilization measures to assess the effect of EHR implementation, interviews were conducted with 100 individuals with a broad array of organizational roles. Interviews led the investigators to hypothesize that ambulatory care delivery had become more efficient by making needed information available during the initial episode of care, thus decreasing the need for follow-up visits and redundant services. Interviews also suggested that quality had improved. These hypotheses formed the basis for the selection of metrics and the quantitative evaluation done in the study (discussed in the Process and Outcome section of this analysis). No further details were provided regarding the data acquired from these interviews or the methodology used to conduct them.

The study design was a retrospective time-series analysis with data analyzed at one-year intervals before and after implementation. Baseline data were used from the three years prior to implementation. For the Kaiser Northwest site, four years of post-implementation data were available, whereas for Colorado, only two years of post-implementation data were available because of later implementation.

The second of the two studies, conducted in the Kaiser Northwest system,⁴⁹ examined the incorporation of guidelines through the EHR to support the decisionmaking process for ordering radiology tests and medications. The two-phase implementation process was described briefly. In the first, a read-only results reporting system that integrated data from departmental systems was implemented. In the second phase, the commercially developed EHR described above for the first Kaiser study was implemented. Together, both phases took approximately three years to complete. Per the authors, attempts were made to present guidelines to providers as efficiency aids that would streamline their workflow; the electronic guidelines were kept simple and integrated smoothly into existing procedures. Provider adherence to guidelines was not mandated; however, the electronic ordering system was designed to make adherence simple, and the guidelines were presented in text form without requiring explicit interaction. No further specific implementation-related information was provided in the paper for either the EHR or the guideline tools.

The study design was a time-series analysis that examined utilization patterns at multiple time points before and after guideline implementation.

Process. The first Kaiser study,⁴⁸ which examined the effects of EHR implementation at two sites in the Kaiser network, examined multiple processes of care.

Three quality indicators from the Health Plan Employer and Data Information Set (HEDIS) were chosen to assess quality. These items were chosen in part because their definitions remained consistent over the time period of the analysis. Each was a process-of-care measure: advice on smoking cessation, cervical cancer screening, and retinal eye examination. No statistically significant differences were found in performance on these process measures from the pre- to the post-implementation period.

However, multiple utilization-related processes were examined and showed considerable change after EHR implementation. In general, they suggest improvements in quality of care through a decrease in redundant health services. Age-adjusted rates of radiology test utilization decreased overall by 4 percent after EHR implementation. The authors note that over this same period, radiology service use increased within the Kaiser system as a whole and nationally as well (quantitative data not provided for either increase). Laboratory test utilization in one site decreased 18 percent four years post-implementation. However, utilization rates subsequently increased 5 to 7 percent annually. In the other site, the rate of laboratory test utilization had risen 14 percent prior to EHR implementation, but decreased by 3 percent over the two post-implementation years included in this study. Comparisons of laboratory utilization with other non-EHR sites in the network were not included in the analysis. The number of telephone encounters physicians scheduled with patients increased substantially after EHR implementation, rising from 1.3 telephone encounters per member per year to 2.1 telephone encounters per year. Per the authors, physicians qualitatively reported that telephone encounters were more effective because their capacity to resolve patient issues was enhanced by accessing the EHR.

The second Kaiser study⁴⁹ focused on processes of care-related adherence for two radiology tests and on formulary adherence for one medication, after guidelines were incorporated into the EHR.

Use of upper gastro intestinal (UGI) radiology testing decreased from 11 UGI per thousand members to 6 UGI per thousand members after guideline implementation (40-percent relative decrease). The number of chest x-rays ordered also decreased 20 percent. Prescription of a nonformulary medication for depression decreased from 4.7 percent of all selective serotonin reuptake inhibitors (SSRIs) to 2.4 percent (SSRIs are the most widely used class of medications for depression and multiple agents are available for prescription in this class). Noted effects were sustained over time. The analysis made no attempt to control for other factors that may have affected utilization of radiology testing or formulary adherence.

Outcome. Outcomes in the first Kaiser study related to efficiency and utilization.⁴⁸ The age-adjusted total office visits decreased by 9 percent in year 2 after initial implementation. Primary care visits decreased by 11 percent and specialty care visits decreased by approximately 5 percent, both of which were statistically significant. Reductions in visits held across patient cohorts, including those with the greatest baseline rates of visits. The number of patients making three or more visits decreased by approximately 10 percent between year 1 and year 2 post-implementation in the Northwest region and by 11 percent in Colorado. In year 4, a further decrease of 2 percent was noted in the Northwest region. No comment was made on whether these decreases in the high-volume use category were statistically significant. Direct comparisons with utilization at non-EHR sites were not possible because of inconsistent definitions of office visits. However, in three other network regions (all of which used independent definitions of a visit) for which visit utilization data were available for the same time period, no similar decreases were noted.

In terms of the statistical analysis, no strict control variables were included in the analysis. The following structural measures were reviewed separately to examine possible confounding: rates of ER visits, ratio of primary care providers to members, ratio of referrals to outside providers. Per the authors report, none changed significantly over the study time frame.

Appointments made for patients after doctor-managed telephone encounters decreased by 7 percent after the EHRs were implemented. However, when telephone contacts reverted to nurses, these appointments "rose" (no quantitative data provided).

EHR Systems in Use in the Netherlands

Structure. A single report details a large multisite study in the Netherlands in which the effects of two different types of EHR laboratory test order interfaces were examined.⁵⁰ Both sought to decrease the number of laboratory tests ordered by providers by presenting a limited set of tests on the primary laboratory order screen in the EHR. While all available tests in a laboratory system cannot usually be presented at once on a computer screen, these interventions did not allow screen size or human factor constraints to dictate which test options were initially made available to providers. Instead, they presented considerably smaller sets of choices. Thus, both interfaces changed provider workflow considerably when compared to paper or to nonrestrictive EHR order interfaces. Although providers could order any tests they wanted, any test not explicitly present on the EHR laboratory screen required additional search time to find and call up.

In one experimental condition, statistical probability was used to select the fifteen most commonly ordered tests overall to present to a provider on the initial order interface. In the other condition, the tests presented to providers depended on the patient's specific diagnosis. Diagnosis-specific tests were presented electronically, based on recommendations from existing guidelines. This intervention altered provider workflow to a greater extent than did the first intervention. First, a menu of guidelines/indications was presented, from which the provider had to select those most relevant to the patient's conditions. Based on the indications for testing entered by the provider, the EHR picked the most relevant to present as possible options for ordering. The guideline set was not comprehensive for all possible tests, and all possible indications for a test were not included in the electronic guidelines. Physicians could override recommended tests and order nonrecommended tests at their discretion by entering, "other indication."

All physician practices in the sample were already using EHRs at the time of the proposed laboratory-ordering intervention. However, prior to the intervention, lab tests were ordered through structured paper-based order forms. All practices in the region using EHRs were offered the opportunity to participate in the experiment and add one of the electronic lab ordering functionalities to their systems. Of 64 practices, 46 (72 percent) agreed. Sixty-two general practitioners worked at those 46 sites. A three-month implementation period was included to familiarize the physicians with the software. Over the course of the study, four practitioners withdrew: one solo practitioner withdrew because the software decreased the performance of his computer, another withdrew because of dissatisfaction with the system, and two other physicians in the same practice withdrew for unspecified reasons. Thus, complete data were available for 44 practices, representing 60 physicians.

Physicians were still left with the option of using paper order forms during the study. In the non-guideline specified cohort, 88 percent of all orders were entered through the software. In the guideline-based electronic order cohort, 71 percent of all tests were ordered through the software. Final data analysis included total lab tests ordered both electronically and through paper forms.

Process. This study⁵⁰ focused primarily on process change: examining the effect of changes in information presentation on test ordering. Physicians randomized to the guideline-based interface ordered 1.4 percent fewer lab tests (5.5 vs. 6.9) than did physicians presented with the list of most commonly ordered tests. This difference translated into a relative decrease of 20 percent in tests ordered. The 20 most commonly ordered tests accounted for 80 percent of all tests. No data on human factors issues or usability were reported. Such data may have been informative, given the different workflows created by each intervention. Further supporting the potential utility of such data are the different rates of use for each software package (in the guideline cohort, 71 percent of all tests were ordered through the software and 29 percent through paper; in the other cohort, 88 percent of all tests were ordered through the software and the remaining 12 percent through paper).

Outcomes. No outcomes were reported for this study.⁵⁰

EHR Systems in Use at Beth Israel Deaconess Medical Center (Boston MA)

Structure. The last EHR study was conducted at the ambulatory care medicine practice at the Beth Israel Deaconess Medical Center in Boston, an academic medical center.⁵¹ Development of their clinical computing system began in the 1970s and was internal. System functionality at the time of the study included documentation, results management, order entry, decision support administrative data management, and electronic communication through email. Electronic documentation and results management capabilities were available through the Internet.

The goal of the study, which began in 1990, was to improve quality of outpatient HIV care by incorporating guideline-based alerts and alarms into the system. At the time of the study, no national consensus guideline on HIV care existed. Thus, as a first step, a set of guidelines was developed internally by a panel of local experts. The guidelines were then automated and incorporated into the EHR.

The alerts and alarms created new provider workflows when compared to a paper-based system. Decision support was given to providers on-line and without provider prompting. Clinicians were given the opportunity to act on the alerts and reminders as they appeared, by sending electronic messages for orders to be executed. The system also allowed providers to decline recommendations; and space was included in the EHR to document the reason. The workflow for each of those options differed. Alerts popped up each time a provider logged on, regardless of the patient being seen or reason for accessing the system. Reminders were shown only at the time of the patient visit. This study, which was conducted over 18 months (from 1992 to 1993), used a controlled clinical trial design. Five practice sites were involved. Coin flips were used to assign practices to the intervention or the control condition. All clinicians at a site were assigned to the same condition over the course of the study. The total sample included 22 providers.

Process. The purpose of this study⁵¹ was to assess the effects on processes of care of incorporating electronic guidelines for outpatient HIV care into the EHR. One year after implementation of the EHR guidelines, the number of eligible patients receiving recommended HIV care in the alerts intervention group was 85 percent vs. 64 percent in the control group. At three months post-implementation enhanced utilization of appropriate services was noted for all measures, including ordering CD4 counts (82 percent vs. 60 percent), starting AZT or DDI when appropriate (86 percent vs. 65 percent), modifying AZT dose (76 percent vs. 62 percent), PCP prophylaxis (88 percent vs. 42 percent), and complete blood counts (89 percent vs. 65 percent). All findings were statistically significant to a p value of 0.05 except for starting AZT/DDI or changing the AZT dose. The median response time for a provider to order appropriate services in response to new clinical information was 11 days in the intervention group and 52 days in the control group.

At one year, the number of eligible patients receiving recommended HIV care in the reminders intervention group was significantly greater than in the control group (68 percent versus 46 percent). Processes of care examined included pneumovax receipt (82 percent vs. 38 percent), TB skin testing (78 percent vs. 62 percent), H. influenza vaccination (41 percent vs. 25 percent), tetanus vaccination (31 percent vs. 17 percent), and referrals to ophthalmology (75 percent vs. 46 percent) (p values were less than 0.05 for all results except TB testing, for which p=0.07 and tetanus vaccination, for which p=0.1). At one year, toxoplasmosis titers were drawn on an equivalent percent of patients (82 percent vs. 81 percent). However, the median response time in the intervention group was 8 days vs. 168 days for the control group. No differences in cervical cancer screening were noted. In the intervention group, the median time for a provider to act on clinical information to order appropriate services was 114 days vs. more than 500 days in the control group.

In the intervention patient cohort, 303 alerts and 432 reminders were generated and sent to clinicians. In the control group, 388 alerts and 360 reminders would have been sent to providers.

Outcome. This study⁵¹ examined rates of visits to primary care, rates of hospitalizations, visits to emergency rooms or walk-in clinics. No statistically significant differences were observed. Rates of pneumocystic disease and one-year mortality also showed no differences.

Conclusions

The studies reviewed in this analysis illustrate a range of ways in which ambulatory EHRs can serve to improve quality of care. In particular, they demonstrate how provider performance can be improved when the clinical information management and decision support tools are available within an EHR system. A recurrent theme in these studies was the capacity of EHRs to store data with high fidelity, to make those data readily accessible, and to help translate them into context-specific information that can empower providers in their work.

This analysis is limited by a number of factors. The small number of studies included in the sample was largely a function of the search criteria. In particular, few systems with the core EHR functionalities of documentation, results management, provider order entry, and decision support have been examined, particularly for commercially developed products. These functional criteria were chosen to make the analysis more pertinent to decisionmakers currently considering EHR adoption. Because of the rapid technical advances in EHR, many of the studies of EHR systems are out of date. This review has focused on EHRs with these core functionalities in order to provide decisionmakers with an overview of the evidence that is most likely to be pertinent to the choices they are making now. Another major limitation is the lack of description (and data) pertaining to the workflow reengineering and organizational change that were required for EHR use. As discussed earlier, the "intervention" in these studies is not only the EHR system but also the manner in which these systems change the way healthcare professionals work, organizations function, and consumers receive care. This information is highly context-specific, and for the findings of research on EHRs to be more widely generalizable, this part of the "intervention" needs to be characterized, described, and measured more accurately and comprehensibly. Without such process implementation data, the applicability of findings from one context to another will be a barrier to informed decisionmaking.

Economic Value of an Electronic Health Record Systems and Health Information Technology Applications

Consumers of the healthcare system, including government in the United States, employers, and patients, are demanding higher quality, safety, consistency, efficiency, and value. In order to meet these demands, interoperable computerized health information technology, especially an EHR system that documents patient care processes and outcomes across the continuum of care, is widely believed to be a critical tool. Ideal use of an EHR system enables improved capture and integration of patient information from diverse sources and allows clinicians to access longitudinal patient-specific information for clinical decisionmaking and disease management. Other commonly used terms referring to aspects of an EHR system include personal health record and electronic medical records. In this review, EHR refers to a HIT element that performs the functions of electronic recording, storage, accessing, and viewing of patient medical information.^{52, 53, 54, 55} An EHR system is a computer application with EHR functionality at minimum. Often, financial data are also included in such a system. Since the system is designed to be used institution-wide to replace paper-based medical records and to aid the efficiency of healthcare processes, many EHR applications also contain other system functions, including prescription and test ordering, care management reminders, and other clinical decision-support capabilities. While the EHR is considered essential technology for improving efficiency and quality of health care, implementation of an EHR system requires substantial capital investments and organizational change. Consequently, many health care organizations are seeking evidence and lessons learned about the costs and benefits of EHR adoption in order to better inform decisions about the timing and strategy for implementation.

EHR is the second most common HIT element among the articles identified that contain economic data. Our literature search identified 92 hypothesis-testing or predictive analysis articles containing information on costs, utilization, or efficiency. Of these, 32 studies assessed a HIT system in which EHR was one of the major system elements. However, only nine articles quantitatively assessed the economic value of an EHR system as a whole. We discuss these articles in further detail below. Most of the remaining studies were tests of certain nonfinancial hypotheses or examination of a subset of functionality, such as decision support, instead of the entire EHR system. Although these studies do not assess the costs and benefits of the entire system, they provide indirect, often empirical, evidence that can support the economic appraisal of the value of an EHR system. Before we begin the review of the nine articles, we first summarize the main findings of the remaining studies. Interested readers are referred to our interactive evidence database to learn more about these studies (http://healthit.ahrq.gov/tools/rand).

Summary of Key Findings from Non-Financially Focused Studies

Among its other functions, an EHR system can facilitate automatic generation of patient reminders for preventive services, screening, and disease management. Five Canadian studies used the same EHR system to generate patient reminders and compared the effectiveness and cost-effectiveness of three strategies--physician reminders, telephone reminders, or letter reminders--to remind patients to get preventive services.^{56, 57, 58, 59, 60} All forms of reminders were effective, with reminders delivered directly to patients being somewhat more effective than reminders to physicians. Another study used computerized pharmacy records to generate patient feedback and compared the effectiveness and cost-effectiveness of two depression care programs.⁶¹ Feedback with care management was significantly better than feedback alone.

Electronic charting is a feature of EHR that has been reported to affect provider productivity.^{62, 63, 64, 65} These studies found that the time needed for development of care plans and documentation initially increased, but preparation time decreased subsequently. The initial loss of productivity was associated with the baseline computer skills of the users (clinicians). Two studies assessed computerized documentation systems used for the ICU.^{66, 67} The authors of the first study asserted that addition of their computer-based nursing documentation required no specific ICU software or bedside workstations because it was implemented in a well-networked information technology environment.⁶⁶ Thus, they found that compared with paper charting, their electronic charting system was relatively inexpensive (although the figures were not provided). In addition, the documentation was more complete because of the presence of reminders for missing entries, and data quality remarkably improved. The other study used work sampling and cost analysis methodologies to show net savings of a vendor-developed bedside documentation system specifically designed for an ICU.⁶⁷

Another potential benefit of EHR systems is avoidance of morbidity because of improved patient safety. One study of ADEs found that building ADE detection and reporting capability into EHR can improve detection and potential reduction of ADE in hospital settings because of the ability of the EHR system to easily identify and confirm patients experiencing ADEs and thus enabled early intervention.⁶⁸ Several studies have shown that severe ADEs were associated with longer hospitalizations and higher hospitalization costs (over $2,000 in 2005 dollars).^{68, 69, 70}

Several studies investigated the impact of point-of-care alerts and reminders imbedded within an EHR system during the process of documentation or entry of orders into the system.^{45, 47, 71, 72, 73, 74, 75, 76} These decision-support functions within an EHR system, when accompanied by the required changes in process and communication, altered physicians' ordering behaviors by facilitating appropriate resource utilization and reducing unnecessary charges.^{45, 47, 71, 72, 73, 74, 75, 76} For example, one study estimated that reducing the ordering of redundant clinical laboratory tests could produce an annual savings of $35,000 in laboratory charges.⁷² A randomized controlled trial that tested the effect of immediately printed summaries of a computerized medical record on physician test ordering rates in an emergency room setting⁷⁷ showed significant improvement in the cost-effectiveness of internists' ordering behaviors. The impact on surgeons' ordering patterns was positive but not significant.

Some EHR systems offer sophisticated CPOE and decision-support functionality. A randomized controlled trial found positive effects of an EHR with integrated CPOE on resource utilization, provider productivity, and care efficiency.⁷⁸ Two additional studies showed that an EHR with integrated decision support helped providers improve the quality of documentation, clinical decisionmaking, and guideline compliance, and resulted in reduced utilization of services and costs of care.^{76, 79} However, another study found that implementation of clinical guidelines via an EHR had no significant effect on clinical outcomes or healthcare costs.³⁰ The benefits of information technology seem to depend greatly on the quality of the implementation and the level and type of decision-support technology.

Analytic Methods to Assess the Economic Value of an EHR System

This section provides a more detailed review of the nine articles that quantitatively assessed the economic value of an EHR system, including summaries of the analytic methods used in the studies. Two articles by the same author described the same ambulatory EHR system, although the economic estimates differ slightly between articles.^{53, 54} Therefore, we refer to them as one study described in two articles. Of the remaining seven articles, four report on evaluations of the economics of an EHR system: two in the ambulatory care setting^{52, 80} and two in an integrated delivery network (IDN) (an IDN comprises providers--both inpatient and outpatient--as well as payers and purchasers, in one connected managed care organization).^{55, 81} Another study concerns health care information exchange and interoperability,⁸² and the remaining two are methodological papers.^{83, 84} A brief summary of the methods and findings of these studies is presented in Table 1.

To assess the value of EHR, one methodological paper described a return-on-investment framework to evaluate the costs and benefits of implementing an ambulatory EHR system, and another presented a spreadsheet tool to help family physicians estimate the costs of implementing an EHR system. Both reports provided illustrative examples. Of the remaining six studies, five used cost-benefit analysis^{52, 53, 54, 55, 81, 82} and one used cost-consequence analysis.⁸⁰ All the cost-benefit analyses adopted the ROI framework, which assesses the difference between the costs of an EHR investment and the benefits reaped from it. In these studies, both costs and benefits are quantified in monetary terms to the extent the authors determined was feasible. For example, one study aimed to justify the cost of EHR by first identifying the goals of the system in order to determine benefits.⁵⁵ To quantify benefits, the authors then performed an extensive literature review, surveyed other institutions that were implementing EHR, interviewed EHR vendors, and conducted process-mapping sessions to identify potential cost savings on work processes affected by EHR. However, cost estimation was based largely on vendor response to a request for information or on current information system costs in the healthcare organization.^{52, 80} The cost-consequence study showed costs in monetary terms but did not quantify the benefits of EHR except for time saved from chart pulling.⁸⁰ All studies except two used the perspective of an organization, either outpatient settings^{52, 53, 54, 83, 84} or IDN.^{55, 81} Of the other two studies, one adopted a societal perspective,⁸⁰ and the other used multiple perspectives, from organization level to national level.⁸²

Six studies reported their data sources.^{52, 55, 80, 81, 82, 83} All used multiple data sources, including primary data collected from an existing EHR system,⁵² published data,^{52, 55, 82, 83} workplace and demonstration site observations,^{80, 81} and surveys or interviews of key informants or EHR users. ^{55, 80} Experts were a primary source of data for some studies,^{52, 81, 82} as were vendors.^{55, 81} One study also used process-mapping sessions.⁵⁵

Cost and benefit variables used in each study are listed in Table 1. The cost variables included in most studies included hardware and network acquisition, software licensing, ongoing technical support and maintenance, and training and other implementation costs. Costs associated with temporary productivity loss due to the EHR implementation were captured by two studies,^{52, 80} but a third study assumed no cost associated with loss of productivity, given a long-term EHR implementation strategy.⁸¹ Other cost variables include installation (which was not quantified in greater detail),⁸³ data entry,⁸⁴ printing,^{53, 54} system integration in IDN setting,⁵⁵ personnel,^{53, 54} and institutional and project management.^{55, 80} The health care information exchange and interoperability study also included interface development cost.

The benefit variables included the following: savings from chart pull and transcription;^{52, 80, 81, 82, 83} time saved to document diagnostic codes;^{81, 83} prevention of ADEs;⁵² reduction in drug^{52, 81} laboratory,^{52, 81} or radiology costs;^{52, 81} improvement of charge capture;^{52, 81} decreased billing errors;⁵² personnel and space savings from reduced existing and future medical record storage requirements;^{53, 54} as well as automated generation of clinical forms,^{53, 54} pharmacy information,⁸¹ and billing data generation.^{53, 54} One study provided a comprehensive list of potential benefits grouped into four categories: data capture and access, decision support, business management, and streamlining patient flow.⁵⁵ The health care information exchange and interoperability study reported quantifiable benefits from health information connectivity for providers, payers, and other stakeholders, including radiology centers, laboratories, pharmacies, and public health departments.⁸² Three studies showed the net ROI.^{55, 82}

Various analytic designs were used, although the reference strategy was always the traditional paper-based medical record. One study assessed the economic value of an EHR system concurrently with its implementation and reported the first-year economic consequences of the system.⁸⁰ Another study predicted a fixed annual savings and expenses based on seven years of EHR implementation experiences in a HMO.^{53, 54} Yet another study constructed a hypothetical primary care provider patient panel using average statistics from one of the nation's leading organizations in EHR implementation.⁵² Both studies of EHR systems in IDN settings are analytic predictions because EHR had not been implemented in the studied organizations.^{55, 81} The health care information exchange and interoperability study for a fully standardized nationwide system is also an analytic prediction based on a conceptual framework describing how health care entities share information and a functional taxonomy reflecting the amount of human involvement required, the sophistication of IT, and the level of standardization.⁸² Sensitivity analyses were performed in three studies.^{52, 80, 82}

Evidence of Economic Costs and Benefits of an EHR System

Our interactive evidence database (http://healthit.ahrq.gov/tools/rand) provides a structured abstract for each of the nine identified studies regarding the costs and benefits of EHR. Main findings are highlighted in Table 1 and summarized below.

Costs of Implementing an EHR System. Five studies quantitatively assessed the costs of implementing an EHR system.^{52, 53, 54, 55, 80, 81} The costs can be divided into two categories: (1) cost of the system itself (hardware, software, license, maintenance, and support) and (2) implementation cost (training, temporary loss of productivity, etc.). The costs vary significantly by the scale of the healthcare organization and the functionality of the EHR system.

(1) Cost of an EHR System. One study estimated the system costs for an ambulatory EHR to be $9,700 per provider (in 2002 dollars), which included $1,600 for the annual software license, $1,500 for annual support and maintenance, and $6,600 for hardware (three computers and network, refreshed every three years).⁵² The estimate was for a hypothetical primary care provider office and was modeled after a well-developed and widely used EHR system at a leading IDN health care system. The component parts of the EHR system include online patient charts, electronic prescribing, laboratory order entry, radiology order entry, and electronic charge capture.

In a Swedish primary health care setting with 50 staff, the system cost of a vendor-developed EHR system was estimated to be $240,000 in the first year (in 1995 U.S. dollars).⁸⁰ The EHR system supported full-text patient records and included a controlled medical terminology, a structured patient database, and tools for the analysis and reporting of patient data. The hardware at the sites comprised one server supporting approximately 40 workstations and 20 printers.

In a large HMO with 13 outpatient care locations in Ohio, a homegrown ambulatory EHR was estimated to have had a system development cost of $10 million (in 1996 dollars) and additional annual expenses of $630,000 (in 1996 dollars) for printing, network expenses, memory, and license renewals.^{53, 54} The EHR system was used routinely by 220 physicians and 110 allied health professionals. It implemented an encounter system that collected and presented such medical data elements as diagnoses, allergies, medications prescribed, immunizations, vital signs, and smoking status at the time of an encounter. The system also generated physician reminders for guideline compliance and patient reminders for preventive services and was linked to centralized clinical data on the mainframe, such as laboratory results, radiology reports, emergency department notes, and hospital discharge summaries.

An academic cancer center with a staff of about 8000 and facilities that included a hospital, outpatient clinics, and remote patient-care sites was interested in implementing an EHR for its IDN as both a clinical and financial information management tool in 1994. The cost estimates for vendor-developed EHR systems to meet the center's organizational needs ranged from $15.8 million to $21 million (in 1994 dollars), which included costs of hardware, software, interface development, network cost, data conversion, training, and annual maintenance.⁵⁵ Additionally, the annual support costs were estimated to range from $3.8 million to $5.3 million.

A 2002 study estimated the costs of implementing an EHR⁸¹ for an IDN that included a medical center with a 280-bed acute care hospital, 16 hospital-based and satellite outpatient clinics, a research institute, and a network of about 400 employed physicians. A vendor-developed EHR was estimated to cost approximately $19 million (in 2001 dollars) for the seven-year implementation period. This included costs for the various software products, server hardware, professional services related to installation and training, as well as desktop devices, monitors, biometric security devices, imaging hardware and software, additional technical-support staff, and other associated costs.

(2) Cost of the Implementation Process. Only two studies provided an estimate of costs associated with the EHR implementation process. Both were for ambulatory settings.^{52, 80} One estimated an implementation cost of $3,400 per provider (in 2002 dollars) in the first year associated with workflow process redesign, training, and historical paper chart abstracting.⁵² It also estimated a revenue loss of $11,200 in the first year due to temporary loss of productivity. The total implementation process cost, $14,600 per provider, is 1.5 times the estimated EHR system cost.

The Swedish study used a societal perspective and included costs of training and unexpected costs pertained to self-training during working hours, loss of normal activities in leisure hours, increase in administrative work load, extra service, and medical records summary.⁸⁰ These costs were estimated at $75,000 (in 1995 U.S. dollars), approximately 30 percent of the EHR system cost.

A third study projected the costs of implementing health care information exchange and interoperability where EHR is a requirement for Levels 3 and 4 implementation,⁸² which, according to the authors' taxonomy, refers to the ability to handle machine-organizable data and machine-interpretable data, respectively. The authors projected costs for multiple stakeholders of the healthcare system for Levels 3 and 4 health care information exchange and interoperability. The national ten-year rollout cost of Levels 3 and 4 were estimated to be $320 billion and $276 billion, respectively. Additionally, the national ten-year annual costs were estimated at $20.2 billion and $16.5 billion, respectively.

Quantified Benefits from an EHR System. Benefits of an EHR system or health care information exchange and interoperability were also quantified in the six studies that assessed the cost of implementing such a system.

One study was based on the Partners HealthCare ambulatory EHR, which not only provided health information and data storage capability but also possessed results management, order entry management, point-of-care decision support, and administrative information management functionalities. Therefore, many benefits were expected. The study divided the benefits into three categories: (1) payer-independent benefits, including savings from chart pulls and transcription; (2) benefits under capitated reimbursement, including averted costs from ADEs, drug utilization, laboratory utilization, and radiology utilization; and (3) benefits under fee-for-service reimbursement, including improvement in charge capture and decreased billing errors.⁵² The authors predicted that savings from chart pulls and transcription would be seen immediately after the EHR implementation, and costs associated with ADEs and drug utilization could be averted from second year on, but other potential savings would not be realized until the fourth year. Five-year total benefits of an EHR implementation were estimated to be $129,300 per provider (in 2002 dollars), or a net savings of $86,400 per provider (in 2002 dollars). Sensitivity analyses showed that the estimates were sensitive to the assumption of the proportion of patients whose care was capitated. The net financial value could range from a $2,300 net cost to a $330,900 net benefit per provider.

The Swedish study examined an EHR system with functionality limited only to health information and data storage. Therefore, the expected benefits were limited and included increase in knowledge capital for the primary health care team, easier and quicker communication for general practitioners during telephone consultations, clearer information to patients, and time saved in retrieving paper-based medical records.⁸⁰ Only the value of the last item was estimated, at a total of approximately $10,500 (in U.S. 1995 dollars) for the first year of EHR implementation.

Despite potential savings from ADE prevention and reduced resource utilization under capitated reimbursement, the study of the ambulatory EHR system in a HMO (13 outpatient clinics) did not quantify this aspect of benefits. Instead, it quantified only the averted costs associated with improved efficiency.^{53, 54} The study estimated an annual savings of $3,700,000 (in 1996 dollars) from reduced medical record room and support staff, elimination of clinical forms, and automatic collection of billing data.

The cancer center study projected the benefits of an EHR over ten years.⁵⁵ This projection made several key assumptions, including no benefit until the third year after implementation, benefits to phase-in as the EHR system became functional, physician acceptance and use of the system, a link between business management benefits and managed care, and productivity changes. The authors divided the benefits into capture and access, decision support, optimization of clinical practice, business management, and streamlining of patient flow. The estimated total quantified benefits were $129.69 million over ten years. Adjusted by the total implementation and system costs, the authors' assigned confidence factor, and 9.5-percent discount rate, the net value was predicted to be $24.9 million (in 1994 dollars).

The other IDN expected even greater benefits from an EHR implementation. The authors predicted approximately $68.5 million in gross quantifiable benefits over a seven-year period; subtracting the cost of the EHR system implementation, the net benefit would be $31.4 million (in 2001 dollars), using a 10-percent discount rate.⁸¹ The authors predicted and quantified benefits from savings in laboratory and radiology order entry, pharmacy order entry, documentation availability of information, and charge capture.

Another study estimated that implementation of a standardized interoperable EHR system by all healthcare organizations in the United States would yield substantial financial benefits. The health care information exchange and interoperability study predicted that investment on a fully standardized, Level-4 nationwide system will have the most financial return, a net value of $77.8 billion per year once fully implemented.⁸² Non-standardized health care information exchange and interoperability also can have positive financial returns, but the returns are smaller compared to the Level-4 implementation.

In summary, despite the heterogeneity in the analytic methods used, all five cost-benefit analyses predicted substantial savings from EHR (and health care information exchange and interoperability) implementation.^{52, 53, 54, 55, 81, 82} In other words, the quantifiable benefits are projected to outweigh the investment costs. However, the predicted time needed to break even varied from three^{52, 81} to six⁵⁵ to perhaps as long as 13 years.⁵⁴

Conclusion. Our evidence review found consistent predictions from five cost-benefit studies that implementation of an EHR system can be financially viable at the individual organization level or through a nationwide implementation with high levels of health care information exchange and interoperability.^{52, 53, 54, 55, 81, 82} However, there are several caveats.

1. All studies are predictive analyses that are based on many analytical assumptions and limited empirical data. The strength of evidence is considered weak.
2. In all studies, the EHR system was assumed to have multiple functionalities that include, at minimum, health information and data storage, administrative processes, decision support, and results management, as well as information exchange capabilities. The functional capability of an EHR system is critical to the benefit accrued.
3. The individual organizations that were the subjects of four studies were all large organizations. Large organizations involve many people, units, and subsystems and have complicated processes and interactions. They can benefit greatly from automated, transparent information processing through HIT, and substantial economies of scale. The literature review did not identify cost-benefit studies for EHR implementation in small organizations.
4. The costs of implementing an EHR system may be underestimated. Only one of the five cost-benefit analyses included the cost of the implementation process,⁵² and it found that this cost was 1.5 times the cost of the EHR system. Implementing an EHR system requires extensive changes in the organizational processes, individual behaviors, and the interactions between the two. These resulting costs are often omitted or not reported from studies but can be substantial.
5. The financial benefits depend on the financing system. As shown in the sensitivity analysis of one study,⁵² the benefit estimates are most sensitive to the assumption of the proportion of capitated patients. Realizing all quantifiable benefits of EHR implementation would require changes to the current health care financing system.
6. Both the cost and the benefit of attaining interoperability among EHR systems are directly proportional to the level of data exchange achieved. For example, the cost of achieving machine-organizable (Level 3) or machine-interpretable (Level 4) interoperability is greatest, but it offers the most potential for increased efficiency, improved healthcare utilization, and reduced costs.

In conclusion, there is some empirical evidence to support the positive economic value of an EHR system and the component parts of EHRs. However, realizing the projected benefits will require proper alignment of the healthcare financing system, strong leadership, effective implementation strategies, and focused efforts to successfully adapt the EHR system.

Health Information Technology and Patient Centeredness

Many advocates of HIT believe that one of its primary goals is to increase the extent to which the patient is at the center of his or her health care. For this report, studies of HIT and "patient centeredness" were defined as those that assessed HIT systems that included the element of patient decision support/consumer health informatics, telemedicine, or data-exchange/community health information networks or that reported patient satisfaction as an outcome. From the database of 256 articles there were 34 unique studies or systematic reviews meeting these criteria.

Ten studies assessed computer-generated reminders. Of these, seven assessed the use of reminder programs to improve the delivery of preventive care such as mammography and immunizations.^{27, 85, 86, 87, 88, 89, 90} All studies reported greater use of preventive services by patients--or the physicians of patients--who received computer-generated reminders. Two other studies assessed the effect of computer-supported or -generated reminder systems for refilling medications,^{91, 92} neither of which reported statistically significant improvements in compliance. A third study assessed the effect of a computer-generated reminder chart on patients' compliance with drug regimens, which reported significant improvements in mean compliance score for patients receiving an automatically generated reminder chart.⁹³

Seven original studies evaluated various aspects of telemedicine, and a review article assessed the role of telemedicine in surgery.⁹⁴ In the context of surgery, telemedicine included tele-mentoring, tele-proctoring, tele-conferencing, and tele-presence surgery, all of which are designed to allow physicians to communicate and improve the technical delivery of remote surgical procedures. Thus, this article was not considered relevant to patient centeredness. Five studies assessed telemedicine in particular contexts, including the intensive care unit,⁹⁵ the control of essential hypertension,⁹⁶ the evaluation of patients in an outpatient pulmonary clinic,⁹⁷ the role of telemedicine as one component of HIT and its importance to child safety,⁹⁸ and the use of a remote video system that allowed nurses and patients to interact in real time for patients with a variety of health conditions.⁹⁹ All of these articles reported benefits from the use of telemedicine technologies. The last article in this group was an assessment of ComputerLink, which was conceived as an alternative to traditional caregiver support services such as support groups and health education programs. It was tested in a 12-month randomized trial in family caregivers of people with Alzheimer's disease. Compared with the control group that did not have access to ComputerLink, caregivers in the experimental group reported more improvement in caregiver strain.¹⁰⁰

Three studies assessed the effect on patient trust and satisfaction of some aspects of HIT that are used during a patient consultation. In a pre-post study of general surgical patients, 96 percent of patients stated that their contact with a doctor was as easy and as personal after installation of a computer in the office as before the installation.¹⁰¹ In a controlled trial, the use of an automatic voice recognition system for transcribing progress notes was not associated with any significant negative effect on patient satisfaction, and was associated with some positive effects in terms of preventive maintenance and patient education.¹⁰² We also identified a study that assessed the effects of a nursing module used at the point of care. In this time-series study, an integrated, menu-driven electronic health record was associated with marked increases in nursing documentation and no effect on patient satisfaction.¹⁰³ One study assessed physician satisfaction, and was excluded.¹⁰⁴

Two studies assessed the effect of computer-guided management of patients. The first study compared the effect of computer-controlled administration of analgesic for post-operative pain to that of patient-controlled administration. The computer-controlled infusion used custom-written software designed to rapidly attain and maintain a theoretical target plasma concentration of the analgesic. In a double-blind randomized trial, the study found that computer-controlled analgesia conferred a more rapid onset of pain relief and was as effective as patient-controlled administration in providing post-operative analgesia.¹⁰⁵ Which of these interventions is more "patient-centric" may be debatable, however. In the other study, computer-guided behavioral therapy that allowed patients to progress through a self-paced workbook was compared with clinician-directed behavioral therapy and with relaxation therapy in a randomized trial of patients with obsessive-compulsive disorder. At ten weeks, the Yale-Brown obsessive-compulsive scale showed significantly greater improvement in the patients receiving clinician-guided behavior therapy than in the group receiving computer-guided behavior therapy, and both of these were significantly greater than the improvement attained with relaxation therapy, which was found to be essentially ineffective. This study concluded that computer-guided behavior therapy was effective and might be a helpful first step in treating patients with obsessive-compulsive disorder when clinical-guided behavior therapy was unavailable.¹⁰⁶

Five studies assessed various aspects of consumer health informatics. The interventions included a clinical trial of an interactive computerized patient education system in family practice;¹⁰⁷ and assessments of the effects of computer tailored smoking cessation in family practice,¹⁰⁸ the effectiveness of a computer-generated patient health summary in changing patients' knowledge, attitudes, and behavior concerning health promotion,¹⁰⁹ and the use of self-administered computerized assessments for psychiatric disorders in patients in primary care.¹¹⁰ All of these studies reported benefits of the computerized health informatics system. A review of 37 studies of computer-generated health behavior interventions intended to motivate individuals to adopt various treatment regimens concluded that such systems are effective.¹¹¹

Two review articles assessed the effect of various HIT systems that are directly accessed by the consumer or patient. One of the reviews assessed ten comparative studies of consumers using the Internet to access health information and services. This review included controlled studies, before-and-after studies, and interrupted time-series analyses of Internet users versus nonusers, or of the use of the Internet versus other communication media. The authors concluded that rigorous research regarding the effects of consumer Internet use on health outcomes is lacking. In the ten studies they assessed, all showed some beneficial effects on health outcomes, although the authors note the methodological quality of many studies was poor.¹¹² A second review article assessed comparative studies evaluating the health or social outcomes of virtual peer-to-peer communities, which they characterized as a type of electronic support group. Among 45 publications describing 38 studies (of which 20 were randomized trials), only six evaluated "pure" peer-to-peer communities without other interventions. The other studies assessed complex interventions, of which a peer-to-peer community was only one component. The authors concluded that no good evidence exists on the effects of consumer-led peer-to-peer communities, partly because most such interventions have been evaluated only as part of a complex intervention or interventions involving health professionals.¹¹³

One review article assessed what is known about email consultations in healthcare. The authors noted that a rapidly expanding proportion of the population has access to email and that, while email consultations have the potential to play an important role in the delivery of preventive health care and facilitation of self-management of chronic disorders, there is little evidence from controlled trials that this potential benefit can be translated to routine clinical care.¹¹⁴

Another review article assessed studies of "electronic communication with patients," which was defined to include studies of computerized communication, telephone follow-up and counseling, telephone reminders, interactive telephone systems, telephone access, and telephone screening. This article concluded that distance medicine technology has benefits in the areas of preventive care, management of osteoarthritis, cardiac rehabilitation, and diabetes care and that "distance medicine technology enables greater continuity of care by improving access and supporting the coordination of activities by a clinician."¹¹⁵

Last, two studies dealt with data exchange networks or community health information networks. One study described the experience of developers of an electronic laboratory reporting system.¹¹⁶ In the second study, researchers developed a cost-benefit model and used published evidence and expert opinion to assess the ten-year rollout and annual cost of healthcare information exchange and interoperability, a development that would allow providers to access patient health care information in any clinical setting. The researchers concluded that a fully standardized interoperable system could save $77.8 billion a year, once fully implemented.⁸²

In summary, evidence for an effect of HIT on patient-centeredness in health care is sparse. The best evidence is the beneficial effect of using computerized patient reminders for preventive care. The evidence for benefits of telemedicine and consumer health informatics is also limited to specific contexts. Finally, the evidence is much more limited for effects of more general interactive HIT (such as the internet or email on health) or the effect of implementing HIT systems (such as the electronic health record) on patient trust and satisfaction.

Barriers to HIT Implementation

All studies initially reviewed were screened for data on barriers to adoption and implementation. For this analysis, qualitative studies that were primarily focused on barriers and studies that collected quantitative data on barriers were included. Studies in which barriers were briefly discussed, but were not a primary focus, were excluded. A primary focus on barriers was identified through reviewer consensus.

We identified 20 publications that focused on the barriers to implementing HIT. Of these, 8 reported the actual or potential barriers encountered with specific HIT implementations,^{55, 62, 116, 117, 118, 119, 120, 121, 122, 123, 124} usually as part of an article discussing the implementation. Two articles were short opinion pieces about potential barriers from the physician perspective.^{125, 126} Two studies assessed the physician time for order entry using CPOE compared to paper methods;^{118, 127} both demonstrated that CPOE took more physician time, although the study by Overhage and colleagues found this additional time to be modest. A third study assessed the effect on primary care physicians' time before and after implementation of an EHR system and reported that the time for a patient visit actually fell by half a minute with EHR use.¹²⁸ Last, one study compared physician user satisfaction with two HIT systems: the VA CPRS system and the Mt. Sinai hospital physician order entry system. This study demonstrated CPRS users to be much more satisfied than Mt. Sinai hospital users on many dimensions and also demonstrated that satisfaction was correlated most strongly with the ability of the HIT system to perform tasks in a "straightforward" manner.¹¹⁷ Finally, one article was a systematic review of physician use of electronic retrieval systems such as Medline.¹²⁹

The other five articles focused more broadly on barriers to HIT implementation. One systematic review¹³⁰ summarized barriers mentioned in the medical and pediatric literature that are significant for pediatric practices. These barriers were divided into four categories. Situational barriers included time and financial pressures, unproven return on investment, insufficient access to the internet or to computer technology in the office setting, the prohibitive cost of information technology for small practices, and software not being supportive of pediatric practice needs. Cognitive and or physical barriers include physical disabilities and insufficient computer skills. Liability barriers included confidentiality concerns. Finally, knowledge and attitudinal barriers included insufficient research about information technology in pediatrics, insufficient knowledge about benefits afforded by information technology, apprehension about change, and philosophical opposition to information technology.

Two studies used surveys to identify barriers in the use of electronic medical records¹³¹ and barriers to implementing CPOE systems in U.S. hospitals.¹³² In the first of these studies, the authors conducted 90 interviews with electronic medical record managers and physician champions in 30 physicians' organizations between 2000 and 2002. Key barriers to electronic medical record use were high initial financial costs, slow and uncertain financial payoffs, and high initial costs in terms of physician time. Additional barriers included difficulties with technology, complementary changes in support, electronic data exchange, financial incentives, and physician attitudes. The authors note that these barriers were most acute for physicians in solo/small group practice, which account for a large proportion of U.S. physicians. The second article¹³² reported the results of 52 interviews at 26 hospitals in various stages of implementation of CPOE--from not considering implementation to fully implemented. Most respondents were Chief Information Officers; the remainder consisted of Chief Financial Officers, Chief Medical Officers, and other management officials. Three main barriers to CPOE adoption were identified. The first was physician and organizational resistance due to the perceived negative impact on the physician's workflow. The authors noted that resistance from physicians could escalate to the point of a "physician rebellion," which could derail the entire implementation process. The second barrier identified was the high cost, with estimates from prior studies for the cost of CPOE ranging from $3 million to $10 million, depending on the hospital's size and the level of existing information technology infrastructure. The third major barrier identified was product/vendor immaturity. Survey respondents reported that many current vendor products did not fit the needs of their hospital, and extensive software modifications were required to accommodate established workflow in the hospital.

We also identified two recent prominent editorials about barriers to HIT implementation that summarized the issues succinctly.^{133, 134} The first of these¹³³ identified several challenges for adoption of electronic health records. These included cost, technical issues, system interoperability, concerns about privacy and confidentiality, and a lack of a well trained clinical informatics work force to lead the process. This author identified financing as the biggest impediment, which he attributed to a misalignment of costs and benefits. He noted that while some studies have suggested a substantially positive return on HIT investment for the health care system as a whole, those who are expected to pay for the systems (physicians and other practice organizations) see only about 11 percent of that return on investment. The rest of the savings go to those who typically do not pay directly for the electronic health record. Another major challenge he identified was system and data interoperability, noting that most health care data (whether on paper or electronic) are trapped in "silos." A third concern was privacy and confidentiality: the author stated that physicians, other health care professionals, and healthcare organizations must be vigilant in protecting patient privacy. The last major barrier identified was the need for a workforce capable of leading the implementation of information technology.

The second editorial¹³⁴ stated that, despite predictions of a "bright and near future" for the use of HIT, this future never seems to be realized. The authors attributed the lack of progress in HIT implementation to a lack of attention to the social component, citing the need to view the clinical workplace as a complex system in which technologies, people, and organizational routines dynamically interact, which leads to the following observations:

"(1) Organizations are simultaneously social (e.g. consisting of people, values, norms and culture) and technical (i.e., without tools, equipment, procedures, technology and facilities the people could not work and the organization would not exist). (2) These social and technical elements are deeply inter-dependent and inter-related--hence the term socio-technical systems. Every change in one element affects the other. (3) Accordingly, good design and implementation is not a technical problem but rather one of jointly optimizing the combined socio-technical system."

The authors also note, "...an information technology in and of itself cannot do anything, and when the patterns of its use are not tailored to the workers and their environment to yield high quality care, the technological interventions will not be productive. This implies that any IT acquisitions or implementation trajectory should, first and foremost, be an organization change trajectory."

In summary, studies have identified a large number of barriers to the implementation of HIT. These barriers can be classified as situational barriers (including time and financial concerns), cognitive and or physical barriers (include physical disabilities and insufficient computer skills), liability barriers (including confidentiality concerns), and knowledge and attitudinal barriers. Cutting across all these categories, however, may be the need for clinical medicine as it is now practiced in the majority of settings to undergo a major structural and ideological reorganization, so it can be integrated with and enjoy the benefits of HIT.

CHAPTER 4. DISCUSSION

Limitations

The primary limitation of this review is the quality and quantity of the available studies. As we have noted, understanding the benefits and costs of implementing a HIT system requires an understanding of the intervention in terms of its technical components as well as its human factors, project management, and organizational component; and understanding the organization implementing the HIT system requires understanding many things about it, including its past and current culture of change and the economic situation in which it operates. Most of this information is absent for most of the published studies of HIT. Past limits on word count in published articles may have prevented some authors from including such information in their published reports, but recognition of the information that is needed and the recent practice of allowing supplementary methodological information to be posted online should obviate the problem.

A second limitation is that not all relevant published studies may have been identified. While our search efforts were comprehensive, it is inevitable that we did not find some relevant studies. An advantage of our interactive database of evidence (http://healthit.ahrq.gov/tools/rand) is that it can be updated easily, so we invite readers to send us the citations for relevant articles we may have missed.

We selected only articles that were classified as systematic reviews, hypothesis testing, or predictive analysis for more detailed review and structured abstracts in our interactive database. These articles tend to have less description about how the HIT actually operated and its implementation processes than do qualitative, descriptive articles. Although in general we did not find good evidence of such critical information in the literature during our review processes, we provide citations of qualitative articles in our interactive database for interested readers. However, it should be noted that while these qualitative articles might contain more contextual information about the HIT systems, they are completely lacking in any generalizable knowledge about the benefits of HIT, such as reduction in errors or quality improvement. Any such studies that compared outcomes (such as error rates) with and without a HIT system would have been classified as hypothesis-testing studies and would have been included in our analyses.

A third limitation is that we considered only published studies. As noted by our TEP, substantially more information regarding implementation may have been obtained by contacting leading HIT implementers and conducting structured interviews with them. An additional limitation of reliance on published studies is that less successful, or failed, HIT attempts may be less likely to be published than successful ones (a result of publication bias).

A final limitation is that many of the costs and financial benefits of EHR will change over the years, because they depend on the changing price of such factors as hardware, software, labor costs, and medical prices. Consequently, it is not easy to translate costs as they were reported in the original article into today's costs.

Conclusions

General Conclusions

1. Predictive analyses, based on statistical modeling techniques, suggest that HIT has the potential to enable a dramatic transformation in the delivery of health care, making it safer, more effective, and more efficient. The empirical research evidence base supporting HIT benefits is more limited.
2. Organizations that have realized some of these major gains through the implementation of multi-functional, interoperable HIT systems built around an electronic health record include the VA, Partners, and Regenstrief Institute.
3. The impact of HIT implementation on cost and quality will not be consistent across institutions, independent of context. The specific context within which HIT is implemented, including the setting, the clinical issues, and the patient populations, greatly influences its use and effects.
4. More widespread implementation of HIT is limited by the lack of generalizable knowledge about what types of HIT and methods for its implementation will result in changes in benefits and costs that are specific for specific health organizations, especially for small practices and small hospitals.
5. The reporting of HIT developments and implementations needs to be improved with greater attention to descriptions of both the intervention and the organizational/economic environment in which it is implemented.
6. A high priority must be placed on establishing standards for the information that needs to be measured and reported in HIT implementation studies, similar to the CONSORT standards for clinical trials of therapeutics.
7. Using existing published evidence, it is not possible to draw firm conclusions about which HIT functionalities are most likely to achieve certain health benefits. The assessment of costs is even more uncertain.
8. Existing evidence is not sufficient to clearly define "who pays for" and "who benefits from" HIT implementation in any healthcare organization except those, such as Kaiser and the VA, that are responsible for paying for and delivering all the care for the defined population.
9. Models can be built to estimate the costs and benefits of interoperable HIT systems within and across health care provider settings, payers/purchasers and cumulatively across the health care continuum, but these models are based on many assumptions.
10. Implementation of HIT faces many barriers.

Pediatrics

1. Limited empiric evidence exists to support a benefit for HIT use in pediatrics in the areas of medication safety, clinical decision-support, process improvement, and cost reduction.
2. Only one scientific study weighed these benefits against the costs or cost-effectiveness of implementing HIT systems in pediatric healthcare settings.
3. A majority of HIT systems for use in pediatric practices were tested and/or developed in academic settings, and the ability to generalize these findings to commercially available systems used in nonacademic settings is limited.

EHRs and the Quality of Ambulatory Care

1. A small set of high quality studies shows that implementation of a comprehensive ambulatory EHR improves quality of care. Available evidence focuses primarily on the impact of ambulatory EHRs on decreasing overused health services by enhancing access to data, providing capabilities for real-time analysis of clinical data, and acting as platforms for decision support.
2. Ambulatory EHRs improve the structure of care delivery, improve clinical processes, and enhance outcomes. Most available evidence shows the effects of ambulatory EHRs on processes of care.
3. Interpreting the precise causal effects of ambulatory EHRs on quality is difficult due to lack of systematic and detailed descriptions of system capabilities, limited data (either qualitative or quantitative) on the workflow redesign and organizational changes that accompanied implementation of an ambulatory EHR (or implementation of a new function in an existing EHR package), use of ad hoc measures to assess quality, and use of study designs that do not explicitly take into account sources of bias and confounding. Thus, while existing evidence may have high internal validity, the generalizability of findings is limited.
4. Although substantial potential exists, evidence for the ability of ambulatory care EHRs to improve quality by making healthcare more consumer- and patient-centered is scant.

Economic Value of an HIT and EHR System

1. The main quantifiable benefits of an EHR system were savings from data capture and access; decision support to improve efficiency, quality, and safety of care; business management related to staffing, billing, and overheads; and streamlining patient flow.
2. Few studies quantitatively assessed the costs to implement an EHR system and the financial benefits reaped from it.
3. All the cost-benefit analyses of an EHR system predicted that the financial benefits would significantly outweigh the costs, in a timeframe that varied from three to thirteen years, but this evidence is limited to large organizations and multi-functional EHR systems.
4. The positive economic estimates for EHR system implementation are encouraging but are based on limited evidence at this time. Only limited empirical evidence supports the assumptions made in the predictive analyses. Most studies omitted the costs of implementing an EHR system that were associated with the temporary loss of productivity and the cost of process redesigns. Moreover, realization of the financial return is highly sensitive to the organization's financial incentives.
5. There is some evidence regarding the positive economic value of implementing component parts of an EHR system, with models suggesting that many of the benefits do not accrue unless a broadly functional system is implemented.

Recommendations for Future Research

This section is divided into two parts: (1) research recommendations and (2) recommendations regarding the appropriate types of researchers to carry out the recommended work.

Recommendations

1. High on the list of future research is the need for agreed-upon standards for reporting HIT implementation studies, similar in purpose to the CONSORT standards for the reporting of clinical therapeutics trials.
2. The organizational change and workflow redesign required by and accompanying HIT implementation (or implementation of a new HIT function) need to be described and measured with greater validity, reliability, and precision in order to understand the impact of HIT on care delivery. Without such information, the true "intervention" remains unclear, and the generalizability of results will remain limited. This kind of reporting will require the development and dissemination of publishing standards.
3. While HIT implementation does not easily lend itself to randomized trials, better use of quasi-experimental study designs and other study designs of high internal validity could greatly enhance the clinical relevance of results, reduce bias and confounding, and increase the generalizability of findings. Currently, the published literature is dominated by simple pre-post implementation designs.
4. Creative, alternative research methodology should be considered to estimate costs and benefits of HIT as a supplement to traditional hypothesis-testing studies. Traditional experimental or quasi-experimental approaches may be impractical because they are expensive, time-consuming, and interfere with HIT implementation. Qualitative studies are often subjective, descriptive, and lack generalizability. Simulation modeling is a promising alternative to generate knowledge and evidence; it is different from analytical modeling where the result functionally depends on the input (a number of parameters). Simulation, or dynamic, modeling uses a set of rules that define how the system being modeled will change in the future, given its present state, existing knowledge, and foreseeable uncertainties. For complex problems like HIT implementation, where time dynamics is important and experimenting with the real system is expensive or impossible, simulation modeling can support estimates of cost, benefit, and net value of HIT systems.
5. The costs and benefits of HIT depend not only on the internal system (the practice environment) but also on the interactions with the external system, including consumers (patients and potential users of the healthcare system), medical service suppliers (laboratories, radiology centers, other healthcare organizations), technology suppliers, and the regulatory and financing systems an organization operates. Multi-perspective studies are needed to investigate the flow of costs and benefits in order to maximize the benefits of HIT in the larger healthcare delivery system. Again, simulation modeling may be the best methodology for this type of research.
6. The conceptual foundation for the impact of EHRs on improving care is strong. More research concerning the efficacy and effectiveness of EHRs across health care settings, providers, and patient populations needs to be carried out. Such research will require focusing on how EHR tools are implemented and utilized in day-to-day practice, a broadening of environments to include nonacademic/nonintegrated network practices, the development of methods and instruments directed at evaluation of externally developed systems, and a broader understanding of the human factors issues relevant to healthcare.
7. More research is needed on which specific components of an EHR are beneficial and also on evaluating new specific components--for example, clinical decision-support. Much of the existing decision support relies on simple rules, and it should be possible to provide substantially better assistance with the use of more-complex rules and models.
8. More research is needed to evaluate the effects of EHRs on improving quality by making care more consumer-centered.
9. Process and outcome benefits of HIT that are important and unique to pediatrics must be better quantified, given the unique workflow and information needs of pediatric organizations and practice settings. A growing body of epidemiologic studies has demonstrated the frequency of medication errors in the pediatric healthcare setting. Well-designed studies are needed to demonstrate empirically the benefit of HIT in improving patient safety, not only in the hospital environment, but also in ambulatory and other settings.
10. Well-designed studies measuring the costs of HIT implementation and resultant benefits in pediatrics and other vulnerable populations (e.g., chronically ill, disabled, etc.) are needed, especially in nonacademic settings and with commercially available HIT systems.

Recommendations Regarding Public and Private Types of Organizations to Perform the Proposed Research and/or Analysis

The assessment of HIT implementations of greatest relevance to most U.S. healthcare institutions will occur in nonacademic settings. Most nonacademic settings have limited research expertise or infrastructure to design and support a research project on HIT. If extramural funds are desired for an evaluation of HIT implementation, the ability to secure funding coincident with the project plan is difficult, if not impossible, especially given the funding cycle of grants. Also, to use a pre-and post-implementation design, the researcher needs funding for an extended period of time to collect enough data to adequately power the study before the HIT system is in place. For financial and pragmatic reasons, this pre-implementation data collection cannot delay the HIT implementation process.

Therefore, we would suggest that for HIT research to be feasible in nonacademic settings with commercial systems, some important steps should be taken:

1. Create incentives (e.g., matching funds) for nonacademic medical centers and provider organizations to perform high-quality evaluations of vendor-based HIT implementation. These projects should be funded by organizational dollars and support should be provided for academic investigators to partner with such organizations. These measures would help organizations that lack a built-in research infrastructure to conduct rigorous research.
2. Provide a number of extramural funding mechanisms (government, state, foundation, or even vendor) to evaluate HIT with limited-funding cycles, allowing for adequate pre-implementation measurements and/or rigorous study design. The investigators typically do not determine the timing of implementation, which is often delayed, and funders much be cognizant of this and not penalize the investigators, by disallowing no-cost extensions.
3. Devise a standard means to adequately assess and describe the "socio-technical" milieu of an organization relevant to HIT implementation.

REFERENCES

1. Kohn LT, Corrigan JM, Donaldson M. To Err is Human: Building a Safer Health System. Washington, DC: National Academy Press. 2000.

2. Office of the Coordination of National Health Information, Department of Health and Human Services. The decade of health information technology: delivering consumer-centric and information-rich health care: framework for strategic action. Federal Register 2004;Sec. 411.357 (u):16142.

3. Ash JS, Gorman PN, Seshadri V, et al. Computerized physician order entry in U.S. hospitals: results of a 2002 survey. J Am Med Inform Assoc 2004;11(2):95-9.

4. Miller RH, Hillman JM, Given RS. Physician use of IT: results from the Deloitte Research Survey. J Healthc Inf Manag 2004;18(1):72-80.

5. Terry K. Doctors and EHRs. Med Econ 2005;82(2):72-4, 77-8, 80-4.

6. Dexter PR, Perkins S, Overhage JM, et al. A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med 2001;345(13):965 70.

7. Sisk JE, Moskowitz AJ, Whang W, et al. Cost-effectiveness of vaccination against pneumococcal bacteremia among elderly people. JAMA 1997;278(16):1333-9.

8. Leape LL, Berwick DM, Bates DW. What practices will most improve safety? Evidence-based medicine meets patient safety. JAMA 2002;288(4):501-507.

9. Shojania KG, Duncan BW, McDonald KM, et al. Safe but sound. Patient safety meets evidence-based medicine. JAMA 2002;288(4):508-513.

10. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003;163(12):1409-16.

11. Johnston ME, Langton KB, Haynes RB, et al. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research. Ann Intern Med 1994;120(2):135-42.

12. Corley ST. Electronic prescribing: a review of costs and benefits. Top Health Inf Manage 2003;24(1):29-38.

13. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293(10):1223-38.

14. Shiffman RN, Liaw Y, Brandt CA, et al. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc 1999;6(2):104-14.

15. American Academy of Pediatrics: Task Force on Medical Informatics. Special requirements for electronic medical record systems in pediatrics. Pediatrics 2001;108(2):513-5.

16. Kaushal R, Bates DW, Landrigan C, et al. Medication errors and adverse drug events in pediatric inpatients. JAMA 2001;285(16):2114-20.

17. Folli HL, Poole RL, Benitz WE, et al. Medication error prevention by clinical pharmacists in two children's hospitals. Pediatrics 1987;79(5):718-22.

18. Jonville Jonville AP, Autret E, Bavoux F, et al. Characteristics of medication errors in pediatrics. DICP 1991;25(10):1113-8.

19. Holdsworth MT, Fichtl RE, Behta M, et al. Incidence and impact of adverse drug events in pediatric inpatients. Arch Pediatr Adolesc Med 2003;157(1):60-5.

20. Koren G, Barzilay Z, Greenwald M. Tenfold errors in administration of drug doses: a neglected iatrogenic disease in pediatrics. Pediatrics 1986;77(6):848-9.

21. Diav-Citrin O, Ratnapalan S, Grouhi M, et al. Medication errors in paediatrics: a case report and systematic review of risk factors. Paediatr Drugs 2000;2(3):239-42.

22. Crossing The Quality Chasm : A New Health System for the 21st Century. Washington, D.C.: National Academy Press; 2001.

23. Mullett CJ, Evans RS, Christenson JC, et al. Development and impact of a computerized pediatric antiinfective decision support program. Pediatrics 2001;108(4):E75.

24. Fortescue EB, Kaushal R, Landrigan CP, et al. Prioritizing strategies for preventing medication errors and adverse drug events in pediatric inpatients. Pediatrics 2003;111(4 Pt 1):722-9.

25. Potts AL, Barr FE, Gregory DF, et al. Computerized Physician Order Entry and Medication Errors in a Pediatric Critical Care Unit. Pediatrics 2004;113(1):59-63.

26. Cordero L, Kuehn L, Kumar RR, et al. Impact of computerized physician order entry on clinical practice in a newborn intensive care unit. J Perinatol 2004;24(2 ):88-93.

27. Ornstein SM, Garr DR, Jenkins RG, et al. Implementation and evaluation of a computer-based preventive services system. Fam Med 1995;27(4):260-6.

28. Szilagyi PG, Rodewald LE, Savageau J, et al. Improving influenza vaccination rates in children with asthma: a test of a computerized reminder system and an analysis of factors predicting vaccination compliance. Pediatrics 1992;90(6):871-5.

29. Chiarelli F, Tumini S, Morgese G, et al. Controlled study in diabetic children comparing insulin-dosage adjustment by manual and computer algorithms . Diabetes Care 1990;13(10):1080-4.

30. Schriger DL, Baraff LJ, Buller K, et al. Implementation of clinical guidelines via a computer charting system: effect on the care of febrile children less than three years of age. J Am Med Inform Assoc 2000;7(2):186-95.

31. Christakis DA, Zimmerman FJ, Wright JA, et al. A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children. Pediatrics 2001;107(2):E15.

32. Margolis CZ, Warshawsky SS, Goldman L, et al. Computerized algorithms and pediatricians' management of common problems in a community clinic. Acad Med 1992;67(4):282-4.

33. Carroll AE, Tarczy-Hornoch P, O'Reilly E, et al. The effect of point-of-care personal digital assistant use on resident documentation discrepancies. Pediatrics 2004;113(3 Pt 1):450-4.

34. Quattlebaum TG, Darden PM, Sperry JB. Effectiveness of computer-generated appointment reminders. Pediatrics 1991;88(4):801-5.

35. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med 2003;348(26):2635-45.

36. Chassin MR, Galvin RW. The urgent need to improve health care quality. Institute of Medicine National Roundtable on Health Care Quality. JAMA 1998;280(11):1000-5.

37. Eisele CW, Slee VN, Hoffmann RG . Can the practice of internal medicine be evaluated? Ann Intern Med 1956;44(1):144-61.

38. Brook RH, Appel FA. Quality-of-care assessment: choosing a method for peer review. N Engl J Med 1973;288(25):1323-9.

39. Knaus WA, Draper EA, Wagner DP, et al. An evaluation of outcome from intensive care in major medical centers. Ann Intern Med 1986;104(3):410-8.

40. Dubois RW, Brook RH, Rogers WH. Adjusted hospital death rates: a potential screen for quality of medical care. Am J Public Health 1987;77(9):1162-6.

41. Hussey PS, Anderson GF, Osborn R, et al. How does the quality of care compare in five countries? Health Aff (Millwood) 2004;23(3):89-99.

42. Hippisley-Cox J, O'Hanlon S, Coupland C. Association of deprivation, ethnicity, and sex with quality indicators for diabetes: population based survey of 53,000 patients in primary care. BMJ 2004;329(7477):1267-9.

43. Barros FC, Vaughan JP, Victoria CG. Why so many caesarean sections? The need for a further policy change in Brazil. Health Policy Plan 1986;1(1):19-29.

44. Donabedian A. Explorations in Quality Assessment and Monitoring. Vol I. The Definition of Quality and Approaches to its Assessment. Ann Arbor, MI: Health Administration Press; 1980.

45. Tierney WM, Miller ME, McDonald CJ. The effect on test ordering of informing physicians of the charges for outpatient diagnostic tests. N Engl J Med 1990 ;322(21):1499-504.

46. Tierney WM, McDonald CJ, Hui SL, et al. Computer predictions of abnormal test results. Effects on outpatient testing. JAMA 1988;259(8):1194-8.

47. Tierney WM, McDonald CJ, Martin DK, et al. Computerized display of past test results. Effect on outpatient testing. Ann Intern Med 1987; 107(4):569-74.

48. Kucher N, Koo S, Quiroz R, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med 2005;352(10):969-77.

49. Chin HL, Wallace P. Embedding guidelines into direct physician order entry: simple methods, powerful results. Proc AMIA Symp 1999;221-5.

50. van Wijk MA, van der Lei J, Mosseveld M, et al. Assessment of decision support for blood test ordering in primary care. a randomized trial. Ann Intern Med 2001;134(4):274-81.

51. Safran C, Rind DM, Davis RB, et al. Guidelines for management of HIV infection with computer-based patient's record. Lancet 1995;346(8971):341-6.

52. Wang SJ, Middleton B, Prosser LA, et al. A cost-benefit analysis of electronic medical records in primary care. Am J Med 2003;114:397-403.

53. Khoury AT. Support of quality and business goals by an ambulatory automated medical record system in Kaiser Permanente of Ohio. Eff Clin Pract 1998;1(2):73-82.

54. Khoury A. Finding value in EMRs (electronic medical records). Health Manag Technol 1997;18(8):34, 36.

55. Kian LA, Stewart MW, Bagby C, et al. Justifying the cost of a computer-based patient record. Healthc Financ Manage 1995;49(7):58-60, 62, 64-7.

56. McDowell I, Newell C, Rosser W. A randomized trial of computerized reminders for blood pressure screening in primary care. Med Care 1989;27(3):297-305.

57. McDowell I, Newell C, Rosser W. Computerized reminders to encourage cervical screening in family practice. J Fam Pract 1989;28(4):420-4.

58. Rosser WW, Hutchison BG, McDowell I, et al. Use of reminders to increase compliance with tetanus booster vaccination. CMAJ 1992;146(6):911-7.

59. McDowell I, Newell C, Rosser W. Comparison of three methods of recalling patients for influenza vaccination. CMAJ 1986;135(9):991-7.

60. Rosser WW, McDowell I, Newell C . Use of reminders for preventive procedures in family medicine. CMAJ 1991;145(7):807-14.

61. Simon GE, VonKorff M, Rutter C, et al. Randomised trial of monitoring, feedback, and management of care by telephone to improve treatment of depression in primary care. BMJ 2000;320(7234):550-4.

62. Cameron S, Regalado M, Quitoles M, et al. Harnessing technology: the creation of an electronic care management record in a social health maintenance organization. Manag Care Q 1999;7(1):11-5.

63. Daly JM, Buckwalter K, Maas M. Written and computerized care plans. Organizational processes and effect on patient outcomes. J Gerontol Nurs 2002;28(9):14-23.

64. Krall MA. Acceptance and performance by clinicians using an ambulatory electronic medical record in an HMO. Proc Annu Symp Comput Appl Med Care 1995;708-11.

65. Bulpitt CJ, Beilin LJ, Coles EC, et al. Randomised controlled trial of computer-held medical records in hypertensive patients. Br Med J 1976;1(6011):677-9.

66. Burkle T, Michel A, Horch W, et al. Computer based nursing documentation means to achieve the goal. Int J Med Inf 1998;52(1-3):71-80.

67. Kilgore ML, Flint D, Pearce R. The varying impact of two clinical information systems in a cardiovascular intensive care unit. J Cardiovasc Manag 1998;9(2):31-5.

68. Evans RS, Pestotnik SL, Classen DC, et al. Prevention of adverse drug events through computerized surveillance. Proc Annu Symp Comput Appl Med Care 1992;437-441.

69. Classen DC, Pestotnik SL, Evans RS, et al. Adverse drug events in hospitalized patients. Excess length of stay, extra costs, and attributable mortality. JAMA 1997;277(4):301-306.

70. Evans RS, Classen DC, Stevens LE, et al. Using a hospital information system to assess the effects of adverse drug events. Proc Annu Symp Comput Appl Med Care 1993;161-165.

71. Tierney WM, Murray MD, Gaskins DL, et al. Using computer-based medical records to predict mortality risk for inner-city patients with reactive airways disease. J Am Med Inform Assoc 1997;4(4):313-21.

72. Bates DW, Kuperman GJ, Rittenberg E, et al. A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests. Am J Med 1999;106(2):144-50.

73. Ornstein SM, MacFarlane LL, Jenkins RG, et al. Medication cost information in a computer-based patient record system. Impact on prescribing in a family medicine clinical practice. Arch Fam Med 1999;8(2):118-21.

74. Wells BJ, Lobel KD, Dickerson LM. Using the electronic medical record to enhance the use of combination drugs. Am J Med Qual 2003;18(4):147-9.

75. Mc Donald CJ. Use of a computer to detect and respond to clinical events: its effect on clinician behavior. Ann Intern Med 1976;84(2):162-7.

76. Jones DL, Kroenke K, Landry FJ, et al. Cost savings using a stepped-care prescribing protocol for nonsteroidal anti-inflammatory drugs. JAMA 1996;275(12):926-30.

77. Wilson GA, McDonald CJ, McCabe GP= Jr. The effect of immediate access to a computerized medical record on physician test ordering: a controlled clinical trial in the emergency room. Am J Public Health 1982;72(7):698-702.

78. Tierney WM, Miller ME, Overhage JM, et al. Physician inpatient order writing on microcomputer workstations. Effects on resource utilization. JAMA 1993;269(3 ):379-83.

79. Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med 1998;338(4):232-8.

80. Arias-Vimarlund V, Ljunggren M, Timpka T. Implementation of computer-based patient records in primary care: the societal health economic effects. Proc AMIA Annu Fall Symp 1996;503-7.

81. Schmitt KF, Wofford DA. Financial analysis projects clear returns from electronic medical records. Healthc Financ Manage 2002;56(1):52-7.

82. Walker J, Pan E, Johnston D, et al. The value of health care information exchange and interoperability. Health Affairs 2005;w5.10.

83. Agrawal A. Return on investment analysis for a computer-based patient record in the outpatient clinic setting. J Assoc Acad Minor Phys 2002;13(3):61-5.

84. Valancy J. How much will that EMR system really cost? Fam Pract Manag 2002;9(4):57-8.

85. Chambers CV, Balaban DJ, Carlson BL, et al. Microcomputer-generated reminders. Improving the compliance of primary care physicians with mammography screening guidelines. J Fam Pract 1989;29(3):273-80.

86. Becker DM, Gomez EB, Kaiser DL, et al. Improving preventive care at a medical clinic: how can the patient help? Am J Prev Med 1989;5(6): 353-9.

87. Moran WP, Nelson K, Wofford JL, et al. Computer-generated mailed reminders for influenza immunization: a clinical trial. J Gen Intern Med 1992;7(5):535-7.

88. Frame PS, Zimmer JG, Werth PL, et al. Computer-based vs manual health maintenance tracking. A controlled trial. Arch Fam Med 1994;3(7):581-8.

89. Barton MB, Schoenbaum SC. Improving influenza vaccination performance in an HMO setting: the use of computer-generated reminders and peer comparison feedback. Am J Public Health 1990;80(5):534-6.

90. McDowell I, Newell C, Rosser W. A follow-up study of patients advised to obtain influenza immunizations. Fam Med 1990;22(4):303-6.

91. Baird TK, Broekemeier RL, Anderson MW. Effectiveness of a computer-supported refill reminder system. Am J Hosp Pharm 1984;41(11):2395-7.

92. Simkins CV, Wenzloff NJ. Evaluation of a computerized reminder system in the enhancement of patient medication refill compliance. Drug Intell Clin Pharm 1986;20(10):799-802.

93. Raynor DK, Booth TG, Blenkinsopp A. Effects of computer generated reminder charts on patients' compliance with drug regimens. BMJ 1993;306(6886):1158-61.

94. Eadie LH, Seifalian AM, Davidson BR. Telemedicine in surgery. Br J Surg 2003;90(6):647-58.

95. Rosenfeld BA, Dorman T, Breslow MJ, et al. Intensive care unit telemedicine: alternate paradigm for providing continuous intensivist care. Crit Care Med 2000;28(12):3925-31.

96. Rogers MA, Small D, Buchan DA, et al. Home monitoring service improves mean arterial pressure in patients with essential hypertension. A randomized, controlled trial. Ann Intern Med 2001;134(11):1024-32.

97. Pacht ER, Turner JW, Gailiun M, et al. Effectiveness of telemedicine in the outpatient pulmonary clinic. Telemed J 1998;4(4):287-92.

98. Johnson KB, Davison CL. Information technology: its importance to child safety. Ambul Pediatr 2004;4( 1):64-72.

99. Johnston B, Wheeler L, Deuser J, et al. Outcomes of the Kaiser Permanente Tele-Home Health Research Project. Arch Fam Med 2000;9(1):40-5.

100. Bass DM, McClendon MJ, Brennan PF, et al. The buffering effect of a computer support network on caregiver strain. J Aging Health 1998;10(1):20-43.

101. Rethans JJ, Hoppener P, Wolfs G, et al. Do personal computers make doctors less personal? Br Med J (Clin Res Ed) 1988;296(6634):1446-8.

102. Threet E, Fargues MP. Economic evaluation of voice recognition for the clinicians' desktop at the Naval Hospital Roosevelt Roads. Mil Med 1999;164(2):119-26.

103. Nahm R, Poston I. Measurement of the effects of an integrated, point-of-care computer system on quality of nursing documentation and patient satisfaction. Comput Nurs 2000;18(5):220-9.

104. O'Connell RT, Cho C, Shah N, et al. Take note(s): differential EHR satisfaction with two implementations under one roof. J Am Med Inform Assoc 2004;11(1):43-9.

105. van den Nieuwenhuyzen MC, Engbers FH, Burm AG, et al. Computer-controlled infusion of alfentanil versus patient-controlled administration of morphine for postoperative analgesia: a double-blind randomized trial. Anesth Analg 1995;81(4):671-9.

106. Greist JH, Marks IM, Baer L, et al. Behavior therapy for obsessive-compulsive disorder guided by a computer or by a clinician compared with relaxation as a control. J Clin Psychiatry 2002;63(2):138-45.

107. Krishna S, Balas EA, Spencer DC, et al. Clinical trials of interactive computerized patient education: implications for family practice. J Fam Pract 1997;45(1):25-33.

108. Strecher VJ, Kreuter M, Den Boer DJ, et al. The effects of computer-tailored smoking cessation messages in family practice settings. J Fam Pract 1994;39(3):262-70.

109. Liaw T, Lawrence M, Rendell J. The effect of a computer-generated patient-held medical record summary and/or a written personal health record on patients' attitudes, knowledge and behaviour concerning health promotion. Fam Pract 1996;13(3):289-93.

110. Lewis G, Sharp D, Bartholomew J, et al. Computerized assessment of common mental disorders in primary care: effect on clinical outcome. Fam Pract 1996;13(2):120-6.

111. Revere D, Dunbar PJ. Review of computer-generated outpatient health behavior interventions: clinical encounters "in absentia". J Am Med Inform Assoc 2001;8(1):62-79.

112. Bessell TL, McDonald S, Silagy CA, et al. Do Internet interventions for consumers cause more harm than good? A systematic review. Health Expect 2002;5(1):28-37.

113. Eysenbach G, Powell J, Englesakis M, et al. Health related virtual communities and electronic support groups: systematic review of the effects of online peer to peer interactions. BMJ 2004;328(7449):1166.

114. Car J, Sheikh A. Email consultations in health care: 1--scope and effectiveness. BMJ 2004;329 (7463):435-8.

115. Balas EA, Jaffrey F, Kuperman GJ, et al. Electronic communication with patients. Evaluation of distance medicine technology. JAMA 1997;278(2):152-9.

116. Overhage JM, Suico J, McDonald CJ. Electronic laboratory reporting: barriers, solutions and findings. J Public Health Manag Pract 2001;7(6): 60-6.

117. Murff HJ, Kannry J. Physician satisfaction with two order entry systems. J Am Med Inform Assoc 2001;8(5):499-509.

118. Overhage JM, Perkins S, Tierney WM, et al. Controlled trial of direct physician order entry: effects on physicians' time utilization in ambulatory primary care internal medicine practices. J Am Med Inform Assoc 2001; 8(4):361-71.

119. Wong DH, Gallegos Y, Weinger MB, et al. Changes in intensive care unit nurse task activity after installation of a third-generation intensive care unit information system. Crit Care Med 2003;31 (10):2488-94.

120. Abookire SA, Teich JM, Sandige H, et al. Improving allergy alerting in a computerized physician order entry system. Proc AMIA Symp 2000;2-6.

121. Ammenwerth E, Mansmann U, Iller C, et al. Factors affecting and affected by user acceptance of computer-based nursing documentation: results of a two-year study. J Am Med Inform Assoc 2003;10 (1):69-84.

122. Gamm LD, Barsukiewicz CK, Dansky KH, et al. Investigating changes in end-user satisfaction with installation of an electronic medical record in ambulatory care settings. J Healthc Inf Manag 1998;12 (4):53-65.

123. Burkle T, Kuch R, Prokosch HU, et al. Stepwise evaluation of information systems in an university hospital. Methods Inf Med 1999;38(1):9-15.

124. Elbourne D, Richardson M, Chalmers I, et al. The Newbury Maternity Care Study: a randomized controlled trial to assess a policy of women holding their own obstetric records. Br J Obstet Gynaecol 1987;94(7):612-9.

125. Lazarus SS. Physicians' use of electronic medical records--identifying and crossing the barriers. Med Group Manage J 1999;46(3):12-4.

126. Berkowitz LL. Breaking down the barriers. Improving physician buy-in of CPR systems. Healthc Inform 1997;14(10):73-6.

127. Bates DW, Boyle DL, Teich JM. Impact of computerized physician order entry on physician time. Proc Annu Symp Comput Appl Med Care 1994;996.

128. Pizziferri L, Kittler AF, Volk LA, et al. Primary care physician time utilization before and after implementation of an electronic health record: a time-motion study. J Biomed Inform 2005;38(3):176-88.

129. Hersh WR, Hickam DH. How well do physicians use electronic information retrieval systems? A framework for investigation and systematic review. JAMA 1998;280(15):1347-52.

130. Johnson KB. Barriers that impede the adoption of pediatric information technology. Arch Pediatr Adolesc Med 2001;155(12):1374-9.

131. Miller RH, Sim I. Physicians' use of electronic medical records: barriers and solutions. Health Aff (Millwood) 2004;23(2):116-26.

132. Poon EG, Blumenthal D, Jaggi T, et al. Overcoming barriers to adopting and implementing computerized physician order entry systems in U.S. hospitals. Health Aff (Millwood) 2004;23(4):184-90.

133. Hersh W. Health care information technology: progress and barriers. JAMA 2004;292(18):2273-4.

134. Wears RL, Berg M. Computer technology and clinical work: still waiting for Godot. JAMA 2005;293(10):1261-3.

APPENDIX A. TECHNICAL EXPERT PANEL

APPENDIX B. INFORMATION TECHNOLOGY IN HEALTH SEARCH STRATEGIES

SEARCH METHODOLOGY -- INFORMATION TECHNOLOGY + EFFECTS/OUTCOMES

DATABASE SEARCHED: PUBMED YEARS OF COVERAGE: 1995-2003 OTHER LIMITERS:    English

SEARCH STRATEGY: automatic data processing[majr] OR medical informatics[majr] OR medical informatics applications[majr] OR public health informatics[majr] OR electronics, medical[majr] OR information technolog* OR information infrastructure* OR ehealth OR e-health

AND

adverse effects[sh] OR outcome and process assessment health care[mh] OR costs and cost analysis OR efficiency, organizational OR risk assessment OR outcome*[ti] OR outcome*[ab] OR cost[ti] OR cost[ab] OR costs[ti] OR costs[ab] OR efficien*[ti] OR efficien*[ab] OR risk*[ti] OR risk*[ab] OR adverse[ti] OR adverse[ab]

AND

systematic[sb] -- NOTE: This is the pre-formulated search developed by Medline to include systematic reviews, meta analyses, etc.

NUMBER OF ITEMS RETRIEVED: 516

================================================================

SEARCH METHODOLOGY -- INFORMATION TECHNOLOGY + IMPROVEMENT

DATABASE SEARCHED: PUBMED YEARS OF COVERAGE: 2002-2004 OTHER LIMITERS:    English

SEARCH STRATEGY: automatic data processing[majr] OR medical informatics[majr] OR medical informatics applications[majr] OR public health informatics[majr] OR electronics, medical[majr] OR information technolog* OR information infrastructure* OR ehealth OR e-health

AND

outcome assessment health care OR process assessment health care OR workplace OR workflow* OR work flow* OR quality indicators, health care

AND

improve*[tiab] OR chang*[tiab]

NUMBER OF ITEMS RETRIEVED: 244

==================================================================

SEARCH METHODOLOGY -- PRELIMINARY PUBMED SEARCH ON INFORMATION TECHNOLOGY BARRIERS

Database Searched and Time Period Covered: PubMed -- 2000-2003

Other Limiters: English only

Search Strategy: automatic data processing[majr] OR medical informatics[majr] OR medical informatics applications[majr] OR public health informatics[majr] OR electronics, medical[majr] OR attitude to computers OR information technolog*

AND

barrier* OR challeng* OR difficult* OR resist OR resisting OR resistance

NUMBER OF ITEMS RETRIEVED: 1662

===================================================================

INFORMATION TECHNOLOGY & HEALTH -- POPULAR LITERATURE

Database: Periodical Abstracts

Query: ((de: information and de: technology) or (de: information and de: systems) or de: computer+ or de: digital or de: electronic or de: internet) and (de: health or de: healthcare or de: medical or de: medicine or de: physician+ or de: doctor+ or de: hospital+) and yr: 2002-2003

NUMBER OF ITEMS RETRIEVED: 433

===================================================================

SEARCH METHODOLOGYHealth Affairs Journal

SEARCH #1 (PERFORMED 12/23/04) DATABASE SEARCHED AND TIME PERIOD COVERED:PUBMED 1966-2004

SEARCH STRATEGY: information technology

AND information systems

NUMBER OF ITEMS RETRIEVED: 98

===================================================================

SEARCH METHODOLOGY COMPUTERIZED MEDICAL RECORDS -- COST BENEFITS

SEARCH #1 (PERFORMED 1/6/04) DATABASE SEARCHED AND TIME PERIOD COVERED:PUBMED 1995-2004

SEARCH STRATEGY:

medical records systems, computerized[majr]

AND

costs and cost analysis OR economics OR financ* OR economics(subheading)

AND

Review

NUMBER OF ITEMS RETRIEVED: 26

===================================================================

SEARCH #2 (PERFORMED 1/6/04) DATABASE SEARCHED AND TIME PERIOD COVERED:PUBMED 1995-2004

SEARCH STRATEGY:

medical records systems, computerized[majr]

AND

costs and cost analysis OR economics OR financ* OR economics(subheading)

NOT

Review

NUMBER OF ITEMS RETRIEVED: 871 NUMBER OF ITEMS SENT TO RESEARCHER (after deleting irrelevant items): 186 (COVERING 2000-2003)

APPENDIX C. HEALTH INFORMATION TECHNOLOGY

This appendix is current available only as a PDF document at http://aspe.hhs.gov/daltcp/reports/2006/HITcb-appendC.pdf.

APPENDIX D. LIST OF TEP MEMBERS WHO PROVIDED COMMENTS

APPENDIX E. REVIEW COMMENTS

APPENDIX F. ARTICLES INCLUDED IN HIT INTERACTIVE DATABASE

1. Abookire SA, Teich JM, Sandige H, et al. Improving allergy alerting in a computerized physician order entry system. Proc AMIA Symp 2000;2-6.

   [Rec#: 54]

2. Agrawal A. Return on investment analysis for a computer-based patient record in the outpatient clinic setting. J Assoc Acad Minor Phys 2002;13(3):61-5.

   [Rec#: 431]

3. Al-Aynati MM, Chorneyko KA. Comparison of voice-automated transcription and human transcription in generating pathology reports. Arch Pathol Lab Med 2003;127(6):721-5.

   [Rec#: 432]

4. Alvis JM, Reves JG, Govier AV, et al. Computer-assisted continuous infusions of fentanyl during cardiac anesthesia: comparison with a manual method. Anesthesiology 1985;63(1):41-9.

   [Rec#: 636]

5. Ammenwerth E, Eichstadter R, Haux R, et al. A randomized evaluation of a computer-based nursing documentation system. Methods Inf Med 2001;40(2):61-8.

   [Rec#: 637]

6. Ammenwerth E, Mansmann U, Iller C, et al. Factors affecting and affected by user acceptance of computer-based nursing documentation: results of a two-year study. J Am Med Inform Assoc 2003; 10 (1):69-84.

   [Rec#: 181]

7. Anderson JG, Jay SJ, Anderson M, et al. Evaluating the capability of information technology to prevent adverse drug events: A computer simulation approach. J Am Med Inform Assoc 2002;9:479-490.

   [Rec#: 172]

8. Arias-Vimarlund V, Ljunggren M, Timpka T. Implementation of computer-based patient records in primary care: the societal health economic effects. Proc AMIA Annu Fall Symp 1996;503-7.

   [Rec#: 182]

9. Baird TK, Broekemeier RL, Anderson MW. Effectiveness of a computer-supported refill reminder system. Am J Hosp Pharm 1984;41(11):2395-7.

   [Rec#: 740]

10. Balas EA, Austin SM, Mitchell JA, et al. The clinical value of computerized information services. A review of 98 randomized clinical trials. Arch Fam Med 1996;5(5):271-8.

    [Rec#: 56]

11. Balas EA, Boren SA, Griffing G . Computerized management of diabetes: a synthesis of controlled trials. Proc AMIA Symp 1998;295-9. 1985;63(1):41-9.

    [Rec#: 185]

12. Balas EA, Jaffrey F, Kuperman GJ, et al. Electronic communication with patients. Evaluation of distance medicine technology. JAMA 1997;278(2):152-9.

    [Rec#: 57]

13. Balas EA, Weingarten S, Garb CT, et al. Improving preventive care by prompting physicians. Arch Intern Med 2000;160(3):301-8.

    [Rec#: 58]

14. Baldwin DR, Beech CA, Evans AH, et al. Principals of design and evaluation of an information system for a department of respiratory medicine. Health Care Anal 1997;5(1):78-84

    [Rec#: 187]

15. Ball A, Goodman M, Evans L, et al. Computer generated discharge summaries and their use as a case mix sensitive audit engine. A tale of two cities, J R Coll Physicians Lond 1999;33(3):260-3.

    [Rec#: 188]

16. Barton MB, Schoenbaum SC. Improving influenza vaccination performance in an HMO setting: the use of computer-generated reminders and peer comparison feedback. Am J Public Health 1990;80(5):534-6.

    [Rec#: 875]

17. Bass DM, McClendon MJ, Brennan PF, et al. The buffering effect of a computer support network on caregiver strain. J Aging Health 1998;10(1):20-43.

    [Rec#: 59]

18. Bates DW, Boyle DL, Teich JM. Impact of computerized physician order entry on physician time. Proc Annu Symp Comput Appl Med Care 1994;996.

    [Rec#: 2]

19. Bates DW, Kuperman GJ, Rittenberg E, et al. A randomized trail of a computer-based intervention to reduce utilization of redundant laboratory tests. Am J Med 1999;106(2):144-50.

    [Rec#: 63]

20. Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998;280(15):1311-1316.

    [Rec#: 4]

21. Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999;6(4):313-21.

    [Rec#: 64]

22. Becker DM, Gomez EB, Kaiser DL, et al. Improving preventive care at a medical clinic: how can the patient help? Am J Prev Med 1989;5(6): 353-9.

    [Rec#: 150]

23. Becker SH, Arenson RL. Costs and benefits of picture archiving and communication systems. J Am Med Inform Assoc 1994;1(5):361-71.

    [Rec#: 7]

24. Bennett JW, Glasziou PP. Computerised reminders and feedback in medication management: a systematic review of randomised controlled trials. Med J Aust 2003;178(5):217-22.

    [Rec#: 192]

25. Bessell TL, McDonald S, Silagy CA, et al. Do Internet interventions for consumers cause more harm than good? A systematic review. Health Expect 2002;5(1):28-37.

    [Rec#: 193]

26. Boldreghini S, Larrabee JH. Difference in nursing documentation before and after computerization: A pilot study. J Nurs Inform [serial online] 1998;2(1).

    [Rec#: 988]

27. Bouhaddou O, Lambert JG, Morgan GE. Iliad and Medical HouseCall: evaluating the impact of common sense knowledge on the diagnostic accuracy of a medical expert system. Proc Annu Symp Comput Appl Med Care 1995;742-6.

    [Rec#: 725]

28. Brettle A. Informaiton skills training: a systematic review of the literature. Health Info Libr J 2003;20 Suppl 1:3-9.

    [Rec#: 197]

29. Brownbridge G, Evans A, Fitter M, et al. An interactive computerized protocol for the management of hypertension: effects on the general practitioner's clinical behaviour. J R Coll Gen Pract 1986;36(286):198-202.

    [Rec#: 638]

30. Brownbridge G, Evans A, Wall T . Effect of computer use in the consultation on the delivery of care. Br Med J (Clin Res Red) 1985;291(6496):639-42.

    [Rec#: 639]

31. Bullpitt CJ, Beilin LJ, Coles EC, et al. Randomised controlled trail of computer-held medical records in hypertensive patients. Br Med J 1976;1(6011):677-0.

    [Rec#: 745]

32. Burack RC, Gimotty PA. Promoting screening mammography in inner-city settings. The sustained effectiveness of computerized reminders in a randomized controlled trial. Med Care. 1997;35(9):921-31.

    [Rec#: 707]

33. Burack RC, Gimotty PA, George J, et al. Promoting screening mammography in inner-city settings: a randomized controlled trial of computerized reminders as a component of a program to facilitate mammography. Med Care 1994;32(6):609-24.

    [Rec#: 159]

34. Burkle T, Kuch R, Prokosch HU, et al. Stepwise evaluation of information systems in an university hospital. Methods Inf Med 1999;38(1):9-15.

    [Rec#: 641]

35. Burkle T, Michel A, Horch W, et al. Computer based nursing documentation means to achieve the goal. Int J Med Inf 1998;52(1-3):71-80.

    [Rec#: 642]

36. Burton ME, Ash CL, Hill DP= Jr, et al. A controlled trial of the cost benefit of computerized bayesian aminoglycoside administration. Clin Pharmacol Ther 1991;49(6):685-94.

    [Rec#: 643]

37. Bushko RG, Havlicek PL, Deppert E, et al. Impact of voice- and knowledge-enabled clinical reporting -- US example. Stud Health Technol Inform 2002;80:265-74.

    [Rec#: 460]

38. Cameron S, Regalado M, Quitoles M, et al. Harnessing technology: the creation of an electronic care management record in a social health maintenance organization. Manag Care Q 1999;7(1):11-5.

    [Rec#: 461]

39. Campbell SM, Roland MO, Gormanly B. Evaluation of a computerized appointment system in general practice. Br J Gen Pract 1996;46(409):477-8.

    [Rec#: 880]

40. Cannon DS, Allen SN. A comparison of the effects of computer and manual reminders on compliance with a mental health clinical practice guideline. J Am Med Inform Assoc 2000;7(2):196-203.

    [Rec#: 167]

41. Car J, Sheikh A. Email consultations in health care: 1 -- scope and effectiveness. BMJ 2004;329( 7463):435-8.

    [Rec#: 950]

42. Carmenates J, Keith MR. Impact of automation on pharmacist interventions and medication errors in a correctional health care system. Am J Health Syst Pharm 2001;58(9):779-83.

    [Rec#: 201]

43. Carroll AE, Tarczy-Hornoch P, O'Reilly E, et al. The effect of point-of-care personal digital assistant use on resident documentation discrepancies. Pediatrics 2004;113(3 Pt 1):450-4.

    [Rec#: 919]

44. Carroll AE, Tarczy-Hornoch P, O'Reilly E, et al. The effect of point-of-care personal digital assistant use on resident documentation discrepancies. Pediatrics 2004;113(3 Pt 1):450-4.

    [Rec#: 989]

45. Casner PR, Reilly R, Ho H. A randomized controlled trial of computerized pharmacokinetic theophylline dosing versus empiric physician dosing. Clin Pharmacol Ther 1993;53(6):684-90.

    [Rec#: 644]

46. Chambers CV, Balaban DJ, Carlson BL, et al. The effect of microcomputer-generated reminders on influenza vaccination rates in a university-based family practice center. J Am Board Fam Pract 1991;4 (1):19-26.

    [Rec#: 882]

47. Chambers CV, Balaban DJ, Carlson BL, et al. Microcomputer-generated reminders. Improving the compliance of primary care physicians with mammography screening guidelines. J Fam Pract 1989;29(3):273-80.

    [Rec#: 149]

48. Chatellier G, Colombet I, Degoulet P. Computer-adjusted dosage of anticoagulant therapy improves the quality of anticoagulation. Medinfo 1998;9 Pt 2:819-23.

    [Rec#: 203]

49. Chatellier G, Colombet I, Degoulet P. An overview of the effect of computer-assisted management of anticoagulant therapy on the quality of anticoagulation. Int J Med Inf 1998;49(3):311-20.

    [Rec#: 204]

50. Chen P, Tanasijevic MJ, Schoenenberger RA, et al. A computer-based intervention for improving the appropriateness of antiepileptic drug level monitoring . Am J Clin Pathol 2003;119( 3):432-8.

    [Rec#: 205]

51. Chertow GM, Lee J, Kuperman GJ, et al. Guided medication dosing for inpatients with renal insufficiency. JAMA 2001;286(22):2839-44.

    [Rec#: 162]

52. Chiarelli F, Tumini S, Morgese G, et al. Controlled study in diabetic children comparing insulin-dosage adjustment by manual and computer algorithms . Diabetes Care 1990;13(10):1080-4.

    [Rec#: 645]

53. Chin HL, Wallace P. Embedding guidelines into direct physician order entry: simple methods, powerful results. Proc AMIA Symp 1999;221-5.

    [Rec#: 465]

54. Chin JM, Muller RJ, Lucarelli CD. A pharmacy intervention program: recognizing pharmacy's contribution to improving patient care. Hosp Pharm 1995;30(2):120, 123-6, 129-30.

    [Rec#: 206]

55. Christakis DA, Zimmerman FJ, Wright JA, et al. A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children. Pediatrics 2001;107(2):E15.

    [Rec#: 732]

56. Christensen-Szalanski JJ, Diehr PH, Wood RW, et al. Phased trial of a proven algorithm at a new primary care clinic. Am J Public Health 1982;72(1):16-21.

    [Rec#: 883]

57. Classen DC, Pestotnik SL, Evans RS, et al. Adverse drug events in hospitalized patients. Excess length of stay, extra costs, and attributable mortality. JAMA 1997;277(4):301-306.

    [Rec#: 11]

58. Coe FL, Norton E, Oparil S, et al. Treatment of hypertension by computer and physician-a prospective controlled study. J Chronic Dis 1977;30(2):81-92.

    [Rec#: 746]

59. Cordero L, Kuehn L, Kumar RR, et al. Impact of computerized physician order entry on clinical practice in a newborn intensive care unit. J Perinatol 2004;24(2 ):88-93.

    [Rec#: 920]

60. Daly JM, Buckwalter K, Maas M. Written and computerized care plans. Organizational processes and effect on patient outcomes. J Gerontol Nurs 2002;28(9):14-23.

    [Rec#: 646]

61. Davidson KW, Kahn A, Price RD. Reduction of adverse drug reactions by computerized drug interaction screening. J Fam Pract 1987;25(4):371-5.

    [Rec#: 885]

62. Day F, Hoang LP, Ouk S, et al. The impact of a guideline-driven computer charting system on the emergency care of patients with acute low back pain. Proc Annu Symp Comput Appl Med Care 1995;576-80.

    [Rec#: 724]

63. Demakis JG, Beauchamp C, Cull WL, et al. Improving residents' compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders. JAMA 2000;284(11):1411-6.

    [Rec#: 729]

64. Dexter PR, Perkins SM, Maharry KS, et al. Inpatient Computer-based Standing Orders vs physician reminders to increase influenza and pneumococcal vaccination rates. JAMA 2004;292(19):2366-2371.

    [Rec#: 948]

65. Dexter PR, Wolinsky FD, Gramelspacher GP, et al. Effectiveness of computer-generated reminders for increasing discussions about advance directives and completion of advance directive forms. A randomized, controlled trial. Ann Intern Med 1998;128(2):102-10.

    [Rec#: 649]

66. Dickinson JC, Warshaw GA, Gehlbach SH, et al. Improving hypertension control: impact of computer feedback and physician education. Med Care 1981;19(8):843-54.

    [Rec#: 743]

67. Donald JB. Prescribing costs when computers are used to issue all prescriptions. BMJ 1989;299( 6690):28-30.

    [Rec#: 651]

68. Durieux P, Nizard R, Ravaud P, et al. A clinical decision support system for prevention of venous thromboembolism: effect on physician behavior. JAMA 2000;283(21):2816-21.

    [Rec#: 71]

69. Eadie LH, Seifalian AM, Davidson BR. Telemedicine in surgery. Br J Surg 2003;90(6):647-58.

    [Rec#: 224]

70. Eccles M, McColl E, Steen N, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 2002;325(7370):941.

    [Rec#: 72]

71. Ehler D, Vacek JL, Bansal S, et al. Transition to an all-digital echocardiography laboratory: a large, multi-site private cardiology practice experience. J Am Soc Echocardiogr 2000;13(12):1109-16.

    [Rec#: 481]

72. Evans RS, Classen DC, Pestotnik SL, et al. Improving empiric antibiotic selection using computer decision support. Arch Intern Med 1994; 154(8):878-84.

    [Rec#: 866]

73. Evans RS, Classen DC, Stevens LE, et al. Using a hospital information system to assess the effects of adverse drug events. Proc Annu Symp Comput Appl Med Care 1993;161-165.

    [Rec#: 17]

74. Evans RS, Larsen RA, Burke JP, et al. Computer surveillance of hospital-acquired infections and antibiotic use. JAMA 1986;256(8):1007-11.

    [Rec#: 74]

75. Evans RS, Pestotnik SL, Classen DC, et al. Prevention of adverse drug events through computerized surveillance. Proc Annu Symp Comput Appl Med Care 1992;437-441.

    [Rec#: 18]

76. Evans RS, Pestotnik SL, Classen DC, et al. Evaluation of a computer-assisted antibiotic-dose monitor. Ann Pharmacother 1999;33(10):1026-31.

    [Rec#: 870]

77. Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med 1998;338(4):232-8.

    [Rec#: 75]

78. Eysenbach G, Powell J, Englesakis M, et al. Health related virtual communities and electronic support groups: systematic review of the effects of online peer to peer interactions. BMJ 2004;328(7449):1166.

    [Rec#: 923]

79. Fihn SD, McDonell MB, Vermes D, et al. A computerized intervention to improve timing of outpatient follow-up: a multicenter randomized trial in patients treated with warfarin. National Consortium of Anticoagulation Clinics. J Gen Intern Med 1994;9(3):131-9.

    [Rec#: 654]

80. Fitzmaurice DA, Hobbs FD, Delaney BC, et al. Review of computerized decision support systems for oral anticoagulation management. Br J Haematol 1998;102(4):907-9.

    [Rec#: 234]

81. Fitzmaurice DA, Hobbs FD, Murray ET, et al. Evaluation of computerized decision support for oral anticoagulation management based in primary care. Br J Gen Pract 1996;46(410):533-5.

    [Rec#: 751]

82. Fortescue EB , Kaushal R, Landrigan CP, et al. Prioritizing strategies for preventing medication errors and adverse drug events in pediatric inpatients. Pediatrics 2003;111(4 Pt 1): 722-9.

    [Rec#: 990]

83. Frame PS, Zimmer JG, Werth PL, et al. Computer-based vs manual health maintenance tracking. A controlled trial. Arch Fam Med 1994;3(7): 581-8.

    [Rec#: 160]

84. Frances CD, Alperin P, Adler JS, et al. Does a fixed physician reminder system improve the care of patients with coronary artery disease? A randomized controlled trial. West J Med 2001;175(3):165-6.

    [Rec#: 733]

85. Friedman CP, Elstein AS, Wolf FM, et al. Enhancement of clinicians' diagnostic reasoning by computer-based consultation: a multisite study of 2 systems. JAMA 1999;282(19):1851-6.

    [Rec#: 78]

86. Gajewski LK, Vanscoy GJ, Dean JC, et al. The development of a computerized LAN-based investigational drug information database. Hosp Pharm 1995;30(12):1113, 1116-8, 1120.

    [Rec#: 240]

87. Gamm LD, Barsukiewicz CK, Dansky KH, et al. Investigating changes in end-user satisfaction with installation of an electronic medical record in ambulatory care settings. J Healthc Inf Manag 1998;12 (4):53-65.

    [Rec#: 488]

88. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293(10):1223-38.

    [Rec#: 994]

89. Garr DR, Ornstein SM, Jenkins RG, et al. The effect of routine use of computer-generated preventive reminders in a clinical practice. Am J Prev Med 1993;9(1):55-61.

    [Rec#: 655]

90. Garrido T, Jamieson L, Zhou Y, et al. Effect of electronic health records in ambulatory care: retrospective, serial, cross sectional study. BMJ 2005;330(7491):581.

    [Rec#: 996]

91. Gehlbach SH, Wilkinson WE, Hammond WE, et al. Improving drug prescribing in a primary care practice. Med Care 1984;22(3):193-201.

    [Rec#: 656]

92. Gilliland A, Mills KA, Steele K. General practitioner records on computer -- handle with care. Fam Pract 1992;9(4):441-50.

    [Rec#: 657]

93. Gioia PC. Quality improvement in pediatric well care with an electronic record. Proc AMIA Symp 2001;209-13.

    [Rec#: 493]

94. Goethe JW, Schwartz HI, Szarek BL. Physician compliance with practice guidelines. Conn Med 1997;61(9):553-8.

    [Rec#: 708]

95. Goldman L, Cook EF, Brand DA, et al. A computer protocol to predict myocardial infarction in emergency department patients with chest pain. N Engl J Med 1988;318(13):797-803.

    [Rec#: 658]

96. Gonzalez ER, Vanderheyden BA, Ornato JP, et al. Computer-assisted optimization of eminophylline therapy in the emergency department. Am J Emerg Med 1989;7(4):395-401.

    [Rec#: 659]

97. Gray J, Majeed A, Kerry S, et al. Identifying patients with ischaemic heart disease in general practice: cross sectional study of paper and computerised medical records. Bmj 2000;321(7260):548-50.

    [Rec#: 246]

98. Greist JH, Marks IM, Baer L, et al. Behavior therapy for obsessive-compulsive disorder guided by a computer or by a clinician compared with relaxation as a control. J Clin Psychiatry 2002;63(2):138-45.

    [Rec#: 249]

99. Harris RP, O'Malley MS, Fletcher SW, et al. Prompting physicians for preventive procedures: a five-year study of manual and computer reminders. Am J Prev Med 1990;6(3):145-52.

    [Rec#: 888]

100. Heden B, Ohlin H, Rittner R, et al. Acute myocardial infarction detected in the 12-lead ECG by artificial neural networks. Circulation 1997;96(6):1798-802.

     [Rec#: 81]

101. Hersh WR, Hickam DH. How well do physicians use electronic information retrieval systems? A framework for investigation and systematic review. Jama 1998;280(15):1347-52.

     [Rec#: 257]

102. Hershey CO, Porter DK, Breslau D, et al. Influence of simple computerized feedback on prescription charges in am ambulatory clinic. A randomized clinical trial. Med Care 1986;24(6):472-81.

     [Rec#: 734]

103. Herzmark G, Brownbridge G, Fitter M, et al. Consultation use of a computer by general practitioners. J R Coll Gen Pract 1984;34(269):649-54.

     [Rec#: 662]

104. Hetlevik I, Holmen J, Kruger O. Implementing clinical guidelines in the treatment of hypertension in general practice. Evaluation of patient outcome related to implementation of a computer-based clinical decision support system . Scand J Prim Health Care 1999;17(1):35-40.

     [Rec#: 728]

105. Hickling K, Begg E, Moore ML. A prospective randomised trial comparing individualised pharmacokinetic dosage prediction for aminoglycosides with prediction based on estimated creatinine clearance in critically ill patients. Intensive Care Med 1989;15(4):233-7.

     [Rec#: 663]

106. Hobbs FD, Delaney BC, Carson A, et al. A prospective controlled trial of computerized decision support for lipid management in primary care. Fam Pract 1996;13(2):133-7.

     [Rec#: 664]

107. Honigman B, Lee J, Rothschild J, et al. Using computerized data to identify adverse drug events in outpatients. J Am Med Inform Assoc 2001;8(3):254-66.

     [Rec#: 83]

108. Hooker RC, Cowap N, Newson R, et al. Better by half: hypertension in the elderly and the 'rule of halves': a primary care audit of the clinical computer record as a springboard to improving care. Fam Pract 1999;16(2):123-8.

     [Rec#: 262]

109. Hutchison BG. Effect of computer-generated nurse/physician reminders on influenza immunization among seniors. Fam Med 1989;21(6):433-7.

     [Rec#: 889]

110. Jha AK, Kuperman GJ, Teich JM, et al. Identifying adverse drug events: development of a computer-based monitor and comparison with chart review and stimulated voluntary report. J Am Med Inform Assoc 1998;5(3):305-14.

     [Rec#: 861]

111. Johnson KB, Davison CL. Information technology: its importance to child safety. Ambul Pediatr 2004;4(1):64 72.

     [Rec#: 939]

112. Johnston B, Wheeler L, Deuser J, et al. Outcomes of the Kaiser Permanente Tele-Home Health Research Project. Arch Fam Med 2000;9(1):40-5.

     [Rec#: 85]

113. Jones DL, Kroenke K, Landry FJ, et al. Cost savings using a stepped-care prescribing protocol for nonsteroidal anti-inflammatory drugs. JAMA 1996;275(12):926-30.

     [Rec#: 890]

114. Kaltenthaler E, Shackley P, Stevens K, et al. A systematic review and economic evaluation of computerised cognitive behaviour therapy for depression and anxiety. Health Technol Assess 2002;6(22):1-89.

     [Rec#: 272]

115. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003;163(12):1409-16.

     [Rec#: 22]

116. Kenkre J, Drury VW, Lancashire RJ. Nurse management of hypertension clinics in general practice assisted by a computer. Fam Pract 1985;2(1): 17-22.

     [Rec#: 747]

117. Kerr EA, Smith DM, Hogan MM, et al. Comparing clinical automated, medical record, and hybrid data sources for diabetes quality measures. Jt Comm J Qual Improv 2002;28(10):555-65.

     [Rec#: 165]

118. Khoury A. Finding value in EMRs (electronic medical records). Health Manag Technol 1997;18(8):34, 36.

     [Rec#: 801]

119. Kian LA, Stewart MW, Bagby C, et al. Justifying the cost of a computer-based patient record. Healthc Financ Manage 1995;49(7):58-60, 62, 64-7.

     [Rec#: 802]

120. Kilgore ML, Flint D, Pearce R . The varying impact of two clinical information systems in a cardiovascular intensive care unit. J Cardiovasc Manag 1998;9(2):31-5.

     [Rec#: 668]

121. King WJ, Paice N, Rangrej J, et al. The effect of computerized physician order entry on medication errors and adverse drug events in pediatric inpatients. Pediatrics 2003;112(3 Pt 1):506-9.

     [Rec#: 991]

122. Kovner C, Schuchman L, Mallard C. The application of pen-based computer technology to home health care. Comput Nurs 1997;15(5):237-44.

     [Rec#: 803]

123. Krall MA. Acceptance and performance by clinicians using an ambulatory electronic medical record in an HMO. Proc Annu Symp Comput Appl Med Care 1995;708-11.

     [Rec#: 804]

124. Kramer TL, Owen RR, Cannon D, et al. How well do automated performance measures asses guideline implementation for new-onset depression in the Veterans Health Administration? Jt Comm J Qual Saf 2003;29(9):479-89.

     [Rec#: 168]

125. Krishna S, Balas EA, Spencer DC, et al. Clinical trials of interactive computerized patient education: implications for family practice. J Fam Pract 1997;45(1):25-33.

     [Rec#: 89]

126. Kucher N, Koo S, Quiroz R, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med 2005;352(10):969-77.

     [Rec#: 995]

127. Kuperman GJ , Teich JM, Bates DW, et al. Detecting alerts, notifying the physician, and offering action items: a comprehensive alerting system. Proc AMIA Annu Fall Symp 1996;704-708.

     [Rec#: 25]

128. Kuperman GJ, Teich JM, Tanasijevic MJ, et al. Improving response to critical laboratory results with automation: results of a randomized controlled trial. J Am Med Inform Assoc 1999;6(6):512-22.

     [Rec#: 92]

129. Larrabee JH, Boldreghini S, Elder-Sorrells K, et al. Evaluation of documentation before and after implementation of a nursing information system in an acute care hospital. Comput Nurs 2001;19(2):56-65; quiz 66-8.

     [Rec#: 670]

130. Larsen RA, Evans RS, Burke JP, et al. Improved perioperative antibiotic use and reduced surgical wound infections through use of computer decision analysis. Infect Control Hosp Epidemiol 1989;10(7):316-20.

     [Rec#: 671]

131. Lewin G, Sharp D, Bartholomew J, et al. Computerized assessment of common mental disorders in primary care: effect on clinical outcome. Fam Pract 1996;13(2):120-6.

     [Rec#: 891]

132. Liaw T, Lawrence M, Rendell J . The effect of a computer-generated patient-held medical record summary and/or a written personal health record on patients' attitudes, knowledge and behaviour concerning health promotion. Fam Pract 1996;13(3):289-93.

     [Rec#: 893]

133. Lisboa PJ. A review of evidence of health benefit from artificial neural networks in medical intervention. Neural Netw 2002;15(1):11-39.

     [Rec#: 287]

134. Litzelman DK, Dittus RS, Miller ME, et al. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. J Gen Intern Med 1993;8(6):311-7.

     [Rec#: 152]

135. Lobach DF. Electronically distributed, computer-generated, individualized feedback enhances the use of a computerized practice guideline. Proc AMIA Annu Fall Symp 1996;493-7.

     [Rec#: 894]

136. Lobach DF, Hammond WE. Computerized decision support based on a clinical practice guideline improves compliance with care standards. Am J Med 1997;102(1):89-98.

     [Rec#: 709]

137. Lobach DF, Hammond WE. Development and evaluation of a Computer-Assisted Management Protocol (CAMP): improved compliance with care guidelines for diabetes mellitus. Proc Annu Symp Comput Appl Med Care 1994;787-91.

     [Rec#: 895]

138. Margolis CZ, Warshawsky SS, Goldman L, et al. Computerized algorithms and pediatricians' management of common problems in a community clinic. Acad Med 1992;67(4):282-4.

     [Rec#: 710]

139. Marr PB, Duthie E, Glassman KS, et al. Bedside terminals and quality of nursing documentation. Comput Nurs 1993;11(4):176-82.

     [Rec#: 673]

140. Mc Donald CJ. Use of a computer to detect and respond to clinical events: its effect on clinician behavior. Ann Intern Med 1976;84(2):162-7.

     [Rec#: 726]

141. McAlister NH, Covvey HD, Tong C, et al. Randomised controlled trial of computer assisted management of hypertension in primary care. Br Med J (Clin Res Ed) 1986;293(6548):670-4.

     [Rec#: 674]

142. McCartney P, Macdowall W, Thorogood M. A randomised controlled trial of feedback to general practitioners of their prophylactic aspirin prescribing. BMJ 1997;315(7099):35-6.

     [Rec#: 737]

143. McCowan C, Neville RG, Ricketts IW, et al. Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma. Med Inform Internet Med 2001;26(3):191-201.

     [Rec#: 731]

144. McDonald CJ. Protocol-based computer reminders, the quality of care and the non-perfectability of man. N Engl J Med 1976;295(24):1351-5.

     [Rec#: 96]

145. McDonald CJ, Hui SL, Smith DM, et al. Reminders to physicians from an introspective computer medical record. A two-year randomized trial. Ann Intern Med 1984;100(1):130-8.

     [Rec#: 141]

146. McDonald CJ, Hui SL, Tierney WM. Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics. MD Comput 1992;9(5):304-12.

     [Rec#: 142]

147. McDonald CJ, Wilson GA, McCabe GP= Jr. Physician response to computer reminders. JAMA 1980;244(14):1579-81.

     [Rec#: 727]

148. McDowell I, Newell C, Rosser W. Comparison of three methods of recalling patients for influenza vaccination. CMAJ 1986;135(9):991-7.

     [Rec#: 147]

149. McDowell I, Newell C, Rosser W. Computerized reminders to encourage cervical screening in family practice. J Fam Pract 1989;28(4):420-4.

     [Rec#: 145]

150. McDowell I, Newell C, Rosser W. A follow-up study of patients advised to obtain influenza immunizations. Fam Med 1990;22(4):303-6.

     [Rec#: 897]

151. McDowell I, Newell C, Rosser W. A randomized trial of computerized reminders for blood pressure screening in primary care. Med Care 1989;27(3):297-305.

     [Rec#: 144]

152. McMichael J, Lieberman R, Doyle H, et al. An intelligent and cost-effective computer dosing system for individualizing FK506 therapy in transplantation and autoimmune disorders. J Clin Pharmacol 1993;33(7):599-605.

     [Rec#: 677]

153. McNeely MD. The use of expert systems for improving test use and enhancing the accuracy of diagnosis. Clin Lab Med 2002;22(2):515-28.

     [Rec#: 304]

154. McPhee SJ, Bird JA, Fordham D, et al. Promoting cancer prevention activities by primary care physicians. Results of a randomized, controlled trial. JAMA 1991;266(4):538-44.

     [Rec#: 158]

155. Mekhjian HS, Kumar RR, Kuehn L, et al. Immediate benefits realized following implementation of physician order entry at an academic medical center . J Am Med Inform Assoc 2002;9(5):529-39.

     [Rec#: 26]

156. Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980-97. Bmj 2001;322(7281):279-82.

     [Rec#: 312]

157. Montani S, Bellazzi R, Quaglini S, et al. Meta-analysis of the effect of the use of computer-based systems on the metabolic control of patients with diabetes mellitus. Diabetes Technol Ther 2001;3(3):347-56.

     [Rec#: 315]

158. Montgomery AA, Fahey T. A systematic review of the use of computers in the management of hypertension. J Epidemiol Community Health 1998;52(8):520-5.

     [Rec#: 316]

159. Montgomery AA, Fahey T, Peters TJ, et al. Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial. BMJ 2000;320(7236):686-90.

     [Rec#: 730]

160. Moran WP, Nelson K, Wofford JL, et al. Computer-generated mailed reminders for influenza immunization: a clinical trial. J Gen Intern Med 1992;7(5):535-7.

     [Rec#: 157]

161. Mullett CJ, Evans RS, Christenson JC, et al. Development and impact of a computerized pediatric antiinfective decision support program. Pediatrics 2001;108(4):E75.

     [Rec#: 27]

162. Mungall DR, Anbe D, Forrester PL, et al. A prospective randomized comparison of the accuracy of computer-assisted versus GUSTO nomogram -- directed heparin therapy. Clin Pharmacol Ther 1994;55(5):591-6.

     [Rec#: 678]

163. Murchie CJ, Kenny GN. Comparison among manual, computer-assisted, and closed-loop control of blood pressure after cardiac surgery. J Cardiothorac Anesth 1989;3(1):16-9.

     [Rec#: 679]

164. Murff HJ, Kannry J. Physician satisfaction with two order entry systems. J Am Med Inform Assoc 2001;8(5):499-509.

     [Rec#: 28]

165. Nahm R, Poston I. Measurement of the effects of an integrated, point-of-care computer system on quality of nursing documentation and patient satisfaction. Comput Nurs 2000;18(5):220-9.

     [Rec#: 680]

166. Nightingale PG, Adu D, Richards NT, et al. Implementation of rules based computerised bedside prescribing and administration: intervention study. BMJ 2000;320(7237):750-3.

     [Rec#: 324]

167. Nilasena DS, Lincoln MJ. A computer-generated reminder system improves physician compliance with diabetes preventive care guidelines. Proc Annu Symp Comput Appl Med Care 1995;640-5.

     [Rec#: 712]

168. Nilasena DS, Lincoln MJ, Turner CW, et al. Development and implementation of a computer-generated reminder system for diabetes preventive care. Proc Annu Symp Comput Appl Med Care 1994;831-5.

     [Rec#: 711]

169. Oren E, Shaffer ER, Guglielmo BJ. Impact of emerging technologies on medication errors and adverse drug events. Am J Health Syst Pharm 2003;60(14):1447-58.

     [Rec#: 329]

170. Ornstein SM, Garr DR, Jenkins RG, et al. Implementation and evaluation of a computer-based preventive services system. Fam Med 1995;27(4):260-6.

     [Rec#: 714]

171. Ornstein SM, Garr DR, Jenkins RG, et al. Computer-generated physician and patient reminders. Tools to improve population adherence to selected preventive services. J Fam Pract 1991;32(1):82-90.

     [Rec#: 151]

172. Ornstein SM, MacFarlane LL, Jenkins RG, et al. Medication cost information in a computer-based patient record system. Impact on prescribing in a family medicine clinical practice. Arch Fam Med 1999;8(2):118-21.

     [Rec#: 569]

173. Overhage JM, Perkins S, Tierney WM, et al. Controlled trial of direct physician order entry: effects on physicians' time utilization in ambulatory primary care internal medicine practices. J Am Med Inform Assoc 2001; 8(4):361-71.

     [Rec#: 30]

174. Overhage JM, Suico J, McDonald CJ. Electronic laboratory reporting: barriers, solutions and findings. J Public Health Manag Pract 2001;7(6): 60-6.

     [Rec#: 103]

175. Overhage JM, Tierney WM, McDonald CJ. Computer reminders to implement preventive care guidelines for hospitalized patients. Arch Intern Med 1996;156(14):1551-6.

     [Rec#: 31]

176. Overhage JM, Tierney WM, Zhou XH, et al. A randomized trial of "corollary orders" to prevent errors of omission. J Am Med Inform Assoc 1997;4(5):364-75.

     [Rec#: 32]

177. Pacht ER, Turner JW, Gailiun M, et al. Effectiveness of telemedicine in the outpatient pulmonary clinic. Telemed J 1998;4(4):287-92.

     [Rec#: 104]

178. Perlini S, Piepoli M, Marti G, et al. Treatment of chronic heart failure: an expert system advisor for general practitioners. Acta Cardiol 1990;45(5):365-78.

     [Rec#: 752]

179. Persson M, Mjorndal T, Carlberg B, et al. Evaluation of a computer-based decision support system for treatment of hypertension with drugs: retrospective, nonintervention testing of cost and guideline adherence. J Intern Med 2000;247(1):87-93.

     [Rec#: 337]

180. Peterson CM, Jovanovic L, Chanoch LH. Randomized trial of computer-assisted insulin delivery in patients with type I diabetes beginning pump therapy. Am J Med 1986;81(1):69-72.

     [Rec#: 683]

181. Pierpont GL, Thilgen D. Effect of computerized charting on nursing activity in intensive care. Crit Care Med 1995;23(6):1067-73.

     [Rec#: 832]

182. Pizziferri L, Kittler AF, Volk LA, et al. Primary care physician time utilization before and after implementation of an electronic health record: a time-motion study. J Biomed Inform 2005;38(3):176-88.

     [Rec#: 2200]

183. Poller L, Wright D, Rowlands M. Prospective comparative study of computer programs used for management of warfarin. J Clin Pathol 1993;46(4):299-303.

     [Rec#: 684]

184. Potts AL, Barr FE, Gregory DF, et al. Computerized Physician Order Entry and Medication Errors in a Pediatric Critical Care Unit. Pediatrics 2004;113(1 ):59-63.

     [Rec#: 163]

185. Pringle M, Robins S, Brown G. Computer assisted screening: effect on the patient and his consultation. Br Med J (Clin Res Ed) 1985;290(6483):1709-12.

     [Rec#: 685]

186. Pringle M, Robins S, Brown G. Timer: a new objective measure of consultation content and its application to computer assisted consultations. Br Med J (Clin Res Ed) 1986;293(6538):20-2.

     [Rec#: 686]

187. Quattlebaum TG, Darden PM, Sperry JB. Effectiveness of computer-generated appointment reminders. Pediatrics 1991;88(4):801-5.

     [Rec#: 903]

188. Ramaswamy MR, Chaljub G, Esch O, et al. Continuous speech recognition in MR imaging reporting: advantages, disadvantages, and impact. AJR Am J Roentgenol 2000;174(3):617-22.

     [Rec#: 576]

189. Raschke RA, Gollihare B, Wunderlich TA, et al. A computer alert system to prevent injury from adverse drug events: development and evaluation in a community teaching hospital. JAMA 1998;280(15):1317-20.

     [Rec#: 864]

190. Raynor DK, Booth TG, Blenkinsopp A. Effects of computer generated reminder charts on patients' compliance with drug regimens. BMJ 1993;306(6886):1158-61.

     [Rec#: 742]

191. Rethans JJ, Hoppener P, Wolfs G, et al. Do personal computers make doctors less personal? Br Med J (Clin Res Ed) 1988;296(6634):1446-8.

     [Rec#: 688]

192. Revere D, Dunbar PJ. Review of computer-generated outpatient health behavior interventions: clinical encounters "in absentia". J Am Med Inform Assoc 2001;8(1):62-79.

     [Rec#: 346]

193. Rind DM, Safran C, Phillips RS, et al. Effect of computer-based alerts on the treatment and outcomes of hospitalized patients. Arch Intern Med 1994;154(13 ):1511-7.

     [Rec#: 865]

194. Robbins JA, Dickinson WA, Bartel AG, et al. Lipid management program: results of applying national guidelines in a private practice. South Med J 1993;86(3):289-92.

     [Rec#: 715]

195. Robson J, Boomla K, Fitzpatrick S, et al. Using nurses for preventive activities with computer assisted follow up: a randomised controlled trial. BMJ 1989;298(6671):433-6.

     [Rec#: 905]

196. Rodman JH, Jelliffe RW, Kolb E, et al. Clinical studies with computer-assisted initial lidocaine therapy. Arch Intern Med 1984;144(4):703-9.

     [Rec#: 689]

197. Rogers MA, Small D, Buchan DA, et al. Home monitoring service improves mean arterial pressure in patients with essential hypertension. A randomized, controlled trial. Ann Intern Med 2001;134(11):1024-32.

     [Rec#: 109]

198. Rollman BL, Hanusa BH, Gilbert T, et al. The electronic medical record. A randomized trial of its impact on primary care physicians' initial management of major depression [corrected]. Arch Intern Med 2001;161(2):189-97.

     [Rec#: 352]

199. Rollman BL, Hanusa BH, Lowe HJ, et al. A randomized trial using computerized decision support to improve treatment of major depression in primary care. J Gen Intern Med 2002;17(7):493-503.

     [Rec#: 110]

200. Rosado B, Menzies S, Harbauer A, et al. Accuracy of computer diagnosis of melanoma: a quantitative meta-analysis. Arch Dermatol 2003; 139(3):361-7; discussion 366.

     [Rec#: 355]

201. Rosenfeld BA, Dorman T, Breslow MJ, et al. Intensive care unit telemedicine: alternate paradigm for providing continuous intensivist care. Crit Care Med 2000;28(12):3925-31.

     [Rec#: 111]

202. Rosenstein AH. Measuring the benefit of performance improvement and decision support. Am J Med Qual 1999;14(6):262-9.

     [Rec#: 357]

203. Rosenstein AH. Measuring the benefits of clinical decision support: return on investment. Pharm Pract Manag Q 2000;20(1):28-43.

     [Rec#: 358]

204. Rosser WW, Hutchison BG, McDowell I, et al. Use of reminders to increase compliance with tetanus booster vaccination. CMAJ 1992;146(6):911-7.

     [Rec#: 146]

205. Rossi RA, Every NR. A computerized intervention to decrease the use of calcium channel blockers in hypertension. J Gen Intern Med 1997; 12(11):672-8.

     [Rec#: 716]

206. Ruiz R, Borches D, Gonzalez A, et al. A new sodium-nitroprusside-infusion controller for the regulation of arterial blood pressure. Biomed Instrum Technol 1993;27(3):244-51.

     [Rec#: 691]

207. Ryff-de Leche A, Engler H, Nutzi E, et al. Clinical application of two computerized diabetes management systems: comparison with the log-book method. Diabetes Res 1992;19(3):97-105.

     [Rec#: 692]

208. Safran C, Rind DM, Davis RB, et al. Guidelines for management of HIV infection with computer-based patient's record. Lancet 1995;346(8971):341-6.

     [Rec#: 717]

209. Sanders DL, Miller RA. The effects on clinician ordering patterns of a computerized decision support system for neuroradiology imaging studies. Proc AMIA Symp 2001;583-7.

     [Rec#: 112]

210. Schriger DL, Baraff LJ, Buller K, et al. Implementation of clinical guidelines via a computer charting system: effect on the care of febrile children less than three years of age. J Am Med Inform Assoc 2000;7(2):186-95.

     [Rec#: 596]

211. Schriger DL, Baraff LJ, Rogers WH, et al. Implementation of clinical guidelines using a computer charting system. Effect on the initial care of health care workers exposed to body fluids. Jama 1997;278(19):1585-90.

     [Rec#: 368]

212. Selmi PM, Klein MH, Greist JH, et al. Computer-administered cognitive-behavioral therapy for depression. Am J Psychiatry 1990;147(1):51-6.

     [Rec#: 753]

213. Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc 1996;3(6): 399-409.

     [Rec#: 118]

214. Shiffman RN, Liaw Y, Brandt CA, et al. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc 1999;6(2):104-14.

     [Rec#: 375]

215. Shojania KG, Yokoe D, Platt R, et al. Reducing vancomycin use utilizing a computer guideline: results of a randomized controlled trial. J Am Med Inform Assoc 1998;5(6):554-62.

     [Rec#: 37]

216. Simkins CV, Wenzloff NJ. Evaluation of a computerized reminder system in the enhancement of patient medication refill compliance. Drug Intell Clin Pharm 1986;20(10):799-802.

     [Rec#: 741]

217. Simon GE, VonKorff M, Ruter C, et al. Randomised trial of monitoring, feedback, and management of care by telephone to improve treatment of depression in primary care. BMJ 2000;320(7234):550-4.

     [Rec#: 738]

218. Singh AK, Moidu K, Trell E, et al. Impact on the management and delivery of primary health care by a computer-based information system. Comput Methods Programs Biomed 1992;37(1):55-64.

     [Rec#: 909]

219. Steele MA, Bess DT, Franse VL, et al. Cost effectiveness of two interventions for reducing outpatient prescribing costs. DICP 1989; 23(6):497-500.

     [Rec#: 735]

220. Strecher VJ, Kreuter M, Den Boer DJ, et al. The effects of computer-tailored smoking cessation messages in family practice settings. J Fam Pract 1994;39(3):262-70.

     [Rec#: 911]

221. Sullivan F, Mitchell E. Has general practitioner computing made a difference to patient care? A systematic review of published reports. Bmj 1995;311(7009):848-52.

     [Rec#: 384]

222. Sun J, Chang MW. Initialization of warfarin dosages using computer modeling. Arch Phys Med Rehabil 1995;76 (5):453-6.

     [Rec#: 385]

223. Szilagyi PG, Rodewald LE, Savageau J, et al. Improving influenza vaccination rates in children with asthma: a test of a computerized reminder system and an analysis of factors predicting vaccination compliance. Pediatrics 1992;90(6):871-5.

     [Rec#: 155]

224. Tape TG, Campbell JR. Computerized medical records and preventive health care: success depends on many factors. Am J Med 1993;94(6):619-25.

     [Rec#: 718]

225. Teich JM, Merchia PR, Schmiz JL, et al. Effects of computerized physician order entry on prescribing practices. Arch Intern Med 2000;160(18):2741-2747.

     [Rec#: 39]

226. Theil DR, Stanley TE= 3rd, White WD, et al. Midazolam and fentanyl continuous infusion anesthesia for cardiac surgery: a comparison of computer-assisted versus manual infusion systems. J Cardiothorac Vasc Anesth 1993; 7(3):300-6.

     [Rec#: 696]

227. Thiru K, Hassey A, Sullivan F . Systematic review of scope and quality of electronic patient record data in primary care. Bmj 2003;326(7398):1070.

     [Rec#: 392]

228. Threet E, Fargues MP. Economic evaluation of voice recognition for the clinicians' desktop at the Naval Hospital Roosevelt Roads. Mil Med 1999;164(2):119-26.

     [Rec#: 615]

229. Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physicians performance versus immediate reminders to perform preventive care. Effects on physician compliance. Med Care 1986;24(8):659-66.

     [Rec#: 43]

230. Tierney WM, McDonald CJ, Hui SL, et al. Computer predictions of abnormal test results. Effects on outpatient testing. JAMA 1988;259(8):1194-8.

     [Rec#: 44]

231. Tierney WM, McDonald CJ, Martin DK, et al. Computerized display of past test results. Effect on outpatient testing. Ann Intern Med 1987; 107(4):569-74.

     [Rec#: 45]

232. Tierney WM, Miller ME, McDonald CJ. The effect on test ordering of informing physicians of the changes for outpatient diagnostic tests. N Engl J Med 1990 ;322(21):1499-504.

     [Rec#: 46]

233. Tierney WM, Miller ME, Overhage JM, et al. Physician inpatient order writing on microcomputer workstations. Effects on resource utilization. JAMA 1993;269(3 ):379-83.

     [Rec#: 47]

234. Timmermans D, van Bockel H, Kievit J. Improving the quality of surgeons' treatment decisions: a comparison of clinical decision making with a computerised evidence based decision analytical model. Qual Health Care 2001;10(1):4-9.

     [Rec#: 395]

235. Turner BJ, Day SC, Borenstein B. A controlled trial to improve delivery of preventive care: physician or patient reminders? J Gen Intern Med 1989;4(5):403-9.

     [Rec#: 143]

236. Turner RC, Peden JG= Jr, O'Brien K. Patient-carried card prompts vs. computer-generated prompts to remind private practice physicians to perform health maintenance measures. Arch Intern Med 1994;154(17):1957-60.

     [Rec#: 719]

237. Valency J. How much will that EMR system really cost? Fam Pract Manag 2002;9(4):57-8.

     [Rec#: 619]

238. van den Hoogen JP, van Ree JW . Preventive cardiology in general practice: computer-assisted hypertension care. J Hum Hypertens 1990;4(4):365-7.

     [Rec#: 744]

239. van den Nieuwenhuyzen MC, Engbers FH, Burm AG, et al. Computer-controlled infusion of alfentanil versus patient-controlled administration of morphine for postoperative analgesia: a double-blind randomized trial. Anesth Analg 1995;81(4):671-9.

     [Rec#: 698]

240. van Wijk MA, van der Lei J, Mosseveld M, et al. Assessment of decision support for blood test ordering in primary care. a randomized trial. Ann Intern Med 2001;134(4):274-81.

     [Rec#: 997]

241. Verner D, Seligmann H, Platt S, et al. Computer assisted design of a theophylline dosing regimen in acute bronchospasm: serum concentrations and clinical outcome. Eur J Clin Pharmacol 1992;43(1):29-33.

     [Rec#: 699]

242. Vincent EC, Hardin PA, Norman LA, et al. The effects of a computer-assisted reminder system on patient compliance with recommended health maintenance procedures. Proc Annu Symp Comput Appl Med Care 1995;656-60.

     [Rec#: 720]

243. Walker J, Pan E, Johnston D, et al. The value of health care information exchange and interoperability. Health Affairs 2005;w5.10.

     [Rec#: 987]

244. Walton R, Dovey S, Harvey E, et al. Computer support for determining drug dose: systematic review and meta-analysis. Bmj 1999;318(7189):984-90.

     [Rec#: 401]

245. Wang SJ, Middleton B, Prosser LA, et al. A cost-benefit analysis of electronic medical records in primary care. Am J Med 2003;114:397-403.

     [Rec#: 171]

246. Warshawsky SS, Pliskin JS, Urkin J, et al. Physician use of a computerized medical record system during the patient encounter: a descriptive study. Comput Methods Programs Biomed 1994;43(3-4):269-73.

     [Rec#: 915]

247. Weingarten MA, Bazel D, Shannon HS. Computerized protocol for preventive medicine: a controlled self-audit in family practice. Fam Pract 1989;6(2):120-4.

     [Rec#: 701]

248. Wells BJ, Lobel KD, Dickerson LM. Using the electronic medical record to enhance the use of combination drugs. Am J Med Qual 2003;18(4):147-9.

     [Rec#: 628]

249. White KS, Lindsay A, Pryor TA, et al. Application of a computerized medical decision-making process to the problem of digoxin intoxication. J Am Coll Cardiol 1984;4(3):571-6.

     [Rec#: 702]

250. White RH, Hong R, Venook AP, et al. Initiation of warfarin therapy: comparison of physician dosing with computer-assisted dosing. J Gen Intern Med 1987;2(3):141-8.

     [Rec#: 703]

251. White RH, Mungall D. Outpatient management of warfarin therapy: comparison of computer-predicted dosage adjustment to skilled professional care. Ther Drug Monit 1991;13(1):46-50.

     [Rec#: 704]

252. Willcourt RJ, Pager D, Wendel J, et al. Induction of labor with pulsatile oxytocin by a computer-controlled pump. Am J Obstet Gynecol 1994;170(2):603-8.

     [Rec#: 705]

253. Willson D, Ashton C, Wingate N, et al. Computerized support of pressure ulcer prevention and treatment protocols. Proc Annu Symp Comput Appl Med Care 1995;646-50.

     [Rec#: 721]

254. Wilson AL, Hill JJ, Wilson RG, et al. Computerized medication administration records decrease medication occurrences. Pharm Pract Manag Q 1997;17(1):17-29.

     [Rec#: 860]

255. Wilson GA, McDonald CJ, McCabe GP= Jr. The effect of immediate access to a computerized medical record on physician test ordering: a controlled clinical trial in the emergency room. Am J Public Health 1982;72(7):698-702.

     [Rec#: 706]

256. Wong DH, Gallegos Y, Weinger MB, et al. Changes in intensive care unit nurse task activity after installation of a third-generation intensive care unit information system. Crit Care Med 2003;31 (10):2488-94.

     [Rec#: 53]

APPENDIX G. LIST OF EXCLUDED ARTICLES

1. Beyond guidelines: tool arms physicians with critical knowledge at the point of care. Dis Manag Advis 2002;8(1):912, 1.

   [Rec#: 173]

2. Capitated groups reaping benefits of EMR systems. Capitation Manag Rep 2003;10(6):81-4.

   [Rec#: 426]

3. CareGroup site takes some of the administrative hassle out of healthcare. Internet Healthc Strateg 2001;3(7):4-6.

   [Rec#: 421]

4. Computerization saves money, time, and trees. Healthc Benchmarks 2000;7(1):1-6.

   [Rec#: 418]

5. Computerized system alerts docs to costs. ED Manag 1999;11(9):100-2.

   [Rec#: 417]

6. Conversion to computerized patient records streamlines ER process, saves money. Health Care Cost Reengineering Rep 1997;2(3):37-40.

   [Rec#: 754]

7. Electronic system places patients in post-acute care. Hosp Case Manag 2003;11 (6):85-6.

   [Rec#: 427]

8. Evidence-based medicine system helps physicians order the right tests. Perform Improv Advis 2003; 7(3):40-4.

   [Rec#: 174]

9. Innovative CareEngine system roots out suboptimal care. Dis Manag Advis 2003;9(1):9-13, 1.

   [Rec#: 175]

10. Innovative charting can slash costs, bolster efficiency, enhance quality. ED Manag 1995;7(11):121-3.

    [Rec#: 756]

11. Medication safety issue brief. Using automation to reduce errors. Part 2. Hosp Health Netw 2001; 75(2):33-4.

    [Rec#: 176]

12. New data storage system saves $20,000 per year. Healthc Benchmarks 2001;8(12):139-40, 142.

    [Rec#: 423]

13. Paperless records will improve quality. Hosp Peer Rev 2002;27(7):99-100.

    [Rec#: 424]

14. Physicians go online for electronic medical records. Data Strateg Benchmarks 2000;4(12):183-5, 177.

    [Rec#: 419]

15. Practice brief. Health informatics standards and information transfer: exploring the HIM role. American Health Information Management Association. J AHIMA 2001;72(1):68A-68D.

    [Rec#: 420]

16. What's happening in information systems technology that is reshaping health care delivery, and how might it assist in infection prevention and control? Am J Infect Control 1996;24(6):476-80.

    [Rec#: 178]

17. Adderley D, Hyde C, Mauseth P. The computer age impacts nurses. Comput Nurs 1997;15(1):43-6.

    [Rec#: 758]

18. Ahearn MD, Kerr SJ. General practitioners' perceptions of the pharmaceutical decision-support tools in their prescribing software. Med J Aust 2003;179(1):34-7.

    [Rec#: 179]

19. Al-Ubaydli M. Handheld computers. BMJ 2004;328(7449):1181-4.

    [Rec#: 930]

20. Allaert FA. The use of personal health information for controlling the costs of delivering health care: does the end justify the means? Int J Biomed Comput 1996;43(1-2):79-82.

    [Rec#: 759]

21. Alper BS, Stevermer JJ, White DS, et al. Answering family physicians' clinical questions using electronic medical databases. J Fam Pract 2001;50(11):960-5.

    [Rec#: 180]

22. Amatayakul M. Making the case for electronic records. Health Data Manag 1997;5(5):56-7, 59, 61-3.

    [Rec#: 760]

23. Ammon D. What works: Ready, set, grow. A local practice implements electronic medical record (EMR) technology, triples patient volume and reduces per patient costs by more than 10 percent. Health Manag Technol 2002;23(12):42-3.

    [Rec#: 434]

24. Anderson B. Document imaging and workflow technology saves $1.2 million annually. Health Manag Technol 1999;20 (11):18-20, 22.

    [Rec#: 435]

25. Anderson G, Knickman JR. Changing the chronic care system to meet people's needs. Health Aff (Millwood) 2001;20 (6):146-60.

    [Rec#: 965]

26. Anderson HJ, Bunschoten B. Creating electronic records. A progress report. Health Data Manag 1996;4(9):36-8, 41-2, 44.

    [Rec#: 761]

27. Anderson JG. Security of the distributed electronic patient record: a case-based approach to identifying policy issues. Int J Med Inf 2000;60(2):111-8.

    [Rec#: 436]

28. Antoine W. Electronic miracle. Memorial Hermann put its patient files online and watch its revenue grow. Health Forum J 2002;45(4):34-5.

    [Rec#: 437]

29. Arikian SR, Mario GG, Doyle JJ . Building a computerized disease management operating system. Oncology (Huntingt) 1996;10(11 Suppl):223-32.

    [Rec#: 183]

30. Asher JR. Time and money. A look at message automation via a computerized patient record. MGMA Connex 2003;3(9):28-9.

    [Rec#: 438]

31. Atkinson JC, Zeller GG, Shah C . Electronic patient records for dental school clinics: more than paperless systems. J Dent Educ 2002;66(5):634-42.

    [Rec#: 439]

32. Aubert BA, Hamel G. Adoption of smart cards in the medical sector: the Canadian experience. Soc Sci Med 2001;53(7):879-94.

    [Rec#: 55]

33. Austin T, Iliffe S, Leaning M, et al. A prototype computer decision support system for the management of asthma. J Med Syst 1996;20(1):45-55.

    [Rec#: 750]

34. Bakken S. An informatics infrastructure is essential for evidence-based practice. J Am Med Inform Assoc 2001;8(3):199-201.

    [Rec#: 184]

35. Bakker A, Hammond E, Ball M. Summary report of observations, conclusions and recommendations. Int J Biomed Comput 1995;39(1):11-5.

    [Rec#: 763]

36. Balas EA, Stockham MG, Mitchell MA, et al. The Columbia Registry of Information and Utilization Management Trials. J Am Med Inform Assoc 1995;2(5):307-15.

    [Rec#: 186]

37. Baldwin G. CIO secrets to calculating return on investment. Health Data Manag 2000;8(7):54-9, 62-4, 66-7.

    [Rec#: 440]

38. Ball MJ, Weaver C, Abbott PA. Enabling technologies promise to revitalize the role of nursing in an era of patient safety. Int J Med Inf 2003;69(1):29-38.

    [Rec#: 441]

39. Baltzer DJ. A realistic take on tech. Hosp Health Netw 2002;76(3):12.

    [Rec#: 442]

40. Barnett GO, Winickoff RN, Morgan MM, et al. A computer-based monitoring system for follow-up of elevated blood pressure. Med Care 1983;21(4):400-9.

    [Rec#: 140]

41. Barrett MJ. The evolving computerized medical record, Part II. Healthc Inform 2000;17(9):83-4, 86, 88.

    [Rec#: 443]

42. Barthell EN, Pemble KR. The National Emergency Medical Extranet project. J Emerg Med 2003;24(1):95-100.

    [Rec#: 189]

43. Basch P. Data control. Primary care practice improves productivity and profitability with electronic medical system. Healthc Inform 1999;16(6):159-62.

    [Rec#: 444]

44. Bates DW. Using information systems to improve practice. Schweiz Med Wochenschr 1999;129(49):1913-9.

    [Rec#: 190]

45. Bates DW, Cohen M, Leape LL, et al. Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc 2001;8(4):299-308.

    [Rec#: 60]

46. Bates DW, Ebell M, Gotlieb E, et al. A proposal for electronic medical records in U.S. primary care. J Am Med Inform Assoc 2003;10(1):1-10.

    [Rec#: 61]

47. Bates DW, Evans RS, Murff H, et al. Detecting adverse events using information technology. J Am Med Inform Assoc 2003;10(2):115-28.

    [Rec#: 191]

48. Bates DW, Gawande AA. Improving safety with information technology. N Engl J Med 2003;348(25):2526-34.

    [Rec#: 62]

49. Bates DW, Kuperman GJ, Jha A, et al. Does the computerized display of charges affect inpatient ancillary test utilization? Arch Intern Med 1997;157(21):2501-2508.

    [Rec#: 3]

50. Bates DW, Makary MA, Teich JM, et al. Asking residents about adverse events in a computer dialogue: how accurate are they? Jt Comm J Qual Improv 1998 ;24(4):197-202.

    [Rec#: 445]

51. Bates DW, Spell N, Cullen DJ, et al. The costs of adverse drug events in hospitalized patients. Adverse Drug Events Prevention Study Group. JAMA 1997;277(4):307-311.

    [Rec#: 5]

52. Beard N, Thomas D. Realising your potential. Health Serv J 1990;100(5203):816-817.

    [Rec#: 6]

53. Bellazzi R, Riva A, Montani S, et al. A web-based system for diabetes management: the technical and clinical infrastructure. Proc AMIA Symp 1998;972.

    [Rec#: 65]

54. Berg M. Patient care information systems and health care work: a sociotechnical approach. Int J Med Inf 1999;55(2):87-101.

    [Rec#: 868]

55. Bergeron BP. What price for your patient data? Medical record freeware may also become shareware. Postgrad Med 2000;108(2):33-5.

    [Rec#: 447]

56. Berkowitz LL. Breaking down the barriers. Improving physician buy-in of CPR systems. Healthc Inform 1997;14(10):73-6.

    [Rec#: 768]

57. Bingham A. Computerized patient records benefit physician offices. Healthc Financ Manage 1997;51(9):68-70.

    [Rec#: 769]

58. Birkmeyer CM, Lee J, Bates DW, et al. Will electronic order entry reduce health care costs? Eff Clin Pract 2002;5(2):67-74.

    [Rec#: 8]

59. Birkmeyer JD, Welch HG. A reader's guide to surgical decision analysis. J Am Coll Surg 1997;184(6):589-95.

    [Rec#: 194]

60. Biskup J, Bleumer G. Cryptographic protection of health information: cost and benefit. Int J Biomed Comput 1996;43 (1-2):61-7.

    [Rec#: 770]

61. Boger E. Electronic tracking board reduces ED patient length of stay at Indiana Hospital. J Emerg Nurs 2003;29(1):39-43.

    [Rec#: 449]

62. Brailer DJ. Translating ideals for health information technology into practice. Health Aff (Millwood) 2004;Suppl Web Exclusives:W4-318-20.

    [Rec#: 954]

63. Brassey J, Elwyn G, Price C, et al. Just in time information for clinicians: a questionnaire evaluation of the ATTRACT project. Bmj 2001; 322(7285):529-30.

    [Rec#: 195]

64. Breitfeld PP, Weisburd M, Overhage JM, et al. Pilot study of a point-of-use decision support tool for cancer clinical trials eligibility. J Am Med Inform Assoc 1999;6(6):466-77.

    [Rec#: 66]

65. Brender J, Nohr C, McNair P. Research needs and priorities in health informatics. Int J Med Inf 2000;58-59:257-89.

    [Rec#: 196]

66. Briggs B. Doctors sound off on I.T. concerns. Health Data Manag 2001;9(1):38-40, 42-4, 46-8.

    [Rec#: 451]

67. Briggs B. Electronic stepping stones. Health Data Manag 2002;10(2):36-8.

    [Rec#: 453]

68. Briggs B. Gathering I.T. intelligence. Health Data Manag 2001;9(4):28-32, 34, 36 passim.

    [Rec#: 452]

69. Briscoe M. Obstacles to the use of computers in British mental health services. Psychiatr Serv 1997;48(3):329-30.

    [Rec#: 772]

70. Broude AL. Build quality by investing in technology. An interview with Alan L. Broude. Healthc Financ Manage 2003;57 (2):36-8.

    [Rec#: 454]

71. Brown EG. Methods and pitfalls in searching drug safety databases utilising the Medical Dictionary for Regulatory Activities (MedDRA). Drug Saf 2003;26(3):145-58.

    [Rec#: 198]

72. Brown SH. No free lunch: institutional preparations for computer-based patient records. Proc AMIA Symp 1999;486-90.

    [Rec#: 456]

73. Brundage MD, Groome PA, Feldman-Stewart D, et al. Decision analysis in locally advanced non-small-cell lung cancer: is it useful? J Clin Oncol 1997;15(3):873-83.

    [Rec#: 199]

74. Brushwood DB. Maximizing the value of electronic prescription monitoring programs. J Law Med Ethics 2003;31(1):41-54.

    [Rec#: 457]

75. Bundt TS. Transcription cost analysis reveals opportunities for savings, efficiencies. J AHIMA 2002;73(8):55-60.

    [Rec#: 458]

76. Burkle T, Kuch R, Passian A, et al. The impact of computer implementation on nursing work patterns: study design and preliminary results. Medinfo 1995;8 Pt 2:1321-5.

    [Rec#: 640]

77. Burstin H. Traversing the digital divide. Health Aff (Millwood) 2000;19(6):245-9.

    [Rec#: 967]

78. Burstin H, Clancy C. Broader adoption of information technology needed . Health Aff (Millwood) 2002;21(6):266-7; author reply 267-8.

    [Rec#: 959]

79. Bush J. Open-source software: just what the doctor ordered? Fam Pract Manag 2003;10(6):65, 69.

    [Rec#: 459]

80. Butler MA, Bender AD. Intensive care unit bedside documentation systems. Realizing cost savings and quality improvements. Comput Nurs 1999;17(1):32-8; quiz 39-40.

    [Rec#: 67]

81. Byrnes JJ. A revolutionary advance in disease management. Healthc Leadersh Manag Rep 2001;9(1):14-20.

    [Rec#: 200]

82. Cantlon G. A new proposal for cooperation in HIT development. Health Aff (Millwood) 1999;18(2):257-9.

    [Rec#: 969]

83. Carlson RW, Tu SW, Lane NM, et al. Computer-based screening of patients with HIV/AIDS for clinical-trial eligibility. Online J Curr Clin Trials 1995;Doc No 179:[3347 words; 32 paragraphs].

    [Rec#: 68]

84. Carroll AE, Saluja S, Tarczy-Hornoch P. Development of a Personal Digital Assistant (PDA) based client/server NICU patient data and charting system . Proc AMIA Symp 2001;100-4.

    [Rec#: 462]

85. Cartwright J, de Sylva S, Glasgow M, et al. Inaccessible information is useless information: addressing the knowledge gap. J Med Pract Manage 2002;18(1):36-41.

    [Rec#: 202]

86. Chambliss ML, Rasco T, Clark RD, et al. The mini electronic medical record: a low-cost, low-risk partial solution. J Fam Pract 2001;50(12):1063-5.

    [Rec#: 463]

87. Chin HL, Krall M. Implementation of a comprehensive computer-based patient record system in Kaiser Permanente's Northwest Region. MD Comput 1997;14(1):41-5.

    [Rec#: 773]

88. Chin HL, Krall MA. Successful implementation of a comprehensive computer-based patient record system in Kaiser Permanente Northwest: strategy and experience. Eff Clin Pract 1998;1(2):51-60.

    [Rec#: 464]

89. Chin TL. Using automation to reduce medication errors. Health Data Manag 1997;5(7):74, 76, 78-83.

    [Rec#: 774]

90. Chrischilles EA, Fulda TR, Byrns PJ, et al. The role of pharmacy computer systems in preventing medication errors. J Am Pharm Assoc (Wash) 2002;42(3):439-48.

    [Rec#: 207]

91. Churgin P, Strawn K. Population health management with computerized patient records. Eff Clin Pract 1998;1(2):61-5.

    [Rec#: 466]

92. Churgin PG. Computerized patient records: the patients' response. HMO Pract 1995;9(4):182-5.

    [Rec#: 775]

93. Classen DC, Burke JP, Pestotnik SL, et al. Clinical and financial impact of intravenous erythromycin therapy in hospitalized patients. Ann Pharmacother 1999;33(6):669-673.

    [Rec#: 9]

94. Classen DC, Pestotnik SL, Evans RS, et al. Computerized surveillance of adverse drug events in hospital patients. JAMA 1991;266(20):2847-2851.

    [Rec#: 10]

95. Coffey RM, Ball JK, Johantgen M, et al. The case for national health data standards. Health Aff (Millwood) 1997;16(5):58-72.

    [Rec#: 980]

96. Coiera E. Four rules for the reinvention of health care. BMJ 2004;328(7449):1197-9.

    [Rec#: 935]

97. Cole L, Houston S. Linking outcomes management and practice improvement. Structured care methodologies: evolution and use in patient care delivery. Outcomes Manag Nurs Pract 1999;3(2):53-9; quiz 59-60.

    [Rec#: 208]

98. Connelly DP, Aller RD. Outcomes and informatics. Arch Pathol Lab Med 1997;121(11 ):1176-82.

    [Rec#: 209]

99. Cooper CP, Williams KN, Carey KA, et al. Advertising campaign on a major internet search engine to promote colorectal cancer screening. BMJ 2004;328(7449):1179-80.

    [Rec#: 928]

100. Corley ST. Electronic prescribing: a review of costs and benefits. Top Health Inf Manage 2003;24(1):29-38.

     [Rec#: 12]

101. Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met? Ann Intern Med 1985;103(4):596-9.

     [Rec#: 69]

102. Cox PMJr, D'Amato S, Tillotson DJ. Reducing medication errors. Am J Med Qual 2001;16(3):81-6.

     [Rec#: 211]

103. Coye MJ, Bernstein WS. Improving America's health care system by investing in information technology. Health Aff (Millwood) 2003;22(4):56-8.

     [Rec#: 468]

104. Coyle GA, Heinen M. Scan your way to a comprehensive electronic medical record. Augment medication administration accuracy and increase documentation efficiency with bar coding technology. Nurs Manage 2002;33(12):56, 58-9.

     [Rec#: 469]

105. Coyle-Toerner P, Collins L. How the clinical customization of an EMR means good business: a case study of Queen City Physicians. J Med Pract Manage 2003;19(1):27-31.

     [Rec#: 470]

106. Cramer H, Katz PA. Electronic medical records: honing the competitive edge. Adm Radiol J 1996;15(2):16, 19-22.

     [Rec#: 776]

107. Cross M. Malpractice insurers endorse electronic records as a way to reduce risk. Health Data Manag 1996;4(6):67-8, 70, 72.

     [Rec#: 778]

108. Currell R, Urquhart C. Nursing record systems: effects on nursing practice and health care outcomes. Cochrane Database Syst Rev 2003;( 3):CD002099.

     [Rec#: 212]

109. Dakins DR. Is the future in the cards? Health Data Manag 2001;9(7):44-8.

     [Rec#: 471]

110. Dambro MR, Weiss BD, McClure CL, et al. An unsuccessful experience with computerized medical records in an academic medical center. J Med Educ 1988;63(8):617-23.

     [Rec#: 647]

111. Darbro DA, Eichenauer J, Darbro MJ. Computerized medical record war story: practical experiences of computer implementation at the front. J Ambul Care Manage 2000;23(2):38-42.

     [Rec#: 472]

112. Dassenko D, Slowinski T. Using the CPR to benefit a business office. Healthc Financ Manage 1995;49(7):68-70, 72-3.

     [Rec#: 779]

113. Davidson EJ , Chismar WG. Planning and managing computerized order entry: a case study of IT- enabled organizational transformation. Top Health Inf Manage 1999;19(4):47-61.

     [Rec#: 13]

114. Davis D, LeMaistre A. Has your organization leveraged the benefits of a computerized patient record? Nurs Case Manag 1997;2(6):240-5.

     [Rec#: 780]

115. Davis W. Nine years of computerized patient records in a small family practice. Fam Pract Manag 1997;4(3):69-70, 73-6.

     [Rec#: 781]

116. de Boer HC, Heijmen BJ. A protocol for the reduction of systematic patient setup errors with minimal portal imaging workload. Int J Radiat Oncol Biol Phys 2001;50(5):1350-65.

     [Rec#: 214]

117. de Boer JC, Heijmen BJ. A new approach to off-line setup corrections: combining safety with minimum workload. Med Phys 2002;29(9):1998-2012.

     [Rec#: 215]

118. Dean JM, Vernon DD, Cook L, et al. Probabilistic linkage of computerized ambulance and inpatient hospital discharge records: a potential tool for evaluation of emergency medical services. Ann Emerg Med 2001;37(6):616-26.

     [Rec#: 473]

119. Destache CJ, Meyer SK, Bittner MJ, et al. Impact of a clinical pharmacokinetic service on patients treated with aminoglycosides: a cost-benefit analysis. Ther Drug Monit 1990;12(5):419-26.

     [Rec#: 648]

120. Destache CJ, Meyer SK, Rowley KM. Does accepting pharmacokinetic recommendations impact hospitalization? A cost-benefit analysis. Ther Drug Monit 1990;12(5):427-33.

     [Rec#: 918]

121. Detmer DE, Deering MJ. Health information: a broader view. Health Aff (Millwood) 1999;18(2):256-9.

     [Rec#: 970]

122. Detmer DE. Building the national health information infrastructure for personal health, health care services, public health, and research. BMC Med Inform Decis Mak 2003;3(1):1.

     [Rec#: 70]

123. DeVore PA. Computer-assisted comprehensive geriatric assessment in a family physician's office. South Med J 1991;84 (8):953-5.

     [Rec#: 886]

124. DeWitt AL, Gunn SR, Hopkins P, et al. Critical care medicine mailing list: growth of an online forum. BMJ 2004;328(7449):1180.

     [Rec#: 929]

125. Diehl M. Health system re-engineering: a CPRS economic decision model. Proc Annu Symp Comput Appl Med Care 1995;688-92.

     [Rec#: 783]

126. Dilts D, Khamalah J, Plotkin A. Using cluster analysis for medical resource decision making. Med Decis Making 1995;15(4):333-47.

     [Rec#: 220]

127. Donald JB. On line prescribing by computer. Br Med J (Clin Res Ed) 1986;292(6525):937-9.

     [Rec#: 650]

128. Doolan DF, Bates DW. Computerized physician order entry systems in hospitals: mandates and incentives. Health Aff (Millwood) 2002;21(4):180-8.

     [Rec#: 963]

129. Dorenfest S. The decade of the '90s. Poor use of IT investment contributes to the growing healthcare crisis. Healthc Inform 2000;17(8):64-7.

     [Rec#: 476]

130. Doupnik AM. An overview of Electronic Document Management System product offerings. Top Health Inf Manage 2002;23(1):62-73.

     [Rec#: 477]

131. Dowie J. What decision analysis can offer the clinical decision maker. Why outcome databases such as KIGS and KIMS are vital sources for decision analysis. Horm Res 1999;51 Suppl 1:73-82.

     [Rec#: 222]

132. Dowie R, Gregory RP, Rowsell KV, et al. A decision analytic approach to commissioning ambulance cardiac services. J Manag Med 1998;12(2-3):81-91, 79.

     [Rec#: 223]

133. Doyle O. Make benefits realization part of software purchase. Mod Healthc 1989;19(37):48.

     [Rec#: 14]

134. Drazen E. Why don't we have computer-based patient records? J AHIMA 1996;67(6):56-8; quiz 59-60.

     [Rec#: 784]

135. In: Drazen E, Fortin J, First Consulting Group. Digital hospitals move off the drawing board. 2003. [Rec#: 916]

136. Dujat C, Haux R, Schmucker P, et al. Digital optical archiving of medical records in hospital information systems -- a practical approach towards the computer-based patient record? Methods Inf Med 1995;34(5):489-97.

     [Rec#: 785]

137. Duke JR, Mandell SF. The benefits and perils of system integration. J Med Syst 1990;14(1-2):43-48.

     [Rec#: 16]

138. Durlach P. Focus on voice recognition. Talking up an emerging technology. Health Manag Technol 2000;21(4):16-8.

     [Rec#: 479]

139. Eckman MH. Patient-centered decision making: a view of the past and a look toward the future. Med Decis Making 2001;21(3):241-7.

     [Rec#: 225]

140. Edsall RL. Toward electronic medical records for the small practice. Fam Pract Manag 2000;7(6):10.

     [Rec#: 480]

141. Eichhorst B. Patient-centric HIS. A healthcare information system based on a longitudinal patient record provides benefits to patients -- and clinicians, administrators and IT staff as well. Health Manag Technol 2002;23(4):40-2.

     [Rec#: 482]

142. Elbourne D, Richardson M, Chalmers I, et al. The Newbury Maternity Care Study: a randomized controlled trial to assess a policy of women holding their own obstetric records. Br J Obstet Gynaecol 1987;94(7):612-9.

     [Rec#: 652]

143. Elson RB, Connelly DP. Computerized decision support systems in primary care. Prim Care 1995;22(2):365-84.

     [Rec#: 226]

144. Essin DJ, Lincoln TL. Implementing a low-cost computer-based patient record: a controlled vocabulary reduces data base design complexity. Proc Annu Symp Comput Appl Med Care 1995;431-5.

     [Rec#: 787]

145. Estabrooks C, Goel V, Thiel E, et al. Decision aids: are they worth it? A systematic review. J Health Serv Res Policy 2001;6(3):170-82.

     [Rec#: 227]

146. Everett JE. A decision support simulation model for the management of an elective surgery waiting system. Health Care Manag Sci 2002; 5(2):89-95.

     [Rec#: 76]

147. Eytan TA, Goldberg HI. How effective is the computer-based clinical practice guideline? Eff Clin Pract 2001;4(1):24-33.

     [Rec#: 228]

148. Fabrey RH= 2nd. Physician computer order entry in a hospital setting. Physician Exec 1996;22(4):31-3.

     [Rec#: 788]

149. Feldman P, Gold M, Chu K. Enhancing information for state health policy . Health Aff (Millwood) 1994;13(3):236-50.

     [Rec#: 984]

150. Feldman PH, Gold M, Chu K. State health policy information: what worked? Health Aff (Millwood) 1997;16(1):207-10.

     [Rec#: 983]

151. Ferguson GH, Hildman T, Nichols B. The effect of nursing care planning systems on patient outcomes. J Nurs Adm 1987;17(9):30-6.

     [Rec#: 653]

152. Fermann GJ, Suyama J. Point of care testing in the emergency department . J Emerg Med 2002;22(4):393-404.

     [Rec#: 229]

153. Fernando B, Savelyich BS, Avery AJ, et al. Prescribing safety features of general practice computer systems: evaluation using simulated test cases. BMJ 2004;328(7449):1171-2.

     [Rec#: 924]

154. Fernandopulle R, Ferris T, Epstein A, et al. A research agenda for bridging the 'quality chasm.'. Health Aff (Millwood) 2003;22(2):178-90.

     [Rec#: 958]

155. Ferner RE. Computer aided prescribing leaves holes in the safety net. BMJ 2004;328(7449):1172-3.

     [Rec#: 925]

156. Fieschi M, Dufour JC, Staccini P, et al. Medical decision support systems: old dilemmas and new paradigms? Methods Inf Med 2003;42(3):190-8. [Rec#: 230]

157. Finfgeld DL. Computer-assisted therapy: harbinger of the 21st century? Arch Psychiatr Nurs 1999;13(6):303-10.

     [Rec#: 231]

158. Finkelstein J, Cabrera MR, Hripcsak G. Internet-based home asthma telemonitoring: can patients handle the technology? Chest 2000;117(1):148-55.

     [Rec#: 77]

159. Finnerty M, Altmansberger R, Bopp J, et al. Using state administrative and pharmacy data bases to develop a clinical decision support tool for schizophrenia guidelines. Schizophr Bull 2002;28(1):85-94.

     [Rec#: 232]

160. Firouzan PA, Coburn K. Research review: Survey of automation in health information management departments. Top Health Inf Manage 1996; 17(1):52-60.

     [Rec#: 789]

161. Fischer S, Stewart TE, Mehta S, et al. Handheld computing in medicine. J Am Med Inform Assoc 2003;10(2):139-49.

     [Rec#: 233]

162. Fleisher GR. What works. Children's Hospital ER automates records to save lives and money. Health Manag Technol 1997;18(4):37.

     [Rec#: 790]

163. Fletcher J, Hicks NR, Kay JD, et al. Using decision analysis to compare policies for antenatal screening for Down's syndrome. Bmj 1995;311(7001):351-6.

     [Rec#: 235]

164. Fordham D, McPhee SJ, Bird JA, et al. The Cancer Prevention Reminder System. MD Comput 1990;7(5):289-95.

     [Rec#: 154]

165. Fors UG, Sandberg HC. Computer-aided risk management -- a software tool for the Hidep model. Quintessence Int 2001;32 (4):309-20.

     [Rec#: 236]

166. Foster RA, Antonelli PJ. Computerized physician-order entry: are we there yet? Otolaryngol Clin North Am 2002;35(6):1237-43, vii.

     [Rec#: 485]

167. Frame PS. Computerized health maintenance tracking systems: a clinician's guide to necessary and optional features. A report from the American Cancer Society Advisory Group on Preventive Health Care Reminder Systems. J Am Board Fam Pract 1995;8(3):221-9.

     [Rec#: 237]

168. Fretschner R, Bleicher W, Heininger A, et al. Patient data management systems in critical care. J Am Soc Nephrol 2001;12 Suppl 17:S83-6.

     [Rec#: 486]

169. Frijling BD, Lobo CM, Hulscher ME, et al. Multifaceted support to improve clinical decision making in diabetes care: a randomized controlled trial in general practice. Diabet Med 2002;19(10):836-42.

     [Rec#: 238]

170. Frisse ME. Computers and productivity: is it time for a reality check? Acad Med 1998;73(1):59-64.

     [Rec#: 487]

171. Gadd CS, Baskaran P, Lobach DF. Identification of design features to enhance utilization and acceptance of systems for Internet-based decision support at the point of care. Proc AMIA Symp 1998;91-5.

     [Rec#: 239]

172. Gardner E. Quantifying system benefits an elusive task. Mod Healthc 1990;20(8):26-27.

     [Rec#: 19]

173. Gawande AA, Bates DW. The use of information technology in improving medical performance. Part II. Physician-support tools. MedGenMed 2000;2(1):E13.

     [Rec#: 940]

174. Gawande AA, Bates DW. The use of information technology in improving medical performance. Part III. Patient-support tools. MedGenMed 2000;2(1):E12.

     [Rec#: 241]

175. Gentile D. Toward growth in transcription solutions. Healthcare providers can gain minutes and even hours of efficiency with transcription technology. Health Manag Technol 2003;24(7):38-40.

     [Rec#: 489]

176. Gillespie G. CIOs making soft sell when calculating ROI. Health Data Manag 2002;10(8):44-8, 50.

     [Rec#: 491]

177. Gillespie G. In paper war, is OCR a good soldier? Health Data Manag 2003 ;11(10):58-60, 62 64, passim.

     [Rec#: 492]

178. Gillespie G. Online clinical guidelines help trim costs. Health Data Manag 2000;8(1):38-45.

     [Rec#: 490]

179. Glandon GL, Shapiro RJ. Benefit-cost analysis of hospital information systems: the state of the (non) art. J Health Hum Resour Adm 1988;11(1):30-92.

     [Rec#: 21]

180. Glowinski A. Integrating guidelines and the clinical record: the role of semantically constrained terminologies. Stud Health Technol Inform 1995;16:207-18.

     [Rec#: 792]

181. Goddard BL. Termination of a contract to implement an enterprise electronic medical record system. J Am Med Inform Assoc 2000;7(6):564-8.

     [Rec#: 494]

182. Goedert J. Practice talks up substantial ROI. Health Data Manag 2003;11(4):54-6.

     [Rec#: 495]

183. Goldsmith J, Blumenthal D, Rishel W. Federal health information policy: a case of arrested development. Health Aff (Millwood) 2003;22(4):44-55.

     [Rec#: 496]

184. Goldstein MK, Coleman RW, Tu SW, et al. Translating research into practice: organizational issues in implementing automated decision support for hypertension in three medical centers. J Am Med Inform Assoc 2004;11(5):368-76.

     [Rec#: 944]

185. Goodman KW. Critical care computing: outcomes, confidentiality, and appropriate use. Crit Care Clin 1996;12(1):109-22.

     [Rec#: 497]

186. Gordon C. Practice guidelines and healthcare telematics: towards an alliance. Stud Health Technol Inform 1995;16:3-16.

     [Rec#: 243]

187. Gordon C, Gray JA, Toth B, et al. Systems of evidence-based healthcare and personalised health information: some international and national trends. Stud Health Technol Inform 2000;77:23-8.

     [Rec#: 244]

188. Gorman C, Looker J, Fisk T, et al. A clinically useful diabetes electronic medical record: lessons from the past; pointers toward the future. Eur J Endocrinol 1996;134(1):31-42.

     [Rec#: 498]

189. Grams RR, Morgan G. Medical record innovations that can improve physician productivity. J Med Syst 1999;23(2):133-44.

     [Rec#: 499]

190. Grams RR, Moyer EH. The search for the elusive electronic medical record system -- medical liability, the missing factor. J Med Syst 1997;21(1):1-10.

     [Rec#: 500]

191. Granados A. Health technology assessment and clinical decision making: which is the best evidence? Int J Technol Assess Health Care 1999;15(3):585-92.

     [Rec#: 245]

192. Grandinetti D. The good news -- and bad -- about Web-based EMRs. Med Econ 2000;77(17):73-4, 77, 80 passim.

     [Rec#: 501]

193. Greatbatch D, Heath C, Campion P, et al. How do desk-top computers affect the doctor-patient interaction? Fam Pract 1995;12(1):32-6.

     [Rec#: 887]

194. Green J, Wintfeld N. How accurate are hospital discharge data for evaluating effectiveness of care? Med Care 1993;31 (8):719-31.

     [Rec#: 79]

195. Greene J. The portal advantage. Hospitals lure docs with easy access to medical records. Hosp Health Netw 2001;75 (10):56-8, 2.

     [Rec#: 502]

196. Greene ZB. Creating and managing a paperless health information management department. Top Health Inf Manage 2002;23(1):26-36.

     [Rec#: 503]

197. Greengold NL. A Web-based program for implementing evidence-based patient safety recommendations. Jt Comm J Qual Improv 2002;28(6):340-8.

     [Rec#: 247]

198. Greenhalgh T, Hughes J, Humphrey C, et al. A comparative case study of two models of a clinical informaticist service. Bmj 2002;324(7336):524-9.

     [Rec#: 248]

199. Grimshaw JM, Russell IT. Achieving health gain through clinical guidelines II: Ensuring guidelines change medical practice. Qual Health Care 1994;3(1):45-52.

     [Rec#: 660]

200. Hagland M. Finding the e in healthcare. Healthc Inform 2001;18(11):21-4, 26.

     [Rec#: 506]

201. Hagland M. Moving gradually toward a paperless world. J AHIMA 2000;71(8):26-31; quiz 33-4.

     [Rec#: 504]

202. Hagland M. Outpatient clinics without the paperwork. Health Manag Technol 1997;18(6):14-9.

     [Rec#: 794]

203. Halley EC, Kambic PM. Concurrent process redesign and clinical documentation system implementation: a 6-month success story. Top Health Inf Manage 1996;17(1):12-7.

     [Rec#: 795]

204. Halpern MT, Palmer CS, Simpson KN, et al. The economic and clinical efficiency of point-of-care testing for critically ill patients: a decision-analysis model. Am J Med Qual 1998;13(1):3-12.

     [Rec#: 250]

205. Hancock WM, Walter PF. Reduce hospital costs with admissions and operating room scheduling systems. Softw Healthc 1986;4(1):42-6.

     [Rec#: 80]

206. Hannan TJ. Detecting adverse drug reactions to improve patient outcomes . Int J Med Inf 1999;55(1):61-4.

     [Rec#: 508]

207. Hannan TJ. Variation in health care -- the roles of the electronic medical record. Int J Med Inf 1999;54(2):127-36.

     [Rec#: 507]

208. Hansen B. Handheld computers in diabetes management. Diabetes Self Manag 2002;19(6):43-4, 46, 48-9.

     [Rec#: 509]

209. Hanson CW3rd, Marshall BE. Artificial intelligence applications in the intensive care unit. Crit Care Med 2001;29(2):427-35.

     [Rec#: 251]

210. Harman J. Topics for our times: new health care data -- new horizons for public health. Am J Public Health 1998;88(7):1019-21.

     [Rec#: 511]

211. Hart JT, Thomas C, Gibbons B, et al. Twenty five years of case finding and audit in a socially deprived community. BMJ 1991;302(6791):1509-13.

     [Rec#: 661]

212. Hasenfeld R , Shekelle PG. Is the methodological quality of guidelines declining in the US? Comparison of the quality of US Agency for Health Care Policy and Research (AHCPR) guidelines with those published subsequently. Qual Saf Health Care 2003;12(6):428-34.

     [Rec#: 937]

213. Hassey A, Gerrett D, Wilson A . A survey of validity and utility of electronic patient records in a general practice. Bmj 2001;322(7299):1401-5.

     [Rec#: 252]

214. Heinrichs W, Monk S, Eberle B . [Automated anesthesia record systems]. Anaesthesist 1997;46(7):574-82.

     [Rec#: 512]

215. Hellbruck RP. [Electronic data processing assisted quality management in general practice networks]. Gesundheitswesen 2000;62(10):511-5.

     [Rec#: 513]

216. Henry SB. Nursing informatics: state of the science. J Adv Nurs 1995;22(6):1182-92.

     [Rec#: 254]

217. Henry SB, Douglas K, Galzagorry G, et al. A template-based approach to support utilization of clinical practice guidelines within an electronic health record. J Am Med Inform Assoc 1998;5(3):237-44.

     [Rec#: 255]

218. Henry SB, Mead CN. Nursing classification systems: necessary but not sufficient for representing "what nurses do" for inclusion in computer-based patient record systems. J Am Med Inform Assoc 1997;4(3):222-32.

     [Rec#: 256]

219. Hicks J, Waldrum M, Wilson V, et al. Transcription system overhaul reaps savings, efficiency. J AHIMA 2000;71 (10):58-63.

     [Rec#: 514]

220. Hodgkins ML. Are you ready for the computer-based patient record? J Ambul Care Manage 1995;18(3):1-8.

     [Rec#: 797]

221. Hoffer A. BPM: antidote to inefficiency. Business process management serves as a foundation for effective workflow automation in healthcare. Health Manag Technol 2003;24(1):40-3.

     [Rec#: 515]

222. Hoffman C, Rice D, Sung HY. Persons with chronic conditions. Their prevalence and costs. JAMA 1996;276(18):1473-9.

     [Rec#: 82]

223. Hofmans-Okkes IM, Lamberts H. The International Classification of Primary Care (ICPC): new applications in research and computer-based patient records in family practice. Fam Pract 1996;13(3):294-302.

     [Rec#: 516]

224. Holmes-Rovner M, Kroll J, Rovner DR, et al. Patient decision support intervention: increased consistency with decision analytic models. Med Care 1999;37(3):270-84.

     [Rec#: 260]

225. Holzman TG, Griffith A, Hunter WG, et al. Computer-assisted trauma care prototype. Medinfo 1995;8 Pt 2:1685.

     [Rec#: 261]

226. Homer CS, Davis GK, Everitt LS. The introduction of a woman-held record into a hospital antenatal clinic: the bring your own records study. Aust N Z J Obstet Gynaecol 1999;39( 1):54-7.

     [Rec#: 665]

227. Hovenga EJ. Nursing information and the use of electronic health records. Aust Nurs J 2001;8(11):39-40.

     [Rec#: 263]

228. Howard WR. Development of an affordable data collection, reporting, and analysis system. Respir Care 2003;48(2):131-7.

     [Rec#: 517]

229. Hozo I, Djulbegovic B. Using the Internet to calculate clinical action thresholds. Comput Biomed Res 1999;32(2):168-85.

     [Rec#: 264]

230. Hughes KK, Dvorak EM. The use of decision analysis to examine ethical decision making by critical care nurses. Heart Lung 1997;26(3):238-48.

     [Rec#: 266]

231. Hurley SF, Dziukas LJ, McNeil JJ, et al. A randomized controlled clinical trial of pharmacokinetic theophylline dosing. Am Rev Respir Dis 1986;134(6):1219-24.

     [Rec#: 666]

232. Hynes DM, Perrin RA, Rappaport S, et al. Informatics resources to support health care quality improvement in the veterans health administration. J Am Med Inform Assoc 2004;11(5):344-50.

     [Rec#: 943]

233. Jacobs I. What works. Never going back. New Jersey ophthalmology practice moves from paper charts to an EMR system. Health Manag Technol 2003;24(8):40-1.

     [Rec#: 518]

234. Jadad AR. Evidence-based decision making and asthma in the internet age: the tools of the trade. Allergy 2002;57 Suppl 74:15-22.

     [Rec#: 268]

235. Jadad AR, Haynes RB, Hunt D, et al. The Internet and evidence-based decision-making: a needed synergy for efficient knowledge management in health care . CMAJ 2000;162(3):362-5.

     [Rec#: 269]

236. Jantos LD, May RR, Sheppard RL. Streamlining medical group operations through an integrated computer-based patient record. J Healthc Inf Manag 1998;12(3):89-101.

     [Rec#: 519]

237. Javitt JC. How to succeed in health information technology. Health Aff (Millwood) 2004;Suppl Web Exclusives:W4-321-4.

     [Rec#: 955]

238. Joch A. EMR the web way. Hosp Pract (Off Ed) 2000;35(5):suppl 12-6.

     [Rec#: 521]

239. Johnson KB. Barriers that impede the adoption of pediatric information technology. Arch Pediatr Adolesc Med 2001;155(12):1374-9.

     [Rec#: 270]

240. Johnson KB, Cowan J. Clictate: a computer-based documentation tool for guideline-based care. J Med Syst 2002;26(1):47-60.

     [Rec#: 271]

241. Joosting AC. Practice administration -- a cost-effective option. S Afr Med J 2001;91(6):444-5.

     [Rec#: 522]

242. Jue J, Jerant AF. Electronic paper & pencil: 6 easy steps to a low-cost electronic medical record. Fam Pract Manag 2001;8(5):33-8.

     [Rec#: 523]

243. Kadas RM. The computer-based patient record is on its way. HMOs, the economy and HIPAA will drive adoption. Healthc Inform 2002;19(2):57-8.

     [Rec#: 524]

244. Kahn RA. Beyond HIPAA: the complexities of electronic record management . J AHIMA 2003;74(4):31-6; quiz 37-8.

     [Rec#: 525]

245. Kantor GS, Wilson WD, Midgley A. Open-source software and the primary care EMR. J Am Med Inform Assoc 2003;10 (6):616; author reply 617.

     [Rec#: 526]

246. Kaplan B. An evaluation model for clinical information systems: clinical imaging systems. Medinfo 1995;8 Pt 2:1087.

     [Rec#: 273]

247. Kaplan B, Lundsgaarde HP. Toward an evaluation of an integrated clinical imaging system: identifying clinical benefits. Methods Inf Med 1996;35(3):221-9.

     [Rec#: 527]

248. Katikireddi SV. HINARI: bridging the global information divide. BMJ 2004;328(7449):1190-3.

     [Rec#: 933]

249. Kaufman DR, Patel VL, Hilliman C, et al. Usability in the real world: assessing medical information technologies in patients' homes. J Biomed Inform 2003;36(1-2):45-60.

     [Rec#: 949]

250. 250. Kazanjian A, Green CJ. Beyond effectiveness: the evaluation of information systems using A Comprehensive Health Technology Assessment Framework. Comput Biol Med 2002;32(3):165-77.

     [Rec#: 274]

251. Keller LS, McDermott S, Alt-White A. Effects of computerized nurse careplanning on selected health care effectiveness measures. Proc Annu Symp Comput Appl Med Care 1991;38-42.

     [Rec#: 667]

252. Kelly B. The future of card technology in health care. Health Data Manag 2002;10(5):64-8.

     [Rec#: 528]

253. Kelly G. Electronics, clinicians, and the NHS. BMJ 2000;321(7265):846-7.

     [Rec#: 529]

254. Kelly JT. 'After the chaos': expected benefits of health information management. Health Aff (Millwood) 1998;17 (6):39-40.

     [Rec#: 975]

255. Kemper DW, Mettler M. Information therapy: a tale. Health Forum J 2002;45(1):16-20.

     [Rec#: 275]

256. Kendall DB, Levine SR. Pursuing the promise of an information-age health care system. Health Aff (Millwood) 1998;17(6):41-3.

     [Rec#: 974]

257. Kerkoulas PS. How to produce EDMS requirements and cost-benefit data. Top Health Inf Manage 2002;23(1):7-15.

     [Rec#: 530]

258. Kerr EA, Krein SL, Vijan S, et al. Avoiding pitfalls in chronic disease quality measurement: a case for the next generation of technical quality measures. Am J Manag Care 2001;7(11):1033-43.

     [Rec#: 164]

259. Kerr EA, Smith DM, Hogan MM, et al. Building a better quality measure: are some patients with 'poor quality' actually getting good care? Med Care 2003;41(10):1173-82.

     [Rec#: 166]

260. Khabir J. What works. Handheld PCs, CPR software help clinic cuts costs, enhance service. Health Manag Technol 1997;18(8):42.

     [Rec#: 800]

261. Khoury AT. Support of quality and business goals by an ambulatory automated medical record system in Kaiser Permanente of Ohio. Eff Clin Pract 1998;1(2):73-82.

     [Rec#: 531]

262. Kim MI, Johnson KB. Personal health records: evaluation of functionality and utility. J Am Med Inform Assoc 2002;9(2):171-80.

     [Rec#: 87]

263. Kinyon CC. One CFO's success with transitioning to an automated patient record. Healthc Financ Manage 2003;57 (2):52-6.

     [Rec#: 532]

264. Kirkby KC, Lambert TJ. Computer aids to treatment in psychiatry. Aust N Z J Psychiatry 1996;30(1):142-5.

     [Rec#: 278]

265. Kleinke JD. Vaporware.com: the failed promise of the health care Internet. Health Aff (Millwood) 2000;19 (6):57-71.

     [Rec#: 966]

266. Kleinke JD. Release 0.0: clinical information technology in the real world. Health Aff (Millwood) 1998;17 (6):23-38.

     [Rec#: 533]

267. Kloss L. An electronic emergency. We need to push the industry toward standardized digital medical records. Mod Healthc 2003;33(19):21.

     [Rec#: 534]

268. Kluge EH. Professional codes for electronic HC record protection: ethical, legal, economic and structural issues. Int J Med Inf 2000;60(2):85-96.

     [Rec#: 535]

269. Kok MR, Boon ME. Consequences of neural network technology for cervical screening: increase in diagnostic consistency and positive scores. Cancer 1996 ;78(1):112-7.

     [Rec#: 88]

270. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005;293(10):1197-203.

     [Rec#: 993]

271. Krall MA, Chin H, Dworkin L, et al. Improving clinician acceptance and use of computerized documentation of coded diagnosis. Am J Manag Care 1997;3(4):597-601.

     [Rec#: 805]

272. Krobock JR. A taxonomy: hospital information systems evaluation methodologies. J Med Syst 1984;8(5):419-429.

     [Rec#: 23]

273. Krohn R. Picture archiving and computing systems: the key to enterprise digital imaging. Manag Care Interface 2002;15(9):36-9.

     [Rec#: 536]

274. Kuebler KK, Bruera E. Interactive collaborative consultation model in end-of-life care. J Pain Symptom Manage 2000;20(3):202-9.

     [Rec#: 90]

275. Kuntz KM, Tsevat J, Weinstein MC, et al. Expert panel vs decision-analysis recommendations for postdischarge coronary angiography after myocardial infarction. Jama 1999;282(23):2246-51.

     [Rec#: 279]

276. Kuperman GJ, Gibson RF. Computer physician order entry: benefits, costs, and issues. Ann Intern Med 2003;139(1):31-9.

     [Rec#: 24]

277. Kuperman GJ, Teich JM, Gandhi TK, et al. Patient safety and computerized medication ordering at Brigham and Women's Hospital. Jt Comm J Qual Improv 2001 ;27(10):509-21.

     [Rec#: 91]

278. Kuruvilla S , Dzenowagis J, Pleasant A, et al. Digital bridges need concrete foundations: lessons from the Health InterNetwork India. BMJ 2004;328(7449):1193-6.

     [Rec#: 934]

279. Laerum H, Ellingsen G, Faxvaag A. Doctors' use of electronic medical records systems in hospitals: cross sectional survey. Bmj 2001;323(7325):1344-8.

     [Rec#: 280]

280. Laing K. The benefits and challenges of the computerized electronic medical record. Gastroenterol Nurs 2002;25(2):41-5.

     [Rec#: 538]

281. Lakhani AD, Avery A, Gordon A, et al. Evaluation of a home based health record booklet. Arch Dis Child 1984;59 (11):1076-81.

     [Rec#: 669]

282. Lambrinoudakis C, Gritzalis S . Managing medical and insurance information through a smart-card-based information system. J Med Syst 2000; 24(4):213-34.

     [Rec#: 539]

283. Landis SE, Hulkower SD, Pierson S. Enhancing adherence with mammography through patient letters and physician prompts. A pilot study. N C Med J 1992;53(11):575-8.

     [Rec#: 156]

284. Langton KB, Horsman J, Hayward RS, et al. A Clinical Informatics Network (CLINT) to support the practice of evidence-based health care. Proc AMIA Annu Fall Symp 1996;428-32.

     [Rec#: 281]

285. Latimer EW. Assessing the impact of ambulatory computer-based medical record systems. MD Comput 1999;16(2):44-6.

     [Rec#: 541]

286. Lawler F, Cacy JR, Viviani N, et al. Implementation and termination of a computerized medical information system. J Fam Pract 1996;42(3):233-6.

     [Rec#: 806]

287. Lazarus R, Kleinman KP, Dashevsky I, et al. Using automated medical records for rapid identification of illness syndromes (syndromic surveillance): the example of lower respiratory infection. BMC Public Health 2001;1(1): 9.

     [Rec#: 542]

288. Lazarus SS. Physicians' use of electronic medical records -- identifying and crossing the barriers. Med Group Manage J 1999;46(3):12-4.

     [Rec#: 543]

289. Leape LL, Bates DW, Cullen DJ, et al. Systems analysis of adverse drug events. ADE Prevention Study Group. JAMA 1995;274(1):35-43.

     [Rec#: 862]

290. Leape LL, Berwick DM, Bates DW. What practices will most improve safety? Evidence-based medicine meets patient safety. JAMA 2002;288(4):501-507.

     [Rec#: 170]

291. Lee FW. Can computer-aided systems engineering tools enhance the development of health care information systems? A critical analysis. Top Health Inf Manage 1996;17(1):1-11.

     [Rec#: 282]

292. Lee IN, Lee SC, Embrechts M. Important variable selection techniques with multiple solutions for medical information applications. Med Inform Internet Med 2002;27(4):253-66.

     [Rec#: 283]

293. Lee ML, Warner HR. Performance of a diagnostic system (Iliad) as tool for quality assurance. Comput Biomed Res 1992;25(4):314-23.

     [Rec#: 93]

294. Lehmann ED, Deutsch T. Application of computers in diabetes care -- a review. I. Computers for data collection and interpretation. Med Inform (Lond) 1995;20(4 ):281-302.

     [Rec#: 749]

295. Lehmann ED, Deutsch T. Application of computers in diabetes care -- a review. II. Computers for decision support and education. Med Inform (Lond) 1995;20(4):303-29.

     [Rec#: 748]

296. Leipzig M. Implementing an electronic medical record system in ambulatory care. Hosp Technol Ser 1996;15(12):2-4.

     [Rec#: 809]

297. Lenert LA, Cher DJ. Use of meta-analytic results to facilitate shared decision making. J Am Med Inform Assoc 1999;6(5):412-9.

     [Rec#: 284]

298. Lepage EF, Gardner RM, Laub RM, et al. Improving blood transfusion practice: role of a computerized hospital information system. Transfusion 1992;32(3 ):253-9.

     [Rec#: 94]

299. Lester J, Prady S, Finegan Y, et al. Learning from e-patients at Massachusetts General Hospital. BMJ 2004;328(7449):1188-90.

     [Rec#: 932]

300. Liang MH, Shadick N. Feasibility and utility of adding disease-specific outcome measures to administrative databases to improve disease management. Ann Intern Med 1997;127(8 Pt 2):739-42.

     [Rec#: 285]

301. Liederman EM, Morefield CS. Web messaging: A new tool for patient-physician communication. J Am Med Inform Assoc 2003;10(3):260-70.

     [Rec#: 95]

302. Lillehaug SI, Lajoie SP. AI in medical education -- another grand challenge for medical informatics. Artif Intell Med 1998;12(3):197-225.

     [Rec#: 286]

303. Linney B. Cardiology practice proves that electronic medical records do raise revenue. Physician Exec 2003;29(3):34-6.

     [Rec#: 545]

304. Lipton HL, Miller RH, Wimbush JJ. Electronic prescribing: ready for prime time? J Healthc Inf Manag 2003;17(4):72-9.

     [Rec#: 546]

305. Liu BJ, Cao F, Zhou MZ, et al . Trends in PACS image storage and archive . Comput Med Imaging Graph 2003;27(2-3):165-74.

     [Rec#: 288]

306. Long PW, Larkins RG, Patterson CG, et al. Facilitating best practice: transferring the lessons of the Clinical Support Systems Program. J Qual Clin Pract 2001;21(4):157-9.

     [Rec#: 289]

307. Lowenstein E, Greenberg D. Next-generation informatics. A new picture of quality medicine. Physician Exec 1996;22(11):14-6.

     [Rec#: 811]

308. Lowes B. Firewalls: cheap and easy protection. Med Econ 2003;80(17):20-1.

     [Rec#: 547]

309. Lowes R. Build an EMR for next to nothing. Med Econ 2003; 80(11):29-30.

     [Rec#: 551]

310. Lowes R. EMR? Not exactly. Good enough? For now. Med Econ 2002;79(15):32-4.

     [Rec#: 550]

311. Lowes R. They're still working on the electronic medical record. Med Econ 2000;77(13):29-32, 38.

     [Rec#: 548]

312. Lowes R. Toward a handheld EMR. Med Econ 2002;79(1):26-8.

     [Rec#: 549]

313. Lubarsky DA, Sanderson IC, Gilbert WC, et al. Using an anesthesia information management system as a cost containment tool. Description and validation. Anesthesiology 1997;86(5):1161-9.

     [Rec#: 812]

314. Lumpkin JR, Richards MS. Transforming the public health information infrastructure. Health Aff (Millwood) 2002;21 (6):45-56.

     [Rec#: 961]

315. Luxenberg SN, DuBois DD, Fraley CG, et al. Electronic forms: benefits drawbacks of a World Wide Web-based approach to data entry. Proc AMIA Annu Fall Symp 1997;804-8.

     [Rec#: 813]

316. Maglaveras N, Koutkias V, Chouvarda I, et al. Home care delivery through the mobile telecommunications platform: the Citizen Health System (CHS) perspective. Int J Med Inf 2002;68(1-3):99-111.

     [Rec#: 291]

317. Mahoney ME. Transforming health information management through technology. Top Health Inf Manage 2002;23 (1):52-61.

     [Rec#: 292]

318. Major LF, Turner MG. Assessing the information management requirements for behavioral health providers. J Healthc Manag 2003;48(5):323-33; discussion 334-5.

     [Rec#: 552]

319. Makela M, Kunnamo L. Implementing evidence in Finnish primary care. Use of electronic guidelines in daily practice. Scand J Prim Health Care 2001;19(4):214-7.

     [Rec#: 293]

320. Mandl KD, Kohane IS, Brandt AM. Electronic patient-physician communication: problems and promise. Ann Intern Med 1998;129(6):495-500.

     [Rec#: 294]

321. Marill KA, Gauharou ES, Nelson BK, et al. Prospective, randomized trial of template-assisted versus undirected written recording of physician records in the emergency department. Ann Emerg Med 1999;33(5):500-9.

     [Rec#: 553]

322. Markle Foundation, The Robert Wood Johnson Foundation. Financial, legal and organizational approaches to achieving electronic connectivity in healthcare: working group on financial, organizational, and legal sustainability of health information exchange. 2004.

     [Rec#: 951]

323. Marks I. A computerized system to measure treatment outcomes and cost. Psychiatr Serv 1996;47(8):811-2.

     [Rec#: 814]

324. Martin-Baranera M, Planas I, Palau J, et al. IMASIS computer-based medical record project: dealing with the human factor. Medinfo 1995;8 Pt 1:333.

     [Rec#: 295]

325. Masys DR. Effects of current and future information technologies on the health care workforce. Health Aff (Millwood) 2002;21(5):33-41.

     [Rec#: 962]

326. Matchar DB, Samsa GP. Using outcomes data to identify best medical practice: the role of policy models. Hepatology 1999;29(6 Suppl):36S-39S.

     [Rec#: 296]

327. Mathieson S. For a few dollars more. Health Serv J 2003;113(5872):suppl 8-9.

     [Rec#: 554]

328. Mathis R, Doyle S. A quality mix: using evidence and experience to evaluate new technologies. J Healthc Qual 2003;25(5):4-6; quiz 6-7.

     [Rec#: 297]

329. Matthews P. Planning for successful outcomes in the new millennium. Top Health Inf Manage 2000;20(3):55-64.

     [Rec#: 298]

330. May S. A CPR system can grow a practice. Improved efficiencies in charting and workflow allow better and faster healthcare delivery. Healthc Inform 2002;19(1):93-4.

     [Rec#: 555]

331. McAlearney AS, Schweikhart SB, Medow MA. Doctors' experience with handheld computers in clinical practice: qualitative study. BMJ 2004;328(7449):1162.

     [Rec#: 922]

332. McArthur J, Dickinson A. Decision making the explicit evidence-based way: comparing benefits, harms and costs. Nurs Prax N Z 1999;14(1):33-42.

     [Rec#: 301]

333. McCormick K, Renner AL, Mayes R, et al. The federal and private sector roles in the development of minimum data sets and core health data elements. Comput Nurs 1997;15(2 Suppl):S23-32.

     [Rec#: 815]

334. McCormick KA. Including oncology outcomes of care in the computer-based patient record. Oncology (Huntingt) 1995;9(11 Suppl):161-7.

     [Rec#: 556]

335. McDonald CJ. Need for standards in health information. Health Aff (Millwood) 1998;17(6):44-6.

     [Rec#: 973]

336. McDonald CJ, Overhage JM, Mamlin BW, et al. Physicians, information technology, and health care systems: a journey, not a destination. J Am Med Inform Assoc 2004;11(2):121-4.

     [Rec#: 938]

337. McDonald CJ, Overhage JM, Dexter P, et al. What is done, what is needed and what is realistic to expect from medical informatics standards. Int J Med Inf 1998;48(1-3):5-12.

     [Rec#: 675]

338. McDonald CJ, Overhage JM, Tierney WM, et al. The Regenstrief Medical Record System: a quarter century experience. Int J Med Inf 1999;54(3):225-53.

     [Rec#: 676]

339. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med 2003;348(26):2635-45.

     [Rec#: 97]

340. McGowan JJ, Winstead-Fry P. Problem Knowledge Couplers: reengineering evidence-based medicine through interdisciplinary development, decision support, and research. Bull Med Libr Assoc 1999;87(4):462-70.

     [Rec#: 303]

341. McKenzie NC, Marks IM. Overcoming interface problems in computerized monitoring of clinical outcomes. MD Comput 1997;14(5):377-81.

     [Rec#: 816]

342. McPhee SJ, Bird JA, Jenkins CN, et al. Promoting cancer screening. A randomized, controlled trial of three interventions. Arch Intern Med 1989;149(8):1866-72.

     [Rec#: 153]

343. McQueen L, Mittman BS, Demakis JG. Overview of the Veterans Health Administration (VHA) Quality Enhancement Research Initiative (QUERI). J Am Med Inform Assoc 2004;11(5):339-43.

     [Rec#: 942]

344. Mehta V, Kushniruk A, Gauthier S, et al. Use of evidence in the process of practice change in a clinical team: a study-forming part of the Autocontrol Project. Int J Med Inf 1998;51(2-3):169-80.

     [Rec#: 305]

345. Memel DS, Scott JP, McMillan DR, et al. Development and implementation of an information management and information technology strategy for improving healthcare services: a case study. J Healthc Inf Manag 2001;15(3):261-85.

     [Rec#: 558]

346. Mendelson DN, Goodman CS, Ahn R, et al. Outcomes and effectiveness research in the private sector. Health Aff (Millwood) 1998;17(5):75-90.

     [Rec#: 977]

347. Mendelson DN, Salinsky EM. Health information systems and the role of state government. Health Aff (Millwood) 1997;16(3):106-19.

     [Rec#: 981]

348. Merkouris AV. Computer-based documentation and bedside terminals. J Nurs Manag 1995;3(2):81-5.

     [Rec#: 306]

349. Messina E. Eliminating serious risks in the outpatient clinic with an electronic medical record. J Healthc Risk Manag 1995;15(4):20-4.

     [Rec#: 817]

350. Metfessel BA. An automated tool for an analysis of compliance to evidence-based clinical guidelines. Medinfo 2001;10(Pt 1):226-30.

     [Rec#: 307]

351. Meyer K, Dimond M, Shaver J, et al. The 'Learnmore' project: information acquisition solutions for the long-term care information system. Medinfo 1995;8 Pt 1:276-9.

     [Rec#: 308]

352. Meyer TJ, Van Kooten D, Marsh S, et al. Reduction of polypharmacy by feedback to clinicians. J Gen Intern Med 1991;6(2):133-6.

     [Rec#: 736]

353. Michael P, Pritchett E. The impact of HIPAA electronic transmissions and health information privacy standards. J Am Diet Assoc 2001;101(5):524-8.

     [Rec#: 559]

354. Middleton G. The electronic revolution. Report writing in the 21st century. Emerg Med Serv 2002;31(8):65-8.

     [Rec#: 560]

355. Mildon J, Cohen T. Drivers in the electronic medical records market. Health Manag Technol 2001;22(5):14-6, 18.

     [Rec#: 561]

356. Miller RH, Sim I. Physicians' use of electronic medical records: barriers and solutions. Health Aff (Millwood) 2004;23(2):116-26.

     [Rec#: 953]

357. Miller RA, Gardner RM. Recommendations for responsible monitoring and regulation of clinical software systems. American Medical Informatics Association, Computer-based Patient Record Institute, Medical Library Association, Association of Academic Health Science Libraries, American Health Information Management Association, American Nurses Association. J Am Med Inform Assoc 1997;4(6):442-57.

     [Rec#: 309]

358. Miller RA, Gardner RM. Summary recommendations for responsible monitoring and regulation of clinical software systems. American Medical Informatics Association, The Computer-based Patient Record Institute, The Medical Library Association, The Association of Academic Health Science Libraries, The American Health Information Management Association, and The American Nurses Association. Ann Intern Med 1997;127(9):842-5.

     [Rec#: 310]

359. Miller SR. I think I scan. Case study: implementing the computerized patient record -- not an EMR. MGMA Connex 2003;3(2):46-9, 1.

     [Rec#: 562]

360. Miller TE, Derse AR. Between strangers: the practice of medicine online. Health Aff (Millwood) 2002;21(4):168-79.

     [Rec#: 964]

361. Millonig MK, Jackson TL, Ellis WM. Improving medication use through pharmacists' access to patient-specific health care information. J Am Pharm Assoc (Wash) 2002;42(4):638-45.

     [Rec#: 311]

362. Milstein A, Galvin RS, Delbanco SF, et al. Improving the safety of health care: the leapfrog initiative. Eff Clin Pract 2000;3(6):313-6.

     [Rec#: 872]

363. Mitchell RD, Jessup S. Staying one step ahead -- development of an electronic record. Caring 1995;14(8):34, 37, 39-41.

     [Rec#: 818]

364. Moehr JR. Informatics in the service of health, a look to the future. Methods Inf Med 1998;37(2):165-70.

     [Rec#: 313]

365. Mohr DN. Benefits of an electronic clinical information system. Healthc Inf Manage 1997;11(4):49-57.

     [Rec#: 819]

366. Moidu K, Wigertz O, Trell E. Multi centre systems analysis study of primary health care: a study of socioorganizational and human factors. Int J Biomed Comput 1992;30(1):27-42.

     [Rec#: 899]

367. Monane M, Matthias DM, Nagle BA, et al. Improving prescribing patterns for the elderly through an online drug utilization review intervention: a system linking the physician, pharmacist, and computer. JAMA 1998;280(14):1249-52.

     [Rec#: 314]

368. Montoya ID, Carlson JW. Point-of-care systems, informatics, and health care delivery. Health Care Superv 1996;15 (2):17-26.

     [Rec#: 822]

369. Moran DW. Health information policy: on preparing for the next war. Health Aff (Millwood) 1998;17(6):9-22.

     [Rec#: 976]

370. Morris AH. Decision support and safety of clinical environments. Qual Saf Health Care 2002;11(1):69-75.

     [Rec#: 317]

371. Morris AH. Rational use of computerized protocols in the intensive care unit. Crit Care 2001;5(5):249-54.

     [Rec#: 318]

372. Morrissey J. Out to set the record. With a push from HHS, the effort to create electronic medical records continues to build momentum. Mod Healthc 2003;33(42):28-32, 35.

     [Rec#: 563]

373. Muller CR. Computer applications in the search for unrelated stem cell donors. Transpl Immunol 2002;10(2-3):227-40.

     [Rec#: 319]

374. Mullooly JP. Misclassification model for person-time analysis of automated medical care databases. Am J Epidemiol 1996;144(8):782-92.

     [Rec#: 320]

375. Munter KH, Schenk JF, Thrun F, et al. The "Phoenix" ADR database of the Drug Commission of the German Medical Profession -- a clinically useful approach to optimize evidence-based medicine in Germany. Semin Thromb Hemost 1999;25(1):57-64.

     [Rec#: 321]

376. Murray D. Electronic records solved this practice's problems. Med Econ 1996;73(10):135-6, 142-4, 147.

     [Rec#: 823]

377. Neame R. Smart cards -- the key to trustworthy health information systems . BMJ 1997;314(7080):573-7.

     [Rec#: 825]

378. Nenov V, Klopp J. Remote analysis of physiological data from neurosurgical ICU patients. J Am Med Inform Assoc 1996;3(5):318-27.

     [Rec#: 322]

379. Nerenz DR. Capacities and limitations of information systems as data sources on quality of care at the end of life. J Pain Symptom Manage 2001;22(3):773-83.

     [Rec#: 323]

380. Nicolosi E, Molino F, Molino G. Computerized approach to active and objective quality control of patient care. Int J Biomed Comput 1996;42(1-2):59-66.

     [Rec#: 826]

381. Nilsson G, Ahlfeldt H, Strender LE. Computerisation, coding, data retrieval and related attitudes among Swedish general practitioners-a survey of necessary conditions for a database of diseases and health problems. Int J Med Inf 2002;65(2):135-43.

     [Rec#: 325]

382. Nixon SJ, Munro MM, Sands I, et al. Is Intranet technology the low cost, quick fix solution for Scotland's clinical information technology chaos? Results of a pilot study. Scott Med J 2000;45(5):140-3.

     [Rec#: 566]

383. Njalsson T. On content of practice. The 391. Ohno-Machado L, Wang SJ, Mar P, et al. advantage of computerized Decision support for clinical trial information systems in family eligibility determination in breast practice. Scand J Prim Health Care cancer. Proc AMIA Symp 1999;3401995;13 Suppl 1:1-102. 4.

     [Rec#: 827]

384. Norrie P, Blackwell RE. Constructing a 392. Oostendorp A, Hoekstra S, Aarts J. Exploring financial case for a computerized factors required for successful patient data management system implementation of healthcare (PDMS) in a cardiothoracic intensive informatics. Stud Health Technol care unit. Med Inform Internet Med Inform 1998;51:146-51. 2000;25(3):161-9.

     [Rec#: 567]

385. O'Brien L. Setting the standard. West Coast healthcare network mandates ERP system to ensure business standardization in four states. Health Manag Technol 2003;24(9):18-21.

     [Rec#: 568]

386. O'Connell RT, Cho C, Shah N, et al. Take note(s): differential EHR satisfaction with two implementations under one roof. J Am Med Inform Assoc 2004;11(1):43-9.

     [Rec#: 917]

387. O'Connor AM, Rostom A, Fiset V, et al. Decision aids for patients facing health treatment or screening decisions: systematic review. Bmj 1999;319(7212):731-4.

     [Rec#: 326]

388. O'Connor AM, Stacey D, Entwistle V, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev 2003;(2):CD001431.

     [Rec#: 327]

389. O'Connor K. Information management for managed care. Med Group Manage J 1995;42(6):52-4, 56, 75.

     [Rec#: 829]

390. O'Neil AC, Petersen LA, Cook EF, et al. Physician reporting compared with medical-record review to identify adverse medical events. Ann Intern Med 1993;119(5):370-376.

     [Rec#: 29]

391. Ohno-Machado L, Wang SJ, Mar P, et al. Decision support for clinical trial eligibility determination in breast cancer. Proc AMIA Symp 1999;340-4.

     [Rec#: 100]

392. Oostendorp A, Hoekstra S, Aarts J. Exploring factors required for successful implementation of healthcare informatics. Stud Health Technol Inform 1998;51:146-51.

     [Rec#: 867]

393. Ornstein S, Bearden A. Patient perspectives on computer-based medical records. J Fam Pract 1994;38(6):606-10.

     [Rec#: 900]

394. Ornstein SM. Translating research into practice using electronic medical records the PPRNet-TRIP project: primary and secondary prevention of coronary heart disease and stroke. Top Health Inf Manage 2001;22(2):52-8.

     [Rec#: 330]

395. Ornstein SM, Garr DR, Jenkins RG. A comprehensive microcomputer-based medical records system with sophisticated preventive services features for the family physician. J Am Board Fam Pract 1993;6 (1):55-60.

     [Rec#: 713]

396. Orson J. Using the Internet to enhance evidence-based practice. Br J Perioper Nurs 2000;10(8):421-7.

     [Rec#: 331]

397. Overhage JM, Middleton B, Miller RA, et al. Does national regulatory mandate of provider order entry portend greater benefit than risk for health care delivery? The 2001 ACMI debate. The American College of Medical Informatics. J Am Med Inform Assoc 2002;9(3):199-208.

     [Rec#: 101]

398. Overhage JM, Tierney WM, McDonald CJ. Design and implementation of the Indianapolis Network for Patient Care and Research. Bull Med Libr Assoc 1995;83(1):48-56.

     [Rec#: 681]

399. Owen RR, Thrush CR, Cannon D, et al. Use of electronic medical record data for quality improvement in schizophrenia treatment. J Am Med Inform Assoc 2004;11(5):351-7.

     [Rec#: 945]

400. Owens DK. Analytic tools for public health decision making. Med Decis Making 2002;22(5 Suppl):S3-10.

     [Rec#: 332]

401. Ozbolt JG. From minimum data to maximum impact: using clinical data to strengthen patient care. MD Comput 1997;14(4):295-301.

     [Rec#: 831]

402. Pace MA. At home with coding. Western healthcare system improves revenue cycle management via home coders. Health Manag Technol 2003;24(9):22-3.

     [Rec#: 570]

403. Pakpahan R, Balas EA, Boren SA. Computable decision modules for patient safety in child health care. Proc AMIA Symp 2002;592-6.

     [Rec#: 333]

404. Papaconstantinou C, Theocharous G, Mahadevan S. An expert system for assigning patients into clinical trials based on Bayesian networks. J Med Syst 1998;22(3):189-202.

     [Rec#: 105]

405. Parente ST. Proprietary data systems: help or hindrance? Health Aff (Millwood) 1997;16(5):218-20.

     [Rec#: 979]

406. Parker B, Harman A. No boundaries. Advanced telecom solutions enable secure storage and transfer of electronic medical records over great distances. Health Manag Technol 2003;24(8):37-8.

     [Rec#: 571]

407. Parker RM, Ratzan SC, Lurie N . Health literacy: a policy challenge for advancing high-quality health care. Health Aff (Millwood) 2003;22(4):147-53.

     [Rec#: 957]

408. Parmigiani G. Measuring uncertainty in complex decision analysis models. Stat Methods Med Res 2002;11(6):513-37.

     [Rec#: 334]

409. Pavlin JA. Investigation of disease outbreaks detected by "syndromic" surveillance systems. J Urban Health 2003;80 (2 Suppl 1):i107-14.

     [Rec#: 106]

410. Pavlopoulos S, Berler A, Kyriacou E, et al. Design and development of a multimedia database for emergency telemedicine. Technol Health Care 1998;6(2-3):101-10.

     [Rec#: 335]

411. Pawlson G. Information technologies and the health care workforce. Health Aff (Millwood) 2002;21(6):266; author reply 267-8.

     [Rec#: 960]

412. Pennachio DL. Saving way more than a buck or two. Med Econ 2003;80(11):84, 87.

     [Rec#: 572]

413. Persson M, Bohlin J, Eklund P . Development and maintenance of guideline-based decision support for pharmacological treatment of hypertension. Comput Methods Programs Biomed 2000;61(3):209-19.

     [Rec#: 336]

414. Petersen LA, Brennan TA, O'Neil AC, et al. Does housestaff discontinuity of care increase the risk for preventable adverse events? Ann Intern Med 1994;121(11):866-72.

     [Rec#: 107]

415. Petrick N, Sahiner B, Chan HP, et al. Breast cancer detection: evaluation of a mass-detection algorithm for computer-aided diagnosis -experience in 263 patients. Radiology 2002;224(1):217-24.

     [Rec#: 108]

416. Philp I, Goddard A, Connell NA, et al. Development and evaluation of an information system for quality assurance. Age Ageing 1994;23(2):150-153.

     [Rec#: 33]

417. Piccoli A. Elementary clinical decision analysis in evidence-based nephrology. J Nephrol 2000;13(6):419-32.

     [Rec#: 338]

418. Pifer EA, Smith S, Keever GW. EMR to the rescue. An ambulatory care pilot project shows that data sharing equals cost shaving. Healthc Inform 2001;18(2):111-4.

     [Rec#: 573]

419. Pincus HA, McQueen L. The computer age: beware the loss of the narrative. Gen Hosp Psychiatry 1996;18(4):284-5.

     [Rec#: 833]

420. Pliskin N, Glezerman M, Modai I, et al. Spreadsheet evaluation of computerized medical records: the impact on quality, time, and money. J Med Syst 1996;20(2):85-100.

     [Rec#: 834]

421. Pollack R. What works. Computerized patient record cuts redundant documentation, improves charting. Health Manag Technol 1997;18(9):29.

     [Rec#: 835]

422. Poon EG, Blumenthal D, Jaggi T, et al. Overcoming barriers to adopting and implementing computerized physician order entry systems in U.S. hospitals. Health Aff (Millwood) 2004;23(4):184-90.

     [Rec#: 952]

423. Powsner SM, Wyatt JC, Wright P. Opportunities for and challenges of computerisation. Lancet 1998;352(9140):1617-22.

     [Rec#: 574]

424. Pringle M, Robins S, Brown G. Computers in the surgery. The patient's view. Br Med J (Clin Res Ed) 1984;288(6413):289-91.

     [Rec#: 902]

425. Pringle M, Robins S, Brown G. Topic analysis: an objective measure of the consultation and its application to computer assisted consultations. Br Med J (Clin Res Ed) 1985;290(6484):1789-91.

     [Rec#: 687]

426. Protti D. What can the American electronic health record (EHR) pioneers tell us about what it takes to be successful? Healthc Manage Forum 2002;15(2):33-5.

     [Rec#: 575]

427. Purves I. Computerised guidelines in primary health care: reflections and implications. Stud Health Technol Inform 1995;16:57-74.

     [Rec#: 340]

428. Rader RW, Toalson TW. Ready or not, handheld prescribing is coming. J Okla State Med Assoc 2001;94(9):403-8.

     [Rec#: 342]

429. Raghupathi W, Tan J. Strategic uses of information technology in health care: a state-of-the-art survey. Top Health Inf Manage 1999;20(1):1-15.

     [Rec#: 343]

430. Ralston JD, Revere D, Robins LS, et al. Patients' experience with a diabetes support programme based on an interactive electronic medical record: qualitative study. BMJ 2004;328(7449):1159.

     [Rec#: 921]

431. Ramos JD, Rai-Chaudhuri A, Neill RW. International online discussion lists on chronic myelogenous leukaemia. BMJ 2004;328(7449):1177-8.

     [Rec#: 927]

432. Ramsaroop P, Ball MJ. The "bank of health". A model for more useful patient health records. MD Comput 2000;17(4):45-8.

     [Rec#: 577]

433. Ramsey SD, McIntosh M, Etzioni R, et al. Simulation modeling of outcomes and cost effectiveness. Hematol Oncol Clin North Am 2000;14(4):925-38.

     [Rec#: 344]

434. Ranke MB, Dowie J. KIGS and KIMS as tools for evidence-based medicine. Horm Res 1999;51 Suppl 1:83-6.

     [Rec#: 345]

435. Reed H. Streamlining patient access with a paperless system. NAHAM Manage J 1997;23(4):12, 15.

     [Rec#: 836]

436. Rehm S, Kraft S. Electronic medical records: the FPM vendor survey . Fam Pract Manag 2001;8(1):45-54.

     [Rec#: 579]

437. Renner K. Electronic medical records in the outpatient setting (Part 1) . Med Group Manage J 1996;43(3):52, 54, 56-7 passim.

     [Rec#: 838]

438. Renner K. Electronic medical records in the outpatient setting (Part 2) . Med Group Manage J 1996;43(4):60-5, 82.

     [Rec#: 839]

439. Retchin SM, Wenzel RP. Electronic medical record systems at academic health centers: advantages and implementation issues. Acad Med 1999;74(5):493-8.

     [Rec#: 580]

440. Rettig RA. Are patients a scarce resource for academic clinical research? Health Aff (Millwood) 2000;19(6):195-205.

     [Rec#: 968]

441. Rhodes H, Dougherty M. Practice brief. Document imaging as a bridge to the EHR. J Ahima 2003;74(6):56A-56G.

     [Rec#: 347]

442. Ridsdale L, Hudd S. Computers in the consultation: the patient's view . Br J Gen Pract 1994;44(385):367-9.

     [Rec#: 904]

443. Robinson A, Thomson RG. The potential use of decision analysis to support shared decision making in the face of uncertainty: the example of atrial fibrillation and warfarin anticoagulation. Qual Health Care 2000;9(4): 238-44.

     [Rec#: 349]

444. Robinson TN, Patrick K, Eng TR, et al. An evidence-based approach to interactive health communication: a challenge to medicine in the information age. Science Panel on Interactive Communication and Health. Jama 1998;280(14):1264-9.

     [Rec#: 350]

445. Rogoski RR. The ABCs of CPRs and EMRs. Definitions, semantics, product differences and the need for demonstrated results hinder wide-scale adoption . Health Manag Technol 2002;23(5):14-6, 19.

     [Rec#: 583]

446. Rogoski RR. Having it your way. Three physicians who were early adopters and who remain vocal EMR advocates share their insights about the utilization, challenges and benefits of EMRs. Health Manag Technol 2003;24(5):12-6.

     [Rec#: 585]

447. Rogoski RR. Opening the floodgates of usability. Clinical information systems allow free flow of patient data to clinicians when and where they need it. Health Manag Technol 2003;24(9):12-7.

     [Rec#: 586]

448. Rogoski RR. You say tomato and... While the practice management landscape has blossomed with progressive technology, still differences exist in issues of EMR integration, and HIPAA endures as the looming challenge of the decade . Health Manag Technol 2002;23(10):24-6.

     [Rec#: 584]

449. Roland MO, Zander LI, Evans M, et al. Evaluation of a computer assisted repeat prescribing programme in a general practice. Br Med J (Clin Res Ed) 1985;291(6493):456-8.

     [Rec#: 690]

450. Rollman BL, Gilbert T, Lowe HJ, et al. The electronic medical record: its role in disseminating depression guidelines in primary care practice. Int J Psychiatry Med 1999;29(3):267-86.

     [Rec#: 351]

451. Roos NP, Black C, Roos LL, et al. Managing health services: how administrative data and population-based analyses can focus the agenda. Health Serv Manage Res 1998;11(1):49-67.

     [Rec#: 353]

452. Roos NP, Black C, Roos LL, et al. Managing health services: how the Population Health Information System (POPULIS) works for policymakers. Med Care 1999;37(6 Suppl):JS27-41.

     [Rec#: 354]

453. Roper WL, Mays G. A maturing market for health information technology? Health Aff (Millwood) 1999;18(2):255-6.

     [Rec#: 972]

454. Rosenfeld S , Zeitler E, Mendelson D. Financial Incentives: Innovative Payment for Health Information Technology. 2004.

     [Rec#: 941]

455. Rosenquist PB, Colenda CC, Briggs J, et al. Riding a Trojan horse: computerized psychiatric treatment planning using managed care principles. Manag Care Q 1996;4(4):89-95.

     [Rec#: 356]

456. Rosenstein AH. Using information management to implement a clinical resource management program. Jt Comm J Qual Improv 1997;23(12):653-66.

     [Rec#: 359]

457. Rosse C. Information technology: value to patients. Health Aff (Millwood) 1999;18(2):256.

     [Rec#: 971]

458. Rosser WW, McDowell I, Newell C. Use of reminders for preventive procedures in family medicine. CMAJ 1991;145(7):807-14.

     [Rec#: 148]

459. Ruffin M. Information Technology, Part 1: A way to streamline medical practice. Physician Exec 1996;22(7):37-40.

     [Rec#: 840]

460. Russell J, Greenhalgh T, Boynton P, et al. Soft networks for bridging the gap between research and practice: illuminative evaluation of CHAIN. BMJ 2004;328(7449):1174.

     [Rec#: 926]

461. Sackett DL, Straus SE. Finding and applying evidence during clinical rounds: the "evidence cart". Jama 1998;280(15):1336-8.

     [Rec#: 360]

462. Safran C, Rind DM, Davis RB, et al. Effects of a knowledge-based electronic patient record in adherence to practice guidelines. MD Comput 1996;13(1):55-63.

     [Rec#: 908]

463. Samsa GP, Reutter RA, Parmigiani G, et al. Performing cost-effectiveness analysis by integrating randomized trial data with a comprehensive decision model: application to treatment of acute ischemic stroke. J Clin Epidemiol 1999;52(3):259-71.

     [Rec#: 361]

464. Sanders GD, Hagerty CG, Sonnenberg FA, et al. Distributed decision support using a web-based interface: prevention of sudden cardiac death. Med Decis Making 1999;19(2):157-66.

     [Rec#: 362]

465. Sandrick K. Calculating ROI for CPRs. Health Manag Technol 1998;19(6):16-20.

     [Rec#: 589]

466. Sarasin FP. Decision analysis and the implementation of evidence-based medicine. Qjm 1999;92(11):669-71.

     [Rec#: 363]

467. Sargent JA. Identifying the value of computers in dialysis. Blood Purif 2002;20(1):11-9.

     [Rec#: 590]

468. Satava RM. Medical applications of virtual reality. J Med Syst 1995;19(3):275-80.

     [Rec#: 364]

469. Sausser GD. Thin is in: web-based systems enhance security, clinical quality. Healthc Financ Manage 2003;57 (7):86-9.

     [Rec#: 591]

470. Scalise D. CPOE (computerized physician order entry). An executive's guide. Hosp Health Netw 2002;76(6):41-6.

     [Rec#: 592]

471. Schadow G, Russler DC, McDonald CJ. Conceptual alignment of electronic health record data with guideline and workflow knowledge. Int J Med Inf 2001;64(2-3):259-74.

     [Rec#: 593]

472. Scharf L. Revising nursing documentation to meet patient outcomes. Nurs Manage 1997;28(4):38-9.

     [Rec#: 693]

473. Schiff GD, Klass D, Peterson J, et al. Linking laboratory and pharmacy: opportunities for reducing errors and improving care. Arch Intern Med 2003;163(8):893-900.

     [Rec#: 113]

474. Schiff GD, Rucker TD. Computerized prescribing: building the electronic infrastructure for better medication usage. JAMA 1998;279(13):1024-9.

     [Rec#: 114]

475. Schleyer T, Spallek H. Dental informatics. A cornerstone of dental practice. J Am Dent Assoc 2001;132(5):605-13.

     [Rec#: 365]

476. Schmidt IK, Svarstad BL. Nurse-physician communication and quality of drug use in Swedish nursing homes. Soc Sci Med 2002;54(12):1767-77.

     [Rec#: 115]

477. Schmitt KF, Wofford DA. Financial analysis projects clear returns from electronic medical records. Healthc Financ Manage 2002;56(1):52-7.

     [Rec#: 594]

478. Schmitz HH. A protocol for evaluating hospital information systems. Hosp Health Serv Adm 1977;22(1):45-56.

     [Rec#: 35]

479. Schneider EC, Eisenberg JM. Strategies and methods for aligning current and best medical practices. The role of information technologies. West J Med 1998;168(5 ):311-8.

     [Rec#: 366]

480. Schneider EC, Riehl V, Courte-Wienecke S, et al. Enhancing performance measurement: NCQA's road map for a health information framework. National Committee for Quality Assurance. JAMA 1999;282(12):1184-90.

     [Rec#: 595]

481. Schneider JH. Online personal medical records: are they reliable for acute/critical care? Crit Care Med 2001;29(8 Suppl):N196-201.

     [Rec#: 367]

482. Schuster DM, Hall SE, Couse CB, et al. Involving users in the implementation of an imaging order entry system. J Am Med Inform Assoc 2003;10(4):315-21.

     [Rec#: 116]

483. Sciamanna CN, Ford DE, Flynn JA, et al. An evidence-based interactive computer program to assist physicians in counseling smokers to quit. MD Comput 1999;16(5):54-60.

     [Rec#: 369]

484. Scott JS. EDI plays major role in coordinating care. Health Manag Technol 1995;16(2):36-8, 40.

     [Rec#: 841]

485. Sennett C. Moving ahead, measure by measure. Health Aff (Millwood) 1998;17(4):36-7.

     [Rec#: 978]

486. Seroussi B, Bouaud J, Dreau H, et al. ASTI: a guideline-based drug-ordering system for primary care. Medinfo 2001;10(Pt 1):528-32.

     [Rec#: 370]

487. Shafazand S, Shigemitsu H, Weinacker AB. A brave new world: remote intensive care unit for the 21st century. Crit Care Med 2000;28(12):3945-6.

     [Rec#: 117]

488. Shaffer DW, Kigin CM, Kaput JJ, et al. What is digital medicine? Stud Health Technol Inform 2002;80:195-204.

     [Rec#: 371]

489. Shaikh I. What works. New VA facility uses paperless medical records systems to compete. Health Manag Technol 1997;18(7):28.

     [Rec#: 842]

490. Shani M. The impact of information on medical thinking and health care policy. Int J Med Inf 2000;58-59:3-10.

     [Rec#: 373]

491. Shea S, Starren J, Weinstock RS, et al. Columbia University's Informatics for Diabetes Education and Telemedicine (IDEATel) Project: rationale and design. J Am Med Inform Assoc 2002;9(1):49-62.

     [Rec#: 119]

492. Shiffman RN, Brandt CA, Liaw Y, et al. A design model for computer-based guideline implementation based on information management services. J Am Med Inform Assoc 1999;6(2):99-103.

     [Rec#: 374]

493. Shojania KG , Duncan BW, McDonald KM, et al. Making health care safer: a critical analysis of patient safety practices. Evid Rep Technol Assess (Summ) 2001;(43):i-x, 1-668.

     [Rec#: 992]

494. Shojania KG, Duncan BW, McDonald KM, et al. Safe but sound. Patient safety meets evidence-based medicine. JAMA 2002;288(4):508-513.

     [Rec#: 169]

495. Sicotte C, Denis JL, Lehoux P . The computer based patient record: a strategic issue in process innovation. J Med Syst 1998;22(6):431-43.

     [Rec#: 597]

496. Sim I, Gorman P, Greenes RA, et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 2001;8(6):527-34.

     [Rec#: 376]

497. Sim I, Sanders GD, McDonald KM. Evidence-based practice for mere mortals: the role of informatics and health services research. J Gen Intern Med 2002;17(4):302-8.

     [Rec#: 377]

498. Sirio CA, Segel KT, Keyser DJ, et al. Pittsburgh Regional Healthcare Initiative: a systems approach for achieving perfect patient care. Health Aff (Millwood) 2003;22(5):157-65.

     [Rec#: 956]

499. Sittig DF, Stead WW. Computer-based physician order entry: the state of the art. J Am Med Inform Assoc 1994;1(2):108-23.

     [Rec#: 121]

500. Siwicki B. Outpatient electronic records. Tackling the challenges. Health Data Manag 1997;5(5):66-8, 79, 74 passim.

     [Rec#: 843]

501. Smith B. Computer-based patient records support organizational redesign . Hosp Technol Ser 1997;16(7):2-4.

     [Rec#: 845]

502. Smith D, Newell LM. Deploying the EMR: Part 2, Implementation. J Med Pract Manage 2003;18(6):304-8.

     [Rec#: 598]

503. Smith J. The case for capitalism in healthcare. Health Manag Technol 2001;22(6):68, 67.

     [Rec#: 599]

504. Solomon GL, Dechter M. Are patients pleased with computer use in the examination room? J Fam Pract 1995;41(3):241-4.

     [Rec#: 910]

505. Sonksen P, Williams C. Information technology in diabetes care 'Diabeta': 23 years of development and use of a computer-based record for diabetes care. Int J Biomed Comput 1996;42(1-2):67-77.

     [Rec#: 846]

506. Soper WD. Why I love my EMR. Fam Pract Manag 2002;9(9):35-8.

     [Rec#: 600]

507. Soto CM, Kleinman KP, Simon SR. Quality and correlates of medical record documentation in the ambulatory care setting. BMC Health Serv Res 2002;2(1):22.

     [Rec#: 601]

508. Staccini P, Joubert M, Quaranta JF, et al. Modelling health care processes for eliciting user requirements: a way to link a quality paradigm and clinical information system design. Int J Med Inf 2001;64(2-3):129-42.

     [Rec#: 602]

509. Staender S, Davies J, Helmreich B, et al. The anaesthesia critical incident reporting system: an experience based database. Int J Med Inf 1997;47(1-2):87-90.

     [Rec#: 379]

510. Stair TO. Reduction of redundant laboratory orders by access to computerized patient records. J Emerg Med 1998;16(6):895-7.

     [Rec#: 122]

511. Stammer L. Chart pulling brought to its knees. Healthc Inform 2001;18(2):107-8.

     [Rec#: 603]

512. Stanfill PH, Weber DG, Wayne C. Clinical patient information capture: the Columbia/HCA cancer care database experience. J Oncol Manag 1998;7(2):17-21.

     [Rec#: 380]

513. Starr P. Smart technology, stunted policy: developing health information networks. Health Aff (Millwood) 1997;16 (3):91-105.

     [Rec#: 982]

514. Starren J, Hripcsak G, Sengupta S, et al. Columbia University's Informatics for Diabetes Education and Telemedicine (IDEATel) project: technical implementation. J Am Med Inform Assoc 2002;9(1):25-36.

     [Rec#: 120]

515. Stausberg J, Koch D, Ingenerf J, et al. Comparing paper-based with electronic patient records: lessons learned during a study on diagnosis and procedure codes. J Am Med Inform Assoc 2003;10 (5):470-7.

     [Rec#: 604]

516. Stefanelli M. Knowledge management to support performance-based medicine. Methods Inf Med 2002;41(1):36-43.

     [Rec#: 381]

517. Stevens B. Development and testing of a pediatric pain management sheet . Pediatr Nurs 1990;16(6):543-8.

     [Rec#: 694]

518. Stricklin ML, Struk CM. Point of care technology: a sociotechnical approach to home health implementation. Methods Inf Med 2003;42(4):463-70.

     [Rec#: 605]

519. Struk C. Critical steps for integrating information technology in home care: one agency's experience. Home Healthc Nurse 2001;19(12):758-65.

     [Rec#: 382]

520. Sujansky WV. The benefits and challenges of an electronic medical record: much more than a "word-processed" patient chart. West J Med 1998;169(3):176-83.

     [Rec#: 383]

521. Sullivan M. The CPR's hidden impact: reducing malpractice risk. Healthc Inform 1996;13(6):36, 38, 40.

     [Rec#: 849]

522. Swanson T, Dostal J, Eichhorst B, et al. Recent implementation of electronic medical records in four family practice residency programs. Acad Med 1997;72(7):607-12.

     [Rec#: 851]

523. Sylvestri MF. Health care informatics: the key to successful disease management. Med Interface 1996;9(5):94-6, 98-9.

     [Rec#: 852]

524. Szynal D. Doctors become palm readers. Health Data Manag 2001;9(6):20, 23, 25.

     [Rec#: 606]

525. Taigman M. Ending the paper trail. Electronic documentation in EMS. Emerg Med Serv 2003;32(3):75-8.

     [Rec#: 607]

526. Takeda H, Endoh H. Commentary on 'health care in the information society. A prognosis for the year 2013'. Int J Med Inf 2002;66(1-3):107-11.

     [Rec#: 386]

527. Takeda H, Matsumura Y, Nakajima K, et al. Health care quality management by means of an incident report system and an electronic patient record system. Int J Med Inf 2003;69(2-3):285-93.

     [Rec#: 608]

528. Tang PC. AMIA advocates national health information system in fight against national health threats. J Am Med Inform Assoc 2002;9(2):123-4.

     [Rec#: 124]

529. Tang PC, Fafchamps D, Shortliffe EH. Tranditional medical records as a source of clinical data in the outpatient setting. Proc Annu Symp Comput Appl Med Care 1994;575-9.

     [Rec#: 125]

530. Tang PC, LaRosa MP, Gorden SM . Use of computer-based records, completeness of documentation, and appropriateness of documented clinical decisions. J Am Med Inform Assoc 1999;6(3):245-51.

     [Rec#: 126]

531. Tang PC, LaRosa MP, Newcomb C, et al. Measuring the effects of reminders for outpatient influenza immunizations at the point of clinical opportunity. J Am Med Inform Assoc 1999;6(2):115-21.

     [Rec#: 127]

532. Teich JM, Glaser JP, Beckley RF, et al. The Brigham integrated computing system (BICS): advanced clinical systems in an academic hospital environment. Int J Med Inf 1999;54(3):197-208.

     [Rec#: 390]

533. Teich JM, Spurr CD, Schmiz JL, et al. Enhancement of clinician workflow with computer order entry. Proc Annu Symp Comput Appl Med Care 1995;459-463.

     [Rec#: 40]

534. Teich JM, Wagner MM, Mackenzie CF, et al. The informatics response in disaster, terrorism, and war. J Am Med Inform Assoc 2002;9(2):97-104.

     [Rec#: 129]

535. Terry K. EMRs boost efficiency, too. Med Econ 2002;79(7):38-40, 43.

     [Rec#: 611]

536. Terry K. EMRS cost too much? This group says no way! Med Econ 2002;79(7):34-9.

     [Rec#: 612]

537. Terry K. EMRs. What you need to know. Med Econ 2003;80(9):TCP5-TCP14.

     [Rec#: 613]

538. Terry K. How connectivity is changing practice. Med Econ 2000;77(13):166-70, 173.

     [Rec#: 609]

539. Terry K. The move to put office orders online. Med Econ 2003;80(15):23-4.

     [Rec#: 614]

540. Terry K. Will the Internet finally put an end to paperwork? Med Econ 2000;77(13):152-4, 159-60, 163-5.

     [Rec#: 610]

541. Terry NP. An eHealth diptych: the impact of privacy regulation on medical error and malpractice litigation. Am J Law Med 2001;27(4):361-419.

     [Rec#: 391]

542. Thompson C, McCaughan D, Cullum N, et al. Research information in nurses' clinical decision-making: what is useful? J Adv Nurs 2001;36(3):376-88.

     [Rec#: 393]

543. Thomson R, Robinson A, Greenaway J, et al. Development and description of a decision analysis based decision support tool for stroke prevention in atrial fibrillation. Qual Saf Health Care 2002;11(1):25-31.

     [Rec#: 394]

544. Tierney WM. Feedback of performance and diagnostic testing: lessons from Maastricht. Med Decis Making 1996;16 (4):418-9.

     [Rec#: 41]

545. Tierney WM. Improving clinical decisions and outcomes with information: a review. Int J Med Inf 2001;62(1):1-9.

     [Rec#: 42]

546. Tierney WM, Murray MD, Gaskins DL, et al. Using computer-based medical records to predict mortality risk for inner-city patients with reactive airways disease. J Am Med Inform Assoc 1997;4(4):313-21.

     [Rec#: 48]

547. Tierney WM, Overhage JM, McDonald CJ. Demonstrating the effects of an IAIMS on health care quality and cost. J Am Med Inform Assoc 1997;4(2 Suppl):S41-6.

     [Rec#: 853]

548. Tierney WM, Overhage JM, Takesue BY, et al. Computerizing guidelines to improve care and patient outcomes: the example of heart failure. J Am Med Inform Assoc 1995;2(5):316-22.

     [Rec#: 697]

549. Tierney WM, Takesue BY, Vargo DL, et al. Using electronic medical records to predict mortality in primary care patients with heart disease: prognostic power and pathophysiologic implications. J Gen Intern Med 1996; 11(2):83-91.

     [Rec#: 49]

550. Timmermans DR, Sprij AJ, de Bel CE. The discrepancy between daily practice and the policy of a decision-analytic model: the management of fever of unknown origin. Med Decis Making 1996;16(4):357-66.

     [Rec#: 396]

551. Tomasulo P. PD(X)MD: a new clinical information tool for patient care. Med Ref Serv Q 2003;22(2):77-83.

     [Rec#: 397]

552. Tomes JP. Legal issues in the acquisition of computerized medical records and payment systems. J Health Care Finance 1996;22(4):88-92.

     [Rec#: 854]

553. Tsui FC, Espino JU, Dato VM, et al. Technical description of RODS: a real-time public health surveillance system. J Am Med Inform Assoc 2003;10(5):399-408.

     [Rec#: 130]

554. Tully M. The impact of information technology on the performance of clinical pharmacy services. J Clin Pharm Ther 2000;25(4):243-9.

     [Rec#: 617]

555. Urden LD. Development of a nurse executive decision support database. A model for outcomes evaluation. J Nurs Adm 1996;26(10):15-21.

     [Rec#: 399]

556. Utz SW. Computerized documentation of case management. From diagnosis to outcomes. Nurs Case Manag 1998;3(6):247-54.

     [Rec#: 618]

557. Valenta AL, Wigger U. Early results of user profiles: physicians' opinions on the use of information technology. Proc AMIA Annu Fall Symp 1996;65-9.

     [Rec#: 914]

558. van Ginneken AM. The computerized patient record: balancing effort and benefit. Int J Med Inf 2002;65(2):97-119.

     [Rec#: 620]

559. van Ginneken AM, Stam H, van Mulligen EM, et al. ORCA: the versatile CPR. Methods Inf Med 1999;38(4-5):332-8.

     [Rec#: 621]

560. Vissers MC, Greve JW, Gouma DJ, et al. Introduction of a computerised protocol in clinical practice: is there anything to gain? Eur J Surg 1997;163(4):245-54.

     [Rec#: 400]

561. von Grey M. Focus on voice recognition. Technology moves into the mainstream. Health Manag Technol 2000;21 (4):20-1.

     [Rec#: 622]

562. Wagner EH. The role of patient care teams in chronic disease management . BMJ 2000;320(7234):569-72.

     [Rec#: 131]

563. Wagner EH, Austin BT, Von Korff M. Organizing care for patients with chronic illness. Milbank Q 1996;74(4):511-44.

     [Rec#: 132]

564. Walker LM. At Kaiser, a vision of the connected future. Med Econ 2002;79(3):52-4, 58.

     [Rec#: 623]

565. Walsh SH. The clinician's perspective on electronic health records and how they can affect patient care. BMJ 2004;328(7449):1184-7.

     [Rec#: 931]

566. Wang YZ, Wang S. Automation of clinical and patient records. South Med J 1999;92(1):16-22.

     [Rec#: 624]

567. Wanlass RL, Reutter SL, Kline AE. Communication among rehabilitation staff: "mild," "moderate," or "severe" deficits? Arch Phys Med Rehabil 1992;73(5):477-81.

     [Rec#: 135]

568. Ward MD. e-Medx, Inc.'s e-POC for cancer. Case Manager 2001;12(2):29.

     [Rec#: 403]

569. Ward MD. InformaCare disease management system. Case Manager 2002;13(3):30-1.

     [Rec#: 404]

570. Ward SR. Health smart cards: merging technology and medical information . Med Ref Serv Q 2003;22(1):57-65.

     [Rec#: 625]

571. Waring N. To what extent are practices 'paperless' and what are the constraints to them becoming more so? Br J Gen Pract 2000;50(450):46-7.

     [Rec#: 626]

572. Warren JR, Posey B, Thornton T, et al. Can computer autoacquisition of medical information meet the needs of the future? A feasibility study in direct computation of the fine grained electronic medical record. Proc AMIA Symp 1999;445-9.

     [Rec#: 627]

573. Watlington AG. Realizing system benefits: one planned step at a time. Comput Healthc 1989;10(2):32-4, 36.

     [Rec#: 50]

574. Watson S. Electronic records pay off twice. Health Data Manag 1996;4(3):112-4.

     [Rec#: 855]

575. Weaver FM, Hatzakis M, Evans CT, et al. A comparison of multiple data sources to identify vaccinations for veterans with spinal cord injuries and disorders. J Am Med Inform Assoc 2004;11 (5):377-9.

     [Rec#: 946]

576. Weekley JS, Smith BJ, Pradhan M. The intersection of health informatics and evidence-based medicine: computer-based systems to assist clinicians. Med J Aust 2000;173(7):376-8.

     [Rec#: 405]

577. Weiner M, Callahan CM, Tierney WM, et al. Using information technology to improve the health care of older adults. Ann Intern Med 2003; 139(5 Pt 2):430-6.

     [Rec#: 51]

578. Weingarten M, Andrew WF. The CPR: balancing quality and cost. Healthc Inform 1996;13(8):44, 46, 48.

     [Rec#: 858]

579. Weingarten SR, Henning JM, Badamgarav E, et al. Interventions used in disease management programmes for patients with chronic illness-which ones work? Meta-analysis of published reports. BMJ 2002;325(7370):925.

     [Rec#: 136]

580. Weinstein MC, O'Brien B, Hornberger J, et al. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices -- Modeling Studies. Value Health 2003;6(1):9-17.

     [Rec#: 406]

581. Welton JM, Jarr S. Automating and improving the data quality of a nursing department quality management program at a university hospital. Jt Comm J Qual Improv 1997;23(12):623-35.

     [Rec#: 407]

582. Westra B, Raup G. Computerized charting -- an essential tool for survival. Caring 1995;14(8):57-61.

     [Rec#: 859]

583. Whitlock WL, Brown A, Moore K, et al. Telemedicine improved diabetic management. Mil Med 2000;165(8):579-84.

     [Rec#: 137]

584. Willey CJ, Struckhoff B. Physicians use EMR to improve quality. Esse Health takes lead in primary care technology, quality. Med Group Manage J 1999;46(6):18-21, 24-7.

     [Rec#: 629]

585. Williams B. The untapped potential of electronic medical records. Pt. 1 -- Road-testing the promises of EMR technology. Tenn Med 2001;94(4):119-21.

     [Rec#: 630]

586. Williams B. The untapped potential of electronic medical records. Pt. 2 -- Solving the problems of EMR technology. Tenn Med 2001;94(5):161-4.

     [Rec#: 631]

587. Williams C, Whitfield G. Written and computer-based self-help treatments for depression. Br Med Bull 2001;57:133-44.

     [Rec#: 408]

588. Wong BK, Sellaro CL, Monaco JA. Information systmes analysis approach in hospitals: a national survey. Health Care Superv 1995;13(3):58-64.

     [Rec#: 52]

589. Wong ET, Abendroth TW. Reaping the benefits of medical information systems. Acad Med 1996;71(4):353-7.

     [Rec#: 409]

590. Wood L, Coulson R. Revitalizing the General Practice Research Database: plans, challenges, and opportunities. Pharmacoepedemiol Drug Saf 2001;10(5):379-83.

     [Rec#: 633]

591. Worth ER, Patrick TB. Do electronic mail discussion lists act as virtual colleagues? Proc AMIA Annu Fall Symp 1997;325-9.

     [Rec#: 139]

592. Wu AW, Pronovost P, Morlock L . ICU incident reporting systems. J Crit Care 2002;17(2):86-94.

     [Rec#: 410]

593. Wu R, Peters W, Morgan MW. The next generation of clinical decision support: linking evidence to best practice. J Healthc Inf Manag 2002; 16(4):50-5.

     [Rec#: 411]

594. Wulff HR. Clinical medicine in the age of the computer. Aust N Z J Med 2000;30(4):503-5.

     [Rec#: 412]

595. Young AS, Mintz J, Cohen AN, et al. A Network-Based System to Improve Care for Schizophrenia: The Medical Informatics Network Tool (MINT). J Am Med Inform Assoc 2004;11(5):358-67.

     [Rec#: 947]

596. Zielstorff RD, Barnett GO, Fitzmaurice JB, et al. A decision support system for prevention and treatment of pressure ulcers based on AHCPR guidelines. Proc AMIA Annu Fall Symp 1996;562-6.

     [Rec#: 722]

597. Zoccali C. The importance of large, high quality clinical databases in nephrology. J Nephrol 2001;14 Suppl 4:S89-93.

     [Rec#: 415]

598. Zurhellen W. The coming of age of the electronic office. Pediatr Ann 2001;30(5):289-97.

     [Rec#: 635]

599. Zvarova J, Tomeckova M, Stefek M, et al. Decision support and data analysis tools for risk assessment in primary preventive study of atherosclerosis . Stud Health Technol Inform 1997;43 Pt B:625-8.

     [Rec#: 416]

NOTES

1. Nonmonetized consequences are merely costs and benefits that are not expressed in dollar terms. It may be easy to express some of them in dollars but difficult to realize the corresponding cash flows. (For example, the time you spend in traffic may be worth $100/hour, but who is going to pay you for it?) Others may resist expression in dollars.

2. Nonprofits may explicitly have commitments to provide the highest quality care, but for-profits also share medical ethics and culture to do the best they can for their patients.

3. However, we recognize that there are barriers to providing this level of specification: For example, prior to that advent of the internet, journals might have been reticent to devote limited space to such descriptions, and the knowledge of what variables need to be included changes over time.

4. Appendixes and Evidence Tables for this report are provided electronically at http://www.ahrq.gov/downloads/pub/evidence/pdf/hitsyscosts/hitsys.pdf.

5. The eight functionalities are documentation (health information and data storage); results management; order entry management; decision support; electronic communication and connectivity; patient support; administrative processes; and reporting and population health management.