Assessment of Approaches to Evaluating Telemedicine. Executive Summary


Telemedicine is part of the expanding use of communications technology in health care, or "telehealth," being used in prevention, disease management, home health care, long-term care, emergency medicine, and other applications. The diversification of such applications and continued advances in communications technologies, including the Internet, are raising expectations for telemedicine. However, the considerable attention focused on the technological aspects of telemedicine during the last decade has been accompanied by a lack of validated or well-demonstrated approaches for evaluating telemedicine. For program funding and policy making, there is increasing need to develop and adapt evaluative frameworks for telemedicine.

In the mid-1990s, the National Library of Medicine (NLM) requested that the Institute of Medicine (IOM) develop a broad framework for telemedicine evaluation. In 1996, based on the deliberations of a 15-member expert committee, the IOM released its report, Telemedicine: A Guide to Assessing Telecommunications in Health Care. The report presented a framework built upon five main evaluation elements: 1) quality of care and health outcomes, 2) access to care, 3) health care costs and cost-effectiveness, 4) patient perceptions, and 5) clinician perceptions (IOM 1996).

Since 1996, the field of telemedicine has continued to evolve and mature. Recently, the DHHS Office of the Assistant Secretary for Planning and Evaluation (ASPE) contracted with The Lewin Group to assess current approaches to evaluating telemedicine. In particular, ASPE requested that Lewin extend or otherwise update the 1996 IOM framework for telemedicine evaluation as it applies to teleconsultations. The purpose of this study is not to evaluate telemedicine, but rather to identify the different kinds of issues on which telemedicine evaluations can focus, and the kinds of information that such evaluations can yield. This report is intended to guide future evaluators and policy makers in selecting the questions that they want to answer regarding the value of telemedicine programs, and in designing evaluations that will best serve their interests and purposes.

This report confirms and provides examples of many of the points raised by the original 1996 IOM framework. In some cases, however, this report provides greater depth or complexity, identifies supplemental issues, and diminishes the importance of ones included in the IOM framework. Some of the differences between this report and the IOM's derive from an additional four years of experience with telemedicine, including practical findings about the barriers to acceptance and use of telemedicine. The following are the main findings of this report.

1. A fundamental consideration in evaluating a telemedicine application is specifying the purpose, target audience, and the scope or focus of evaluation. Although these often are not straightforward decisions, each evaluation should specify a minimum set of elements.

2. Patient satisfaction with telemedicine has consistently been demonstrated to be high. As such, resources for future evaluations may be better allocated to areas of higher priority.

3. Lack of reimbursement for telemedicine services has been a significant confounder in past evaluations of telemedicine. Future evaluation efforts (e.g., demonstration projects) should seek to establish comparable reimbursement environments for telemedicine and the usual care comparators whenever differences in reimbursement might affect study results.

4. The findings and utility of a telemedicine evaluation are likely to be influenced by the selection of economic perspective(s) of evaluation. To be of practical use, evaluations should account for one or more of multiple relevant economic perspectives, e.g., of clinicians, patients, hospitals, payers, or society-at-large.

5. Telemedicine comprises an evolving portfolio of technologies and applications. As such, any prospective evaluation must allow for and be prepared to assess the impact (on efficacy or effectiveness, cost, cost-effectiveness, etc.) of applications that may not have been foreseen during the evaluation design.

6. Plans for evaluation of telemedicine programs should make explicit their assumptions regarding the relationship between the timing of evaluation and the maturity of the telemedicine program, and the evaluations should be designed accordingly.

7. Given the need to minimize the influence of known as well as unknown sources of bias in comparative studies involving telemedicine, it is desirable to use randomized designs whenever possible. Depending upon the investigation, it may be appropriate to randomize one or more of patients, physicians, or delivery sites. However, randomization is often impractical or impossible for evaluating telemedicine applications.

8. A recurrent weakness in telemedicine evaluations has been the lack of clearly defined control groups. In general, a comparator should be the standard or level of care that would be provided in the absence of the telemedicine intervention.

9. The time horizon for a telemedicine evaluation should be sufficiently long to capture the stream of relevant health and economic effects and to detect any differences in these effects between the intervention and control groups.

10. In order to be successful, telemedicine must be integrated as smoothly as possible into existing, routine clinical and administrative functions, including facilities, scheduling and appointments, patient records, coding, and billing.

11. Independent financial viability of a telemedicine program will increase its prospects for integration into the health care mainstream and for long-term success.

View full report


"index.pdf" (pdf, 198.19Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®