Assessment of Approaches to Evaluating Telemedicine. 1. Iterated Points of the IOM Evaluation Framework


Much of the literature on evaluation of telemedicine written since the IOM report has concurred with or further elucidated the information provided in the 1995 framework. Broad categories raised in the recent literature that enhance the IOM framework include:

  • development of appropriate outcomes,
  • the necessity of a sensitivity analysis, and
  • challenges inherent in setting up an evaluation of a telemedicine program.

These points offer initial guidance in further developing a conceptual framework to supplement the IOM evaluative framework.

a) Development of appropriate outcomes

The issue of choosing the appropriate outcomes is addressed in the IOM framework, though the issue may be approached from several new perspectives not directly addressed by that report. The recent literature echoes outcomes noted in the IOM framework, such as clinical outcomes, health outcomes, patient and provider satisfaction, long-term versus intermediate outcomes, and others.

A point raised in the recent literature addresses more specifically the level at which the outcomes are assessed. Several articles call for a move away from assessment of individual technologies toward assessment of how a telemedicine program would work at the health care system level, or societal level. Bashshur (1998) argues that "a number of these [technology specific] issues are no longer of concern .... the question of clinical safety should be put to rest." Siwicki (1997) concurs, arguing that the technology behind the medicine has been adequately demonstrated. What is needed now is "a vast number of legitimate, in-depth studies that spell out that telemedicine delivers quality health care that is cost-effective."

Taylor (1998) addresses at length the issue of appropriate outcomes. Levels of assessment may include an improvement in the well-being of a population, a reduction in the costs of providing a service, an increase in the knowledge of general practitioners, an improvement in the quality of information received, or increased patient compliance. Taylor's general argument is that evaluations of specific technologies and pilots are anecdotal and do not greatly increase the level of knowledge with regard to system evaluation. More useful, Taylor argues, is an assessment of the effects of telemedicine systems, rather than the more narrowly focused assessments of individual technologies.

b) The necessity of sensitivity analysis

As mentioned in the IOM discussion and restated several times in the literature, telemedicine is a dynamic field. Technology is constantly improving, and new, sometimes unintended applications are continually arising. However, as in the evaluation of any technology, a static evaluation may be obsolete by the time it is completed. Given this situation, it is essential to integrate into any evaluation a sensitivity analysis that would attempt to account for such potential changes in the applications, conditions, use, or costs of telemedicine technology, and how these might affect outcomes or costs of interest, as well as other unintended uses and consequences (Bashshur 1998, Crowe 1998, Sisk and Sanders 1998). Further, an ongoing evaluation built into a telemedicine program may be most effective in assessing the true success or lack of success of a maturing program. Such a sensitivity analysis reflects the essential principles for an evaluation framework.

c) Challenges to developing an evaluation of a telemedicine program

Mintzer (1997) proposes a number of lessons learned from analysis of a program involving thirteen telemedicine networks funded in 1994 by the HRSA Office of Rural Health Policy's Rural Telemedicine Program. While many of these points do not have direct implications for evaluation of telemedicine, they collectively represent the necessary backdrop for conducting an effective evaluation. Knowledge of these challenges integrated into an evaluative framework would allow those conducting evaluations to account for, and possibly avoid, similar pitfalls. These include the following, although some of these points were also made in the IOM framework.

  • Expect to expend considerable effort in training and convincing practitioners to try telemedicine.
  • Utilization is as likely to be initiated by specialists as by rural practitioners.
  • Look for non-conventional clinical applications.
  • Conduct a thorough needs assessment and have regularly scheduled telemedicine clinics.
  • Transmission costs are high and need to be factored into long-term plans for sustaining a telemedicine network.
  • Confidentiality and privacy may be bigger concerns in theory than in practice.

This same article contains a comparison of start-up challenges versus operational challenges. While start-up issues centered on delays in obtaining equipment, phone connections, and properly working hardware and software, operational issues included problems such as equipment residing in inaccessible areas (e.g., far away from emergency room staff), or off-hours inaccessibility (i.e., equipment located in a room that is normally locked during the night shift).

Other start-up issues cited by Mintzer included provider reluctance to use telemedicine and lack of provider comfort with equipment. An article written as an interim report of a telehome health project evaluation cited as additional barriers the concerns among staff that the technology would replace the nurses, and that the physical distance between patient and provider would threaten their professional relationship with patients (Johnston 1997). To alleviate some of these concerns, those conducting the study implemented a communications plan to keep staff involved in project development, and attempt to preempt misperceptions regarding the program.

As mentioned, these lessons learned should be viewed as precursors to carrying out an evaluation of telemedicine that will both be implemented and executed in an efficient fashion and yield worthwhile results.

View full report


"index.pdf" (pdf, 198.19Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®