Assessment of Approaches to Evaluating Telemedicine. 1. Technological Maturity


The evaluation of technology is usually influenced, and sometimes triggered, by its progress through its lifecycle. One set of terms that is used to describe stages of technological maturity includes "future," "experimental," "investigational," "established," and "obsolete" (The Lewin Group 2000). Future technologies are in a conceptual stage, anticipated, or in the earliest stages of development. Experimental technologies are undergoing bench or laboratory testing. Investigational technologies are undergoing initial clinical evaluation with patients. Established technologies are considered to be standard or mainstream approaches to management of a particular indication or set of clinical circumstances. Obsolete, outmoded, or abandoned technologies have been superseded by other technologies or found to be ineffective or harmful.

Technology companies, state and federal regulatory agencies, payers, clinicians, and others tend to make decisions about technologies' particular junctions in their lifecycles. Indeed, the determination of a technology's stage of development may be the primary purpose of an assessment. For payers, technologies deemed experimental or investigational are usually excluded from coverage, but those that are established and fall within the set of covered benefits are typically eligible for coverage. Other legal and regulatory requirements may affect these considerations for telemedicine as well as other services, e.g., definitions of what constitutes a physician service and licensing as it pertains to out-of-state services.

There are tradeoffs inherent in the timing of evaluation. At an early stage, evaluation may curtail diffusion of a telemedicine application that is ineffective. However, as noted elsewhere in this report, the findings of an early evaluation may not be definitive or may be misleading. An investigational telemedicine application may not yet be perfected, clinicians may not have honed their skills with the technology, and its costs may not stabilized. One telemedicine program reported to us that it took approximately 1,000 cases of telemedical pathology consults for a physician to become proficient at using the technology. The application may not have been used in enough clinical or geographical circumstances to recognize its potential benefits, and its long-term impact on health outcomes and costs may not be known.

Premature evaluation of telemedicine technology may miss evolving and/or unanticipated applications of the technology. For example, we learned from our site visit to Allina Health Systems that, although the intended primary application of its telemedicine system was to be teleconsultations, the use of the system for continuing education for clinicians and other providers and managers has emerged as a major application. As noted by various of our interviewees, ongoing program evaluation can be integrated into planning and adjusting of telemedicine programs over time.

Some observers consider that telemedicine in the form of video-based teleconsultations facilities will be eclipsed by other applications. In its 2010 forecast, the Institute for the Future anticipates that:

(Telemedicine) over dedicated videoconferencing facilities .... will stay a fringe activity, but some telemedicine will transmute into the use of groupware (computer communications) to share information between care teams and the use of online environments for collaboration between clinicians. In another example, the production of digital signals from imaging equipment will dramatically increase radiologists' ability to use computers to analyze and abstract information from X-rays, MR images, and other imaging devices. -- Institute for the Future (2000)

Decision makers should recognize that studies of telemedicine applications that are prototypes or are not integrated into the health care mainstream provide only interim findings about the feasibility of an application but not how well it operates as a mature application (Ohinmaa et al. 1999). Further, as noted by DeChant et al. (1996), "Methods appropriate for mature technologies may not be suitable for emerging ones, and indeed, may risk stifling their development with premature negative conclusions."

The lack of technological or programmatic maturity has profound impacts on cost evaluations in particular. Because telemedicine often is underutilized early in deployment relative to its subsequent steady-state use, the average cost (e.g., per patient or per teleconsultation) at this early stage may appear to be unacceptably high if it is taken to represent costs of the application at a more mature stage. As noted below with respect to the time horizon of analysis, this may be compounded by methods of cost accounting for the capital equipment, facilities, and staff required for the local and remote sites involved in teleconsultations. Our respondents emphasized the value of data collection beginning at the inception of a telemedicine program, which allows tracking of effectiveness, costs, satisfaction, and other parameters over time and for setting realistic expectations for other new initiatives. However, they also stressed that critical evaluations of the success of a telemedicine application should be based to the extent possible on performance at steady-state levels. This view is consistent with that of others in the literature who address the value of distinguishing between pilot and steady-state evaluations of telemedicine (Crowe 1998; Mintzer et al. 1997).

One approach to evaluating telemedicine programs that accounts for technological maturity is taking a staged approach modeled after the paradigm used for pharmaceuticals, i.e., preclinical testing followed by evaluation at phases I, II, III, and IV. DeChant et al. (1996) propose an analogous set of stages for telemedicine where, "in each stage of the analysis, the evaluation is tailored to the technology's state of development." These results would then be used to improve the technology before it is more widely diffused. The method addresses to varying degrees the three primary elements outlined in the IOM framework of quality, access, and cost. DeChant et al. argue that not all three of these components would play a role in each stage of the process, but should be considered only as appropriate. The intent is not only to adapt the evaluation to the respective stages of maturity of the technology, but to better "capture telemedicine's potential to produce system-wide change." Further work is required to develop or adapt evaluation designs that take staged approaches commensurate with technological maturity. As suggested above, this may be analogous to stages or phases of evaluation used for other types of health care technology. However, given important differences in the nature of the technologies and their respective regulatory requirements, the particular evaluation models used for pharmaceuticals and medical devices themselves are largely inappropriate for telemedicine.

As noted elsewhere in this report, the moving target problem of technology is subject to the timing of evaluation. By the time an evaluation of a telemedicine application is conducted, reviewed, published (or otherwise disseminated), and incorporated into a clinical protocol or payment policy, its findings may be outdated by further data collection or changes in the component technologies, how the application is used, or competing technologies (Goodman 1996).

View full report


"index.pdf" (pdf, 198.19Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®