Assessment of Approaches to Evaluating Telemedicine. 2. Focus of Evaluation


A fundamental consideration in evaluating a telemedicine application is specifying the scope or focus of evaluation. Doing so may not be as straightforward as in the case of a new drug or new medical device. In a narrow sense, an evaluation may focus on a particular store-and-forward technology or a two-way interactive television system. In a broader sense, an evaluation may encompass a full teleconsultation network, including the component technologies as well as the related facilities, protocols, staffing, reimbursement, etc. In the latter instance, telemedicine is a broadly encompassing technology, as suggested by Ohinmaa et al.:

The scope of telemedicine as a technology is considerably wider than the telecommunications equipment and systems that enable exchange of information at a distance. Telemedicine should be regarded in terms of the interaction of the equipment and the information transmitted with the activities of the health care professionals who use them, and the consequences for patients and others who are their clients (Ohinmaa et al. 1999).

Telemedicine often is a means to facilitate or transmit care, or is used in conjunction with other technologies, and thus can be difficult to assess as an independent intervention. Grigsby, Schlenker, et al. (1995) developed two conceptual models that incorporate this distinction to facilitate research on telemedicine. The first is a more narrowly focused framework for studying the efficacy of telemedicine as a diagnostic medium, based on the analysis of sensitivity and specificity to establish the accuracy of telemedicine in relation to usual care. The second model is a scheme for classifying broader telemedicine applications based on processes of care rather than on specialties or disorders; this is intended to account for such variables as cost, access, acceptability, and effects on practice patterns.

Farand (1997) conducted a study designed to examine, in part, the "clinical problem-solving processes in the context of a telemedical consultation, in order to verify to what extent the technological environment preserves the characteristics of medical reasoning that are known to occur in more traditional clinical settings." This represents a more unusual approach by focusing on the clinical interaction and its consequences, rather than on the effectiveness or utility of one or more of its component technologies. Among Farand's conclusions was that an evaluation should account for the interacting problem-solving modalities that may be encountered in the context of telemedicine consultation - that is, the reasoning that a health professional may use to make a clinical determination - and the changes, if any, in the interaction between the physician and the patient.

Another dimension of evaluation focus has to do with the extent to which the findings are intended to pertain to a particular setting or telemedicine programs in general. That is, an evaluation may be designed more to yield findings with external validity or findings with internal validity. For example, a multi-center trial may be designed to demonstrate that hub-and-spoke telepsychiatry programs can increase accessibility and reduce long-term health care events and costs. Even if this is established, however, the manager of a hospital or health network must consider whether implementing such a program would be a good investment for that organization. That is, the placement of a new hub-and-spoke telepsychiatry program in a particular hospital or health network may need to demonstrate that it is technically feasible, is acceptable to clinicians and patients, and can attract enough users and realize enough revenue to meet its costs.

There is no single correct way to describe the scope or focus of a teleconsultation evaluation. In general, though, any such evaluation should specify at least the following elements, each of which can be broken down or described in greater detail:

  • health care problem(s), e.g., diagnosis of dermatological problems, diabetes management, hypertension management, psychiatry, trauma, neurosurgical emergencies;
  • patient population(s), e.g., children, non-elderly adults, elderly;
  • technology(ies), e.g., particular videoconferencing system, teleradiology system, hub-based multi-site teleconsultation network;
  • practitioners or users, including referring clinicians (e.g., general practitioners, mid-level practitioners) and consulting clinicians (e.g., radiologists, pathologists, dermatologists, surgeons);
  • setting(s) of care: e.g., ambulances and emergency room, rural clinics and university-based teaching hospital; and
  • properties (or impacts or health outcomes) to be assessed, e.g., efficacy or effectiveness, cost, cost-effectiveness, cost-utility, physician and patient satisfaction.

Describing the scope or focus of a telemedicine evaluation with elements such as these helps to strengthen claims for internal and external validity. For example, the internal validity of a controlled trial comparing a telemedicine intervention and usual care can be strengthened by specifying the particular conditions of a health care problem, patient population, setting of care, etc., thereby controlling for factors that might otherwise confound the causal effect of a telemedicine intervention on the endpoints of interest. External validity can be strengthened in that the generalizability of the findings of a particular study can be constrained or specified in terms of particular health care problems, patient populations, technologies, settings, etc.

Any evaluation should make explicit the purpose of the evaluation and the intended users or target audience of the evaluation. Knowledge of the intended users should affect the objectives and scope of the evaluation. Clinicians, health care managers, patients, payers, policy makers, and others have different interests and levels of expertise. As noted with regard to evaluation perspective, they tend to have different concerns about the effects or impacts of teleconsultation systems. They also have different needs regarding the scientific or technical level of reports, the presentation of evidence and findings, and the format of reports.

View full report


"index.pdf" (pdf, 198.19Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®