Assessment of Approaches to Evaluating Telemedicine. 2. Supplementary Points to the IOM Evaluation Framework


Several main issues were raised in the recent literature that supplement the IOM framework in important ways. These points are outlined below.

a) Pilot Versus "Steady State" Evaluation

A telemedicine program in the very early stages of maturity will have very different costs and results than a program that has matured to a steady state. Evaluation of a telemedicine program past the initial pilot phase and into the steady state phase of implementation is necessary to assess fully and accurately the viability of such a program. As one measure, the costs associated with a pilot program are different, and often greater, than those associated with a more mature program. In some cases, this is due to the lack of economies of scale, or early cost burdens associated with extensive training, staff familiarization, and equipment set-up. As Crowe (1998) states regarding communication costs, "The collection of data on communication costs, often the major part of system costs, in a pilot telemedicine project may not necessarily reflect the costs likely to be incurred in a mature telemedicine system." Mintzer (1997) cites other challenges of evaluating a pilot telemedicine program versus a mature program. These include:

  • delays in obtaining equipment;
  • delays in getting telephone connections made;
  • equipment and software technical difficulties;
  • training of new or inexperienced staff;
  • provider reluctance to use telemedicine; and
  • developing comfort among staff who will be using the telemedicine.

The point at which a program matures into a steady state varies for each program. Further, it may be difficult to place a time limit on the pilot phase, as it has been shown that even two years may not be enough for a program to reach a steady state (Mintzer 1997). As a supplement to the steps for evaluation planning, this aspect of telemedicine evaluation should be taken into account to ensure reliable outcomes.

b) Cost-effectiveness Evaluation

Because cost structures and expenditures change over time, the issue of how to best carry out a cost-effectiveness evaluation of telemedicine is closely related to evaluation of programs at the pilot versus mature stage. Crowe provides extensive consideration of this topic in a cost-effectiveness analysis of telemedicine published in 1998. Specifically addressing the issue of evaluation of pilot versus mature programs, Crowe states, "There is a problem that a health-related telemedicine service may be evaluated in isolation as a pilot project, but, as a mature service, may be integrated with other services such as tele-education and telebanking for a rural community." Exhibit 1 provides a breakdown of cost types, as according to the article by Crowe (1998).

Cost Type Cost Elements
Exhibit 1: Cost Types for Telemedicine Evaluation
Project establishment costs
  • Preparation of submissions for funding approval

  • Selection processes to decide which projects are to proceed

  • Recruitment of staff

  • Feasibility studies

  • Preparation of tenders for equipment

  • Selection and installation of equipment

  • Revision of organizational arrangements

  • consultation with staff

  • Training of staff in new systems and procedures and in use of equipment

  • Establishment of an evaluation framework involving procedures for the collection and analysis of data for both the status quo and the new initiative and often involve computer staff

Equipment costs
  • Computers and associated hardware (modems and video boards)

  • Videoconferencing and document display software

Maintenance costs
  • Suggested that maintenance charges be calculated at 10-15% per year of the capital cost of the equipment

  • Travel times and costs

  • Downtime loss

Communication costs
  • Because of economics of scale, communications costs should decrease substantially in a mature program
Staffing costs
  • A successful telemedicine program (in a steady state) should make demands on staff time less, and should therefore cost less
  • Suggested that an hourly rate is used for staff specialists and an appropriate fee for visiting consultants
Source: Crowe (1998)

Sisk and Sanders (1998) also address the issue of cost-effectiveness analysis of telemedicine programs, citing the need to specify the full range of actual alternatives and the unique barriers to cost-effectiveness analysis: "Multiple uses of a telemedicine system may have joint costs that are difficult to apportion to one service, the existence of a system may lead to expanded indications of use, and technological change may make an evaluation outdated." Sisk and Sanders outline some of the potential cost implications of a telemedicine program, briefly summarized here as follows:

  • any savings and expenditures incurred in treating a patient earlier in the course of the condition;
  • changes in the productivity of health professionals;
  • patient time saved; and
  • changes in transportation costs.

The above costs and benefits of a telemedicine program accrue both to society, in general, and to the party responsible for payment of the relevant health care services, in particular. However, the costs and benefits to payers of telemedicine are particularly dynamic, as changing times in the health insurance marketplace illuminate opportunities for savings among payers. Historically, a lack of insurance coverage for telemedicine services has been an impediment to adoption with fee-for-service payment. Under capitation payment and fixed budgets, however, providers have financial incentives to use the most efficient method to deliver services. With the expansion of integrated health care delivery systems and such capitated payment arrangements, plans and providers are likely to weigh a broader range of costs against potential benefits in deciding about investments in telemedicine. While some of these points are made within the IOM framework, the articles by Sisk and Crowe add value to the framework by considering these points from an economic analysis standpoint.

c) Rigorous Methodology

In the context of higher standards for evidence-based health care, relatively few studies have been conducted that apply a rigorous methodology to the study of telemedicine. This is a necessary first step in developing a framework for evaluating telemedicine programs.

Problems mentioned in the literature include small sample sizes, flawed study design, and inaccurate and imprecise measurement (Bashshur 1998). Suggestions to improve current methodology include pooling of data across programs, using randomized controlled trials (RCTs), and case control studies with relevant meta-analyses (Yellowlees 1998). Nitzen et al. (1997) attempted to ensure methodological rigor by establishing a gold standard, requiring that each patient be examined by multiple physicians, conducting the in-person visits and teleconsults within a very short time span, and conducting matched-pair analyses on all study data. Finally, the researchers calculated kappa coefficients, both for comparison of their findings with other studies and as a check on their success in reducing bias in the study design (Nitzen et al. 1997).

In the first of a two-part series, Taylor (1998) proposes a comprehensive set-up of telemedicine evaluation. The article broadly outlines an evaluation of telemedicine consisting of three phases:

1. identification of the technical specification of equipment required for the particular telemedicine application;

2. tests to ensure that the evaluation is being conducted in the appropriate settings; and

3. establishment of a set of standards and guidelines to ensure that the telemedicine system is used to the best advantage.

By considering a specific study (which is generalized here), the evaluative process specifics are broken into four elements, each of which has key issues associated with it, as summarized in Exhibit 2.

Element Key Issues
Exhibit 2: Elements and Key Issues of a Sample Evaluation
Select a set of cases to provide a suitable basis for answering the questions of interest.
  • An adequate number of cases must be used.
  • Awkward or difficult cases must be included.
  • The range of cases should reflect the specific questions addressed in the study.
Interpret cases both 1) using telemedicine (the study condition) and 2) not using telemedicine (the control condition).
  • The roles of the study and control groups must be clearly distinguished.
  • The situations in the study and control groups should be comparable.
  • Any possibility of confounding or transfer between the conditions should be minimized.
  • Subjects should be given clear instructions and, if appropriate, training in the use of the new technology.
Interpret cases to develop a "gold standard."
  • If it is not possible to establish a gold standard, then a design, which does not require a gold standard, may be better.
  • Any effect whereby determining the gold standard systematically excludes cases should be minimized.
  • If a gold standard is required, it should be established independently of the control and the study conditions.
Compare the conclusions of interpreters in the study and the control conditions to the gold standard and indices of diagnostic accuracy.
  • The statistical analysis used should be appropriate to the question being answered.
  • The conclusions drawn should be clearly warranted by the analysis.
  • Statistics should not be used unnecessarily.
Source: Taylor (1998)

d) Staged Approach to Evaluation

As mentioned in the IOM framework and the recent literature, a sensitivity analysis is an essential aspect of any telemedicine evaluation. Taking this need into account, one possible approach to evaluating telemedicine programs at the technology level may be a staged approach similar to that currently in practice in the pharmaceutical industry (i.e., preclinical testing, Phases I, II, III, and IV). This is presented by DeChant et al. (1996) in an article titled "Health systems evaluation of telemedicine: a staged approach," in which comparisons are made to the method by which pharmaceuticals are developed.

DeChant et al. propose an analogous set of stages for telemedicine, and that "in each stage of the analysis, the evaluation is tailored to the technology's state of development." These results would then be used to improve the technology before dissemination occurs. The method entails addressing to varying degrees the three primary concerns outlined in the IOM framework: quality, access, and cost. DeChant et al. argue that not all three of these components would play a role in each stage of the process, but should be considered only as appropriate. The intent is not only to adopt the evaluation to the maturity of the technology, as appropriate, but to integrate into this methodology aspects from the evaluation of pharmaceuticals in order to better "capture telemedicine's potential to produce system-wide change." Such a staged approach may provide a method for taking into account the maturation of telemedicine technologies over time, and the resulting effects on important outcomes.

e) Other Issues

Other issues that were included in the recent literature, but not expanded upon extensively include the following.

  • Sisk and Sanders (1998) raised the issue of economic discounting when conducting evaluations. This "reflects the fact that people place a higher value on events in the present than in the future, and that funds (or effort) invested in the present can reap interest over time." While this is a somewhat less critical point in conducting an evaluation and is not specific to telemedicine, evaluators of telemedicine should be aware of and take into account this added factor.
  • Bashshur (1998) and others point out that a "triage system" may be necessary to avoid potential over-utilization of telemedicine technology. This should include establishment of specific telemedicine-related protocols to reduce arbitrary or frivolous use of the technology. While this is primarily a program development issue, an implication for evaluation is the determination of appropriateness and necessity of technology utilization. That is, one aspect of an evaluation should be whether telemedicine is being used in an appropriate fashion, and when necessary. Appropriateness evaluation may be done retrospectively (e.g., through medical record review) or prospectively, as a method to supplement clinical decision making.
  • Farand et al. (1997) conducted a study designed to examine, in part, the "clinical problem-solving processes in the context of a telemedical consultation, in order to verify to what extent the technological environment preserves the characteristics of medical reasoning that are known to occur in more traditional clinical settings." This represents a somewhat unique approach to evaluating telemedicine, focusing on the actual interaction and subsequent consequences of the interaction rather than the effectiveness or utility of a technology. They concluded that an evaluation should take into account the interacting problem-solving modalities that may be encountered in the context of telemedicine consultation, that is, the reasoning that a health professional may use to make a clinical determination, and the changes, if any, in the interaction between the physician and the patient.

View full report


"index.pdf" (pdf, 198.19Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®