Assessment of Approaches to Evaluating Telemedicine. Main Findings


Much of the present study serves to confirm and reinforce the 1996 IOM evaluation framework. The recent literature and interviews with telemedicine providers and other telemedicine experts provided examples of, and otherwise helped to elucidate many of the points raised by that framework.

In some cases, however, the present study provides greater depth or complexity, identifies supplemental issues, and calls into question the importance of ones included in the IOM framework. Some of these differences between this report and the IOM's derive from an additional four years of experience with telemedicine, including practical findings about the barriers to acceptance and use of telemedicine.

For example, this report addresses greater depth or complexity in the matters of identifying appropriate comparators for telemedicine evaluations (particularly the crucial role of reimbursement inequities) and in the types of incentives and disincentives that arise from different economic perspectives of evaluation. Examples of supplemental issues identified in this report include the need to implement and interpret evaluations with due consideration of technological maturity and time horizons that do not produce misleading results, considerations for randomized design, the need to evaluate progress toward moving telemedicine programs into the health care mainstream, and the importance of independent financial viability as a prospective evaluation criterion. An aspect of the IOM framework that is viewed as being of lower priority is the need for continued emphasis on measuring patient satisfaction. The main findings described below incorporate and emphasize these issues of departure from the IOM framework.

As in evaluation of any health care technology, the evaluation of telemedicine can entail various combinations of properties or impacts (access, technical properties, safety, efficacy or effectiveness, cost, etc.) and methodological aspects (evaluation perspective, selection of comparator, time horizon, etc.). Among the telemedicine programs we examined and experts we interviewed, the properties or impacts viewed as being of highest evaluation priorities were patient access and "quality" (comprising some combination of technical properties, efficacy or effectiveness, and appropriateness of care). These were followed by clinician acceptance and cost and other economic impacts. Of lower priority was patient satisfaction, not because it is unimportant, but because it has so consistently been demonstrated to be high that continuing to emphasize it in evaluation would be redundant. Safety was generally regarded as not being at issue for telemedicine.

Among methodological issues, respondents emphasized the need to identify valid control groups to represent standard or usual care, as well as the challenges of doing so. Of the many factors that could confound a comparison of the impacts of a telemedicine program and standard or usual care, respondents most often pointed to differences in reimbursement, where usual care is reimbursed in a routine fashion and telemedicine services are not reimbursed at all, inadequately, or via non-routine or inefficient means.

The main findings of this report are as follows.

1. A fundamental consideration in evaluating a telemedicine application is specifying the purpose, target audience, and the scope or focus of evaluation. Although these often are not straightforward decisions, each evaluation should specify a minimum set of elements.

- Telemedicine technology often is a means to facilitate or transmit care, or is used in conjunction with other technologies; therefore, it can be difficult to assess as an independent intervention. The technological scope of an evaluation may range from a particular store-and-forward technology or a two-way interactive television system to a full teleconsultation network.

- In general an evaluation should specify at least such elements as: health care problem(s), patient population(s), technology(ies), practitioners or users, setting(s) of care, and properties (or impacts or health outcomes) to be assessed.

- Evaluations should make explicit their purposes and intended users or target audiences. Knowledge of the intended users should affect the objectives, scope, and presentation of findings of the evaluation.

2. Patient satisfaction with telemedicine has consistently been demonstrated to be high. As such, resources for future evaluations may be better allocated to areas of higher priority.

- The great majority of studies to date indicate very high levels of patient satisfaction, as patients have given virtually universal positive responses to receiving treatment to which they would otherwise not have access.

- Patient satisfaction with telemedicine may now have been over-studied. Multi-question surveys of satisfaction can be a nuisance to patients in settings where clinicians, patients, and other participants in telemedicine are seeking to establish conditions that are as routine as possible.

3. Lack of reimbursement for telemedicine services has been a significant confounder in past evaluations of telemedicine. Future evaluation efforts (e.g., demonstration projects) should seek to establish comparable reimbursement environments for telemedicine and the usual care comparators whenever differences in reimbursement might affect study results.

- Inequitable reimbursement conditions for telemedicine vs. usual care may confound findings about clinician or hospital acceptance, access, utilization, health outcomes (if dependent on utilization), and other evaluation measures. Reimbursement differences might not affect certain telemedicine evaluations, e.g., of the technical performance of a system, ease of use, or operating costs.

- The administrative process for reimbursement should be the same as it is for usual care; that is, there should not be an added administrative burden or less convenience for securing reimbursement for telemedicine services. Non-existent or separate billing procedures for telemedicine constitute further departure from the health care mainstream.

- Reimbursement inequities pose disincentives that contribute to underutilization of telemedicine services, including initial and follow-up encounters, which in turn affects determinations of their cost-effectiveness.

- Lack of conventional reimbursement procedures (e.g., capturing services data via CPT codes) can hinder data collection and evaluation.

4. The findings and utility of a telemedicine evaluation are likely to be influenced by the selection of economic perspective(s) of evaluation. To be of practical use, evaluations should account for one or more of multiple relevant economic perspectives, e.g., of clinicians, patients, hospitals, payers, or society-at-large.

- Costs and outcomes or benefits of telemedicine programs accrue differently to multiple stakeholders. Accordingly, these stakeholders have different incentive structures for participating in or supporting telemedicine. Evaluations should account for perspectives of one or more of referring and consulting clinicians, patients, hospitals, managed care organizations, third-party payers, society-at-large, or others as appropriate.

- Due in part to the relative difficulty of accounting for patients' direct non-medical and indirect costs, evaluations from their perspective have been insufficient. Given the central importance of patient participation in telemedicine applications, it is essential that evaluations from this perspective be properly undertaken.

- Physician willingness to participate in, or satisfaction with, a telemedicine program may depend upon the physician's form of compensation (e.g., salary vs. fee-for-service). Persuading third-party payers of the cost-effectiveness of telemedicine may require demonstrating that current reimbursement for teleconsultations may diminish downstream adverse health events and utilization of services.

- Determining appropriate evaluation perspectives should entail consideration of the multiple applications of many telemedicine programs, including for different health care departments and for educational and managerial purposes.

5. Telemedicine comprises an evolving portfolio of technologies and applications. As such, any prospective evaluation must allow for and be prepared to assess the impact (on efficacy or effectiveness, cost, cost-effectiveness, etc.) of applications that may not have been foreseen during the evaluation design.

- Traditional evaluation methodology stresses prospective measurement of predetermined endpoints. This approach is generally appropriate for mature technologies that have reached steady-state applications. However, this approach does not account for evolving uses of technologies, such as those used in telemedicine, that change their utility in practice.

- In many instances, the originally intended applications of telemedicine programs (e.g., teleconsultations or telepathology) have been overtaken or accompanied by other applications (e.g., continuing education or management) that originally were unanticipated or considered to be of minor importance.

- Evaluations of telemedicine programs that maintain focus only on the performance of predetermined applications (some of whose utility may be diminished during pilot or demonstration stages) may ignore alternative applications, thereby yielding findings that underestimate the actual value of the telemedicine program.

5. Plans for evaluation of telemedicine programs should make explicit their assumptions regarding the relationship between the timing of evaluation and the maturity of the telemedicine program, and the evaluations should be designed accordingly.

- Decision makers should recognize that studies of telemedicine applications that are prototypes or are not integrated into the health care mainstream provide only interim findings about the feasibility of such application, not how well they operate as mature applications.

- The lack of technological or programmatic maturity has profound impacts on cost evaluations in particular. When start-up costs are high and initial utilization is low, the cost-effectiveness of a telemedicine application may appear to be unacceptably high if it is taken to represent cost-effectiveness at a more mature stage.

- Further work is required to develop or adapt evaluation designs that take staged approaches commensurate with technological maturity. This may be analogous to stages or phases of evaluation used for other types of health care technology, although the evaluation models themselves that are used for pharmaceuticals and medical devices are largely inappropriate for telemedicine.

7. Given the need to minimize the influence of known as well as unknown sources of bias in comparative studies involving telemedicine, it is desirable to use randomized designs whenever possible. Depending upon the investigation, it may be appropriate to randomize one or more of patients, physicians, or delivery sites. However, randomization is often impractical or impossible for evaluating telemedicine applications.

- Telemedicine presents challenges to randomized design. Telemedicine interventions are not always discrete or self-contained technologies (as in the instance of many pharmaceutical therapies). The causal effects of teleconsultations may be confounded by differences among the participating physicians, among the participating delivery sites, or among other factors.

- In addition to randomizing patients to either teleconsultation or standard care, it may be desirable to randomize participating referring physicians to teleconsultations or standard care, and to randomize participating clinics to providing teleconsultations or standard care.

- For randomization to be successful, the number of patients (or other subjects) to be assigned must be large enough to achieve a high probability of evenly distributing any prognostic factors. This may not be practical for many telemedicine programs.

- If randomization of patients is not incorporated into an evaluation design, investigators should provide a rationale for this. If it is methodologically desirable but impractical to randomize physicians or delivery sites, investigators should provide a rationale for this, including their assumptions about the similarities among physicians or among the delivery sites.

8. A recurrent weakness in telemedicine evaluations has been the lack of clearly defined control groups. In general, a comparator should be the standard or level of care that would be provided in the absence of the telemedicine intervention.

- The evaluation design should specify, and justify, the comparator. For teleconsultations, alternatives might include one or more of: no care, inadequate or underspecialized in-person care locally, in-person care remotely (requiring patient travel), delayed in-person care remotely (requiring patient travel), or delayed in-person care locally (requiring physician specialist travel).

- Rather than methodologically preferable contemporaneous controls, telemedicine evaluations have too often relied on historical controls. Historical controls can be sufficient if there is strong reason to assume that prevailing conditions have not changed over time, and that the relationship between usual care and the outcomes of interest has remained virtually constant.

- Similarly, designs other than contemporaneous, randomized controls, such as matched controls or pre- and post-test designs, may be more convenient, but have methodological weaknesses. Investigators should provide rationale for using these types of control groups, and address their implications for the validity of study findings.

- As noted above, the existence of reimbursement for usual care and its absence for telemedicine services may undermine the validity of usual care as a comparator.

9. The time horizon for a telemedicine evaluation should be long enough to capture the stream of relevant health and economic effects that are sufficient to detect any differences in these between the intervention and control groups.

- Given the novelty of telemedicine applications, it is inevitable that provider institutions, physicians, patients, and other participants will require some time and practical experience to reach a steady-state of operation.

- Similarly, given the inefficiencies of resource use that arise when installing any new technology or program, the costs of operating a start-up telemedicine operation will not reflect the true, longer-term running costs of the program. The results of evaluations of cost or cost effectiveness of telemedicine operations can be very sensitive to the time span of the evaluation.

- The time horizon of a comparative evaluation of telemedicine should depend upon the endpoints or outcome measures of interest. For example, the time horizons required to determine differences in access, patient satisfaction, and the effect of early interventions on longer-term health outcomes and costs will differ.

10. In order to be successful, telemedicine must be integrated as smoothly as possible into existing, routine clinical and administrative functions, including facilities, scheduling and appointments, patient records, coding, and billing.

- Unlike most new technologies that diffuse smoothly into health care delivery, implementing telemedicine programs often presents departures from standard means of health care delivery, administration, and financing.

- Telemedicine evaluations should distinguish between any inefficiencies or lack of acceptance that are inherent in telemedicine applications themselves, as opposed to those that derive from an awkward fit between the telemedicine application and the clinical mainstream.

- In order to assess the integration of a telemedicine program into the health care mainstream, one should consider the program's impact on practice patterns, patient flow, and revenue streams, along with any resulting physician or institutional resistance to the program. These findings can contribute to modifications toward achieving better integration of programs into delivery environments.

11. Independent financial viability of a telemedicine program will increase its prospects for integration into the health care mainstream and long-term success.

- The ability of most programs to achieve financial viability will depend on their ability to secure appropriate coverage and adequate third-party payment for their services.

- The single most important evaluation criterion for any telemedicine program may be the extent to which it achieves independent financial viability.

- Proposed telemedicine programs should include multi-year business plans that describe how the program will progress toward financial viability as outside funding from grants or other temporary sources diminishes. Such plans should be considered in any grants review process, and financial status with respect to these plans should be regularly monitored.

View full report


"index.pdf" (pdf, 198.19Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®