The majority of telemedicine programs are in the earliest stages of usage. As asserted by Bashshur (1998), the absence of mechanisms for reimbursement and related funding for telemedicine programs will continue to constrain the maturation of such programs, in turn preventing appropriate evaluation.
The findings from the recent literature are broken into two broad categories:
- Iterated points of the IOM evaluation framework.
- Supplementary points of the IOM evaluation framework.
In general, little new information is offered by the recent literature. Most often, articles restate or echo the findings of the IOM framework. Given that the IOM framework is a comprehensive study, includes an extensive review of the literature, and was completed in 1995, this is not unexpected.
Several broad issues arose from the literature review that could add to the IOM framework, and which are incorporated into the present report. First, evaluations of telemedicine should take into account the maturity of the program being evaluated (e.g., pilot versus a "steady state" programs). Second, integrated into any evaluation should be a more substantial and specific cost-effectiveness analysis to adequately take into account the unique nature of telemedicine applications. Third, an appropriately rigorous methodology should be applied to the evaluative process to ensure that the data gathered are useful to the health care community and those that it serves, providing evidence-based findings that can be used to support coverage decisions as appropriate. Finally, a staged approach to evaluation, similar to that used for pharmaceuticals, is suggested. These four points are addressed in following sections.
Among the points arising in the recent literature that reinforce the IOM framework are: 1) the need for a sensitivity analysis to take into account potential changes in the applications, conditions of use or cost of a technology, and how these might affect outcomes or costs of interest, 2) the necessity of developing appropriate outcomes, and 3) the unique challenges to developing an evaluation of a telemedicine program. These points are addressed below.
-
1. Iterated Points of the IOM Evaluation Framework
-
Much of the literature on evaluation of telemedicine written since the IOM report has concurred with or further elucidated the information provided in the 1995 framework. Broad categories raised in the recent literature that enhance the IOM framework include:
- development of appropriate outcomes,
- the necessity of a sensitivity analysis, and
- challenges inherent in setting up an evaluation of a telemedicine program.
These points offer initial guidance in further developing a conceptual framework to supplement the IOM evaluative framework.
a) Development of appropriate outcomes
The issue of choosing the appropriate outcomes is addressed in the IOM framework, though the issue may be approached from several new perspectives not directly addressed by that report. The recent literature echoes outcomes noted in the IOM framework, such as clinical outcomes, health outcomes, patient and provider satisfaction, long-term versus intermediate outcomes, and others.
A point raised in the recent literature addresses more specifically the level at which the outcomes are assessed. Several articles call for a move away from assessment of individual technologies toward assessment of how a telemedicine program would work at the health care system level, or societal level. Bashshur (1998) argues that "a number of these [technology specific] issues are no longer of concern .... the question of clinical safety should be put to rest." Siwicki (1997) concurs, arguing that the technology behind the medicine has been adequately demonstrated. What is needed now is "a vast number of legitimate, in-depth studies that spell out that telemedicine delivers quality health care that is cost-effective."
Taylor (1998) addresses at length the issue of appropriate outcomes. Levels of assessment may include an improvement in the well-being of a population, a reduction in the costs of providing a service, an increase in the knowledge of general practitioners, an improvement in the quality of information received, or increased patient compliance. Taylor's general argument is that evaluations of specific technologies and pilots are anecdotal and do not greatly increase the level of knowledge with regard to system evaluation. More useful, Taylor argues, is an assessment of the effects of telemedicine systems, rather than the more narrowly focused assessments of individual technologies.
b) The necessity of sensitivity analysis
As mentioned in the IOM discussion and restated several times in the literature, telemedicine is a dynamic field. Technology is constantly improving, and new, sometimes unintended applications are continually arising. However, as in the evaluation of any technology, a static evaluation may be obsolete by the time it is completed. Given this situation, it is essential to integrate into any evaluation a sensitivity analysis that would attempt to account for such potential changes in the applications, conditions, use, or costs of telemedicine technology, and how these might affect outcomes or costs of interest, as well as other unintended uses and consequences (Bashshur 1998, Crowe 1998, Sisk and Sanders 1998). Further, an ongoing evaluation built into a telemedicine program may be most effective in assessing the true success or lack of success of a maturing program. Such a sensitivity analysis reflects the essential principles for an evaluation framework.
c) Challenges to developing an evaluation of a telemedicine program
Mintzer (1997) proposes a number of lessons learned from analysis of a program involving thirteen telemedicine networks funded in 1994 by the HRSA Office of Rural Health Policy's Rural Telemedicine Program. While many of these points do not have direct implications for evaluation of telemedicine, they collectively represent the necessary backdrop for conducting an effective evaluation. Knowledge of these challenges integrated into an evaluative framework would allow those conducting evaluations to account for, and possibly avoid, similar pitfalls. These include the following, although some of these points were also made in the IOM framework.
- Expect to expend considerable effort in training and convincing practitioners to try telemedicine.
- Utilization is as likely to be initiated by specialists as by rural practitioners.
- Look for non-conventional clinical applications.
- Conduct a thorough needs assessment and have regularly scheduled telemedicine clinics.
- Transmission costs are high and need to be factored into long-term plans for sustaining a telemedicine network.
- Confidentiality and privacy may be bigger concerns in theory than in practice.
This same article contains a comparison of start-up challenges versus operational challenges. While start-up issues centered on delays in obtaining equipment, phone connections, and properly working hardware and software, operational issues included problems such as equipment residing in inaccessible areas (e.g., far away from emergency room staff), or off-hours inaccessibility (i.e., equipment located in a room that is normally locked during the night shift).
Other start-up issues cited by Mintzer included provider reluctance to use telemedicine and lack of provider comfort with equipment. An article written as an interim report of a telehome health project evaluation cited as additional barriers the concerns among staff that the technology would replace the nurses, and that the physical distance between patient and provider would threaten their professional relationship with patients (Johnston 1997). To alleviate some of these concerns, those conducting the study implemented a communications plan to keep staff involved in project development, and attempt to preempt misperceptions regarding the program.
As mentioned, these lessons learned should be viewed as precursors to carrying out an evaluation of telemedicine that will both be implemented and executed in an efficient fashion and yield worthwhile results.
-
-
2. Supplementary Points to the IOM Evaluation Framework
-
Several main issues were raised in the recent literature that supplement the IOM framework in important ways. These points are outlined below.
a) Pilot Versus "Steady State" Evaluation
A telemedicine program in the very early stages of maturity will have very different costs and results than a program that has matured to a steady state. Evaluation of a telemedicine program past the initial pilot phase and into the steady state phase of implementation is necessary to assess fully and accurately the viability of such a program. As one measure, the costs associated with a pilot program are different, and often greater, than those associated with a more mature program. In some cases, this is due to the lack of economies of scale, or early cost burdens associated with extensive training, staff familiarization, and equipment set-up. As Crowe (1998) states regarding communication costs, "The collection of data on communication costs, often the major part of system costs, in a pilot telemedicine project may not necessarily reflect the costs likely to be incurred in a mature telemedicine system." Mintzer (1997) cites other challenges of evaluating a pilot telemedicine program versus a mature program. These include:
- delays in obtaining equipment;
- delays in getting telephone connections made;
- equipment and software technical difficulties;
- training of new or inexperienced staff;
- provider reluctance to use telemedicine; and
- developing comfort among staff who will be using the telemedicine.
The point at which a program matures into a steady state varies for each program. Further, it may be difficult to place a time limit on the pilot phase, as it has been shown that even two years may not be enough for a program to reach a steady state (Mintzer 1997). As a supplement to the steps for evaluation planning, this aspect of telemedicine evaluation should be taken into account to ensure reliable outcomes.
b) Cost-effectiveness Evaluation
Because cost structures and expenditures change over time, the issue of how to best carry out a cost-effectiveness evaluation of telemedicine is closely related to evaluation of programs at the pilot versus mature stage. Crowe provides extensive consideration of this topic in a cost-effectiveness analysis of telemedicine published in 1998. Specifically addressing the issue of evaluation of pilot versus mature programs, Crowe states, "There is a problem that a health-related telemedicine service may be evaluated in isolation as a pilot project, but, as a mature service, may be integrated with other services such as tele-education and telebanking for a rural community." Exhibit 1 provides a breakdown of cost types, as according to the article by Crowe (1998).
Cost Type Cost Elements Exhibit 1: Cost Types for Telemedicine Evaluation Project establishment costs -
Preparation of submissions for funding approval
-
Selection processes to decide which projects are to proceed
-
Recruitment of staff
-
Feasibility studies
-
Preparation of tenders for equipment
-
Selection and installation of equipment
-
Revision of organizational arrangements
-
consultation with staff
-
Training of staff in new systems and procedures and in use of equipment
-
Establishment of an evaluation framework involving procedures for the collection and analysis of data for both the status quo and the new initiative and often involve computer staff
Equipment costs -
Computers and associated hardware (modems and video boards)
-
Videoconferencing and document display software
Maintenance costs -
Suggested that maintenance charges be calculated at 10-15% per year of the capital cost of the equipment
-
Travel times and costs
-
Downtime loss
Communication costs - Because of economics of scale, communications costs should decrease substantially in a mature program
Staffing costs - A successful telemedicine program (in a steady state) should make demands on staff time less, and should therefore cost less
- Suggested that an hourly rate is used for staff specialists and an appropriate fee for visiting consultants
Source: Crowe (1998) Sisk and Sanders (1998) also address the issue of cost-effectiveness analysis of telemedicine programs, citing the need to specify the full range of actual alternatives and the unique barriers to cost-effectiveness analysis: "Multiple uses of a telemedicine system may have joint costs that are difficult to apportion to one service, the existence of a system may lead to expanded indications of use, and technological change may make an evaluation outdated." Sisk and Sanders outline some of the potential cost implications of a telemedicine program, briefly summarized here as follows:
- any savings and expenditures incurred in treating a patient earlier in the course of the condition;
- changes in the productivity of health professionals;
- patient time saved; and
- changes in transportation costs.
The above costs and benefits of a telemedicine program accrue both to society, in general, and to the party responsible for payment of the relevant health care services, in particular. However, the costs and benefits to payers of telemedicine are particularly dynamic, as changing times in the health insurance marketplace illuminate opportunities for savings among payers. Historically, a lack of insurance coverage for telemedicine services has been an impediment to adoption with fee-for-service payment. Under capitation payment and fixed budgets, however, providers have financial incentives to use the most efficient method to deliver services. With the expansion of integrated health care delivery systems and such capitated payment arrangements, plans and providers are likely to weigh a broader range of costs against potential benefits in deciding about investments in telemedicine. While some of these points are made within the IOM framework, the articles by Sisk and Crowe add value to the framework by considering these points from an economic analysis standpoint.
c) Rigorous Methodology
In the context of higher standards for evidence-based health care, relatively few studies have been conducted that apply a rigorous methodology to the study of telemedicine. This is a necessary first step in developing a framework for evaluating telemedicine programs.
Problems mentioned in the literature include small sample sizes, flawed study design, and inaccurate and imprecise measurement (Bashshur 1998). Suggestions to improve current methodology include pooling of data across programs, using randomized controlled trials (RCTs), and case control studies with relevant meta-analyses (Yellowlees 1998). Nitzen et al. (1997) attempted to ensure methodological rigor by establishing a gold standard, requiring that each patient be examined by multiple physicians, conducting the in-person visits and teleconsults within a very short time span, and conducting matched-pair analyses on all study data. Finally, the researchers calculated kappa coefficients, both for comparison of their findings with other studies and as a check on their success in reducing bias in the study design (Nitzen et al. 1997).
In the first of a two-part series, Taylor (1998) proposes a comprehensive set-up of telemedicine evaluation. The article broadly outlines an evaluation of telemedicine consisting of three phases:
1. identification of the technical specification of equipment required for the particular telemedicine application;
2. tests to ensure that the evaluation is being conducted in the appropriate settings; and
3. establishment of a set of standards and guidelines to ensure that the telemedicine system is used to the best advantage.
By considering a specific study (which is generalized here), the evaluative process specifics are broken into four elements, each of which has key issues associated with it, as summarized in Exhibit 2.
Element Key Issues Exhibit 2: Elements and Key Issues of a Sample Evaluation Select a set of cases to provide a suitable basis for answering the questions of interest. - An adequate number of cases must be used.
- Awkward or difficult cases must be included.
- The range of cases should reflect the specific questions addressed in the study.
Interpret cases both 1) using telemedicine (the study condition) and 2) not using telemedicine (the control condition). - The roles of the study and control groups must be clearly distinguished.
- The situations in the study and control groups should be comparable.
- Any possibility of confounding or transfer between the conditions should be minimized.
- Subjects should be given clear instructions and, if appropriate, training in the use of the new technology.
Interpret cases to develop a "gold standard." - If it is not possible to establish a gold standard, then a design, which does not require a gold standard, may be better.
- Any effect whereby determining the gold standard systematically excludes cases should be minimized.
- If a gold standard is required, it should be established independently of the control and the study conditions.
Compare the conclusions of interpreters in the study and the control conditions to the gold standard and indices of diagnostic accuracy. - The statistical analysis used should be appropriate to the question being answered.
- The conclusions drawn should be clearly warranted by the analysis.
- Statistics should not be used unnecessarily.
Source: Taylor (1998) d) Staged Approach to Evaluation
As mentioned in the IOM framework and the recent literature, a sensitivity analysis is an essential aspect of any telemedicine evaluation. Taking this need into account, one possible approach to evaluating telemedicine programs at the technology level may be a staged approach similar to that currently in practice in the pharmaceutical industry (i.e., preclinical testing, Phases I, II, III, and IV). This is presented by DeChant et al. (1996) in an article titled "Health systems evaluation of telemedicine: a staged approach," in which comparisons are made to the method by which pharmaceuticals are developed.
DeChant et al. propose an analogous set of stages for telemedicine, and that "in each stage of the analysis, the evaluation is tailored to the technology's state of development." These results would then be used to improve the technology before dissemination occurs. The method entails addressing to varying degrees the three primary concerns outlined in the IOM framework: quality, access, and cost. DeChant et al. argue that not all three of these components would play a role in each stage of the process, but should be considered only as appropriate. The intent is not only to adopt the evaluation to the maturity of the technology, as appropriate, but to integrate into this methodology aspects from the evaluation of pharmaceuticals in order to better "capture telemedicine's potential to produce system-wide change." Such a staged approach may provide a method for taking into account the maturation of telemedicine technologies over time, and the resulting effects on important outcomes.
e) Other Issues
Other issues that were included in the recent literature, but not expanded upon extensively include the following.
- Sisk and Sanders (1998) raised the issue of economic discounting when conducting evaluations. This "reflects the fact that people place a higher value on events in the present than in the future, and that funds (or effort) invested in the present can reap interest over time." While this is a somewhat less critical point in conducting an evaluation and is not specific to telemedicine, evaluators of telemedicine should be aware of and take into account this added factor.
- Bashshur (1998) and others point out that a "triage system" may be necessary to avoid potential over-utilization of telemedicine technology. This should include establishment of specific telemedicine-related protocols to reduce arbitrary or frivolous use of the technology. While this is primarily a program development issue, an implication for evaluation is the determination of appropriateness and necessity of technology utilization. That is, one aspect of an evaluation should be whether telemedicine is being used in an appropriate fashion, and when necessary. Appropriateness evaluation may be done retrospectively (e.g., through medical record review) or prospectively, as a method to supplement clinical decision making.
- Farand et al. (1997) conducted a study designed to examine, in part, the "clinical problem-solving processes in the context of a telemedical consultation, in order to verify to what extent the technological environment preserves the characteristics of medical reasoning that are known to occur in more traditional clinical settings." This represents a somewhat unique approach to evaluating telemedicine, focusing on the actual interaction and subsequent consequences of the interaction rather than the effectiveness or utility of a technology. They concluded that an evaluation should take into account the interacting problem-solving modalities that may be encountered in the context of telemedicine consultation, that is, the reasoning that a health professional may use to make a clinical determination, and the changes, if any, in the interaction between the physician and the patient.
-
View full report
"index.pdf" (pdf, 198.19Kb)
Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®