As noted above, limitations in the number and rigor of evaluations in the field of telemedicine prompted the NLM to request that the IOM develop a broad framework for such evaluation. In 1996, based on the deliberations of a 15-member expert committee, the IOM released a report intended to encourage evaluations that would guide policymakers, reassure patients and clinicians, inform health policy managers, and help those who had invested in telemedicine to identify shortcomings in, and improve upon, their programs.
The IOM framework identified the following four main components as being essential to the design of a telemedicine evaluation.
- Evaluation principles
- Steps for evaluation planning
- Elements of an evaluation
- Evaluation questions (five categories)
The key points and questions of the each of the main components of the IOM framework are listed below.
1. Evaluation Principles
- Evaluation should be viewed as an integral part of program design, implementation, and redesign.
- Evaluation should be understood as a cumulative and forward-looking process for building useful knowledge and as guidance for program or policy improvement rather than as an isolated exercise in project assessment.
- The benefits and costs of specific telemedicine applications should be compared with those of current practices or reasonable alternatives.
- The potential benefits and costs of telemedicine should be broadly construed to promote the identification and measurement of unexpected and possibly unwanted effects and to encourage an assessment of overall effects on all significant strategies.
- The accent should be on identifying the least costly and most practical ways of achieving desired results rather than investigating the most exciting or advanced telemedicine options.
- By focusing on the clinical, financial, and social objectives and needs of those who may benefit or suffer from telemedicine, evaluations can avoid excessive preoccupation with the characteristics and demands of individual technologies.
2. Steps for Evaluation Planning
- Establish evaluation objectives.
- Set priorities for the selection of specific applications to be evaluated.
- Assess the probable feasibility of an evaluation, including the availability of adequate funding and the likelihood of adequate cooperation from relevant parties.
- Identify the particular intervention to be evaluated, the alternatives to which it will be compared, the outcomes of interest, and the level and timing of evaluation.
- Specify the expected relationships between interventions and outcomes and the other factors that might affect these relationships.
- Develop an evaluation strategy that includes a credible and feasible research design and analysis plan.
3. Elements of an Evaluation
- Project description and research question(s). The description identifies the application being evaluated and the alternative to which it is being compared. Research questions are to serve as the link between the program intervention and desired outcomes.
- Strategic objectives. State the intended effects of the project on the organization's or sponsor's goals and how the evaluation strategy relates to these goals.
- Clinical objectives. State the intended effects of the project on the individual or population health by changing the quality, accessibility, or cost of care.
- Project management plan or business plan. A management plan functions to outline project's leadership and management structures, its workplan and schedule, and its budget; while a business plan is ideally more extensive and incorporates a detailed financial analysis and appraisal of the program's fit with the organization's strategic plan.
- Level and perspective of evaluation. Perspectives may be clinical, institutional, or system/societal.
- Research design and analysis plan. These evaluation elements must take into consideration the following: (1) characteristics of experimental and comparison groups; (2) technical, clinical, and administrative processes; (3) measurable outcomes; and (4) sensitivity analysis.
- Documentation of methods and results.
4. Evaluation Questions
- Evaluating Quality of Care and Health Outcomes
- What were the effects of the telemedicine application on the clinical process of care compared to the alternative(s)?
- What were the effects of the telemedicine application on immediate, intermediate, or long-term health outcomes compared to the alternative(s)?
- Evaluating Access to Care
- Did telemedicine affect the use of services or the level or appropriateness of care compared to the alternative(s)?
- Did the application affect the timeliness of care or the burden of obtaining care compared to the alternative(s)?
- Evaluating Health Care Costs and Cost-Effectiveness - What were the costs of the telemedicine application for participating health care providers or health plans compared to the alternative(s)?
- What were the costs of the telemedicine application for patients and families compared to the alternative(s)?
- What were the costs for society overall compared to the alternative(s)?
- How did the cost of the application relate to the benefits of the telemedicine application compared to the alternative(s)?
- Evaluating Patient Perceptions
- Were patients satisfied with the telemedicine service compared to the alternative(s)?
- Evaluating Clinician Perceptions
- Were attending and/or consulting clinicians satisfied with the telemedicine application compared to the alternative(s)?
Our report uses the IOM framework as a base and attempts to build on its points. Of the four main IOM components shown above, our report devotes greatest attention to the fourth one, as requested by the Task Monitor.