There are several approaches for evaluating consumer health tools. The first tactic is to conduct an evaluation of their quality relative to pre-established criteria, such as accuracy of information or ease of use. A second method is to gauge user satisfaction. Building on consumer satisfaction surveys are surveys that ask individuals whether they changed their behavior because of exposure to the tool. The final and most compelling—although also the most difficult to come by—type of evaluation involves randomized control trials and evidence of improved health outcomes or reduced costs or avoidable medical use. This section will discuss each of these approaches in turn.
Quality - There are several different dimensions of health-website quality that are discussed in the literature:
- Accuracy (evidence-based, up-to-date, peer-reviewed)
- Credibility and transparency of source (including listing conflicts of interest)
- Ease of use (reading level appropriate for audience, readable format, easy navigation)
- Privacy and security of personal data
As is the case with research on prevalence rates, the most data on the quality of web tools are available related to websites that provide health information. About 70% of studies included in one meta-analysis identified problems with quality of information on the internet, with only 9% coming to a positive conclusion and the rest staying neutral. Problems with accuracy and lack of completeness were often cited by the authors of the studies in that analysis. The authors point out that there is also a good deal of inaccurate or misleading information about health presented in other media, like television, and that internet sites on other topics also often have quality problems.
One potential indicator of quality is whether a site explains where its information is coming from. A very small portion of websites that discuss health topics include the source and date of that information, and only about one-fifth to one-quarter of people who look for health information online report that they look for the source at least most of the time. However, some research suggests that there is no correlation between having the source and date on a page and the quality of the information contained on it. Another discouraging finding comes from a study of websites devoted to smoking cessation, which found that the three sites that were deemed of highest quality were viewed by only 7% of individuals who visited websites on those topics. Health web sites can seek accreditation by the nonprofit organization URAC to certify that the content meets certain standards, but it is not a universal practice.
Although a substantial number of quantitative studies have sought to evaluate the quality of health-related information online, there is little consensus about the criteria that should be used to define a high-quality site. Because of this, as well as the lack of rigor of some studies and the fact that different studies focus on information about different health topics, it is difficult to draw concrete conclusions about the overall quality of health information online.
Satisfaction - Several studies have revealed high levels of satisfaction about the information found online. For example, one survey found that 55% of people who sought information online to help them make treatment decisions were very satisfied with the information they found and another 38% were somewhat satisfied. In another study looking broadly at searches about health topics, 70% said they felt satisfied with the information their search revealed, with an equal share saying that after they went online they were more comfortable with the information that a health provider had given them. Yet, some individuals report negative experiences, with one-quarter reporting feeling overwhelmed by the quantity of online health information and slightly smaller shares expressing frustration (22%) and confusion (18%). Some evidence suggests that individuals’ experience with online health information gets worse with age.
Moving beyond looking for health information, most studies suggest that people generally report high levels of satisfaction for a range of e-health tools. For example, in several studies the percentage of people reporting they were satisfied with tools that are meant to encourage healthy behaviors, such as smoking cessation or nutrition sites, was in the 90s.
Surveys of users of provider portals also reveal a good amount of satisfaction. Several portals report that more than 90% of surveyed users are satisfied with the sites. Particularly high percentages of users said they were very satisfied with ability to refill medications (81%), engage in secure messaging with providers (65%), and view test results (55%). It is important to note that most of these analyses are only looking at people who decide to enroll in the patient portals, which may introduce self-selection bias and inflate satisfaction ratings.
Self-reported impact on consumers - In one study, 42% of respondents said that the information they found online in their most recent search had no impact on the health care they received or that they helped others with; only 11% reported that it had a major impact. (The percentage of searchers who experienced a major impact is slightly higher among those who reported a serious diagnosis or health crisis than among those who did not.) Roughly 20% of respondents to a survey about one provider portal said they had changed their behavior because of the suite of tools offered through their provider. One online study found that approximately 65% of respondents reported being more compliant with the treatments prescribed by their doctors because of their online health activities.
Health IT can affect health treatment by altering the communication between patients and health care providers. Depending on the survey, between one-third and a little over one-half of people who look for information online say that they discuss that information with their health care providers. However, another study found that fewer than 10% said that using the internet or email affected the number of doctor visits or phone calls in the previous year.
There is little evidence suggesting that access to publicly reported quality information on providers succeeds in steering patients to providers of a higher caliber. (Some studies suggest such reporting does improve the quality of care at hospitals that had been under-performing.) One study looked more broadly at self-reported behavior upon learning about providers and treatments (not necessarily quality data). Roughly one-third of people who found information about available treatments or their costs said that information affected the treatment they received.
All of these findings should be viewed cautiously as they represent perceived changes among individuals who participate in a survey.
Randomized trials and outcomes - In contrast to the efforts described above, some research has more rigorously examined clinical or other outcomes of specific computer-based applications. A 2006 overview of research in this area reached several broad conclusions:
- There are a variety of tools related to a range of health topics that have been linked with positive outcomes in randomized control studies, but the evidence on improved health outcomes is not as consistent as research on self-reported increases in understanding about conditions or improved attitudes and intentions to change behavior.
- The tools that tend to be used the most frequently are not often the ones that have undergone rigorous evaluation by third parties; the latter are typically developed as part of research projects and not widely available to the public. In particular, although there are many commercial weight loss sites available online, there are limited data on their effectiveness.
- Tools to encourage behavior change that are tailored to match how ready individuals are to change or the user’s initial behavior and knowledge, have had positive results.
- With some exceptions, many studies rely on convenience samples and include a disproportionately high number of white women with higher education. Several studies have found that individuals with less education or minorities might benefit more than those with more education.
A meta-analysis of studies regarding computer-based applications that include both health information and either social support, decision support, or behavior-change support, found evidence of positive improvements on knowledge, social support, and clinical outcomes, and mixed results on behavior change outcomes. A summary of research comparing the ability of web-based interventions to affect behavior change relative to the ability of non-web-based interventions, found that 16 of the 17 studies reviewed were in favor of web-based approaches. (However, the effect was only significant in six studies.)
In addition to looking at clinical outcomes, some studies have focused on financial implications. For example, one randomized clinical study found that a small sample of children and adolescents reduced their emergency and urgent visits after playing a video game that included education about diabetes management. Another study noted that a managed care organization discovered that hospitals’ use of remote disease monitoring devices allowed patients with chronic heart failure to return home sooner and reduced the number of cardiac events, ultimately yielding a nearly 175% return on investment. Remote monitoring might also save costs if it allows patients to assume control over tasks that would otherwise need to be performed by health care providers. Additionally, there is a small amount of evidence suggesting that decision aids may reduce the likelihood of invasive procedures, while at the same time improving consumer knowledge and expectations of harms and benefits, and decreasing their internal conflict about the decision.
Because of the novelty of PHRs, there is little empirical evidence about their financial benefits or the extent to which they improve the quality of care.