Literature Review and Environmental Scan: Evaluation of Personal Health Records Pilots for Fee-for-Service Medicare Enrollees. Current Knowledge about PHR Usability


Current best practices call for spending about 10% of a design project's budget on usability. On average, this will more than double a website's desired quality metrics and slightly less than double its intranet's quality metrics.

Jakob Nielsen

In a 2007 report entitled ‘Evidence Base for Personal Health Record Usability’, Gary Marchionini et al. stated that the number of empirical studies on PHR usability has been small. Their extensive literature review of 52 studies found only eight that focused on patients as users, while 44 focused on health care professionals. Marchionini believes, however, that the number of published studies may not be representative of the actual number of usability studies conducted. Large vendors–Microsoft (HealthVault) and Google (Google Health), for example–employ usability testing, but their results are usually proprietary, and vendors of all sizes have little incentive to publish their findings in academic journals.[132]

During the development of different PHR applications, the extent to which consumers have been engaged in usability testing has been limited, largely because of pressure for vendors to put their product on the market quickly and to manage the process within a limited budget. Many health IT developers believe it will cost them more to incorporate consumer-centric usability design into their product development. To the contrary, this type of usability testing is expected to substantially increase consumer traffic (e.g., website visitors) and user performance, providing substantial returns to the cost of usability testing.[133]

Conversely, health IT projects that forgo or de-emphasize usability testing early in the development process should expect lower usability. The usability rule, ‘test early; test often’ should be followed, as the cost of revising the IT after its release will be considerably more than if usability testing had been done during its initial development.[134] A revised product is a burden for consumers to relearn and those who tried the first version may be difficult to lure back.

Usability Testing Methodologies: Findings from Expert Discussions

NORC’s discussions found several organizations going to considerable efforts to ensure the usability of their PHRs. These include two large health systems, the Veterans Health Administration and Kaiser Permanente, the software developer, Intuit (maker of Quicken Health and Quicken Medical Expense Manager), and Whatcom County Washington State PHR system. All four of these organizations consider the consumer-centric usability testing of their PHR to be integral to their development protocol.

VHA, My HealtheVet

Usability testing of the VHA’s My HealtheVet includes full-functioning mock-ups early in the design process; one-on-one and small group testing in the laboratory (e.g., monitoring key strokes and eye movement); and other descriptive tests (users provide their thoughts with their actions tracked as they use the PHR). After the initial version is considered satisfactory, alpha and beta tests are conducted at pilot sites before its large-scale release. Web metrics are then used to monitor website usage. Additional functions are developed, tested, and released one-at-a-time. Although its guidelines and features are not geared specifically to older users, the VHA had made accommodations for older users. For example, it allots more time before the system ‘times out’ because older users generally require more time to complete a task.

Kaiser Permanente, HealthConnect

Kaiser Permanente has a key testing unit, which brings in small groups of users that undergo testing in a single day. It found it most effective to do in-depth testing with small numbers. These users are assigned tasks, filmed, and their keystrokes recorded. After this information is analyzed, the IT staff makes the necessary changes and then test again. Kaiser Permanente had initially released a version of HealthConnect that offered members the ability to make notes in their records. However, it discontinued this feature because one-in-four users did not understand that their doctor does not see the notes entered on their record and, therefore, no longer valued its use.

Intuit, Quicken Health

After it has completed user-centered design efforts, Intuit presents a low-resolution prototype to 10-15 consumers. Once this feedback has been obtained, the information is incorporated into the design of a higher resolution prototype, with the product and features drawn in more detail. Only after this is done do they begin feature-by-feature detailed usability tests. As of the winter of 2008, Quicken Health was being tested at a beta site, in order to assess its end-to-end elements. Its usability test approach was also used to develop Quicken and Turbo Tax. When asked what insights a focus group might offer, Intuit representatives said that focus groups were helpful for understanding consumers’ needs, and the extent to which a solution is desired.

Intuit has found that there are two segments of the population most interested in a health expense management application[3]: 1) healthy young families who make a lot of transactions online; and 2) recently retired persons with large medical expenses. They have done extensive testing with 50-64 year olds and have found them very engaged in these application concepts. This population receives a considerable number of medical bills and has little confidence paying their bills.

Whatcom County, SharedCare Plan

Whatcom County employed a user-centered design specialist who interviewed consumers to determine what information they wanted to track and how they would use this information. They interviewed consumers over age 50 with diabetes or congestive heart failure. Whatcom then designed a paper prototype (in Microsoft Word) and discussed the prototype with consumers, their family members, and health professionals. Changes were made to the prototype based on this feedback. After numerous iterations, the electronic screen-versions were designed in detail.

The PHR was developed in sections. Demographics, ‘about me’, diagnosis, medications, and allergies were the core pages. The medications from the hospital systems, electronic prescribing, and the advanced directives page were built later. At the request of consumers, a family history page was added, and refinements were made to the patient-controlled privacy capabilities. Online registration was then designed and tested. Whatcom County found that consumers prefer that the names of medications be written in English only (no Latin abbreviations). Consumers also requested that both the generic and brand-names names be displayed (to prevent consumers from unintentionally taking a double dose of their medications).

View full report


"litreview.pdf" (pdf, 998.47Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®