Evaluation of the Personal Health Record Pilot for Medicare Fee-For Service Enrollees from South Carolina. Study Limitations.


The findings presented in this report have several limitations in terms of the generalizability of study findings, limitations of the MyPHRSC tool and usage data constraints. These limitations are discussed in further detail below.

Applicability of Results. The population for this study consisted of fee-for-service beneficiaries in South Carolina that self selected to use MyPHRSC. Therefore, this population is not representative of Medicare beneficiaries as a whole and the findings outlined in this report may not be applicable to the broader Medicare community.

This pilot involved only a single PHR product. Consequently, the study evaluates the experiences of beneficiaries using MyPHRSC.  There are many different types of PHRs offered by health plans, vendors and providers, and the results of this evaluation may not inform the experience of beneficiaries who are using PHRs that are tethered to Electronic Health Records (EHRs) and that are not claims-based or offer more information than claims. Additionally since the data in the PHR was generated entirely from claims there were issues of accuracy and/or reliability of the information.  Respondents to a study conducted by Grossman et al. noted that claims data had inherent weaknesses relative to clinical data[28].

Small Sample Size.  NORC experienced numerous obstacles in recruiting participants for the various evaluation activities and worked with the CMS contractor, QSSI, in all recruitment efforts. Due to challenges in recruiting participants, the sample sizes for the qualitative activities of the evaluation are limited.

Limitations of MyPHRSC.  The evaluation also revealed several limitations of the MyPHRSC tool itself. Firstly, since our population was fee-for-service beneficiaries, the PHR did not offer a comprehensive source of information to facilitate coordination of care across providers[7]. Specifically, this pilot did not involve providers and the PHR did not provide access or tools specifically for providers.  MyPHRSC did not include features to allow beneficiaries to communicate electronically with their providers, such as secure messaging or emailing. Such features may serve to address communication barriers between physicians and patients and contribute to physicians’ ability to provide a more personalized level of care. PHRs with the capability to directly communicate with providers can also serve to improve the speed and quality of communication between physicians and their patients. Additionally, MyPHRSC did include a mechanism for providers to integrate the PHR with an EHR system. Features such as these provide several advantages and can have the further effect of increasing the value of the PHR.[29][30]

An issue beneficiaries identified during discussion groups and the observational studies was that the PHR lacked comprehensive, longitudinal health information.  MyPHRSC was pre-populated with two years of claims data, however many beneficiaries desired an extended or lifetime view of their health care histories. Additionally, MyPHRSC excluded information on laboratory results, and beneficiaries indicated they desired a PHR to view their test results and track results over time. Beneficiaries participating in the pilot study indicated that auto-population of laboratory results would be a useful supplement to claims data in MyPHRSC.

During the pilot, a federal rule prohibiting the transmission of Medicare Part D data prevented access to medication data for pre-populating MyPHRSC. As a result, medications were not listed in MyPHRSC, thus requiring users to manually enter that data into the PHR. Beneficiaries participating in the pilot indicated that auto-population of medications would have been very useful and would have made the medication function less challenging to use.

Palmetto facilitated importation of CMS claims data into the PHR and identified limitations in pre-populating claims data from their existing claims system into MyPHRSC. Utility and usability could be improved in designing a PHR to incorporate a variety of data types including claims and clinical data, such as seen in electronic medical record systems.  

Provider Discussion Group.  Given that NORC only spoke with a small group of providers in South Carolina, findings from this discussion may not be applicable to all providers.  Participants in this discussion may not have represented the opinions of the whole group, or of future groups of providers.

Usage Data Constraints. The registration and usage data analysis is constrained by a number of limitations.  First, the South Carolina Medicare beneficiaries who registered for MyPHRSC self-selected into the pilot by choosing to register. This results in a biased study sample.  Beneficiaries who chose to register may have been more technologically inclined than the general beneficiary population, or may differ in other respects.  As a result, findings cannot necessarily be generalized to the entire South Carolina or broader Medicare population. 

Second, the data on individual users’ page views was too coarse for meaningful analysis.  Because page view data was available only as an aggregated quarterly total for each user, it was not possible to tie pages viewed to specific logins.  Furthermore, because so few page views exist for many users in our data collected, there is insufficient statistical power to perform comparisons in many cases.

Third, session data on individual users' navigational path (i.e., the sequence of pages viewed) when using the PHR was not available.

Fourth, information on illnesses/conditions was available only as lay user descriptions which may have resulted from a number of different underlying diagnostic codes.  It is not possible to determine with certainty that every beneficiary identified in the data as having a particular condition does have that condition in the same way it is defined elsewhere and in the literature.  Similarly, it is not possible to determine with certainty that users who are not identified in the data as having a particular condition do not have that condition.

Although limited in the generalizability of the results, this study expands the limited knowledge base regarding PHR use and user-friendliness, expands the limited understanding of how the elderly relate to technology, and will aid CMS in determining their role regarding PHRs and beneficiaries for future projects.

View full report


"report.pdf" (pdf, 1.89Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®