The Feasibility of Using Electronic Health Data for Research on Small Populations. Part I: The Challenge of Small Populations for Research on Health and Health Care: Examples from Four Under-Studied Populations

A vast body of research shows important differences among segments of the population on virtually all aspects of health and health care, including patterns of disease and disability, use of services, and quality and outcomes of care. Documenting such differences is an essential starting point for a wide array of policies and interventions to improve peoples’ health. Biological, cultural, historical, and socioeconomic differences among different segments of the population may create distinctive patterns of health care needs and differences in the use of and responses to medical services. Understanding the patterns and differences is impossible unless researchers can separate and compare data from various segments of the population. That is difficult when those population segments are small or difficult to identify. This is a particular concern when the small population in question has special vulnerabilities or may be subject to inequitable treatment. To date, the federal government’s very substantial data collection efforts have not generated adequate data about some subpopulations because of their small size or their distribution (either great concentration or lack of concentration) or because of insufficiently standardized ways of identifying the population in a survey context.

The small size of some populations means they may not be included in numbers sufficient for separate analyses in federal surveys. Also, information identifying some small populations may not be routinely included in the medical records and insurance claims that are another source of data. To illustrate the different research and methodological challenges facing research on small populations, this report focuses on four case examples—Asian-American subpopulations; lesbian, gay, bisexual, and transgender (LGBT) populations; adolescents with autism spectrum disorders (ASDs); and residents of rural areas. This report is about why research is needed about small populations such as those that we have chosen and about the challenges that small populations pose for research; we make no attempt here to report comprehensively on the health and health care needs of the four populations. We also recognize that many other relatively small populations may have special health care needs or pose particular challenges to the health care system. Our cases are illustrative of a more general set of issues.

Advocacy organizations, as well as some researchers and policymakers, have pushed for the collection of more data about various small populations, including the examples we focus on in this report. With the growing use of electronic health data in the provision of medical care, the possibility that such data might be used for research that complements or supplements existing federal data collection activities merits consideration. That is the topic of Part II of this report. For purposes of this report, we define “research” broadly as addressing issues traditionally addressed through clinical, pharmaceutical, health services, public health, public policy, and evaluation research.

In selecting our example small populations, we targeted those that would illustrate a broad range of health and health care questions, as well as challenges encountered in conducting research to answer them, with existing federal data sources and potential with electronic sources generated in medical care.

Small populations that need study share characteristics with what are typically considered underserved populations: “poor; uninsured; have limited English language proficiency and/or lack familiarity with the health care delivery system; or live in locations where providers are not readily available to meet their needs.”¹⁵ To focus our study, we consulted with government officials at the Agency for Healthcare Research & Quality (AHRQ), the Center for Disease Control’s (CDC’s) National Center for Health Statistics (NCHS), and the Health Resources and Services Administration (HRSA) about populations about which information requests have been received that could not be answered from existing federal data sources. We have also reviewed some related National Institutes of Health (NIH) projects, like the Health Care System Research Collaboratory program.

Once the four study populations were selected, we reviewed past federal surveys regarding the extent to which they could be identified in available data sources, and we examined existing literature for information about their characteristics, health and health care issues, as well as reasons why they have been difficult to study in existing federal surveys and with other sources of data.

In addition, we conducted tailored interviews with 16 expert informants whose work has focused on one of our small populations (see Table I.1). Topics in the interview guide were based on issues and concerns raised in available literature and by organizations that serve the populations in question. An initial purposive sample of experts was identified from published sources, advice from the governmental sources mentioned above, and the research team’s knowledge of the field, followed by some snowballing based on suggestions by the experts we were interviewing. Each person gave permission to have the interview recorded, and the interviews were summarized thematically. Particular attention was paid to areas of convergence and divergence among interviews, as well as between interviews and the literature.

There are a number of strengths to primary survey data compared to other primary data sources (e.g., focus groups, case studies) and secondary data (e.g., administrative and claims data). Survey data allows the researcher more control over who is included (i.e., sample frame and sample), the kinds of information that is collected from them (e.g., data domains, elements or specific questions), and key aspects of data elements (e.g., standardization and quality) compared to administrative, claims, or other secondary data sources. Consequently, it is often easier to generalize to the nation or other large populations and to replicate survey research.

All research approaches and data sources have limitations, and that is true of survey research. Although many important research questions (e.g., about outcomes of treatment or the consequences of being uninsured) require longitudinal data, most surveys are designed to collect cross-sectional data at a point of time. The Medical Expenditure Panel Survey (MEPS) is a two-year panel and a rare example of a study that attempts to follow cohorts (of households) over time. Such efforts are few and expensive. There are also limitations regarding the kinds of data that can be collected via survey research. For health matters, for example, surveys most often are limited to collecting self-reports about individual’s overall health status, so the resulting data do not include the kinds of clinical information (e.g., about diagnoses, service and procedures, laboratory results, drugs, genetic information) needed for some kinds of studies. Selection bias, which results from survey respondents’ decisions about whether to participate or not, can lead to misleading data.¹⁶ Self-reported survey data have weaknesses resulting, for example, from limitations in knowledge or from recall bias. Finally, with the exception of highly specialized studies, surveys generally obtain data from too few people to break out separate results for small populations. As a result even valid inferences drawn about the population (or major segments thereof) based on well-designed survey samples may not apply to small populations such as we are considering in this report.

General problems with small populations do not necessarily stem from the absolute size of the population, but rather its size relative to the total population (or sampling frame) from which the survey sample is drawn. Sample sizes calculated to collect information on the general population of Americans often lack ability to accurately detect small populations. This problem only increases when wanting to study specific health conditions within these small populations. There are standard approaches to increasing the chances of including people from small populations, such as using a list of group members to specifically target or screening questions to increase representation of the groups. However, these strategies are not typically used in national surveys.

Standard “solutions” for getting adequate numbers for analysis from small populations include oversampling¹⁷ and combining data from multiple years. But oversampling subgroups may require the researcher to screen out large numbers of people who do not fit the category in order to obtain the sought-after number of those who do. This becomes more costly as the target group’s presence in the population being screened becomes smaller and as the number of needed subgroups (e.g., age, gender, or those using different languages) increases. The smaller a group’s presence in the population being screened, the more calls are needed to obtain the desired number of respondents. Combining data from multiple years becomes problematic if year-to-year changes are taking place within that population or if survey questions change. A third alternative, sampling from an organization that specializes in service to the population in question, raises questions of representativeness.

In general, the limitations of national surveys for studying small populations can be summarized as issues related to coverage of the target population and issues related to data collection.¹⁸ These issues as they relate to our four example populations are presented in Table I.2 and are discussed in greater detail later in this report.

“Asians” are one of the five race categories that must be used in the federal government’s surveys and administrative forms under rules of the Office of Management and Budget, but the Asian-American population is quite internally diverse. The 15.5 million Asian Americans who compose about 4.4 percent of the American population include more than 50 different Asian ethnicities and 100 languages. Asian Americans are concentrated in urban areas, particularly in California, New York, and Texas. Which Asian-American subpopulations are found in particular areas varies. Urban areas in California like Los Angeles and San Francisco, as well as eastern areas like New York City have larger Chinese populations than any other Asian subpopulation, while urban areas in Texas have higher concentrations of Asian Indians and Vietnamese.²⁶ Other local concentrations of Asian subpopulations can increasingly be found throughout the country.²⁷ Between 2000 and 2010, there was a 46 percent increase in the Asian-American population, making them the fastest growing racial group.²⁸

It has been well documented that racial and ethnic minorities receive lower quality health care than non-minorities even after accounting for access-related factors,²⁹ but little of the research on racial/ethnic disparities has focused on Asian Americans. Their health care needs remain poorly understood due to inconsistent definitions used in data collection, lack of disaggregated data about ethnic subgroups, and the uneven geographic distribution of the Asian-American population.³⁰

The commonplace view of Asian Americans as self-sufficient, educated, and upwardly mobile fails to recognize the health needs of Asians overall, as well as their diversity in terms of ethnic background, country of origin, length of time in the United States, and other factors that may affect health and health care.³¹

Figure I.1, which comes from the Palo Alto Medical Foundation Research Institute’s Pan Asian Cohort Study (National Institutes of Health, National Institute of Diabetes and Digestive Kidney Diseases grant 5R01DK81371), which primarily utilizes electronic health record (EHR) data, shows diabetes prevalence among men in the San Francisco Bay area and provides a vivid example of the differences in health problems among sub-groups of the Asian-American population.³² The prevalence rate among Filipino men is more than three times that of Japanese men. It is apparent from these and other data, that health needs vary greatly within what is often treated in research as a single racial population.³³

Figure I.1. Pan Asian Cohort Study—Preliminary Findings for Diabetes Prevalence

Source: Pan Asian Cohort Study. “Preliminary Findings for Diabetes Prevalence.” Palo Alto Medical Foundation. Accessed March 1, 2013. http://www.pamf.org/pacs/men.jpg.

There is also evidence of health care–related differences within the Asian-American population. Asian immigrants to the United States are less likely than U.S.-born Asians to have health insurance and use health care services.³⁴ Linguistic isolation (living in a household in which no one above age 14 speaks English) may contribute to this. About one-quarter of Asian Americans live in linguistically isolated households, with rates ranging from 10 percent among Filipinos to 45 percent of the Vietnamese.³⁵ Not surprisingly, linguistically isolated households tend to be of low socioeconomic status and have poorer access to care and more depravation of various kinds than do households in which English is spoken. New immigrants from all countries tend to locate near earlier immigrants. This pattern may facilitate access to various kinds of culturally specific goods and services but may produce isolation from the larger society as well as shared exposure to any environmental risk factors that are proximate to their locale.³⁶

The language barriers and cultural differences associated with immigrant status create various complexities, including communications difficulties with health care providers, advice that is inconsistent with cultural beliefs and practices, and dissatisfaction with or distrust of medical advice.³⁷ Imperfect language translation and nuance can create confusion. Language and cultural isolation of immigrant or non-English speaking groups may present barriers to care-seeking and treatment.³⁸ Behavioral health issues—stress, smoking, domestic violence, alcohol abuse—may also be associated with these factors.

There is need for better information about subpopulations of Asian Americans, as can be can be illustrated by considering the examples of Vietnamese and Filipinos in the United States.

The health and health needs of lesbian, gay, bisexual, and transgender people are not well documented. Even basic information is hard to come by. As a recent Institute of Medicine report puts it, “it has been an ongoing challenge for researchers to collect reliable data from sufficiently large samples to assess the demographic characteristics of LGBT populations.”⁸² This project mainly focuses on the health and health needs of lesbian, gay, and bisexual people. The transgender population has a host of separate issues around classification, health problems, and provider relations that are not well researched.⁸³

To start with the basics, federal and non-federal survey-based estimates of numbers of lesbian, gay, bisexual, and transgender people have varied by gender, over time, and according to survey methods and question wording (see TableI.4 in the Appendix to Part I). Recent estimates puts the percentage of the adult population who identify as homosexual, gay, lesbian, or bisexual at about 3.5%).⁸⁴ No such information is available about transgender people. The percentage of adults who identify themselves as lesbian, gay, or bisexual to survey researchers is smaller than the percentage who report having same sex partners or who report some desire for or attraction to a person of the same sex. The small size of LGBT populations and the sensitivity of results to the wording of questions are among the challenges to studying health issues in these populations via survey research. However, there are many indications that such research is needed.

Autism spectrum disorders (ASDs) are a group of developmental disabilities that range from mild to severe and are characterized by social impairment, difficulty communicating, and repetitive motions or other unusual behaviors.¹⁰⁶ These characteristics are usually noticeable before the age of 3 and remain as a lifelong chronic condition with both medical and psychological implications.¹⁰⁷ ASDs include autistic disorder, Asperger’s disorder, pervasive developmental disorder–not otherwise specified (PDD-NOS), Rett syndrome, and childhood disintegrative disorder.¹⁰⁸ Based on 2008 data from the 14 sites in its Autism and Developmental Disabilities Monitoring Network, the Centers for Disease Control estimates 1 in 88 8-year-old children have ASDs.¹⁰⁹ Prevalence in these sites had increased 23 percent from two years earlier and 78 percent since 2002. Although there is disagreement about whether the true prevalence has increased (since guidelines for diagnosis have changed, more services are available, and awareness of ASD has increased), the CDC numbers are based on evaluation records, not parental reports. Measuring ASD prevalence continues to be a challenge due to the complexity of the disorder, the lack of consistent and reliable diagnostic standards, and changes in the definition of such conditions.¹¹⁰ ASD prevalence is about five times higher in boys than in girls (ratio of 4.5 boys to 1 girl). Prevalence is also significantly higher among non-Hispanic white children than among black and Hispanic children. Intellectual ability is highly variable, with 38 percent reported as intellectually disabled, 24 percent as borderline, and 38 percent with average or above average intellectual ability.

There are controversies about what should be included in the category of autism spectrum disorders. The NIH classifies Rett syndrome as an ASD, but some argue that it is more similar to non-autistic spectrum disorders such as fragile X syndrome or Down syndrome. Unlike other ASDs, Rett syndrome is also almost always in girls.¹¹¹ There is also debate over whether Asperger’s disorder is a separate disorder or simply a less severe form of autism.¹¹² The next revision of the American Psychiatric Association’s Diagnostic and Statistical Manual (DSM) will drop individual classifications for autistic disorder, Asperger’s disorder, childhood disintegrative disorder and PDD-NOS, grouping all of them under “autism spectrum disorder”—a term that is already widely used. APA has said this change will help “more accurately and consistently diagnose children with autism.” Rett syndrome will be dropped from the DSM altogether. There is concern among the Asperger’s and Rett communities that these changes will result in a loss of identity among individuals with these specific disorders and that it may affect health insurance coverage and school funding for special education.¹¹³

The exact causes of ASDs remain unknown, but research suggests genetics and environment both play important roles. Researchers are studying factors such as family medical conditions, parental age and other demographic factors, exposure to toxins, and complications during birth or pregnancy. CDC and IOM studies have found no link to childhood immunizations.^{114, 115, 116, 117}

Depending on the definition used—particularly degree of proximity to urban areas—the proportion of the U.S. population described as rural ranges from 17 to 49 percent.^{150, 151, 152}     Rural communities are far from uniform, but they are generally less densely populated and more geographically isolated than urban areas. These characteristics result in limited access to services and economic opportunities.¹⁵³ Compared to the rest of the population, people in rural areas are more likely to live in poverty as a result of low wage jobs and less likely to be highly educated.¹⁵⁴ Many rural areas face declining numbers due to the out-migration of younger residents.

This report has focused on need for health information data about small populations and the challenges that meeting that need has posed for researchers. To explore these challenges we considered populations defined by four types of characteristics―sexual orientation and behavior, geography, race and ethnicity, and a health-related condition—that were selected to illustrate the range of problems that face researchers when using existing federal surveys (see Table I.6). In a Part II of this report, we examine the potential of data based on electronic health records and related electronic data sources to complement these surveys and overcome some of the problems researchers have historically faced.

In each of our four illustrative populations, we have presented evidence of distinctive health and health care issues that could usefully be better understood by research. Some of these issues pertain to problems and concerns that may characterize the population itself—as with the high rates of diabetes among Filipino Americans, the distance from specialty care that some rural populations face, or the problems posed by the transition to adulthood for adolescents with autism spectrum disorders. Some issues pertain to possible differences and possibility disparities from other populations or the population at large regarding health conditions, services, or outcomes of care.

Research to address questions about small populations depends on several things. The most fundamental is the ability to identify the population of interest in the data. The second is having data on the independent and dependent variables of interest, as well as relevant co-variates (e.g., education, income) that need to be controlled for. Third, the value of many data sources can be enhanced if researchers are able to link to other data sources. Such linkage requires availability of a unique identifier or a matching algorithm that uses multiple variables. Fourth, some research questions require longitudinal data in which data about the same people can be linked over time. Finally, given resource realities and constraints, ways are needed to conduct research as efficiently and effectively as possible. Primary data collection strategies for getting sufficient numbers of people from small populations can be very expensive.

Some national health survey data sets (including the National Survey of Family Growth, National Health and Nutrition Examination Survey, National Health Information Survey, and Behavioral Risk Factor Surveillance System) contain information about the LGBT population or Asian subpopulations. Although such data may be collected, issues exist that make it difficult to use for research on small populations. Information (e.g., zip codes) that is needed to characterize an individual’s degree of rural-ness is not available in federal public use data sets because of concerns that deductive identification of individual people might be possible. Additionally, validity concerns can be raised about information reported by a parent in household surveys about a condition such as a child’s autism. Survey data may also not include the dependent variables and co-variates needed to answer questions about the health and health care of small populations. Data analysis also requires sufficient numbers, and this can be a problem in survey research and secondary data analysis for people in categories that appear only in small numbers in a large population. This is particularly true when co-variates are considered. The common solutions for this problem all have important drawbacks.

Combining data from surveys conducted in multiple years may yield a sufficiently large analytic sample, but it can produce misleading results if changes are occurring within the population over time. Oversampling a small population in survey research is often feasible, but it can be expensive. Two-stage sampling, starting with a targeted survey, and then a follow-up survey of the target population, can be expensive, and can only be used when the target population is stable and easily identified.²⁰¹ Web-based surveys are another potential approach, but these are also limited by self-selection bias (due to high nonresponse rates), representativeness issues, and concerns about the reliability and validity of the data collected.^{202, 203} Finally, focusing the study on a region or setting in which there is a concentration of people who fit the category is an oft-used option for obtaining sufficiently large numbers, but the resulting data may not be representative of the larger population.

Available data sources also have other important limitations. Federal survey research is typically cross-sectional, lending itself poorly to research questions that have a longitudinal dimension. Additionally, survey domains, questions, and response categories may change over time, limiting the ability to use the data longitudinally. Data based on insurance claims may permit data analysis that has a longitudinal dimension, but insurance claims do not typically include information that would permit identifying someone as from a LGBT or an Asian-American subpopulation and the data are limited to billed services from particular payers.

In sum, policymakers, advocates, or researchers interested in the health and health needs of small populations encounter various barriers to research using existing federal surveys.

A great deal of hope has been placed in the possibility that electronic information generated in the patient care process in organizations that have electronic health records will provide data that can be used for research on small populations, even though the organizations that collect such information at this time are hardly representative. Electronic health records and associated electronic data (e.g., patient reported health behavior or laboratory or prescription information) have a number potential benefits, such as the possible inclusion of large numbers of individuals from small populations, the collection of rich information about key process of care and outcome variables of interest, the potential for longitudinal study of cohorts of people (e.g. regarding outcomes of care), and the ability to do these relatively inexpensively.

In Part II of this report, we explore these possibilities on how electronic health records and other electronic data can be used to strengthen research on these patient populations.

Table I.1. Key Informant Interviews

Table I.2. Limitations of National Surveys for Small Populations

Table I.3. The Ability of Key National Surveys to Study Four Target Populations

Table I.4. Estimated Percentage of People by Sexual Orientation and Behavior from Selected Federal and Non-Federal Sample Surveys

This table does not display the most recent estimates, but rather is presented to illustrate how federal and non-federal survey-based estimates of numbers of lesbian, gay, bisexual, and transgender people have varied by gender, over time, and according to survey methods and question wording. For more discussion, see the “Population #2: Lesbian, Gay, Bisexual, and Transgender People” section in Part I.

Table I.5. Common Rural Taxonomies Used by the Federal Government

Table I.6. Potential Areas for Further Research