Assessing the Need for a National Disability Survey: Final Report

Executive Summary

Gina Livermore, Denise Whalen, and David C. Stapleton

Mathematica Policy Research

September 29, 2011

This report was prepared under contract #HHSP23320095642WC between the U.S. Department of Health and Human Services (HHS), Office of Disability, Aging and Long-Term Care Policy (DALTCP) and Mathematica Policy Research. For additional information about this subject, you can visit the DALTCP home page at http://aspe.hhs.gov/_/office_specific/daltcp.cfm or contact the ASPE Project Officer, Hakan Aykan, at HHS/ASPE/DALTCP, Room 424E, H.H. Humphrey Building, 200 Independence Avenue, S.W., Washington, D.C. 20201. His e-mail address is: Hakan.Aykan@hhs.gov.

The opinions and views expressed in this report are those of the authors. They do not necessarily reflect the views of the Department of Health and Human Services, the contractor or any other funding organization.

Purpose of the Report

This is the final report of a project that assesses the need for developing and fielding another national disability survey data collection effort. It presents the findings from three principal project activities designed to assess whether existing data are sufficient to answer key disability-related research questions identified by the staffs of various federal agencies, and options for addressing the unanswered questions, including the pursuit of a national disability survey. These activities include:

• Obtaining feedback from federal and state agencies regarding their disability data needs and unanswered questions,
• Reviewing the disability-related information that is available in 40 existing national surveys, and
• Obtaining ideas and feedback from experts on disability data and research regarding how to address existing limitations in national disability data.

The summary provided in this report includes a review of existing disability data collection efforts and their limitations, and discusses a wide range of potential options to address these limitations. While the general purpose of the project is to assess the need for a national disability survey, we do not offer conclusions or recommendations regarding whether a national disability survey is warranted and if so, what its specific nature should be. Rather, this report provides objective information intended to be a resource for staff of the U.S. Department of Health and Human Services (HHS) and other federal agencies to inform their discussions and consideration of options for addressing their disability-related data needs, including the decision whether to conduct a new national disability survey.

Current State of Disability Data

Review of Existing National Surveys. Key findings from a review of 40 national surveys include the following:

• The specific measures of disability and wording of questions designed to elicit information about a particular type of disability (for example, visual impairment), differ markedly across surveys. Nearly all of the national surveys reviewed have questions that can be used to identify people with disabilities, but a few do not.

• The National Health Interview survey (NHIS) and Survey of Income and Program Participation (SIPP) are frequently used to address disability-related issues because they contain a relatively large amount of disability content, are representative of the national household population, and include large samples of people with disabilities.

• A concerted effort is being made to use a uniform set of disability indicators across surveys. The six-question series included in the American Community Survey (ACS), Current Population Survey (CPS), American Housing Survey (AHS), and NHIS supplement is part of this effort. A new supplement in the 2011 NHIS, which will include a battery of questions related to disability, represents an effort to make more uniform, descriptive information on disability available across countries.

• Few surveys contain measures to specifically identify individuals with cognitive or intellectual disabilities. Measures to identify specific health conditions underlying disability also are uncommon.

• There are numerous surveys of specific subpopulations (for example, students, inmates, nursing home residents, and beneficiaries of specific programs) on a variety of disability-related topics. This likely reflects the specific data and information needs of particular agencies and the fact that the general population surveys do not contain large enough samples of individuals in these specific target populations.

• Many national surveys have longitudinal components, though most cover a specific subgroup rather than a general population. The Panel Survey of Income Dynamics and Health and Retirement Study represent the most extensive efforts in that they regularly interview individuals from the time of sampling until death.

• New disability supplements to the SIPP and CPS will increase the information available on employment, economic well-being, and program participation. Two new longitudinal surveys are also in the works. The National Longitudinal Transition Survey 2012 will provide new longitudinal information on transitions from school to work, and the National Health and Aging Trends Study will monitor changes in daily life and activities throughout the aging process of Medicare beneficiaries over age 65.

• Consumption, finances, expenditures, transportation, environmental barriers, accommodations, and community supports are the general topic areas least commonly addressed for people with disabilities in the surveys we reviewed.

• Administrative data have been linked with several of the national surveys. The most frequently linked administrative data sources are from Social Security and Centers for Medicare and Medicaid Services (CMS) programs, which have been linked with several health-focused national surveys.

A final key observation is that a considerable amount of survey data is already being collected on disability issues and populations. Finding ways to improve, augment, and coordinate these existing mechanisms is an important means for addressing the limitations of survey data on disability-related issues and populations.

Disability Data Limitations. A number of limitations of existing disability data were identified by state and federal agency staff, Technical Advisory Group (TAG) members, and through our review of national surveys. Principal among these include:

• Disability measurement is inadequate. The health and functional measures used to identify people with disabilities vary markedly in concept, detail, and quality across surveys. Except for the surveys that focus on health, most do not capture in-depth information on the health issues and functional limitations experienced by people with disabilities. Measures corresponding to mental, emotional, cognitive, and intellectual disabilities were identified as being particularly poor.

• Sample sizes are small. Small numbers of people with disabilities in the large, general population surveys limit the ability to examine subgroups of people with disabilities. Analysis of residents of specific regions, states, or substate geographic areas is also limited due to small sample sizes.

• Administrative data are inaccessible. Administrative data are often hard to access due to legal, privacy, and security concerns, and data-sharing agreements are sometimes difficult to obtain. Linking the data across sources is also problematic because of technical issues and resource constraints.

• Quality is poor for some data. Respondent reports of program participation are believed to be inaccurate. Proxy responses and stigma related to disability also might contribute to poor data quality.

• Disability topics are inadequately addressed. Topics believed to be inadequately addressed include: disability onset and dynamics; program participation and the adequacy of support; supports needed to live in the community and environmental barriers to community participation; supports needed to obtain and maintain employment; disability-related expenditures; and health care access among employed people with disabilities.

• Data are not timely. Many sources of disability-related data are old or are produced very infrequently.

• There is inadequate longitudinal data. Existing longitudinal data fail to reasonably identify persons with disabilities and do not provide a lifetime perspective on disability.

Incremental Options for Addressing Disability Data Limitations

There are a number of ways that existing disability data collection efforts could be strengthened to address data gaps and limitations. Here, we describe a variety of strategies that build on existing data collection efforts, and thus, might be considered incremental in nature.

Increase Awareness of Existing Data. The perception of several TAG members was that a lot of good disability-related data exists, but that knowledge and use of some data sources are limited. Improving available information about existing data and making it easier for federal and state agency staff to access it is one potential way disability data limitations might be addressed. Reports that summarize a great deal of information in a format that is easily accessible, are one way this can occur and periodic cross-agency panels or work groups that exchange information about the nature and limitations of disability data are another.

Improve Disability Measures in National Surveys. Although disability is a difficult concept to define and measure, efforts to develop a succinct but comprehensive set of questions to identify people with disabilities have great potential value. Such efforts led to a six-question series developed by an interagency work group, which is becoming the new standard for identifying disability in national surveys. Despite the considerable efforts to develop these questions, however, they appear to not capture a significant number of people. Though not without its limitations, the addition of the six common questions across surveys was viewed as an important first step. While supportive of efforts to use a common set of disability questions, TAG and Federal Advisory Expert Panel (FAEP) members believed that complementary efforts were important and needed to be pursued. These include: preserving the ability to produce trends by at least temporarily including old disability questions when new questions are added; conducting research to better understand who is being captured by the six-question disability series; and strategically adding disability questions to selected surveys that will provide information to inform the findings from other surveys that do not include the additional questions.

Facilitate Increased Use of Administrative Data. There are many possibilities for enhancing the use of administrative data. Several appealing options include the following:

• Maintain and strengthen efforts to link survey and administrative data. Survey and administrative data linkages currently exist for many national surveys, and additional efforts in this area are an important way to address many disability data limitations. An example of a new effort would be to develop a match between the ACS and Social Security, Medicare, and Medicaid data. Improving the match rates in the survey-administrative data links conducted by the National Center for Health Statistics (NCHS) is another means of strengthening such efforts.

• Maintain and strengthen efforts to match administrative data across agencies. Cross-program data linkages would be useful for developing a better understanding of how the target populations of programs overlap and the extent to which service needs are met or are duplicative. Many interagency data use agreements are already in place; developing more of these and creating multilateral agreements would expand access to administrative data. Creating a stand-alone data workroom is another possibility. The workroom would provide access to data without personal identifiers that are linkable to data from numerous program sources.

• Make greater use of Unemployment Insurance (UI) records. Employment and earnings information collected by states for the UI program are reported quarterly and, as such, represent the most frequent and consistently collected administrative data source on employment and earnings. Developing linkages to federal administrative data or survey data and making the UI data accessible for more research purposes would address some of the disability data limitations surrounding the issue of employment.

• Develop the capacity to use electronic health records. The emerging use of electronic health records offers future opportunities to use these data on their own or to supplement other data for disability research and policy purposes, but work needs to begin now to facilitate this. Of particular importance is developing a standard set of items that measure disability and functioning. Social Security’s current efforts to identify people with disabilities who meet their programs’ disability criteria from such records might provide a good foundation for a broader effort to develop measures of disability and functioning from such records.

• Improve general accessibility. Greater accessibility could be achieved through improved documentation, technical assistance, and the creation of variables that are useful for research as opposed to program administration purposes. Access also could be facilitated through the use of a contractor to administer data use agreements (under a model similar to that used by the CMS’s Research Data and Assistance Center).

Improve Collection of Longitudinal Information. Options for improving longitudinal data include collecting such information using event history calendars or diaries, and asking retrospective questions (for example, about key events surrounding disability onset). Improving the ability to efficiently measure and interpret changes in disability status over time might also lead to disability questions being added to existing longitudinal surveys. New or enhanced longitudinal survey efforts could also provide information on other unanswered disability-related questions. A new effort could sample people who apply to a program and who exit from the same programs. A more ambitious approach would be to develop a longitudinal disability sample. The SIPP could be used to identify respondents with disabilities and these respondents could be followed for an extended period (for example, 10 years).

Enhance Disability-Related Content in Existing Surveys. Inclusion of a basic set of disability questions in all surveys is the most important way disability-relevant content in national surveys could be improved. Among surveys that already include questions to identify people with disabilities, FAEP and TAG members believed it would be useful to add content related to: disability severity and onset, program participation and service use, barriers to independent living, work accommodations, accessible public transportation, assistive technology, disability-related costs, and the emotional and social impacts of disability.

Augment Samples in Existing Surveys. There are several possible ways to augment survey samples and address issues related to small sample sizes of people with disabilities:

• Pool data across survey years. This is a useful means for increasing sample sizes when data have been collected consistently over time. The Medical Expenditure Panel Survey (MEPS), NHIS, CPS, and SIPP are good candidates for pooling multiples years to increase sample sizes.

• Modify survey sample frames and methods to identify more people with disabilities in surveys. The exclusion of people residing in group quarters or institutions may inadvertently omit a large number of people with disabilities. Improvements in the manner with which surveys interview respondents might also help to identify more people with disabilities.

• Purposefully augment existing survey samples with individuals in the subpopulation of interest. An example is Social Security’s collaboration with the Census Bureau to include an additional sample of Social Security disability beneficiaries in the 2001 SIPP.

• The addition of the six-question disability series to federal surveys might also offer future opportunities to augment sample sizes in surveys that obtain their samples from larger national surveys. The screener questions from the parent survey could be used to augment samples of people with disabilities in the supplemental survey.

• Oversample from identifiable groups with high disability prevalence. The SIPP used this approach to oversample low-income households. For example, use ACS data pooled across years to identify small geographic areas where disability prevalence is relatively high, and then oversample those areas for the SIPP or NHIS.

Field Periodic Supplements. Adding a topical supplement to an existing national survey is a useful approach when a large amount of new information is required or when there is a need to study a specific subpopulation that cannot be easily identified with existing information. An existing, large national survey would act as the screener and this “parent” survey would also provide additional information that enhances the supplement. Three general models for fielding topical supplements to existing surveys are currently used and offer models for adding disability-related content:

• Topical modules are supplementary questionnaires administered during a longitudinal survey that contains information on other topics, such as employment. The planned disability supplement to the CPS is an example.

• Topical question batteries are sets of questions that can be added to a core survey questionnaire but, unlike topical modules, are only asked of a subsample. An example of this is the Behavioral Risk Factor Surveillance System, where topical supplements can be used in concert with the core national survey and administered in a single interview.

• Topical surveys appear to be stand-alone surveys, but in fact derive their samples from a parent survey and so are essentially extensive topical modules of the parent survey. For example, the MEPS sample is derived from the NHIS. The supplemental interviews are conducted separately from the original interviews but the data from the original survey can be combined and used with the topical survey data.

Conduct Periodic Surveys of Subpopulations. Over half of the 40 surveys reviewed for this report represent surveys of subpopulations. The need for these types of surveys is driven by the specific information needs of particular agencies and the inability of general population surveys to identify or include in their samples sufficient numbers of individuals for which disability information is needed. Most subpopulation surveys focus on those already in a subpopulation, rather than those transitioning into or out of the subpopulation, such as applicants. Hence, adding special surveys that start with transitioning populations might be of considerable value (for example, transitions from education to employment, changes in residences, or from institutional to and from noninstitutionalsettings, entry to and exit from public programs, and across changes in health care systems).

Designate a Lead Agency or Group to Coordinate Disability Data Efforts. Designating a lead organization or agency to advocate for and coordinate disability data improvements efforts was perceived to be necessary. Proposed organizations include the HHS Office of the Assistant Secretary for Planning and Evaluation, the NCHS, the Office of Management and Budget, and the National Institute on Disability and Rehabilitation Research.

Addressing Disability Data Limitations with a National Disability Survey

Another approach to addressing the disability data limitations would be to design and conduct a national survey focused specifically on disability issues and populations. Pursuing such an option might be desirable if: pursuit of other, more incremental options are infeasible or do not adequately address the disability data limitations considered to be of significance to federal agencies; there is a desire to collect a large amount and wide variety of data; and there is a goal to make disability issues more prominent and to establish and ongoing mechanism for periodically and consistently collecting data to assess the experiences and economic well-being of people with disabilities. Potential features of such a survey include the following:

Purpose. There are many different data limitations a national disability survey could be designed to address and the primary purpose of the survey will dictate its design features. Potential purposes include: providing a broad range of detailed information about disability and acting as the gold standard for general population prevalence and disability measurement; focusing on a subgroup neglected in other surveys; providing important disability-related information not captured in existing surveys; and ensuring periodic and consistent collection of key information on people with disabilities.

Stand-alone versus Supplement. A basic decision to be made with regard to conducting a national disability survey is the choice to establish a stand-alone survey or create a supplement to an existing survey. The only national disability survey ever conducted in the United States, the National Health Interview Survey on Disability (NHIS-D), was a supplement to an existing survey. Relative to a stand-alone survey, this approach imposes lower costs and burden and has the benefit of providing comparison data for people without disabilities at no additional cost. However, deriving a sample from an existing survey imposes the sampling and administration methods on the new survey. A stand-alone survey offers the opportunity to tailor sampling and administration methods to better capture people with disabilities who may be excluded from existing surveys.

Frequency of Administration. Ideally, data collection would occur frequently, perhaps annually or biennially, which would allow for an analysis of trends and short-term changes in the experiences of people with disabilities. An alternative model would be to administer a core set of questions on a regular basis with a more comprehensive disability survey occurring less frequently (for example, every 5-10 years).

Population Coverage and Level of Estimates. A national disability survey provides the opportunity to improve on data limitations related to the population sampled. Possible opportunities to improve or modify data collection in this respect include:

• Sample all Americans, including people who reside in the community, group quarters, and institutions, as well as those who are homeless.

• Improve coverage of people with disabilities and their families. Such information might be important in understanding the support system available to respondents with disabilities.

• Include all age groups and disability questions that are appropriate for different ages. The inclusion of all age groups in one survey would be important to a national disability survey intended to provide information on disability across the life cycle.

• Oversample or create new surveys focused on disability subpopulations neglected in other surveys. This could include people making transitions, racial and ethnic minorities, people with rare disabling conditions, and employment subgroups.

• Create a survey with a sample size large enough to produce state or local-level estimates. This will assist states in obtaining basic prevalence rates, assessing access to and unmet need for services, and understanding the impacts of state-level policies and programmatic changes.

• Focus on communities as the unit of observation. A focus on communities, rather than on individual experiences and perspectives, might be valuable for purposes of gathering information about the community environment, including supports for people with disabilities and how the environment affects outcomes for individual consumers.

Longitudinal Data. A longitudinal survey can address disability-related issues that cannot be addressed by other means (for example, capturing information about the dynamics of disability and the characteristics and outcomes of key transitions). Longitudinal information about working-age people with disabilities is particularly scarce. In collecting longitudinal data, panel length and survey frequency must be balanced against financial costs and the risk of sample attrition. Alternative methods to collecting longitudinal data include the use of retrospective questions and linkages with administrative data.

Disability Measures. A national disability survey could address problems with the inconsistency and quality of disability measures. A step to ensure consistency across other surveys measuring disability is to include the six-question series used to identify disability in the ACS, AHS, and CPS, which appears to be the new standard for identifying disability. Including other measures of disability as well would allow for a better understanding of which individuals are being identified and overlooked by the six-question series. A battery of more detailed questions to measure onset, severity, functioning, underlying causes of disability, and use of accommodations and supports could also be included. Short general health scales, such as the SF-12 and more detailed SF-36, might also be included to assess how these scales operate in the context of disability.

Topical Content. A national disability survey could potentially cover a wide variety of general topics, including health, health care, education, employment, program participation, participation in an array of social activities, housing, disability services, supports, and quality of life. A new data collection effort could also address disability-specific topic areas, such as the role of the surrounding environment and transitions in and out of federal disability programs. Finally, a national disability survey could reach beyond a cursory level of understanding to know more about the motivation behind outcomes.

Links to Administrative Data. A national disability survey, regardless of its other features, would significantly benefit from linkages with administrative data. People with disabilities participate in public programs at high rates; creating linkages with administrative data from federal programs, particularly the Social Security disability programs, Medicare, and Medicaid, would provide accurate historical information on program participation and usage.

There are several advantages and drawbacks to conducting a national disability as a means for addressing existing disability data limitations.

Advantages. A national disability survey provides the flexibility to design a comprehensive survey that moves beyond the current level of coverage of disability-related issues. The key advantages of a national disability survey include:

• As a new data collection effort, it could be designed to specifically address particular existing data limitations.

• A national disability survey would be focused on disability and thus would be open to many questions, including those that may be of relevance only to people with disabilities.

• Statistics would be based on a single set of disability identifiers.

• In a new national disability survey, the identification of people with disabilities who are often overlooked (for example, people with specific disabilities, with difficulties responding to a particular survey format, or living in institutional settings) could be prioritized.

• A national survey has the potential to provide a complete life cycle perspective on disability, which seem to be missing from existing data.

• An ongoing national disability survey would ensure regular and consistent information over time about the experiences and well-being of people with disabilities and in doing so, inform disability issues and policies and make them more prominent.

Disadvantages. Although theoretically appealing, the reality of a national disability survey comes with several disadvantages. Disadvantages of a national disability survey include:

• It is potentially expensive to design and field.

• There might be significant political and practical challenges in attempting to coordinate, design, and implement a large undertaking.

• There is too much potential topical disability-related content to be incorporated in one survey. A survey lengthy enough to cover all relevant content would be prohibitively long, unless questions were spread across topical modules in a longitudinal survey.

• If designed to cover a broad set of topic areas, rather than focused on filling an existing information gap, might be viewed as duplicative of existing efforts.

• If designed as an extensive supplement to an existing survey, the parent survey may be averse to adding extensive screening questions or changing the sampling method to identify people with disabilities who may be overlooked.

• If designed as a stand-alone survey, comparative information on people without disabilities would either be unavailable or would require an even larger sample size.

Conclusions and Potential Next Steps

A substantial amount of survey and administrative data relating to disability is currently collected in the United States and significant progress has been made in improving the nature of the disability-related data collected. But despite this progress, important limitations to existing disability data persist. In this report, we have provided numerous ideas for ways in which disability data limitations might be addressed. Some potential next steps include:

• Conducting analyses to better understand the advantages and limitations of the six-question disability series with the intent of developing ways to improve the standard measure.

• Facilitating the use of linked survey and administrative data by developing procedures to improve match rates, improving documentation, and developing summary variables useful to administrative data users.

• Pursuing opportunities to develop the capacity to use electronic health records for disability-related purposes by engaging an organization involved in health information technology advancement to support such an effort and developing the means to identify people with disability in health records.

• Encouraging those responsible for existing national surveys to reassess their sampling frames and data collection methods to determine if there are ways to increase the likelihood that people with disabilities will be captured in national surveys.

• If HHS and other agencies are interested in pursuing a national disability survey, the critical next step is to define the specific scope and purpose of the survey.

A national disability survey can address many existing disability data limitations, but there also are numerous incremental means to do so. Final decisions regarding which specific disability data collection efforts to pursue will be determined by federal policy makers and program administrators whose agencies’ needs are diverse. It is unlikely that any single effort would address all limitations and disability data needs. But there are many possible strategies for ameliorating current disability data limitations and there is considerable room for multiple efforts, large and small.