The Technical Advisory Group (TAG) meeting for the Assessing the Need for National Disability Survey project was held in June 2011. The meeting included two discussion sessions: (1) the extent to which data limitations identified by federal and state agencies overlap with the TAG feedback and (2) options for improving disability-related data. Each session included discussion of additional topic areas not identified prior to the TAG meeting and the importance and feasibility of various suggestions. The sections that follow summarize the discussions that took place during the TAG meeting.
A. Unanswered Questions and Data Limitations
Prior to the meeting, TAG participants were asked to review Disability Data in National Surveys, a draft report summarizing the written input provided by federal and state agencies and describing 40 surveys with disability-related content. Participants were also asked to review a memorandum summarizing the proceeding of a May 2011 FAEP meeting. The first session began with a brief presentation that summarized the pre-meeting input regarding unanswered disability-related questions, inadequately addressed subgroups, and data limitations. Unanswered disability-related questions were presented as broad topic areas including program participation, education, employment, housing/independence, quality of life, health status, and health care. Similarly, inadequately addressed subgroups were defined by age, other demographic characteristics, veterans, homeless individuals, specific types of disabilities, employment subgroups, program participants, long-term care users, and informal care providers and users. Finally, the presentation provided details about noted data limitations, including poor and inconsistent measures of disability, small sample sizes, difficulty accessing and linking administrative data, poor data quality, inadequate survey content, and timing issues.
The consensus among TAG participants was that the unanswered questions, inadequately-covered subgroups, and data limitations identified by federal and state agencies were mostly accurate and valid concerns. Participants also acknowledged that the issues identified were varied and often disjoint from one another, and that it would be challenging to determine the overlapping data needs and priorities in determining which data improvement options to pursue. Specific issues discussed during the session are summarized below.
1. Unanswered Questions
Program Participation. Several topics related to program participation were mentioned by TAG members. One member perceived program participation to be the topic area for which accessible data were most lacking and for which incremental changes to existing survey data might be least likely to address. Specifically, program take-up rates were mentioned as receiving inadequate data coverage. With available data it is difficult to identify which segments of the population are eligible for various programs, making it impossible for program administrators to predict demand, assess whether their programs are reaching the intended target populations, and estimate unmet need. The inability to assess program effectiveness and outcomes was also identified as an issue due to a lack of data. Although it was noted that such topics might be addressed using the Survey of Income and Program Participation, TAG members acknowledged that information is still lacking with regard to many programs that serve people with disabilities, such as vocational rehabilitation. In addition, TAG members indicated that addressing program-specific policy and research questions through survey data may not be the most feasible approach.
Community Living. One participant discussed data limitations related to community living. He stressed that community living applies to all age groups and goes beyond simply program participation, noting that most surveys do a poor job of measuring variables that broadly represent this domain, such as family circumstances and local resources. The issue will continue to grow in importance with the increase in the size of the United States population requiring assistance to remain living in the community.
Transitions. TAG participants noted that the lack of consistency in samples and disability measurement across multiple data sources makes it difficult to follow individuals as they make transitions. Identifying program participants before enrollment (when interventions may be particularly fruitful), and capturing the dynamic process of disability were mentioned as being nearly impossible to do with existing data. Following children with disabilities for long periods after they leave school was also discussed as being necessary but not done currently.
2. Data Limitations
Disability Measurement. Throughout the discussion, issues related to the measurement of disability were mentioned several times, echoing earlier feedback from federal and state agencies. Omission of a basic set of disability measures from federally funded surveys was regarded as problematic among meeting attendees. In surveys that include measures of disability, TAG participants lamented that little information beyond the basic measure is typically collected. Participants described measures of severity as being noticeably absent in most surveys. Participants also noted the inability of existing measures to identify psychiatric and cognitive disabilities well. There was discussion of the six-question disability sequence used on the ACS, and its likely promulgation to other federal surveys given the requirement under the Affordable Care Act that a disability demographic be included in federal surveys. Participants agreed that having a standard disability measure like the six-question sequence was critical, but many had concerns about who the current questions were identifying as having disabilities and several suggested that more work should be done to understand and improve the standard measures.
Timing Issues. Several issues related to data timing were mentioned during the meeting. The first was the lack of longitudinal data covering people with disabilities, making it difficult to understand the dynamics of disability across the lifespan and to measure episodic recoveries and relapses. One participant suggested that this information might be ascertained by piecing together the experiences of larger samples over shorter time frames as opposed to following individuals over long periods of time. Delays in the release of data were also noted as a shortcoming, contributing to difficulty assessing current events, such as the impact of the recession on people with disabilities.
Small Samples. The inability of many federal surveys to produce state-level estimates was noted as an important limitation of existing data. It was mentioned that for many programs, such as Medicaid, there is direct involvement of states in the provision of services, and the nature of the programs varies substantially across states. Without state-level data, it is difficult to get basic prevalence rates, assess access to and unmet need for services, or understand the impacts of state-level policies and programmatic changes. The Behavioral Risk Factor Surveillance System (BRFSS) was noted as a possible vehicle for improving disability data to support state policy issues.
B. Options for Improving Disability-Related Data
TAG participants suggested a number of constructive options for improving disability-related data. The second discussion session began with a brief summary of options for improving disability-related data that were discussed in the draft report. These options include: increasing awareness and use of existing data, improving disability measures, collecting longitudinal information, increasing survey and administrative data linkages, enhancing disability-related content, fielding periodic survey supplements, augmenting samples in existing surveys, conducting periodic surveys of subpopulations, and fielding a periodic national disability survey. Below is a summary of discussion surrounding the aforementioned options as well as another option proposed by TAG members.
Improving Disability Measurement. Disability measurement was identified as a major limitation of disability-related data, but several options were proposed to improve on this metric. There was agreement that all federally funded surveys should include measures of disability. Most TAG members believed that there should be at least one standard measure applied consistently across all surveys. There was less agreement about what the measure of disability should be. Many members expressed discontent with the ACS six-question series, citing failure to adequately identify people with behavioral health conditions, aspects of cognition, social functioning, and mobility problems. One participant pushed for research to validate the six-question series, and another discussed developing a completely new set of questions to identify people with disabilities, perhaps based on the WHO classification of disability. It was noted that substantial research had already been conducted to validate the six-question series and one participant believed that starting over with completely new measures was unnecessary and a waste of resources. Regardless of their views on the quality of the six-question series, participants agreed that the recent addition of these questions to several surveys is a positive step for the state of disability-related data. Those who were unhappy with the measure suggested that surveys should include additional questions to identify people with disabilities. Examples of additional questions include those used to identify behavioral health conditions, measures of functioning, difficulty with specific activities, and work-limitations. Opportunities to assess the six-question series in the future using the NHIS and the re-designed Survey of Income and Program Participation (SIPP) were noted, as both of these surveys will include the ACS six-question series in addition to a larger set of health and disability measures. Another suggestion was to include measures of disability that relate to program definitions. However, several participants noted the large number of definitions of disability and thought that approach would be infeasible, and one noted that the design of the six-question ACS series was intended to capture this in a very broad fashion.
Increasing Access to Administrative Data. TAG participants noted that several limitations of existing disability-related data could be eased with use of administrative data. Administrative data related to program participation is generally considered to be more accurate than survey data. Participants suggested that comparisons of administrative and survey measures could serve as a check of data accuracy and potentially alleviate concerns about survey measure validity. Administrative data also provide a longitudinal perspective on program participation without having to repeatedly interview respondents. One participant expressed that his ideal administrative data system would be similar to those in Scandinavian countries, where cross-program linkages are more frequently conducted, but recognized that this was unlikely due to privacy concerns. In order to make administrative data more accessible, TAG members suggested that administrative data should be made more user-friendly, perhaps through improved documentation, technical assistance, or the creation of variables that are useful for research as opposed to program administration purposes. Even if a national disability survey were to be established, one participant suggested that a cross-sectional survey would benefit from links to administrative data such as earnings records. However, only those individuals participating in a given program are included in administrative data.
Increasing Awareness and Use of Existing Data. The perception among several TAG participants was that a lot of good disability-related data exists, but that knowledge and use of some data sources are limited. Several Census Bureau efforts were noted as being underutilized, such as small area estimates, which can calculate prevalence rates at the school district level, and research data centers, where researchers can go to gain access to results run on restricted data. The Census is also developing synthetic data files, which use variable imputations to mask the identity of survey respondents while making previously sensitive data available to the public. One participant also mentioned the opportunity to coordinate state data collection efforts to produce a large, national database. Electronic health records were cited as having potential for research in the future, but currently have a long way to go in terms of standardizing content in a way that would be useful for research. Dissemination of these efforts to the research community might help many researchers answer questions currently perceived to be unanswerable with existing data. One suggestion was to organize a disability statistics group, similar to the former Interagency Committee on Disability Research, which would hold monthly phone calls and periodic meetings to share information.
Field a National Disability Survey. Although the majority of meeting attendees agreed that a lot could be done with existing data, the group supported the idea of fielding a national disability survey. One participant noted that the United States is one of the only industrialized nations without a national disability survey, to which another responded that it was time for an ambitious project in the United States. Administrative data, survey data, and efforts to advance the state of disability research were described as fragmented. A combined national project was identified as having the power to unify this fragmented system and to provide a variety of information on people with disabilities without having to link across many surveys, which may not be feasible. A national survey would also have the potential to provide a lifetime perspective of disability that seems to be missing in current data.
TAG members had varying ideas about the ideal national disability survey. The National Health Interview Survey on Disability (NHIS-D) was described mostly as a random collection of disability-related questions that were unaddressed by the surveys that existed at the time. It was suggested that a future disability survey be more focused and have an overarching theme. Inclusion of information on specific conditions and program effectiveness was considered necessary by one member. There was disagreement about the ideal frequency of a national survey, but many participants noted that longitudinal data were needed. The model of the National Cancer Institute, which funds a 15-minute supplement to an existing survey every five years, could be adopted. Others suggested that a longer time frame would be sufficient, such as a large national survey conducted every 10 or 20 years. Yet others wanted a yearly survey, perhaps with boosted samples every 10 years to provide state-level estimates. One participant said his ideal survey would be similar to the longitudinal Health and Retirement Study (HRS), but inclusive of all age groups, or minimally, the younger age groups not covered by the HRS. Many TAG participants liked the NHIS-D model of using an existing survey as the basis for sampling; the NHIS and Medical Expenditure Panel Survey (MEPS) were suggested as potential starting points. Another suggestion was to take advantage of the event history calendar that will become a part of the re-designed SIPP. The calendar allows respondents to indicate important milestones that might be the cause or consequence of specific events for which SIPP data are collected. Question prompts specific to disability could be added.
Several obstacles to a national survey were mentioned. There was skepticism about the potential to keep people in a longitudinal survey over extended periods of time. It was also noted that a stand-alone survey targeted to people with disabilities would not allow for comparisons to people without disabilities. One participant pointed out that there is interest in more than just the health of people with disabilities; the affect of disabilities on all aspects of life are important to consider. Accordingly, a disability survey would need to include a large number of topics. Another participant agreed and noted that it is the responsibility of each government agency and survey to include disability measures and content. Finally, there was concern about funding. It was noted that securing the funding for such a large project would be difficult and disability advocates may not be in favor of federal spending on survey research, but rather, would prefer the resources to be devoted to programs and services.
Designate an Agency to Coordinate Efforts. Having an organization to spearhead efforts was mentioned during discussion of each of the above options. TAG members acknowledged that improvements in data covering older individuals were achieved through an active research community and strong relationships between the government and universities. Finding a way to mirror their success in the disability arena would facilitate disability data improvement efforts. Participants believed that the inclusion of the ACS six-question series in several surveys marked a rise in interest in improving disability data and that now is a good time to build on that momentum. One TAG member called for capacity building and suggested that funding research on people with disabilities and disability-related data might help towards this end. Designating a lead organization or agency to advocate for and coordinate such efforts was perceived to be necessary. Proposed organizations include the Office of the Assistant Secretary for Planning and Evaluation, the National Center for Health Statistics, the Office of Management and Budget, and the National Institute on Disability and Rehabilitation Research.
Input received from federal agencies with program and policy responsibilities, federal agencies with survey and statistical responsibilities, state agencies, private contractors, and academic researchers received via written survey, an FAEP meeting, and a TAG meeting suggests that there are many limitations to existing disability-related data. These limitations span many topical areas and have a variety of sources. However, the discussions that occurred during the TAG meeting suggest that significant improvements could be made through relatively modest and well coordinated actions.
The most ambitious effort to improve disability-related data discussed during the TAG meeting was the fielding of a national disability survey. Participants acknowledged that while attractive, frequent cross-sectional national disability surveys or one longitudinal national disability survey would require significant resources. Still, if funding were made available, most TAG members showed a great deal of enthusiasm for this option.