Non-Elderly Disabled Category 2 Housing Choice Voucher Program: An Implementation and Impact Analysis. C. Identifying Treatment and Comparison Samples


Our empirical analysis was based on identifying nursing facility residents in treatment and comparison areas and comparing differences in community transition rates before and during the period of voucher availability. Accordingly, we begin this subsection by describing the selection of treatment and comparison areas. We then describe the process of selecting a sample of individuals from the treatment and comparison areas who were likely to use NED2 vouchers.

1. Defining Treatment and Comparison Areas

For purposes of analysis, we pooled within-state treatment areas. We combined the Baltimore County and Baltimore City PHAs into one area (hereafter "Baltimore") and Snohomish and Tacoma into another area (hereafter "Washington") for analysis. The Cincinnati PHA was the only NED2 PHA analyzed in Ohio.30 Pooling provided us with more statistical power to detect a significant effect of voucher availability on transitions to community-based settings.31 Accordingly, we also present a specification for which we pooled samples across all treatment sites and another for which we pooled samples for the two sites with the most similar baseline characteristics: Baltimore and Cincinnati.

We assessed the comparability of treatment areas along several dimensions before pooling, including two housing-related factors: rental vacancy rates and fair market rents. Rental vacancy rates are available through the Census Bureau, based on the 2011 American Community Survey. Fair market rents are determined annually by HUD as the 40th percentile of the sum of rent and utilities for a two-bedroom apartment in a given area. This index is calculated for metropolitan areas and non-metropolitan counties. Several areas in our analysis were considered to be part of the same area and accordingly to have the same fair market rent.

Baltimore City and Baltimore County are contiguous and were similar based on several housing-related measures. Both areas had low rental vacancy rates (7.4 percent and 5.5 percent, respectively) and were considered to have the same value for fair market rent. The proximity and similarity between the two areas were noticeable in the overlap of transition behavior; we observed residents who resided in the county but used city vouchers to help facilitate their transitions to the community. Furthermore, Baltimore City and Baltimore County had low community transition rates at baseline: 16 percent and 24 percent, respectively. We calculated transition rates based on the analysis sample, the development of which is discussed below.

Although not contiguous, Snohomish County and Tacoma are both close to Seattle, with Snohomish directly to the north and Tacoma directly south. The two areas had similar rental vacancy rates (4.6 percent and 7.7 percent, respectively) and similar values of fair market rent ($1,176 and $1,018). In addition, both areas were observed to have high baseline transition rates among likely voucher users (75 percent in Snohomish and 67 percent in Tacoma).

We identified comparison areas for each PHA included in our analysis based on population size, rental vacancy rates, fair market rents, and baseline community transition rates from nursing homes. Like treatment areas, comparison areas were similar to the treatment areas within the same state along many dimensions and were pooled (Table III.4). For example, the fair market rent value in Cincinnati was within $26 of that in the pooled comparison areas in Ohio. Larger differences emerged on some characteristics across other areas, however. For example, the Washington comparison areas had a higher fair market rent value than the Washington treatment areas. At the same time, the baseline community transition rate was similar in the two areas. Indeed, the difference between baseline community transition rates in the three treatment areas and the pooled samples of corresponding comparison areas was not statistically significant. Although not all comparison areas aligned with corresponding treatment areas on all criteria, those selected represented the most comparable within-state areas and generally exhibited similar housing opportunities. The estimation methodology assumes that, in the absence of NED2, observed differences in transition rates across treatment and comparison areas would have been very similar in the post-intervention period after adjusting for changes in the observed characteristics of individuals in the sample.

TABLE III.4. Characteristics of Treatment and Comparison Areas
    Population     Rental Vacancy Rate  
  Fair Market Rent  
  Pre-Intervention Period  
Transition Rate
Baltimore 1,429,434 6.3 1,263 17.9
Anne Arundel, Montgomery, & Prince George's counties   2,405,240 5.3 1,416 15.9
Cincinnati 296,223 17.1 752 24.1
Akron, Cleveland, & Dayton 734,356 9.4 726 32.0
Washington 923,078 5.3 1,142 69.6
Bellingham, Spokane County, & Vancouver 720,382 6.4 784 67.4
SOURCES: Population estimates are available from the Census Bureau. Rental vacancy rates are based on the 2011 American Community Survey. Fair market rents are determined annually by HUD. The rental vacancy rates and value of fair market rents in Baltimore and Washington are based on population-weighted averages. Transition rates are based on authors' calculations using MDS data among the analytical sample in 2010, before NED2 vouchers were made available.

2. Identifying the Analytical Sample

Given the small number of vouchers relative to the size of the full samples in the treatment and comparison areas, detecting even substantively large impacts on transitions would not be possible if we included the full samples in the analysis. Holding the number of voucher users constant, the ability of the analysis to detect impacts increases as the treatment sample size decreases. To illustrate why, consider a site in which 20 vouchers were used. If the treatment sample contained 10,000 individuals, the maximum potential effect, which is the ratio of used vouchers to the voucher-eligible population, would be just 0.2 percentage points--that is, if all vouchers were used by people who would not have made transitions without vouchers, the impact on the percentage making transitions would be 0.2 percentage points. In contrast, suppose we could exclude 9,500 individuals from this sample because we know they did not meet voucher eligibility criteria or were otherwise unlikely to use NED2 vouchers (based on characteristics observed in the MDS, for instance). That would reduce the treatment sample size to 500 individuals, making the maximum potential effect size 4 percentage points.

Unfortunately, the data did not allow for definitive identification of subjects who met the eligibility criteria used to allocate NED2 vouchers in a given area. Although we could limit our sample based on two of the criteria--age (under 62 years) and residence (institutionalized)--we were unable to make exclusions based on income (unavailable in the MDS) or on disability status, which was not precisely defined. We could, however, identify the subjects in each area who were extremely unlikely to be eligible for, or to use, NED2 vouchers by comparing the characteristics of voucher users to those who did not use them.

We identified subjects for the analytical sample in several stages. First, we identified all people residing in nursing facilities in designated treatment and comparison areas at two distinct times: the period of voucher use and data availability (April 1, 2011, through December 31, 2011) and an analogous time period before vouchers were available (April 1, 2010, through December 31, 2010). The pre-intervention and post-intervention samples were treated as independent, that is, we did not track individuals over time. Next, we excluded nursing facility residents with characteristics exhibited by none of the voucher users. In all sites we excluded individuals who were under age 19 or over age 62, were in hospice care, were comatose, had difficulty communicating with others, were unable to perform bed mobility, were unable to transfer between surfaces, were unable to dress, were unable to use the toilet, were unable to eat or had extreme difficulty doing so, were unable to maintain personal hygiene or had extreme difficulty doing so, had difficulty walking in their rooms, had difficulty walking in facility corridors, or had any of the following conditions: Parkinson's disease, tuberculosis, Alzheimer's disease, or cerebral palsy. We also excluded anyone who indicated he or she did not intend to make a transition to the community--including one voucher user in Washington.

Additional site-level exclusions were based on characteristics of the voucher users in each site. Because nearly all voucher users from Baltimore and Cincinnati also met the eligibility criteria for MFP--they were Medicaid enrollees who had resided in nursing facilities for at least 90 days--we limited the analytical samples in both sites to people who met these criteria. Two voucher users in Baltimore were excluded based on nursing facility length of stay as of the date they leased their vouchers. The Washington sample, reflecting the characteristics of voucher users in the site, included some who were not enrolled in Medicaid, as well as many who had resided in nursing facilities between 20 and 90 days and beyond 90 days.32 Other exclusions were based on functional status and health conditions.

We then identified our analytical sample using propensity score matching. Propensity score matching is an econometric method that helps address the concern that program voucher users are different from non-users, and that individual differences, rather than the program itself, drive differences in outcomes. This method calculates the likelihood of program participation based on observed characteristics and summarizes that information in one number, called a propensity score. Users and non-users are then matched based on the propensity score, and impact estimates are based on differences in outcomes for the matched cases. We estimated propensity scores using a logit model of the likelihood that a subject who resided in a nursing facility in the treatment area during the intervention period used a voucher:

Equation: prob (voucher use) = 1 over 1 + e superscript -y subscript i, y subscript i = beta subscript 0 + beta subscript 1 chi subscript i

where Xi is a vector of demographic characteristics, functional status, health conditions, and institutional stay-related characteristics.33 The equation was estimated separately for each of the three sites (Baltimore, Cincinnati, and Washington).

Propensity scores were calculated as the predicted probability of voucher use in treatment areas during the intervention period based on the above logit regression and matched to the corresponding pre-intervention treatment group and both control groups using the stratification and interval matching method (see Caliendo and Kopeinig 2005). First, the treatment and comparison samples were restricted to the common support, so that individuals with propensity scores outside the range of scores for NED2 voucher users were excluded. Based on this methodology, we were able to reduce our sample sizes to levels at which the probability of detecting an impact on transitions that is equal to the maximum potential effect was reasonably high. Although this was encouraging, it did not guarantee that our analytical sample would detect an impact of that size, and if the impact were smaller, the chance it would be detected was even smaller. We refer to the set of observations we used for the impact analysis as our analytical sample (Table III.5).

For purposes of impact estimation, within each site, all remaining observations were weighted based on propensity scores. The weighted samples for each comparison and pre-treatment group within each site summed to the size of the corresponding treatment sample. We proceeded as follows. First, observations were partitioned into a set of intervals based on propensity score. Intervals were defined so that the intervention period treatment sample was uniformly distributed across intervals, and at least one observation from the pre-intervention treatment sample and one observation from each period of the comparison sample were included in each interval. For each interval in a sample, we defined the weight as the number of treatment group subjects in that interval divided by the number of subjects in the sample in that same interval. The resulting weights were used to produce all of the difference-in-difference impact estimates.

TABLE III.5. Treatment Area Sample Sizes in Intervention Period and Maximum Potential Effect on Transitions
Site   Full Treatment  
Area Sample
Analytical Treatment Area
  Sample for Impact Analysis  
Maximum Potential
  Effect on Transitions  
(percentage points)
Pooled 9,227 1,216 9.3
Pooled (Maryland & Ohio) 6,004 519 10.6
Baltimore City & Baltimore County, Maryland   3,825 362 9.7
Cincinnati, Ohio 2,179 157 12.7
Snohomish & Tacoma, Washington 3,223 697 8.3
SOURCE: HUD administrative data linked to MDS.

A comparison of the characteristics of voucher users and non-users in the analytical treatment sample made apparent that, as intended, the sample selection methodology resulted in an analytical treatment sample in which the observable characteristics of voucher users were quite similar, on average, to those of non-users, although some differences remained (first two columns of Table III.6). There were marginally significant differences in some functional limitations, with users having fewer limitations than non-users. There were also significant differences in the distribution of length of stay in a nursing facility. Finally, a lower proportion of NED2 users had dual Medicaid/Medicare coverage relative to treatment area non-users.

The method for selecting an analytical sample also produced treatment and comparison samples that were generally similar, with a few exceptions. There were significant differences in the unweighted treatment and comparison samples in several domains, including race, health conditions, functional status, and length of stay. Some of these differences were insignificant once the samples were weighted (last two columns), but somewhat surprisingly, most differences in functional status remained. This was an important reason to include the characteristics as covariates when estimating the impact of voucher availability on community transition rates.

TABLE III.6. Characteristics of Voucher Users and Intervention Period Treatment and Comparison Samples
    NED2 Voucher User  
(N = 113)
Non-NED2 Users
  in Treatment Area  
(N = 1,103)
  Treatment Areas,  
(N = 1,216)
  Comparison Areas,  
(N = 784)
  Treatment Areas,  
(N = 1,216)
  Comparison Areas,  
(N = 784)
Age 50.8 52.4 52.3 52.1 52.3 52.7
Female 38.1 41.3 41.0 39.7 41.0 45.5
Currently married 8.0 10.9 10.6 10.1 10.6 10.0
Non-White 42.5 35.5 36.2 45.8*** 36.2 30.7**
Condition: heart/circulatorya 82.3 78.8 79.1 78.3 79.1 77.7
Condition: infectionb 19.5 16.6 16.9 18.6 16.9 18.7
Condition: metabolicc 41.6 43.4 43.3 43.1 43.3 45.3
Condition: musculoskeletald 15.0 20.8 20.2 16.8* 20.2 21.2
Condition: cognitivee 6.2 9.9 9.5 5.6*** 9.5 6.1
Condition: motorf 19.5 18.8 18.8 19.8 18.8 18.0
Condition: neurologicalg 31.0 28.8 29.0 34.1** 29.0 31.9
Condition: psychologicalh 64.6 62.6 62.8 67.3** 62.8 67.5**
Condition: cancer 4.4 2.4 2.5 1.3 2.5 2.4
Condition: renal disease 10.6 10.3 10.4 10.2 10.4 11.2
Adequate visioni 93.8 93.3 93.3 92.0 93.3 92.4
Adequate hearingj 97.3 97.3 97.3 97.7 97.3 97.4
Able to make self understood 95.6 92.7 92.9 94.9* 92.9 94.0
Bed mobility--independentk 58.4 46.8* 47.9 38.9*** 47.9 37.8***
Transfer ability--independentk 58.4 45.1*** 46.3 36.4*** 46.3 35.8***
Walk in room--independentk 42.5 38.8 39.1 30.4*** 39.1 29.0***
Walk in corridor--independentk 36.3 33.4 33.6 24.6*** 33.6 24.0***
Locomotion on unit--independentk 68.1 56.5** 57.6 47.1*** 57.6 50.9***
Locomotion off unit--independentk 62.8 51.7** 52.7 43.1*** 52.7 45.2***
Dressing--independentk 50.4 36.4*** 37.7 28.4*** 37.7 29.8***
Eating--independentk 66.4 61.4 61.8 54.1*** 61.8 54.4***
Toilet use--independentk 54.9 39.7*** 41.1 33.5*** 41.1 31.9***
Personal hygiene--independentk 53.1 39.6*** 40.9 32.3*** 40.9 36.6*
Bathing--independentk 27.4 20.7 21.3 18.9 21.3 18.1*
Days in nursing facility 386.0 367.0 368.7 488.8*** 368.7 376.8
Distribution of length of stay in nursing facility
   <90 days (WA only) 15.0 33.7*** 32.0 17.6*** 32.0 33.7***
   90-180 days 23.9 17.7*** 18.3 20.4*** 18.3 18.9***
   180-365 days 23.9 15.3*** 16.1 19.1*** 16.1 15.0***
   365-730 days 22.1 18.9*** 19.2 20.2*** 19.2 16.5***
   >730 days 15.0 14.4*** 14.5 22.7*** 14.5 16.0***
Public Health Insurance Receipt
   Medicaid-only 64.6 51.7*** 52.9 53.2 52.9 46.3
   Dual Medicaid/Medicare 26.5 31.4*** 30.9 35.7 30.9 35.5
Entered nursing facility from other facility   88.5 90.8 90.5 92.9* 90.5 94.5***
Transition intent missing 68.1 58.9** 59.8 65.3** 59.8 53.9
SOURCE: HUD administrative data linked to MDS.
NOTES: Chi square tests of significance were conducted on length of stay in nursing facility and Medicaid/Medicare status; two-sample t-tests for significance were conducted on all other variables.
  1. Heart/circulatory conditions include anemia, dysrhythmia, coronary artery disease, deep venous thrombosis, pulmonary embolus, heart failure, hypertension, hypotension, peripheral vascular disease, and peripheral arterial disease.
  2. Infections include multidrug-resistant organisms, pneumonia, septicemia, tuberculosis, urinary tract infection, viral hepatitis, and wound infection.
  3. Metabolic conditions include diabetes and thyroid disorder.
  4. Musculoskeletal conditions include arthritis, osteoporosis, and fracture.
  5. Cognitive conditions include Alzheimer's disease, aphasia, and dementia.
  6. Motor conditions include cerebral palsy and paralysis.
  7. Neurological conditions include stroke, multiple sclerosis, seizure disorder, epilepsy, and traumatic brain injury.
  8. Psychological conditions include anxiety, depression, manic depression, and schizophrenia.
  9. Vision adequacy is defined as ability to see fine detail in adequate light with the aid of visual appliances.
  10. Hearing adequacy is defined as ability to hear in normal conversation with hearing appliances.
  11. Activity of daily living independence is self-reported and defined as capability to complete an activity without help or oversight.

*Indicates characteristic is statistically different from that of NED2 voucher users (Column 2) or treatment areas (Columns 4 and 6) at the 10% level.
**Indicates characteristic is statistically different from that of NED2 voucher users (Column 2) or treatment areas (Columns 4 and 6) at the 5% level.
***Indicates characteristic is statistically different from that of NED2 voucher users (Column 2) or treatment areas (Columns 4 and 6) at the 1% level.

View full report


"Cat2Housing.pdf" (pdf, 1.44Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®