Report on Study Methods: Analysis of the Effect of Regulation on the Quality of Care in Board and Care Homes

Publication Date

Jan 18, 1996

U.S. Department of Health and Human Services

Report on Study Methods: Analysis of the Effect of Regulation on the Quality of Care in Board and Care Homes

Catherine Hawes, Judith Wildfire, Vince Iannacchione, Linda Lux, Angela Greene, Vincent Mor and Linda Laliberte

Research Triangle Institute

January 19, 1996

This report was prepared under contract #HHS-100-89-0031 between HHS's Office of Family, Community and Long-Term Care Policy (now the Office of Disability, Aging and Long-Term Care Policy) and the Research Triangle Institute. For additional information about this subject, you can visit the DALTCP home page at http://aspe.hhs.gov/_/office_specific/daltcp.cfm or contact the office at HHS/ASPE/DALTCP, Room 424E, H.H. Humphrey Building, 200 Independence Avenue, S.W., Washington, D.C. 20201. The e-mail address is: webmaster.DALTCP@hhs.gov. The Project Officer was Floyd Brown.

SECTION 1. INTRODUCTION

1.1 Goals of the Study

1.2 Overview of Study Methods

SECTION 2. OVERVIEW OF STUDY DESIGN

SECTION 3. SELECTION OF STUDY STATES

3.1 Review of Regulations

3.2 Ranking the States and Study State Selection

SECTION 4. ELIGIBILITY CRITERIA FOR INCLUSION OF FACILITIES: DEFINITION OF LICENSED AND UNLICENSED BOARD AND CARE HOMES

4.1 Criteria for Inclusion of Licensed Board and Care Homes

4.2 Criteria for Inclusion of Unlicensed Board and Care Homes

SECTION 5. SAMPLING APPROACH

5.1 Overview of Sampling Design

5.2 Sample Allocation

5.3. Sample Selection

5.4 Participation Rates

SECTION 6. DATA SOURCES: INSTRUMENTATION AND DATA COLLECTION

6.1 Operator Interviews

6.2 Staff Interviews

6.3 Resident and Proxy Respondent Interviews

6.4 Structured Walk-Through Observation

SECTION 7. MEASUREMENT APPROACH

7.1 Measurement Approach: Assessing Quality

7.2 Descriptive Measures

7.3 Covariates

SECTION 8. ANALYTIC METHODS

8.1 Analytic Approach to Description of Homes, Operators, Staff, and Residents

8.2 Multivariate Analysis

SECTION 9. REFERENCES

APPENDIXES

Appendix A. Additional Exhibits

Appendix B. Instrumentation and Measures

Appendix C. Reliability and Validity of Measures

EXHIBITS

Exhibit 1. Study States

Exhibit 2. First-Stage Sample Allocation to Study States

Exhibit 3. Distribution by Network Source Listings of Potential Unlicensed Facilities

Exhibit 4. Snowball Sampling Results

Exhibit 5. Estimated Coverage of Network Lists in 50-County Subsample

Exhibit 6. Estimated Number of Eligible Board and Care Homes in the Ten-State Study Population

Exhibit A-1. Eligibility and Participation Status of Sample Facilities

Exhibit A-2. Resident Participation Summary

Exhibit A-3. Staff Participation Summary

Exhibit B-1. Summary of Content of Onsite Data Collection Surveys

Exhibit B-2. Research Issues, Data Requirements, and Sources of Data

Exhibit B-3. Summary of Quality Measures, Definitions, and Sources

Exhibit C-1. Summary of Interrater Reliability Analyses: Distribution of Item Measure of Association Scores by Type of Measure

Exhibit C-2. Content of Constructs or Domains of Environmental Quality

Exhibit C-3. Internal Consistency of Environmental Quality Constructs

Exhibit C-4. Relationship of Environmental Quality Measures to the Reputational Ratings of Board and Care Homes By "Knowledgeables"

This study had three major goals growing out of the Office of the Assistant Secretary for Planning and Evaluation's (ASPE's) interest in understanding the role that board and care can play in providing long-term care to the elderly and disabled:

To examine the effect of State regulation on the quality of care in board and care homes
To explore the differences between licensed and unlicensed homes, particularly in terms of quality of care
To provide descriptive information about board and care homes, their operators, their staff, and the residents who reside in them.

1.2 Overview of Study Methods

To accomplish these goals, the study design incorporated several activities, including a major collection of new data. However, all activities focused on the main study goals of facilitating cross-sectional comparisons among facilities and residents based on the licensure status and regulatory environment under which the homes operated.

The major components of the study included:

Review of current State regulatory approaches, using information from studies by the Office of the Inspector General and Research Triangle Institute's (RTI's) 50-State survey for the American Association of Retired Persons (AARP), to create a ranking of State regulatory environments
Implementation of a sampling plan with the following key features:
- Selection of 10 study States that represent the extremes on a continuum of regulatory systems ranging from very extensive to very limited regulation
- Selection of probability-based samples of homes, staff, and residents using a stratified, multistage, cluster design
Creation of a sampling frame of unlicensed homes using the Social Security Administration's State Data Exchange Tapes (SDX) and network sampling of State and local agencies to identify eligible unlicensed homes
Development of valid and reliable measures to capture key aspects of quality and to describe residents, homes, staff, and operators
Primary data collection in 385 licensed and 129 unlicensed board and care homes, including interviews with 490 operators, 1,138 staff, and 3,257 residents and observations of the physical environment and care of residents in each home.

This report provides a summary of the study methods and complements the technical reports of the study findings, which are presented in the Executive Summary (Hawes et al., 1995a), A Description of Board and Care Facilities, Operators, and Residents (Wildfire et al., 1995) and the Report on the Effect of Regulation on Quality of Care (Phillips et al., 1995).

SECTION 2. OVERVIEW OF STUDY DESIGN

This study's main objective was to determine whether regulation affects the quality of care in licensed and unlicensed board and care homes. Specifically, the study was designed to determine whether an extensive regulatory system is associated with better quality of care and, if such an association is found, to determine whether regulation affects licensed and unlicensed homes differently.

Prior analyses of the effect of regulation found little effect beyond improving the fire safety in facilities (Dittmar and Smith, 1983; Reschovsky and Ruchlin, 1993). However, these analyses were limited by the fact that the homes included in these studies operated in only five States. Furthermore, although it was asserted that these States varied in the nature and extent of their regulatory approaches, there was little systematic data available to support the contention that these States represented significant differences in regulatory systems. Thus, the current study included a very systematic effort to identify key features of State regulatory systems and to select States with real differences in their approach.

To examine the effect of regulation on quality, we chose a "polarized" approach to test the main hypothesis. That is, if regulation affects quality of care, the association will be most apparent when one compares the extremes on the continuum of regulatory environments. One has the best chance of detecting regulatory effects on quality by restricting the sample to homes located in States with the most extensive and most limited regulatory systems.

In the sections that follow, we discuss the definition of the study population and the sample design, including the selection of States and the selection of probability-based samples of board and care homes, their staff, and their residents. We also summarize our strategies for sample frame construction, data collection, measurement construction, and statistical analysis.

SECTION 3. SELECTION OF STUDY STATES

3.1 Review of Regulations

Our first task involved the selection of the 10 study States. Based on prior studies, we knew there was significant variation among States in several aspects of their regulatory structure (GAO, 1989; Hawes et al., 1993; Stone and Newcomer, 1987; U.S. DHHS, 1982 and 1990). The plethora of different names by which board and care homes are known across the States--from personal care and domiciliary care homes to homes for the aged, adult foster care, and assisted living--is only one reflection of this variability. States differ in how they define places that must be licensed, in what services and level of care homes may provide, and in the types of residents homes may admit and retain (Hawes et al., 1993; McCoy and Conley, 1989; Reichstein and Bergofsky, 1980). States also vary in the type of regulatory programs they have in place, from licensure to certification to registration. Some States have standards, inspections, and enforcement protocols. Others simply place a home on a list based only on the home's assertion of what it provides.

Because of this variability, our first task was to identify all the regulatory programs in each State and to collect information on the key components of their systems. We focused on the three major components of regulatory systems:

Licensure standards
Nature of the inspection process
Availability and use of compliance or enforcement mechanisms.

We then used existing data from two recent studies of State board and care regulatory systems to evaluate each State's regulatory system. The first study was a survey of State regulatory agencies by the Office of the Inspector General of the U.S. Department of Health and Human Services (DHHS). The second effort was a telephone survey of State regulatory and payment agencies, ombudsmen, and local area agencies on aging conducted by RTI for AARP. To the degree possible, both studies attempted to determine not merely what existed "on paper" in State regulatory systems but how the system actually functioned during the year preceding the survey (Hawes et al., 1993; U.S. DHHS, 1990).

3.2 Ranking the States and Study State Selection

Using information from the DHHS and RTI studies, we ranked States according to their regulatory environment. In assessing the States' systems, we arrayed the 50 States along a continuum from extensive to limited regulatory system, using the three dimensions of State regulatory environments discussed above.

We used two methods to array the 50 States along a continuum from extensive to limited regulation. First, we identified "minimum" standards of facility regulation that were present in at least two-thirds of all States. These included such features as licensing requirements that address physical structure, fire and safety standards, plan of care, residents' rights, and at least annual inspections of homes. A State received a "negative" point for each of these features that was not addressed in its licensure standards. We then identified elements that appeared to represent a more extensive regulatory environment. These relatively "rare" features included standards that addressed minimum per-resident staffing ratios, operator training and certification, and the availability and use of a range of intermediate sanctions or enforcement remedies. If States had these relatively more "rare" features, they received a separate, "positive" score. The scores (positive and negative points) for each State were then summed so that each State received a total score, and States were arrayed along the resulting continuum of scores from "high/extensive" scores to "low/limited" scores.

As part of our second ranking strategy, we surveyed the study's technical advisory group (TAG), which included board and care operators, a resident, consumer advocates, regulators, and other outside experts in quality of care. We also surveyed local ombudsmen involved in board and care at a conference sponsored by the National Association of State Units on Aging. We asked the TAG members and ombudsmen to identify the elements of a regulatory system that they thought would produce better quality of care. Using these recommendations, we developed a separate weighting system for the various features of States' regulatory systems. Thus, a State was identified as having an "extensive" regulatory system if it included all of the "minimum" elements (those features present in two-thirds of the States) and it was ranked in the top third of all States using the combined positive scores for relatively "rare" features and the features considered important by our experts. A State was defined as having a "limited" system if some "minimum" standards were not present and it ranked in the bottom third of the combined positive scores and experts' scores.

These two methods of ranking State regulatory systems produced remarkable convergence, with many States consistently falling in the "extensive" range or the "limited" range. Moreover, we tested the sensitivity of these rankings by trying different weights for the various components of the regulatory systems and found the distribution of homes in the extensive and limited categories was remarkably robust. Exhibit 1 lists all the States that these methods produced.

To further test the validity of our rankings, we also independently surveyed the TAG and ombudsmen from across the country, asking them to identify States that they believed were effective or ineffective in regulating board and care homes. Again, this list largely overlapped the one generated by our more systematic approach.

This exercise produced a comparative ranking of the regulatory environments among the 50 States. We are confident that the States in the extensively regulated range have more extensive regulations, monitoring, and enforcement policies than do those on the limited end of the continuum. However, we did not assess the States' regulatory environment with respect to a "gold standard" of very strong regulations. Further, we cannot say whether the regulatory "distance" between limited and extensive systems represents a large or small difference. Neither can we say that the extensive systems are, in fact, adequate to ensure acceptable quality of care. It is clear, for example, that extensive regulatory systems for the board and care sector do not approach the minimum Federal standards in place for nursing homes. What we can say is that there are measurable differences between the two ends of the continuum for board and care regulation. It is the effect of these differences on quality that is the focus of the study.

EXHIBIT 1. Study States
State	Number of Licensed Homes	Monthly SSI/SSP Payments	Census Region
Extensive Regulatory Systems^a
California^b	4,176	$709	West
Florida^b	2,196	583	South
Oregon^b	3,008	388	West
Minnesota	1,374	943	Midwest
Hawaii	570	838	West
New York	1,320	821	Northeast
New Jersey^b	385	536	Northeast
Vermont	175	637	Northeast
New Mexico	155	392	West
Utah	93	472	West
Oklahoma^b	82	450	South
Indiana	30	789	Midwest
Limited Regulatory Systems^a
Georgia^b	1,283	$386	South
Kentucky^b	669	655	South
Ohio	584	550	Midwest
Texas^b	206	386	South
Illinois^b	145	718	Midwest
South Dakota	113	589	Midwest
Nebraska	51	479	Midwest
Arkansas^b	104	386	South
North Dakota	38	386	Midwest
Montana	154	480	West
Wyoming	24	406	West
Regulatory system type as of 1991 or early 1992. Some States, such as Ohio and Texas, have since changed key elements of their systems, although such changes did not take place in the study States until after our data collection. Study States.

The final selection of States from among the candidates shown in Exhibit 1 was made on the basis of several additional factors. First, we attempted to achieve some regional distribution of States. Second, we attempted to balance the level of State supplemental payments (SSP) made to facilities caring for recipients of Federal Supplemental Security Income (SSI). We attempted to select some "extensive" States with relatively low SSP and some "limited" States with relatively higher SSP so as not to confound the effect of regulation. Finally, in selecting States, we considered the number of homes (and beds per 1,000 elderly) in each State, as summarized in Exhibit 1. We eliminated States with fewer than 55 licensed facilities because of the need for a sufficiently large facility population from which to sample.

After considering these additional factors and making preliminary State selections, we surveyed the licensing agencies in each proposed study State to verify that no significant changes had occurred in their regulatory systems between the times of the surveys (1989 and 1990-91) and the time of State selection (1992). One State, Ohio, was in the process of implementing significant changes that would have altered its ranking, so it was eliminated as a possible study State. We similarly made calls to each study State's licensing agency at the end of data collection to verify that no major changes had been implemented during the period between State selection and data collection. Although two States were in the process of implementing new regulations, none had been implemented during the study period.

SECTION 4. ELIGIBILITY CRITERIA FOR INCLUSION OF FACILITIES: DEFINITION OF LICENSED AND UNLICENSED BOARD AND CARE HOMES

Each study State had different definitions or criteria for licensure of board and care homes, and, even within States, there were multiple categories of homes and multiple agencies that licensed homes. Because of this variation, we adopted decision rules about inclusion and exclusion for both licensed and unlicensed homes.

4.1 Criteria for Inclusion of Licensed Board and Care Homes

4.1.1 Definition of Licensure

Our first task was to determine whether a facility was licensed using our criteria for "licensure." As noted earlier, States differ in their regulatory approaches. Moreover, there is some variability within some States between multiple agencies with responsibility for "regulating" board and care homes. For example, the AARP study found a total of 62 agencies in 50 States and the District of Columbia that regulated board and care homes. In some cases, the regulation was called licensure. In other cases, it was called "registration" or "certification." However, certification in some States referred to a facility's qualifying to receive a certain type of payment, while in another State certification was the equivalent of licensure. Thus, we defined "licensure" as applying only to those regulatory programs that promulgated standards, conducted regular inspections to determine whether homes complied with those standards, and had some type of enforcement remedies for use with noncomplying facilities. Any program, regardless of its name, that had these components was considered a "licensure" agency, and the facilities it regulated were included in our sample frame of licensed homes.

4.1.2 Exclusion of Homes Licensed for Special Populations

As a first step, we limited the study population by excluding homes specifically licensed to serve only special populations--children, the chronically mental ill, mentally retarded/developmentally disabled (MR/DD), or substance abusers. These facilities often operate under different licensure standards and have different programmatic funding compared to the vast majority of homes that are licensed to serve an unrestricted population. Because the main goal of the study was to assess the effectiveness of regulation and to describe the most prevalent homes and residents, this exclusion was necessary.

4.1.3 Exclusion of Licensed Nursing Homes

We also excluded places in which all beds were licensed as a nursing home (or other facility, such as an inpatient rehabilitation unit). If a facility was part of a multilevel facility or campus, we included only those beds not licensed as a nursing home.

Once these exclusions were made, we included in the sample all other facilities licensed as board and care homes in the State. However, the construction of the sampling frame for licensed homes was complicated by the need to "capture" all relevant licensed homes, even those referred to by other names and those licensed by more than one agency or division. This meant, for example, that in some States, such as Oregon, we included three types of board and care homes serving an elderly/mixed population: adult foster care homes, residential care homes, and assisted living facilities. However, in California we included only residential care facilities for the elderly (RCFEs) and excluded residential care facilities (RCFs) that served only persons younger than 60 with chronic mental illness or developmental disabilities.

4.2 Criteria for Inclusion of Unlicensed Board and Care Homes

Given the variety of definitions of licensure across the 10 study States, developing criteria for the inclusion of unlicensed facilities was even more challenging. Because of licensure standard variations, for example, homes that were legally unlicensed in Texas (e.g., adult foster care homes with five or fewer beds) were required to be licensed in other study States, such as California and Oregon. Similar variation was found for facilities, often referred to as "assisted living," that housed residents in apartments while they received services similar to those in more traditional board and care homes. In some States, "assisted living" facilities were required to be licensed under the board and care regulations. In other States, they were exempted or specifically excluded from these licensure requirements. Finally, in all States, we expected to find some places that ignored the licensure requirements and were operating "illegally."

To make valid comparisons between licensed and unlicensed homes, we needed some defining principles that were consistent across the study States rather than relying on the States' definitions of "unlicensed," which usually focused only on illegally operating unlicensed facilities. As a result, we developed an operational definition for an eligible unlicensed board and care home that we used across the States. This definition rested on the services the home provided or described itself as providing. Thus, we did not count as unlicensed any places that housed residents who might require services or assistance with activities of daily living (ADLs)--unless the home provided (or said it provided) "eligible" services. Thus, some residential settings that ombudsmen or others might regard as an "unlicensed board and care home" based on the presumed care needs of residents were not part of our universe of eligible facilities.

Inclusion of "Traditional" Board and Care Homes. A facility was eligible if it provided room, meals, some type of 24-hour protective oversight or supervision, and one or more eligible services (e.g., personal care, transportation to medical and dental appointments, organized recreational activities, medication reminders) to two or more adults who were not related to the operator/owner. Provision of meals, laundry, and housekeeping only was not sufficient for a facility to qualify as an unlicensed board and care home. These criteria basically followed the requirements for licensure as a board and care home in several study States, such as California and Oklahoma.

Because of the lack of consensus across the States on how to treat "apartments" in which residents received services comparable to those provided by traditional board and care homes, we also defined specific criteria for inclusion of "assisted living" facilities and other places that included apartments. This was made more complex by the fact that some of these places provided extensive supportive services while not offering our "core" eligible services, such as three meals a day or 24-hour supervision or oversight. Thus, we modified the criteria somewhat for such facilities.

Inclusion of "Assisted Living" Facilities and Apartments. A place with only apartments was considered eligible if it provided all the "core" criteria (i.e., three meals, 24-hour staff supervision, eligible services). If it did not provide at least two meals a day or 24-hour supervision but provided a significant or intensive level of supportive services (e.g., medication storage and medication "passes" to residents, money management, assistance with ADLs), the apartment facility was considered eligible.

SECTION 5. SAMPLING APPROACH

5.1 Overview of Sampling Design

The sampling plan had two purposes: to facilitate the testing of the study hypotheses and to produce samples of homes, residents, and staff that could be used to make inferences to the study population with acceptable levels of accuracy and cost. The resulting sample design was a stratified, three-stage, cluster design. First-stage sampling units (FSUs) were counties, second-stage units were homes within selected FSUs, and third-stage sampling units were residents and staff of selected homes.

Board and care homes were the primary analytic units. However, the sample of homes as drawn from a sample of 80 FSUs, each comprised of one or more counties. The sample was clustered within FSUs to facilitate construction of the second-stage sampling frame of unlicensed board and care homes. In addition, clustering the sample within a predetermined number of FSUs was an effective way to control the onsite data collection costs.

We used stratification to control the distribution of the sample. FSUs were explicitly stratified by State and implicitly by urbanicity. Explicit stratification was also used to control the distribution of the facility sample with respect to licensure status and regulatory environment. Additionally, because we expected home size to affect quality, we stratified licensed facilities to control the distribution of small, medium, and large facilities. Unlicensed facilities were not stratified because, based on anecdotal evidence and the results of prior studies, we expected there would be less variation in home size among the unlicensed facilities (i.e., most were expected to be small).

The design also had to accommodate analysis of resident and staff-level measures of quality. Thus, the sampling design had to balance the need to make accurate estimates of differences both for analyses focusing on facilities and for analyses focusing on individual resident and staff measures. To support the analytic objectives for resident-level measures of quality, the resident sample was stratified and allocated within the three size categories for licensed homes and the one size stratum for unlicensed homes. The purpose of the size stratification was to enable approximately an equal number of residents to be selected from each size category without unduly increasing the design effect at the resident level. Thus, selection probabilities of residents were made approximately equal within each size stratum. No further stratification of residents or staff into special subpopulations was used.

5.2 Sample Allocation

5.2.1 First-Stage Sample Allocation

The study's ability to examine the effect of State regulation on the quality of care in board and care homes depended heavily on the first-stage sample allocation strategy. Although an allocation proportional to the number of homes in each State would maximize the statistical efficiency of the sample estimates, a proportionately allocated sample of facilities would have been concentrated in large States. ASPE and the TAG felt that this could make the study insensitive to regulatory effects in small States. Alternatively, if the sample were equally divided among the study States, the statistical efficiency of the sample estimates would suffer. Clearly, a sample allocation strategy somewhere between these extremes was needed.

Exhibit 2 displays the first-stage sample allocation to the study States. The allocation strategy modified the proportional allocation by controlling the number of homes selected in extremely large and very small States. In particular, a minimum of two FSUs (5 percent of the subpopulation) and a maximum of 14 FSUs (35 percent of the subpopulation) were selected from any one State.

EXHIBIT 2. First-Stage Sample Allocation by Study States
State	Proportional Allocationa		Actual Allocation
State	FSUs	%	FSUs	%
States with Extensive Regulation
California	19.4	48.4	14	35.0
Florida	10.1	25.2	11	27.5
Oregon	8.6	21.5	10	25.0
New Jersey	1.6	4.0	4	7.5
Oklahoma	0.4	0.9	2	5.0
	40.0	100.0	40	100.0
States with Limited Regulation
Georgia	21.5	53.7	14	35.0
Kentucky	10.9	27.2	13	32.5
Texas	3.7	9.2	6	15.0
Illinois	2.3	5.8	4	10.0
Arkansas	1.6	4.1	3	7.5
	40.0	100.0	40	100.0
FSU = First-stage sampling units. In general, a single county corresponded to an FSU if it had at least 10 licensed homes. Otherwise, two or more counties were combined to form an FSU. Allocation proportional to the estimated total number of licensed board and care homes in each State.

5.2.2 Second-Stage Sample Allocation

The power to detect true differences in quality of care in the subpopulations (extensive regulation vs. limited regulation; licensed vs. unlicensed) was crucial to the study. Given the cross-sectional comparisons that were allowed for in the study design, we focused heavily on structural and process measures of quality. Thus, many of the key measures of quality were facility-level measures. Given this, the power to detect quality differences depended on the number of facilities in the sample.

We made the following assumptions in allocating the facility sample:

The sample should support detection of differences in the range of 11 to 12 percent for the main effects (performance of homes operating under extensive regulation versus limited regulation and licensed versus unlicensed homes).
The sample should support detection of differences in the range of 14 to 19 percent for any interaction between the main effects (regulatory system type and licensure).
The direction of the hypothesis was assumed to be one-tailed (directional), with a significance level of 0.05, power of 0.75, and an assumed design effect of 1.40.

After review of the expected differences in sample estimates, we concluded that a sample of 600 homes would be sufficient to detect quality differences. The homes were allocated among the four cells in the design as follows:

Type of Home	Regulatory Environment				Total
	Extensive		Limited		Total
	Expected	Actual	Expected	Actual	Expected	Actual
Licensed	200	195	200	191	400	386
Unlicensed	100	33	100	93	200	126
	300	228	300	284	600	512

The table shows that, except for unlicensed homes in the extensively regulated States, we nearly achieved the desired allocation.

To support the analytic objectives at the resident and staff member levels, the licensed home sample was further allocated to three size strata within each State: small (2-10 beds), medium (11-50 beds), and large (51 or more beds). The allocation enabled approximately equal numbers of residents to be selected from each size category without unduly increasing the design effect at the resident level.

5.2.3 Third-Stage Sample Allocation

Although quality of care is primarily a characteristic of homes, we developed other quality measures at the resident level (e.g., resident satisfaction, unmet care needs, autonomy) and staff level of analysis (e.g., staff knowledge of basic care routines). Data on these measures were obtained directly from residents and staff. Thus, as noted above, the resident sample was allocated so that approximately equal numbers of residents were selected from each home size in licensed facilities, with another reporting domain for residents of unlicensed facilities.

5.3 Sample Selection

5.3.1 First-Stage Sampling Units: Counties

The lack of a centralized list of unlicensed homes was a major consideration in our development of a sample selection strategy for the 10-State survey. Thus, we selected cluster sampling to screen (probabilistically) geographic areas for concentrations of unlicensed homes. We selected a two-phase probability sample of FSUs, equally divided between the States with extensive and limited regulation.

First Phase of FSU Selection. A total of 80 FSUs comprising 128 counties made up the Phase 1 sample. The FSUs were selected with probabilities proportional to the weighted sum of the number of small, medium, and large licensed homes in the county or counties constituting the FSU. In some cases where population was sparse, several counties were combined into one FSU. In a few cases, large urban counties such as Dade County, Florida (Miami) and Harris County, Texas (Houston) constituted more than one FSU. Licensure lists obtained from State licensing agencies provided the numbers of licensed facilities in each county.

Second Phase of FSU Selection. At the second phase, our sampling approach was to reduce the first-phase sample to a more manageable number of counties for the development of the unlicensed sampling frame. At the same time, we wanted to account for most of the unlicensed homes in the 128 counties or 80 FSUs. To accomplish this, we needed knowledgeable local sources who could provide us with estimates of the number of unlicensed homes in each county.

We surveyed long-term care ombudsmen in all 128 counties selected at the first phase and asked them to provide an estimate of the total number of unlicensed places operating in their area. We used these estimates to categorize the counties into low- and high-yield sampling strata; however, to avoid possible selection bias, all 128 counties were assigned positive selection probabilities, even those with estimates of zero unlicensed homes.

We selected a second-phase subsample of 40 FSUs that spanned 50 counties and accounted for 1,309 or 83 percent of the 1,580 unlicensed homes estimated by the ombudsmen to be operating in the 128 first-phase FSU counties. We confined all further sampling activities for unlicensed homes to these counties.

5.3.2 Selection of Second-Stage Sample: Board and Care Homes

Licensed Homes. All licensed homes in the 80 FSUs constituted the sampling frame of licensed homes. We created the frame from licensure lists provided by the State licensure agencies in each study State. Because the quality and currency of the lists of licensed homes varied from State to State, we made calls to each agency immediately prior to sample selection to update the lists with any "births" (newly licensed facilities) since the list was generated. However, we anticipated that some attrition would occur from the lists of licensed homes as they went out of business or otherwise became ineligible between the time of sample selection and data collection. To ensure that the desired number of participating licensed homes was attained, we selected a replicated backup sample of licensed homes from each subpopulation. The designated backup was activated whenever a primary home was found to be ineligible or refused to participate. Our target of 400 licensed homes was evenly divided among the two types of States (extensive/limited). In the 80 primary FSUs, we selected 798 licensed homes for screening and recruitment. The results of this screening and the survey participation rates among eligible licensed homes are discussed below.

5.3.3 Creation of Unlicensed Home Sample Frame

Within the 40-FSU/50-county subsample, we used network sampling (Sudman et al., 1988) to construct the sampling frame of unlicensed homes. Network sampling is a statistical technique that is often used for locating and measuring the size of rare populations. Its objective is simple: to increase the amount of information obtained during a screening by interviewing a related group or network that is knowledgeable about members of the rare population, in this case unlicensed board and care homes.

Three basic sources were used to construct a list of potential unlicensed homes: local knowledgeables; the Social Security Administration, which provided addresses of SS1 recipients who appeared to be living in a group setting; and other sources of information about retirement housing and residential settings for the elderly and disabled. The listing of main sources is shown in Exhibit 3.

For the population of unlicensed homes, the network consisted, in part, of persons in agencies who would be likely to be aware of such places. For example, hospital social workers and discharge planners are often responsible for recommending residential settings to frail elderly or persons with disabilities who are being discharged from the hospital setting. Thus, we interviewed discharge planners or case managers from acute and psychiatric hospitals, as well as Veterans' Administration (VA) hospitals in the 40 selected FSUs. We also interviewed mental health case workers or case managers in local community mental health centers. Others, such as local offices of the State licensing agency, attorney general's office, or the ombudsmen, may receive complaints about unlicensed homes. Thus, we interviewed such individuals in each of the counties in our subsample of FSUs. Because we assumed that a significant proportion of residents of board and care homes are referrals from local health and social service organizations, we believed that the combined knowledge of these organizations would account for the overwhelming majority of homes in a county. In addition, we interviewed local licensing officials if the locality or State licensed boarding homes or rooming houses and building inspectors.

EXHIBIT 3. Distribution by Network Source Listings of Potential Unlicensed Facilities
Source of Candidate Facility	Places Listed^a	Places Screened^b	Eligibles Found^c	Eligibility Rate (%)^c
State Data Exchange (SDX)	1,214	486	85	17.5
Retirement Directory	673	485	81	16.7
Life Styles Directory (TX)	17	17	7	41.2
Yellow pages	35	34	7	20.6
Hospital discharge planners	81	58	24	41.4
Adult Foster Care Program (TX)	108	33	20	60.6
State attorney general offices	18	15	15	100.0
Ombudsmen programs	24	16	15	93.8
Boarding and rooming home list (NJ)	584	200	2	1.0
Residential hotel inspectors (TX)	36	35	3	8.6
Mental health case workers	14	13	0	0.0
Other	21	21	9	42.9
Total	3,190	1,744	376	21.6
Nonunique places. Places with known telephone numbers. Eligibility rate among places screened.

We also assumed that the few homes overlooked by the referral organization network or that did not attract the attention of potential regulators or ombudsmen would be small unlicensed homes that primarily rely on word-of-mouth referrals. To account for these homes, we obtained a list from the Social Security Administration of all addresses with two or more unrelated SSI recipients in each study State. This is known as the State Data Exchange (SDX) list. We added these addresses to those provided by the network of organizations.

Finally, we added listings from the telephone book yellow pages for retirement housing, retirement apartments, personal care homes, etc.; entries from commercial books listing residential retirement settings; and facilities advertising in local newspapers. We concatenated the lists, purged duplicate entries, and eliminated licensed homes. As Exhibit 3 shows, we compiled 3,190 candidate places using the network sampling approach.

5.3.4 Screening the List of Potential Unlicensed Homes

Screening the list of potential unlicensed places was critical, since we expected that some listings (e.g., from the SDX tape) might not meet our eligibility criteria and others might be out of business. The screening to determine eligibility for inclusion in the final sampling frame was conducted by telephone. Because we relied on the operators of candidate places to self-report their licensure status, we took the following measures to guard against possible misrepresentation (both intentional and unintentional) of their licensure status. First, we categorized our study as a study of places providing a residential setting for elderly and disabled individuals rather than a study of unlicensed homes. Second, we asked whether the candidate place met the core eligibility criteria by asking about the meals, supervision, and services it provided. For places with apartments, we made further inquiries about the level of their supportive services, as discussed in the criteria for eligibility.

If a candidate place met the core criteria, we then asked about its licensure status. Although we explicitly referred to board and care licensure status as opposed to other forms of licensure (e.g., kitchen sanitation or business license), about 18 percent of the unlicensed facilities claimed to be licensed but were not by our criteria for licensure. Some were unlicensed board and care home sections of licensed nursing homes. Others reported being licensed, registered, or certified by some agency other than the State agencies that were our relevant regulatory agencies (e.g., by the VA or registered Adult Foster Care homes in Texas). To avoid excluding these eligible unlicensed homes, we probed for the name of their licensing agency, what they were licensed for, and for the length of time they had been licensed. Most of these places were found to be eligible unlicensed homes under study definitions.

Another 18 percent of the unlicensed homes identified during screening met the core criteria but claimed to be licensed with the appropriate board and care agency. We surveyed the appropriate licensing agencies to determine whether any facilities had been added to their licensure list. For these homes we were either unable to verify their claim of licensure or found that their license had expired. Thus, we classified all of these homes as eligible unlicensed facilities.

Unfortunately, we were unable to screen all of the places on our original list of potential unlicensed board and care homes. When our sources of the initial listing were unable to provide a telephone number for a candidate place, which was the case for most of the 3,000+ places listed, we sent the address to a telephone matching service. If a match could not be found, we tried local crisscross directories, and, in some cases, nontraditional sources such as credit agencies. In all, we secured telephone numbers for approximately 2,000 of the 3,190 candidate places. Some of these numbers were no longer in service; others were never answered, even though we made at least 10 telephone calls to each number. Further, for some places, the initial network informant was unable to provide sufficient information to allow us to identify an address or telephone number for the listed member (e.g., Mrs. Smith on 4th Street). Thus, we were subsequently able to complete screening for 1,744 (55 percent) of the total original listings. (Our evaluation of the sample frame coverage is discussed in Section 5.3.5 of this report.)

We were unable to determine whether unlicensed homes occurred more or less frequently on the unscreened portion of the frame (i.e., candidate places without telephones or listed numbers) than on the screened portion. Therefore, to reduce the potential selection bias between the unscreened places and the screened places, we calculated adjustment factors based on the eligibility or "hit" rate of each of the list sources shown in Exhibit 3.

Within each type of source, we assumed that the eligibility rate among unscreened places was the same as the screened places. Because we tended to screen sources with high eligibility rates more completely than other sources, we estimated an overall eligibility rate of 21.6 percent for the screened portion compared with 16.5 percent for the unscreened portion. We examined the sensitivity of our population estimates to these assumptions and found that, if the true eligibility rate is 25 percent higher/lower than what we assumed, the estimated total number of unlicensed homes increases/decreases by only 9.1 percent.

5.3.5 Evaluating Coverage of Unlicensed Sample Frame

We identified a total of 329 unlicensed homes in the 50-county subsample using the telephone screening procedures described above. Most (286) of these came from a single network source, while 39 came from two sources and only 4 from three sources. This lack of overlap among the network sources indicated that the network was not as close-knit as we had hoped. However, our subsequent evaluation of the completeness of the network lists provided some evidence that they did account for the vast majority of unlicensed homes in the subsample.

To provide some evidence of the completeness of the network lists, we used another technique for sampling rare populations known as snowball sampling (Kalton and Anderson, 1986). The object of snowball sampling is to create (or, in our case, enlarge) a sampling frame by asking known members of a rare population to identify other members of the population. If the members know each other, then repeating the process among newly identified members should produce a "snowball" effect as more and more population members are identified.

In our case, we assumed that board and care operators, like most business people, would be aware of "competition" from other nearby facilities. Our plan was to take advantage of this awareness by having operators identify one or more nearby unlicensed homes. Then, if we found that we had already identified these homes through network sampling, we would obtain some evidence of the completeness of the frame.

We began the snowball process during our in-person interviews with participating operators by asking them to list any other board and care homes (licensed or unlicensed) that they were aware of in their county. Later, we purged the lists of known ineligibles (e.g., licensed board and care homes, nursing homes, homes in nonstudy county) and then compared the remaining candidate facilities with the network lists used to construct the frame. If the candidate was not on the network lists, we attempted to contact it by telephone to determine if it was a "new" unlicensed home, that is, one not previously identified either on the screened or unscreened portions of the network lists.

As Exhibit 4 shows, a total of 316 candidate facilities were identified. However, only 79 of the candidates required further screening to determine their eligibility. The rest were either known ineligibles or were already accounted for by the network lists.

We were able to contact 58 (73 percent) of the 79 candidates and found 17 eligible unlicensed facilities not previously identified by the network lists. Only one of the 17 "new" facilities was reported by more than one operator. We attribute this apparent "isolation" among operators to the preponderance of large urban areas in the subsample.

We adjusted the snowball survey findings to account for nonresponse and then weighted the counts to estimate the number of unlicensed facilities in the 50-county subsample not accounted for by the network lists. Exhibit 5 shows that the weighted estimate of 43 "new" homes implies a 92 percent coverage rate for the network lists.

EXHIBIT 4. Snowball Sampling Results
Survey Questionnaire Results:
Licensed homes, nursing homes	192
Other ineligible places already identified	11
Unlicensed homes already identified	34
Candidates for telephone screening	79
Total	316
Telephone Screening Results:
Unable to contact	21
Ineligible places not previously identified	41
Eligible unlicensed homes not previously identified	17
Total	79

Our estimate of 585 unlicensed homes in the 50-county subsample is less than half of the 1,309 estimated by the ombudsmen. Most of this difference occurred in large urban areas where the ombudsmen estimated far more unlicensed homes than we were able to identify. We speculate that this may have been caused by the way they projected their counts in the urban areas (e.g., one unlicensed for every licensed) versus a firsthand knowledge of homes in the rural areas. In any event, the difference between the ombudsmen's estimate and the final study estimate was not statistically significant when weighted to reflect the 10-State population.

EXHIBIT 5. Estimated Coverage of Network Lists in 50-County Subsample
Estimated Number of Unlicensed Homes Found:
Screened portion of list	329
Unscreened portion of list	213
	542 (92.6%)
Estimated Number of Unlicensed Homes Missed:
Identified via snowballing	17
Identified but among the unable to contact, refusals, & eligibles not selected for the study	26
Total	43 (7.4%)
Estimated 50-County Total of Unlicensed Homes	585 (100%)

5.3.6 Estimating Size of Survey Population

As previously stated, we limited the study's survey population to licensed and unlicensed board and care homes operating within the 10 study States. To estimate the characteristics of this population, we assigned design-consistent estimation weights to the sample of participating homes. We based the weights on the probability structure used to select the sample (Iannacchione, 1992) and then made adjustments to compensate for survey nonresponse and noncoverage.

To estimate the number of licensed board and care homes eligible for the survey, we estimated the eligibility rates among licensed homes by State and size of home and then applied the rates to counts of homes supplied by the 10 State licensing agencies. Because the only ineligible licensed homes were those with only one bed or with no residents at the time of data collection, almost all (97 percent) of the licensed homes we selected were eligible for the survey.

To estimate the number of unlicensed board and care homes, we multiplied the FSU-level sampling weights by the corresponding estimated total number of unlicensed homes in each of the 50 counties selected for the subsample. Overall, we estimated a total of 1,555 unlicensed homes operating in the 10 study States in the Fall of 1993. By applying the design-consistent estimate of the standard error to this estimate, we calculated a one-sided 95 percent confidence interval with an upper bound of 2,052 unlicensed homes.

Exhibit 6 shows the estimated total number of licensed and unlicensed board and care homes by regulatory system and type of home. We estimated the overall size of the survey population to be 13,189 licensed and unlicensed homes with over 300,000 beds. Unlicensed homes accounted for less than 12 percent of total homes and about 27 percent of the total beds. On average, unlicensed homes were larger than licensed homes (52 beds per unlicensed home compared with 19 beds per licensed home). This is because most assisted living/apartment-type facilities were unlicensed. In fact, assisted living/apartments accounted for a third of all unlicensed homes and well over 80 percent of all unlicensed beds.

Finally, the population estimates indicate that the regulatory environment may influence the frequency of unlicensed homes. For example, over 25 percent of the homes in the five States with limited regulation are unlicensed compared to less than 7 percent in the five States with extensive regulation. Our research team speculates that the relative scarcity of unlicensed homes in the extensively regulated States is caused by regulatory pressures not found in the States with limited regulation, as discussed in the technical reports, Executive Summary (Hawes et al., 1995a) and Report on the Effect of Regulation on Quality of Care (Phillips et al., 1995).

Once a potentially unlicensed place was determined to be eligible, it was placed in the final sampling frame. In the 40FSU subsample, we identified 329 unlicensed homes; all were initially selected for recruitment and a second screening.

EXHIBIT 6. Estimated Number of Eligible Board and Care Homes in the Ten-State Study Population(Standard Errors of Estimates Show in Parentheses)
Regulatory Environment and Type of Home	Licensed				Unlicensed
Regulatory Environment and Type of Home	Number of Homes		Mean Beds per Home		Number of Homes		Mean Beds per Home
Five States with Extensive Regulation
Traditional Homes	8,807	(366)	15.9	(1.9)	313	(110)	22.2	(8.4)
Assisted Living Apartments	109	(61)	120.2	(26.0)	312	(118)	90.4	(14.3)
	8,916	(377)	17.2	(2.1)	625	(136)	56.3	(8.9)
Five States with Limited Regulation
Traditional Homes	2,671	(142)	22.8	(3.3)	727	(179)	17.9	(4.7)
Assisted Living Apartments	47	(13)	112.7	(37.6)	202	(45)	171.6	(17.2)
	2,718	(143)	24.3	(3.4)	929	(270)	51.3	(10.7)
Ten-State Total
Traditional Homes	11,478	(393)	17.5	(1.7)	1,040	(222)	18.6	(4.2)
Assisted Living Apartments	156	(63)	117.9	(21.5)	515	(105)	137.6	(17.2)
	11,634	(403)	18.8	(1.8)	1,555	(303)	52.5	(8.4)

5.3.7 Selection of Third-Stage Sample: Staff and Residents

Our third-stage sample consisted of staff and residents in the selected board and care facilities. We restricted our sampling frame of staff to those who were paid staff and provided some kind of direct care or services (e.g., not just meals or housekeeping). Staff were selected randomly from this frame. In addition, we administered the staff interview to 11 operators who reported that they provided some direct care or services to residents. We selected a random sample of residents within each facility.

5.4 Participation Rates

5.4.1 Overall Facility Participation Rates

As noted above, our goal was to recruit and secure participation of 400 licensed board and care homes in the 10 study States (i.e., 200 in States with extensive regulatory systems and 200 in States with limited regulatory systems). Recruitment continued until we had 479 licensed facilities that agreed to participate. The additional 79 were needed because we believed that some facilities might change status between the time of recruitment and actual field data collection. The response rate for the licensed sample was 89 percent.

We also recruited from the final listing of 329 eligible unlicensed board and care homes. At the point of telephone recruitment, we achieved a response rate of 86 percent. Two factors intervened to change these numbers and rates. First, as noted above, our goal was to recruit a total of 200 unlicensed homes, which would be evenly allocated between States with extensive regulatory systems and States with limited regulatory systems. However, the vast majority of the homes we identified and that were eligible for inclusion in the study at the time of recruitment were in the States with limited regulatory systems. Further, the vast majority of unlicensed homes in the States with limited regulatory systems were in Texas. To reduce the disproportionate presence of unlicensed homes in Texas and still preserve the original sampling objective, we selected a subsample of the Texas board and care homes. A total of 156 unlicensed homes were actually sent to the field for data collection, 146 of which were eligible at the time of field data collection.

The status of facilities in terms of eligibility and participation is summarized in Appendix A as Exhibit A-1.

5.4.2 Operator Participation

All operators who agreed that their facility would participate completed the interview. In addition, "supplementary" questionnaires were mailed to the operators that were to be completed prior to the interviewer's visit. Of the 512 operators who agreed to participate in the study, 490 (96 percent) completed the operator supplement and either mailed it to RTI or gave it to the field interviewer during his or her visit to the home.

5.4.3 Resident Participation

Residents were contacted in person by RTI field interviewers and asked to participate in the study. We contacted a total of 4,368 residents or their guardians. The overall participation rate, with the loss of both refusals and those not available, was 75 percent, with residents in licensed homes having consistently higher participation rates in terms of both availability and consent.

Residents were lost to participation for several reasons. The most significant reason was that selected residents were not available to be interviewed (17.5 percent of the sampled residents). As displayed in Appendix A, Exhibit A-2, residents were temporarily gone from the facility (e.g., visiting family, attending a day treatment program), hospitalized, deceased, or too ill for the interviewer to even seek consent for a proxy respondent interview. In addition, operators refused access to 62 residents who were selected. Refusals by the resident, a guardian, or a proxy respondent constituted only 8 percent of the total loss.

5.4.4 Staff Participation Rates

As noted earlier, all paid staff who provided direct resident care were eligible for the survey. The overall participation rate was 74 percent. As with the residents, the actual refusal rate was quite small (less than 0.4 percent), but several staff were not available during the site visit period (e.g., part-time or night-shift staff not on duty during the site visit). These data are also displayed in Exhibit A-3, Appendix A.

SECTION 6. DATA SOURCES: INSTRUMENTATION AND DATA COLLECTION

During the summer and early fall of 1993, RTI field interviewers (FIs) conducted inperson interviews in the sampled facilities with operators, staff, and residents in 10 States. In addition, interviewers followed a protocol to conduct a structured observational "walkthrough" of the home, rating various qualities of the facility. We conducted site visits to 386 licensed and 126 unlicensed board and care homes. In these visits, we conducted interviews with 490 operators, 1,138 staff, and 3,257 residents and observed the physical environment and care of residents in each facility.

This section of the report describes the data collection activity, the main topics covered during each interview, and the source of the instrumentation for each interview, as well as for the structured "walk-through" observation of the home's environment and certain care practices. A more detailed summary of the topics and items covered during each interview and the structured walk-through observation can be found in Appendix B, Exhibit B-1 and Exhibit B-2. Copies of both the English and Spanish versions of the operator, staff, resident, and resident proxy respondent interviews are available on request from ASPE. In addition, a more detailed description of facility and field interviewer recruitment, including the screening calls, interviewer training, and field operations are provided in Field Data Collection Report, Analysis and Comparison of State Board and Care Regulations and Their Effects on the Quality of Care in Board and Care Homes (Johnson and Greene, 1994).

6.1 Operator Interviews

The in-person operator interview lasted 20 to 30 minutes. Operators were also asked to complete a 15- to 20-minute mail questionnaire covering some aspects of the facilities' operations and policies. These two parts of the operator survey drew on items used in two prior board and care studies (Dittmar and Smith, 1983; Sherwood et al., 1981) and covered a variety of factors, including:

Ownership and organizational affiliation
Admission and discharge policies
Services provided or arranged by the home
Staffing levels and staff turnover
Training requirements for staff
Payer mix and average monthly charge by payer
Resident case mix in the home
Operator demographics, education, training, and experience.

6.2 Staff Interviews

Each staff interview required about 15 minutes. The interview included both questions and scenarios designed to elicit staff knowledge of how to respond to situations or provide appropriate care. The interview included questions on such topics as:

Demographics, education, and prior work experience
Training received
Length of time on the job
Knowledge of basic care procedures
Knowledge of the characteristics of "normal aging"
Knowledge of medication supervision (for staff who passed medications)
Policies and practices related to the use of physical restraints Management of behavior problems
Use of punishment and physical or verbal abuse of residents by the staff member or other staff.

In constructing the instrument, we drew on prior work by Pillemer and Moore (1989) on interviewing staff about resident abuse. We used work by Avorn and colleagues (1989) and a pharmacy consultant to construct questions and scenarios to test staff's knowledge of medication monitoring and supervision. And we drew on work by the National League of Nursing and tests developed by two licensing agencies for certification of aides working in long-term care to develop scenarios designed to test staff knowledge of basic care and monitoring. Finally, Palmore's work (1986) that tests individual's knowledge of whether particular signs and symptoms (e.g., incontinence, depression, confusion) are a normal part of aging was used.

6.3 Resident and Proxy Respondent Interviews

A 20- to 30-minute interview with residents was administered to those who consented to be interviewed (or whose legal guardian provided such consent). We attempted to complete the cognitive performance items with all residents. For residents who scored as intact on the test of cognitive performance, the full interview was administered. If a resident scored as moderately to severely cognitively impaired, or if the resident was too ill or cognitively impaired to complete even the brief set of cognitive performance items, we asked the resident's consent to interview a proxy respondent about the resident's health status. The proxy respondent was someone from the facility (the operator or a staff member) who provided daily care to the resident. About one-third of the residents had a proxy respondent.

6.3.1 Resident Questionnaire

For resident interviews, the questionnaire included five basic domains:

Health status and health care use, including mental health
Functional status (ADLs, IADLs, continence, cognitive performance)
Demographics and income
Items addressing satisfaction, social interaction, activities, and unmet needs
Medications used in the past 7 days.

In developing the items for the Resident Interview Questionnaire, we drew on several existing instruments and scales.

Cognitive Status. We used the 6-item Short Blessed Test (Katzman et al., 1983), which assesses short-term memory, orientation, and concentration.

Mental Health/Mood. We used the 5-item Mental Health Index (Ware and Sherbourne, 1992) developed by RAND that is part of the Short Form-36 (SF-36) and is widely used in medical outcomes studies to determine the presence of sad or anxious mood.

Self-Performance in Activities of Daily Living (ADLs). We used a combination of items from the National Long-Term Care Survey (NLTCS) and the federally mandated nursing home resident assessment instrument (RAI/MDS) to assess ADL function and continence of bowel and bladder. The RAI/MDS items were used to facilitate comparisons with a nursing home population (Hawes et al., 1995b).

Unmet Care Needs. We used items from the NLTCS to address residents' reports of need for new or additional assistive devices. We used two consultants--a geriatrician and a geropsychiatrist--to develop questions aimed at identifying conditions that suggested unmet health care needs, accelerated loss of function, or inappropriate care (e.g., medical conditions combined with receipt of certain medications that indicate a need for frequent monitoring by a physician; inappropriate patterns of wheelchair use). Finally, we used our own work in geriatric assessment and long-term care quality assurance to specify items aimed at determining the presence of unmet need in terms of ADL assistance.

Medications. We used items we developed on another project for the Health Care Financing Administration (HCFA) to identify all medications the resident took during the 7 days preceding the interview (prescription, over-the-counter, and long-acting medications), as well as all medications prescribed to be administered or taken "as needed" (PRN). This instrument contains the name of each medication, the dosage, and the route of administration. For residents who self-administered their medications, the resident was asked for this information. If the resident was assisted in the administration of medications or if there was central storage of medications, the facility staff who were responsible for storage of medications were asked to complete a 5- to 10-minute questionnaire listing the relevant information.

Health Conditions, Health Care Use, Demographics. Most of the items were drawn from the NLTCS.

Satisfaction, Social Interaction, Activities. Questions about contacts with family and friends were modeled on instruments developed by Sherwood and her colleagues (Sherwood et al., 1981). Items about "satisfaction" and activities were developed by the project team and focused less on "how satisfied are you" questions and more on asking for specific information about practices and life in the facility (e.g., are you allowed to get up and go to bed when you choose? can you use the kitchen to fix a snack or get something to drink if you want to? do the meals contain enough fresh fruit and vegetables? do you have access to a private place to use the telephone? what activities do you enjoy? have you done those activities in the last 14 days?).

6.3.2 Interviews with Proxy Respondent for the Resident

As noted, we selected a facility staff member who was a direct caregiver as a proxy respondent. First, we needed a respondent who had daily contact with the resident and who knew the resident's level of cognitive and physical performance. Second, many residents did not have family members or did not have family who saw them on a daily basis. Thus, facility staff seemed the best informed proxy. Using a proxy respondent, however, meant limiting the items covered and, in some cases, changing the questions designed to cover a specific functional area (e.g., cognition). The major eliminations were questions only residents could answer: "satisfaction," unmet care needs, and reports of physical or verbal abuse. The Proxy Interview Questionnaire included items in the following topic areas:

Health status and health care use, including mental health
Functional status (ADLs, IADLs, continence, cognitive performance)
Limited demographics and source of board and care payment
A limited set of items on activities and contact with family and friends
Medications used in the past 7 days.

Unless otherwise noted below, the items in these areas were the same as those used in resident interviews, although rephrased to reflect the fact that we were asking the question of a proxy respondent (e.g., "In the last 7 days, did _____ [resident's name] receive any help from another person with bathing?").

Sad or Anxious Mood. We used the RAI/MDS Version-2 items on sad or anxious mood. This allowed us to get reports of signs and symptoms (rather than self-reports of feelings) of sad or anxious mood. These have been tested for reliability and validity (Hawes et al., 1995b; Morris et al., forthcoming).

Cognitive Status. We used the RAI/MDS items on cognitive status, which measure essentially the same constructs as the Short-Blessed, including short-term memory and orientation, and add an item on cognitive skills for daily decision-making. These items have been tested for reliability (Hawes et al., 1995b) and validity (Hartmaier et al., 1995; Morris et al., 1994). Moreover, the scale can produce the same ratings (intact, moderately impaired, severely impaired) as the Short Blessed. In addition, for nearly half the residents who had a proxy respondent, we also obtained an interview with the resident that included the Short Blessed items. Thus, we were able to validate the ability of the RAI/MDS cognitive performance items to produce the same broad ratings of cognitive status.

6.4 Structured Walk-Through Observation

In addition to the interviews and mail questionnaire for the operator, RTI field interviewers were trained to conduct a structured walk-through observation of the home and its environment. This took between 45 minutes and 2.5 hours, depending on the size of the home. We used an instrument largely based on the Multiphasic Environmental Assessment Protocol (MEAP), developed by Moos and Lemke (1992). It included observation of the cleanliness and maintenance in the home, safety features, physical amenities and recreational aids, an environmental diversity. We added items asking for the FIs' assessment of the activities, grooming, and hygiene of the residents, as well as an assessment of how many residents they observed using a wheelchair or "using" physical restraints or similar supportive devices (e.g., posey vests, trunk restraints, Geri-chair with a fixed lap-board that prevents the resident from rising).

Because the structured observation of the physical environment and certain aspects of resident care relied on a different training method than the one suggested by the developers of the MEAP, we tested the instrument for reliability, using our proposed training protocols. This part of the field test involved dual assessments of the facility using what we referred to as the Walk-through Instrument. In all, dual assessments of 35 facilities were performed independently by similarly trained FIs.

Using well-established criteria, we viewed a correlation between the two raters' responses for an item as adequate for research purposes if it attained a value of .4 or higher. Similarly, we considered a value above .7 to be excellent (Fleiss, 1981; Fleiss and Cohen, 1973). In Appendix C, Exhibit C-1 summarizes the correlations for the items. As indicated, 94 percent of the nominal items, 91 percent of the ordinal items, and 100 percent of the continuous items attain correlations of .4 or higher. In general, aggregated measures or multi-item scales will show a higher level of reliability than individual items. Since both the percent agreement and the correlation coefficients were acceptable for the vast majority of the individual Walk-Through Observation items, we did not calculate the correlations for aggregate measures. We eliminated items (e.g., ratings of odors in various parts of the facility) that did not attain acceptable item reliabilities.

SECTION 7. MEASUREMENT APPROACH

This section of the report summarizes our general approach to measuring key aspects of the board and care environment, including descriptions of the homes and residents, the development of quality measures, and the construction of covariates used in our analysis of quality of care. The first section describes our conceptual approach to measuring quality and the way in which key quality measures were developed. It is followed by sections describing how key descriptive measures and covariates were constructed.

7.1 Measurement Approach: Assessing Quality

The development of measures to describe residents, homes, operators, and staff and to capture key elements of quality was another major task. We developed three basic types of measures: (1) those used to evaluate the quality of care and life in the homes, (2) those used to describe homes and residents, and (3) those used as covariates in the analysis of the effect of regulation and licensure on quality. This section describes our conceptual approach to measuring quality.

7.1.1 Overview: Defining Quality

Donabedian (1966 and 1980) and others have articulated the utility of different types of measures of quality, including the triad that often forms the framework on quality measures: structure, process, and outcome. Outcomes are considered by many to be the sine qua non of measures, which may be true in acute and ambulatory care. However, in long-term care, measures of structure and process quality are especially relevant because residents not only receive care and services, they also live in the care environment, often for years (Kane and Kane, 1988; Lohr, 1988).

Moreover, research indicates that the physical and architectural features of group living settings can influence the behavior and well-being of residents and that residents have clear preferences for certain types of physical features in the residential environment (Brennan et al., 1988; Lawton, 1977; Lemke and Moos, 1986; Moore et al., 1986; Nasar and Farokhpay, 1985; Newman, 1989; Reigner and Gelwicks, 1981). Thus, we have included several measures designed to capture the characteristics of the environment. These represent important features from the perspective not only of experts in environmental design but, more important, from the perspective of residents living in such places as congregate apartments and personal care homes (Moos and Lemke, 1988; Newman, 1989).

In general, our measures can be grouped into three categories:

Structural measures of quality, which were measured at the facility level
Process measures of quality, which were measured at both the facility and staff level
Process quality and cross-sectional outcome measures, which were measured at the resident level of analysis.

Within these three broad categories, we developed specific constructs. Below, we describe the basic constructs that showed sufficient variability to be included in the analyses. Greater detail on all the quality measures we examined is provided in Appendix B; Exhibit B-3 displays the measures we used, gives a summary description of the individual elements or items in the constructed measure, and indicates the instrument that is the source of the measure. These measures differ somewhat from the constructs contained in the Moos and Lemke structure of environmental constructs. We did not use all of their measures, but we added items of our own to capture key aspects of a facility's performance. In addition, we created constructs modeled on Moos and Lemke's that we modified based on our analyses of the items that showed variability across facilities and that scaled together.

The aspects of quality that were the focus of our analysis can be summarized in terms of the following major dimensions:

Facility-Level Structural Quality
- Safety of physical environment: This construct includes such individual items as smoke detectors, sprinklers, fire extinguisher in the kitchen, nonskid surfaces on steps and in bathrooms, and secure handrails by all stairs. Moos and Lemke's (1988) survey of residents of various residential settings shows this to be considered by residents to be the most important aspect of their environment.
- Adequacy of physical environment: This domain includes three major constructs.
  1. Availability of supportive devices (which the MEAP refers to as prosthetic aids), such as lift bars next to toilet, grab-bars in the shower/tub, and call buttons in the bathroom
  2. Availability of social/recreational aids, such as reading materials, working television, working radio/CD/cassette player
  3. Availability of physical amenities, such as game/card tables, outside sitting area, picnic table, seating area in lounge/community room
- Adequacy of staffing and services: This domain includes five dimensions. However, one dimension (facility policies on resident autonomy) showed too little variation across facilities to be useful in the analysis. The four remaining dimensions are:
  1. Availability of licensed nurses on staff, including full or part-time registered nurses (RNs) or licensed practical or vocational nurses (LPNs, LVNs).
  2. Operator policies on training of staff, which consists of two items--relating to whether the operator requires preservice or a combination of preservice and inservice training for direct care staff
  3. Operator training, which is a single item concerning whether the operator was trained in care of the elderly and disabled prior to beginning to operate a board and care home
  4. Service availability in the home, which measures whether the home makes a range of supportive services available to residents who need them, including nursing care, special diets, transportation, organized activities, and ADL assistance.
- Very low quality: We also constructed four measures of very low structural quality, based on the premise that one major goal of regulation is to prevent homes from performing below a certain "minimum" threshold. Thus, we tried various strategies for developing such a measure, concluding after investigating several options that scores in the bottom 20 percent of the facility distribution for each measure were a reasonable operational definition of the "lowest" quality. We constructed four such measures of low structural quality: safety, availability of supportive devices, availability of social/recreational aids, and availability of physical amenities.
Process Quality Measures at Facility and Staff Level of Analysis
- Staff Knowledge: included four constructs:
  1. Staff knowledge of basic care and monitoring, which included scenarios that tested their knowledge of how to respond to hypothetical cases (e.g., a resident with chest pains, a resident with new onset of incontinence, a resident taking a new medication who developed hives).
  2. Staff knowledge of ombudsmen program, which included knowledge of the name and telephone number of the ombudsman, willingness to refer families or residents to the ombudsman, willingness to call the ombudsman about problems, and whether the staff member had ever called the ombudsman.
  3. Staff knowledge of normal aging included knowledge of whether certain signs and symptoms of functional and health conditions were a normal part of aging (e.g., incontinence, memory loss, sad mood).
  4. For any staff who reported passing medications, knowledge of appropriate administration and monitoring, which included questions about how to measure an appropriate dose of a liquid medication as prescribed and signs of adverse drug reactions.
- Use of physical restraints and reasons for their use
- Staff reports of witnessing or participating in physical or verbal abuse: included yelling, threats, cursing, imposing punishment such as withholding food, and hitting or throwing things at a resident.
- Facility cleanliness and attractiveness: included cleanliness of the rooms and cleanliness, comfort, and condition of the furniture, beds linens, and towels.
- Facility environmental diversity: included variation in furniture throughout the facility, personalization of residents' bedrooms, lighting source (e.g., fluorescent or incandescent), variation in wall and floor colors and coverings.
- Very low environmental process quality: this measure captured the bottom 20 percent of scores on cleanliness/attractiveness and environmental diversity.
Process Quality and Cross-Sectional Outcome Measures at Resident Level of Analysis
- Use of medications: included three constructs:
  1. Prescription of psychotropic/antipsychotic medications, that is, the resident had a prescription for an antipsychotic, possibly PRN, whether or not it was used during the past 7 days
  2. Use of antipsychotic medications
  3. Use of medications contraindicated for the elderly by an elderly resident, using well-established criteria for medications contraindicated because of adverse interactions with other drugs the resident received, incorrect dosages, etc., or medications that, in general, are not indicated for use with elders (Beers et al., 1991; Stuck et al., 1994).
- Unmet care needs: included three basic constructs, only one of which (unmet need for assistive devices) had sufficient prevalence to be used in the analyses
  1. Unmet need for assistive devices, resident self-report (or proxy report) of need for new or additional devices, such as eyeglasses, hearing aid, cane, walker, wheelchair
  2. Unmet need for ADL assistance, which was resident self-report of need for more assistance with ADL activities or of a problem with the "quality" of the assistance (e.g., did you ever wet yourself because you had to wait too long when you requested help getting to the toilet? were you every unhappy with the way you looked when someone helped you dress? do you get to bathe as often as you want?) However, because this was asked only in the resident interview and because so few of these residents needed ADL assistance, even reports of unmet need at the 10 to 12 percent level represented too low a frequency for comparative analysis.
  3. Unmet need for health care, which included several items, including inappropriate use of a wheelchair (e.g., no functional disability that indicated need and no prior use before entering the board and care home for newly admitted residents); no physician visit in the past 12 months; no physician visit in the past 3 months for residents with conditions or a combination of conditions and medications requiring regular monitoring by a physician; and no mental health professional visit for residents with a current psychiatric condition (excluding Alzheimer's disease), but these were relatively low-prevalence events.
- Resident "satisfaction": included both traditional satisfaction items (e.g., do you get to participate in activities outside the home as often as you want to?) and more objective items (e.g., for each activity a resident reported enjoying, "how many times in the last 14 days did you get to do _____ activity?")
  1. With activities, e.g., activities the resident enjoys, outside the home.
  2. With food, e.g., how much of the time are meals tasty, well-seasoned, enough food.
  3. With the physical environment, e.g., cleanliness, comfort of the furniture, maintenance of the facility.
- Resident involvement in activities: a report of several items, including percent of time involved in activities, objective reports about trips outside the facility (other than for medical/dental appointments), number of different activities involved in during past 14 days (e.g., including both individual activities, such as reading a book, watching television, listening to music, gardening, and group or organized activities, such as playing cards, religious activities, crafts).
- Resident autonomy and choices: included questions about kitchen access, choices over such daily activities as time to arise in the morning, time to go to bed, leaving the facility during the day, skipping a meal, fixing a snack (e.g., do you get to decide when to get up and go to bed; do you have access to a kitchen to make a snack if you want to?).
- Residents' rights: freedom from physical or verbal abuse, treated with respect by staff, private use of telephone, receive mail unopened, staff knock on door to room before entering.
- Resident's council: presence of a resident's council that participates in decisions about life in the facility.

7.1.2 Construction of Quality Measures

Listed below are the general steps that we took to create the composite quality measures described in Section 7.1.1. Although we have tried to present a series of orderly steps, it is important to note that the process we used to create the measures was actually iterative, with subsequent steps often informing and suggesting modifications for measures already created.

First, we examined all items of interest to determine if they were all scaled in the same direction and rescaled items as needed. Items from the walk-through questionnaire were aggregated based upon work completed by Moos and Lemke (1978, 1992). Second, we calculated the correlation matrix for the items to determine which should be aggregated into a single measure. While doing this, we examined patterns of missing data for each individual item to determine a strategy for handling missing data in the aggregated composite measures.

Third, we aggregated single items into a relevant quality construct. The method of aggregation varied depending on the structure of the individual items. For measures derived from the series of items on the walk-through instrument, we calculated percentage scores based on the total number of positive responses possible. This resulted in a continuous quality measure for safety, physical amenities, physical attractiveness, and other structural measures. For these characteristics we also created a second measure based on the distribution of the continuous measure. We categorized facilities as being in the bottom 20 percent of the distribution versus the top 80 percent. As noted above, we used this measure to determine whether licensure and regulatory extensiveness had an impact on facilities by ensuring that facilities performed above a minimal level of quality.

For resident outcomes (e.g., satisfaction with activity, autonomy, residents' rights), we standardized the individual item scores and then added them to form aggregated measures. If these composite measures were not normally distributed, we then categorized the scores as above or below the median. Next, we examined the constructs to determine whether their internal validity was acceptable using Cronbach's Alpha. An Alpha that was greater than or equal to 0.6 was considered acceptable. If the Alpha was not acceptable, we reconstructed the measure by:

Deleting items that reduced the reliability of a scale, or
Standardizing the scores when the alpha for the standardized variables was acceptable.

We also calculated correlations between separate quality measures to determine if they were measuring different constructs or whether they could be aggregated into a single measure.

7.2 Descriptive Measures

We used fairly straightforward measures to describe residents and homes. Most come directly from individual items on the various survey instruments. In addition to these variables, we constructed several composite measures (e.g., facility occupancy rate, average payment per month per resident, counts of the number of different services offered by the facility, and an aggregate case mix classification describing the bulk of facility residents). Additionally, we used several scales to characterize residents' physical and emotional functioning: a count of the number of ADL dependencies, the five-item Mental Health Inventory (McHorney et al., 1994) and a three-level mental health status measure that was constructed from the Blessed Scale, and, for residents with proxy respondents, ratings of the resident's cognitive function using the Cognitive Performance Scale (Morris et al., 1994).

7.3 Covariates

In addition, we constructed some variables for inclusion in the multivariate analyses in which we examined the effect of regulation and licensure on quality. For example, we constructed the following variables for use as covariates in the analysis:

Proxies for resident case-mix:
- We dichotomized residents as aged (65 or older) or nonaged, with a facility defined as having a high elderly mix if 90 percent of the residents were aged.
- We dichotomized residents as having or not having a "mental, emotional or nervous condition" based on their self-report, and used the same calculation as above to denote facilities with a high mental health resident mix.
Facilities were classified as apartments only if the board and care home had only private apartments.
Facility size was used both as an implicit stratification in the allocation and selection of homes and as a covariate in the analyses. We arrived at our definitions through a process of compromise. The definitions used were: small (2-10 beds), medium (11-50 beds), and large (51 or more beds). Many of the earlier studies and several State licensing laws define small homes as up to five or six beds. However, when we viewed the distribution of licensed homes in the study States, which ranged from two beds to more than 1,000, we realized that we either had to modify the definition of "small" or have more than three size strata. Because creating more strata based on facility size would have meant a substantial increase in the size of the samples (for homes and residents), we retained only three size strata. Based on the distribution of licensed facilities in the universe of our study States, these size categories were most representative.

SECTION 8. ANALYTIC METHODS

We used basic descriptive statistics and multivariate models to analyze the data and address the primary research questions.

8.1 Analytic Approach to Description of Homes, Operators, Staff, and Residents

Licensure status and the extensiveness of regulation in the two groups of States were the independent variables against which facility and resident variables were compared. The effect of licensure status and regulatory extensiveness on facility and resident variables was examined both overall and within facility size. The statistical significance of comparisons using categorical variables was assessed using the log-likelihood ratio chi-square, a generalized chi-square statistic. The statistical significance of a comparison using a continuous variable was made based on the size of the t-statistic associated with the regression coefficient for the independent variable of interest (licensure status or regulatory extensiveness). Consistent with the sample design and original study power calculations, we used a probability level of 0.05 as the determination of statistical significance.

All analyses were conducted using weighted estimates of the number of homes and residents. To account for the multistage, complex cluster sampling techniques used, we used software that adjusts the standard errors of estimate for the intercorrelation among sampled units within clusters--SUDAAN. SUDAAN produces unbiased variance estimators for linear (or non near) statistics no matter how subsampling occurs within FSUs.

8.2 Multivariate Analyses

To study the relationship between regulation, licensure, and quality of care constructs at both the facility and the resident level, we used multivariate modeling techniques that control for multiple explanatory variables. For continuous quality measures we used linear regressions. When the outcome was binary, logistic regression was the technique of choice. Because no single measure of quality of care summarized all aspects of care, we fit multiple models with dependent variables that characterized the different aspects of care. As with the descriptive statistics, because the data used in these multivariate analyses were derived from a complex multistage sampling design, we conducted the analysis with SUDAAN.

At the facility and staff level we first fit models that included the main effects (licensure status and regulatory environment), covariates of interest (size of the facility and an indicator of whether the facility contained only apartments), and interaction terms for the following interactions: licensure status with size, regulatory environment with size, and licensure status with regulatory environment. Using a Satterwaite adjusted F statistic, we assessed the significance of the interaction terms. If an interaction term was not significant (p0.05), we dropped it from the model. If at least one of the interaction terms was significant, we retained the interaction in the model and calculated the contrasts between the different levels of the significant interaction. This enabled us to calculate odds ratios for significant comparisons of binary outcomes.

Analyses for resident-level measures were guided by a similar strategy with one exception--both resident-level and facility-level covariates were included in the models. The facility-level covariates in these models included size of the facility, whether the facility contained "apartments only," and a categorical variable that described the payor mix of the resident population. Resident-level covariates described the age, cognitive status, and mental health status of each resident. These variables were entered into the resident models as design variables that indicated whether a resident had a certain characteristic (e.g., older than 65, cognitively impaired).

We modified our modeling strategy for the resident level to include one additional step. We first fit models that included the main effects and all the covariates, allowing us to assess the significance of the covariates and determine whether to include them in an interaction model. If a covariate was significant, we retained it in a model that also included the significant interactions.

Finally, we implemented the same modeling strategy, at both the resident and the facility level, a subset of facilities that included only licensed facilities. These analyses allowed us to examine the impact of regulatory environment separately from the licensure effect and to take advantage of the more powerful sample design for licensed facilities.

SECTION 9. REFERENCES

Avorn, J., P. Dreyer, K. Connely, and S.B. Soumerai. 1989. Use of psychoactive medication and the quality of care in rest homes. New England Journal of Medicine 320(4):227232.

Beers, M.H., J.G. Ouslander, I. Rollingher, D.B. Rueben, J. Brooks, and J.C. Beck. 1991. Explicit criteria for determining inappropriate medication use in nursing home residents. Archives of Internal Medicine 151:1825-1832.

Brennan, P.L., R.H. Moos, and S. Lemke. 1988. Preferences of older adults and experts for physical and architectural features of group living facilities. The Gerontologist 28(l):84-90.

Dittmar, N.D., and G. Smith. 1983. Evaluation of the Board and Care Homes: Summary of Survey Procedures and Findings. Report to the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services, Denver Research Institute. University of Denver, Denver, Colorado.

Donabedian, A. 1966. Evaluating the quality of medical care. Milbank Memorial Fund Quarterly/Health and Society 4(181):166206, July.

Donabedian, A. 1980. Explorations in Quality Assessment and Monitoring. Vols. 1-3. Health Administration Press. Ann Arbor, MI.

Fleiss, J.L. 1981. Statistical Methods for Rates and Proportions, 2nd edition. New York, NY: John Wiley and Sons, Inc.

Fleiss, J.L., and J. Cohen. 1973. The evidence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational Psychology Measures 33:613-619.

GAO (General Accounting Office, United States Congress). 1989. Insufficient Assurances That Residents' Needs Are Identified and Met. Report to Congressional Requesters.

Hartmaier, S., P. Sloane, H. Guess, G. Koch, C.M. Mitchell, and C. Phillips. 1995. Validation of the minimum data set cognitive performance scale: agreement with the mini-mental status examination. The Journals of Gerontology: Medical Sciences 5OA(2):Ml28-Ml33.

Hawes, C., V. Mor, J. Wildfire, V. Iannacchione, L. Lux, R. Green, A. Greene, V. Wilcox, D. Spore, and C. Phillips. 1995a. Executive Summary, Analysis of the Effect of Regulation on the Quality of Care in Board and Care Homes. Prepared for U.S. Department of Health and Human Services. Research Triangle Institute, Research Triangle Park, NC. [http://aspe.hhs.gov/daltcp/reports/care.htm]

Hawes, C., J.N. Morris, C.D. Phillips, V. Mor, and B. Fries. 1995b. Reliability estimates for the minimum data set for nursing home facility resident assessment and care screening (MDS). The Gerontologist 35(2):172-178.

Hawes, C., J. Wildfire, and L. Lux. 1993. The Regulation of Board and Care Homes: Results of a Survey in the 50 States and District of Columbia--National Summary. American Association of Retired Persons, Washington, DC.

Iannacchione, V.G. 1992. ASPE Study of the Effect of Regulation on the Quality of Care in Board and Care Homes Deliverable #15: Final Sampling Plan. Research Triangle Institute. Research Triangle Park, NC.

Johnson, I., and A. Greene. 1994. Field Data Collection Report, Analysis and Comparison of State Board and Care Regulations and Their Effects on the Quality of Care in Board and Care Homes. Center for Survey Research, Research Triangle Institute, Research Triangle Park, NC.

Kane, R.A., and R.L. Kane. 1988. Long-term care: Variations on a quality assurance theme. Inquiry 25(l):132-146.

Katzman, R., T. Brown, P. Fuld, et al. 1983. Validation of a short orientation memory concentration test of cognitive impairment. American Journal of Psychiatry 140:734-739.

Lawton, M.P. 1977. The impact of the environment on aging and behavior. In J.E. Birren & K.W. Schaie (eds.), Handbook of the Psychology of Aging. Van Nostrand Reinhold Co., New York.

Lemke, S., and R.H. Moos. 1986. Quality of residential settings for elderly adults. Journal of Gerontology 41:268-276.

Lohr, K.N. 1988. Outcome measurement: concepts and questions. Inquiry 25(l):37-50.

McCoy, L., and R.W. Conley. 1989. Surveying Board and Care Homes: Issues and Data Collection Problems.

Moore, G.T., D.P. Tuttle, and S.C. Howell. 1986. Environmental Design Research Directions: Process and Prospects. New York: Praeger.

Moos, R.H., and S. Lemke. 1978. Assessing the physical and architectural features of sheltered care setting. Journal of Gerontology 35(4):571-583.

Moos, R.H., and S. Lemke. 1988. Multiphase Environmental Assessment Procedure Manual. NIMH Grant MH28177. Veterans' Administration Health Services Research and Development Service, Washington, DC.

Moos, R.H., and S. Lemke. 1992. Physical and Architectural Features Checklist Manual. Center for Health Care Evaluation Department of Veterans Affairs and Stanford University Medical Center, Palo Alto, CA.

Mor, V., S. Sherwood, and C. Gutkin. 1986. A national study of residential care for the aged. The Gerontologist 26(4):405-417.

Morris, J., B. Fries, D. Mehr, C. Hawes, C. Phillips, V. Mor, and L. Lipsitz. 1994. MDS cognitive performance scale. The Journals of Gerontology: Medical Sciences 49(4):Ml74-M182.

Morris, J.N., C. Hawes, C.D. Phillips, V. Mor, and B.E. Fries. Forthcoming. Reliability Estimates for Version-2 of the Minimum Data Set for Nursing Home Resident Assessment and Care Screening (MDS). Working paper. Hebrew Rehabilitation Center for Aged, Boston, MA.

Nasar, J.L., and M. Farokhpay. 1985. Assessment of activity priorities and design preferences of elderly residents in public housing: A case study. The Gerontologist 25(3):251.

National Citizens Coalition for Nursing Home Reform. 1985. A Consumer Perspective on Quality Care: The Residents' Point of View. Washington, DC.

Newcomer, R.J., and L.A. Grant. 1988. Residential Care Facilities. Understanding Their Role and Improving Their Effectiveness. Policy Paper No. 21. Institute for Health and Aging, University of California at San Francisco.

Newman, S.J. 1989. The bounds of success: What is quality in board and care homes? In M. Moon, G. Gaberlavage, and S.J. Newman (eds.) Preserving Independence, Supporting Needs: The Role of Board and Care Homes. American Association of Retired Persons, Washington, DC.

Palmore, E. 1986. Facts on aging: A short quiz. The Gerontologist 17(4):315-320.

Phillips, C., J. Wildfire, C. Hawes, V. lannacchione, R. Green, L. Lux, A. Green, G. Dunteman, V. Mor, and D. Spore. 1995. Report on the Effect of Regulation on Quality of Care: Analysis of the Effect of Regulation on the Quality of Care in Board and Care Homes. Prepared for U.S. Department of Health and Human Services. Research Triangle Institute, Research Triangle Park, NC. [http://aspe.hhs.gov/daltcp/reports/b&crpt.htm]

Pillemer, K., and D. Moore. 1989. Abuse of patients in nursing homes: Findings from a survey of staff. The Gerontologist 29(3):314-320.

Reichstein, K.J., and L. Bergofsky. 1980. Report on the National Survey of State Administered Domiciliary Care Programs in 50 States and the District of Columbia. Horizon House Institute, New York, NY.

Reigner, V., and L.E. Gelwicks. 1981. Preferred supportive services for middle to higher income retirement housing. The Gerontologist 21:54-58.

Reschovsky, J., and H. Ruchlin. 1993. Quality of board and care homes serving low income elderly: Structural and public policy correlates. The Journal of Applied Gerontology 12(2):225-245.

Sherwood, S., V. Mor, and C. Gutkin. 1981. Volumes 1-3: Domiciliary Care Clients and the Facilities in Which They Reside. Hebrew Rehabilitation Center for Aged, Boston, MA.

Spore, D., V. Mor, J. Hiris, E.P. Larrat, and C. Hawes. Forthcoming. Psychotropic use among elderly residents of board and care facilities. Journal of the American Geriatrics Society.

Stone, R., et al. 1982. Descriptive Analysis of Board and Care Trends in the 50 States. Aging Health Policy Center, University of California, San Francisco, CA.

Stone, R., and R.J. Newcomer. 1985. The state role in board and care housing. In Long Term Care of the Elderly: Public Policy Issues, C. Harrington, R.J. Newcomer, C.L. Estes (eds.). Sage Publications, Beverly Hills, CA.

Stone, R., and R.J. Newcomer. 1987. Board and care housing and the role of state governments. In Housing and Aging Society: Issues, Alternatives, and Policy, R.J. Newcomer, M.P. Lawton, and T.O. Byerk (eds.), Van Nostrarn and Reichold Company, New York, NY.

Stuck, A.E., M.H. Beers, A. Steiner, H.U. Aronow, LZ Rubenstein, and J. Beck. 1994. Inappropriate medication use in community-residing older persons. Archives of Internal Medicine 154:21952200.

U.S. DHHS (U.S. Department of Health and Human Services). 1982. Board and Care Homes. A Study of Federal and State Actions to Safeguard the Health and Safety of Board and Care Residents. Office of the Inspector General, Washington, DC.

U.S. DHHS (U.S. Department of Health and Human Services). 1990. Board and Care. Regional Inspector General for Region II, Office of Evaluation and Inspection, New York, NY.

U.S. House of Representatives, Select Committee on Aging. 1989. Board and Care Homes in America: A National Tragedy. A Report by the Chairman of the Subcomrnittee on Health and Long-Tenn Care, House of Representatives, Comm.Pub.No. 101-711.

Ware, J.E., Jr., and C.D. Sherbounie. 1992. The MDS 36-Item Short-Form Health Survey (SF-36): Conceptual framework and item selection. Medical Care 30:473-483.

Wildfire, J., C. Hawes, V. Mor, V. Iannacchione, L. Lux, R. Green, A. Greene, V. Wilcox, D. Spore, and C. Phillips. 1995. A Description of Board and Care Facilities, Operators, and Residents. Prepared for U.S. Department of Health and Human Services. Research Triangle Institute, Research Triangle Park, NC. [http://aspe.hhs.gov/daltcp/reports/bcdesces.htm]

APPENDIX A. ADDITIONAL EXHIBITS

EXHIBIT A-1. Eligibility and Participation Status of Sample Facilities
Status	LicensedFacilities	UnlicensedFacilities	Overall
Selected for recruitment	695	329	1,024
Unable to contact	74	46	120
Eligibility at recruitment^a	554	249	803
Refused	74	37	111(13.8%)
Agreed to participate	479(86.5%)	212(85.1%)	691(86.1%)
Selected subsample^b		156
Eligible at field data collection	448	146	594
Participated	389^a(86.8%)	123^a(84.2%)	513(86.2%)
Overall facility participation rate^c	75.1%	71.7%	74.2%
The number of licensed homes includes three facilities that later became unlicensed and were included in the unlicensed sample. Thus, the final sample is 386 licensed facilities and 126 unlicensed homes. A subsample of 156 unlicensed facilities was selected from the 212 that agreed to participate. Product of the participation rate among eligibles at recruitment and at field data collection.

EXHIBIT A-2. Resident Participation Summary
Status	LicensedFacilities	UnlicensedFacilities	Overall
Participating facilities	386	126^a	512
Residents selected	3,045	1,323	4,368
Moved/deceased	NC	NC	48
Inpatient/hospital	NC	NC	68
Operator refused	NC	NC	62
Resident too ill for FI to seek consent	NC	NC	41
Temporarily gone; other	NC	NC	605
Residents available for interview	2,513(82.5)	1,031(77.9%)	3,544(81.1%)
Resident or guardian or proxy refused	173(6.8%)	113(11.0%)	286(8.1%)
Complete resident or proxy form	2,335	910	3,245
Partial resident or proxy form	5	8	13
Total	2,340(76.9%)	914(69.4%)	3,258(74.6%)
NC = Not calculated. Includes three facilities that were selected as licensed but were unlicensed at data collection.

EXHIBIT A-3. Staff Participation Summary
Status	LicensedFacilities	UnlicensedFacilities	Overall
Staff selected	1,242	292	1,534
Staff available for interview	912(73.4%)	232(79.5%)	1,144(74.6%)
Participated	908(73.1%)	230(78.8%)	1,138(74.2%)

APPENDIX B. INSTRUMENTATION AND MEASURES

EXHIBIT B-1. Summary of Content of Onsite Data Collection Surveys

Focus of Interviews with Operators

Characteristics of the operator and training received
Characteristics of the home and staffing
Training requirements for staff
Characteristics of the residents
Types of services the home provides to residents
Admission and discharge criteria
Policies and procedures for caring for residents
Payment sources and rates

Focus of Interviews with Staff

Basic sociodemographic characteristics
Training received
Policies on use of physical restraints
Practices related to abuse and punishment
Knowledge of basic care procedures
If responsible for passing medications, knowledge of dosage and monitoring

Focus of Interviews with Residents

Physical functioning in ADLs and some IADLs
Need for additional assistance
Cognitive functioning
Health conditions
Sad or anxious mood
Use of health care services
Use of home and community-based care services (e.g., home health, senior center, sheltered workshops)
Daily activities
Contact with family and friends
Satisfaction
Basic demographics

Focus of the Walk-Through Observation

Characteristics of the Home
- Cleanliness and maintenance
- Safety features
- Physical amenities and recreational aids
- Environmental diversity: homelike vs. institutional
Observation of residents--e.g.
- Activities
- Grooming and hygiene
- Use of wheelchairs

EXHIBIT B-2. Research Issues, Data Requirements, and Sources of Data
	Sources of Data
	Walk-ThroughObservation	OperatorIn-PersonInterview	OperatorMailSurvey	StaffInterview	ResidentInterview
TO DESCRIBE BOARD AND CARE RESIDENTS, STAFF, OPERATORS, AND HOMES
RESIDENT CHARACTERISTICS
Demographics
-- sex					x
-- age					x
-- race					x
-- education					x
-- marital status					x
-- income					x
Pycho-social well-being and mental health status
-- mental retardation/developmental disability					x
-- emotional or nervous condition					x
Health status
-- need for hospital treatment					x
-- need for acute care in doctor's office					x
-- current medications					x
Physical level of impairment
-- presence of: -- diabetes -- paralysis -- arthritis -- high blood pressure -- cerebral palsy, epilepsy, MS -- cancer -- stroke -- heart attack -- foot, leg amputated					xxxxxxxxx
-- use of: -- oxygen -- wheelchair -- urinary cathetar -- ostomy/colostomy -- adult diapers					xxxxx
Falls in last year					x
Physical functioning
-- need help eating					x
-- confined to bed					x
-- confined to chair					x
-- need help getting out of bed					x
-- need help getting around home					x
-- need help toileting					x
-- need help dressing					x
-- need help bathing					x
Reluctance to report complaints					x
Family interactions
-- number of relatives living nearby					x
-- visits with relatives					x
-- visits with friends					x
STAFF CHARACTERISTICS
Demographics
-- sex				x
-- race				x
-- age				x
-- education				x
Compensation: salary, benefits			x	x
Length of time in home				x
Past work experience				x
Training			x	x
Duties				x
OPERATOR CHARACTERISTICS
Demographics
-- sex		x
-- age		x
-- race		x
-- education		x
-- marital status		x
-- income		x
Background
-- previous training		x
Opinion on regulations
-- most important		x
-- least important		x
-- complaints about regulations or inspections		x
FACILITY CHARACTERISTICS
Ownership
-- profit or not-for-profit		x
-- proprietorship		x
-- own other b&c homes or nursing homes		x
History of home
-- past penalties or sanctions		x
-- previous type of institution		x
Bed size and current occupancy
-- number of current residents		x
-- number of other people living in home		x
Licensure status and licensure agency		x
Inspection agencies		x
Staffing of home: number and training of staff			x
-- daytime			x
-- nightime			x
-- volunteers			x
Distribution of home residents by:
-- incontinence			x
-- bedfastness			x
-- chairfastness			x
-- confusion			x
-- substance abuse			x
-- AIDS			x
-- mental illness			x
-- age			x
-- in need of nursing home care			x
Types of services received by residents		x
Number of resident discharges to:
-- acute care hospital			x
-- psychiatric hospital			x
-- nursing home			x
-- own home, friend's home, or relative's home			x
-- other board and care home			x
Number of residents who died in the home			x
Cost of care		x
Source of payment for residents' care			x
Monthly rate		x
-- personal needs allowance			x
-- services not covered by monthly rate			x

EXHIBIT B-2. Research Issues, Data Requirements, and Sources of Data
	Sources of Data
	Walk-ThroughObservation	OperatorIn-PersonInterview	OperatorMailSurvey	StaffInterview	ResidentInterview
TO ANALYZE THE EFFECT OF REGULATIONS ON QUALITY OF CARE IN BOARD AND CARE HOMES
STRUCTURAL DIMENSIONS
Staffing
-- qualifications: experience, licensed			x
-- staffing levels: -- types and number of staff: paid volunteer, full-time, part-time -- duties of staff			xxx
-- staff training: types, amount, when -- medication management		xx	xx
-- staff knowledge of: -- incontinence in elderly -- emergency heart condition -- cognitive decline -- medication reaction				xxxxx
-- staff attitudes toward conditions of aging and disability				x
Availability of services
-- home: supervision, laundry, money management, personal care, special meals, medications		x			x
-- community based: home health, senior center, advocacy, nursing care		x			x
-- case management		x			x
-- therapies		x
Availability of activities
-- In-home: television, games, reading					x
-- Out of home: shopping, transportation					x
Safety of physical environment
-- presence of smoke detectors, fire extinguishers	x
-- presence of secure handrails, grab-bars	x
-- presence of call buttons	x
Adequacy of physical environment
-- pleasantness/comfort -- availability and condition of furniture -- size of common rooms -- bedrooms - size and occupancy - closet space available - privacy curtain available - adjustable temperature - personal possessions in room -- bathrooms - types - locks available	xxxxxxxxxxxx
-- cleanliness -- presence of odors, pests	xx
-- adequate maintenance -- obstruction in hallways -- security of handrails, grab-bars -- quality of nonskid surfaces	xxxx
-- adequate lighting	x
-- treatment of window areas	x
PROCESS DIMENSIONS
Resident's rights
-- freedom from abuse				x	x
-- visitation by family and friends					x
-- presence of resident's council					x
-- presence of process for reporting complaints				x
-- knowledge of appropriate advocacy agencies				x
Autonomy
-- management of own money					x
-- make phone calls privately					x
-- access to kitchen					x
-- freedom to set own schedule					x
-- required to do chores					x
-- privacy of mail					x
Provision of care
-- supervision of medications		x			x
-- unmet care needs					x
-- need for aids or assistive devices					x
-- inadequate assistance with: -- eating -- getting out of bed -- getting around home -- toileting -- dressing					xxxxxx
-- use of needed services					x
-- appearance of residents: grooming, dress, cleanliness	x				x
-- use of restraints	x			x	x
-- use of pychotropic drugs	x			x	x
OUTCOMES
Quality of life
-- time involved in activity					x
-- social isolation -- family involvement -- freedom to have visitors					xxx
-- safety of possessions					x
Resident satisfaction with
-- meals					x
-- environment					x
-- activities and social interaction					x

EXHIBIT B-3. Summary of Quality Measures, Definitions, and Sources
Quality Measure	Definition	Source
FACILITY-LEVEL STRUCTURAL QUALITY MEASURES
Operator policies on training of staff	Preservice or preservice plus in-service training required	OP
Training of operator	Received training on how to operate and care for elderly/disabled prior to becoming operator	OP
Staff type available	Any licensed nurse (RN/LPN/LVN) available (full or part time)	OP
Facility policies on resident autonomy	Resident can east when they want to, flexibility of visiting days/hours, resident can leave home without permission (total score possible = 4)	OP
Service availability in home	Count of services (e.g., transportation, activities, personal care, nursing care) (total = 8)	OP
Facility safety	Percent of safety features present (e.g., smoke detectors, fire extinguisher in kitchen, secure handrail on stairs) (total of 100)	WK
Prevalence of supportive/ prosthetic devices	Percent of devices present (e.g., grab-bars in toilet, shower, call buttons in bathroom) (total score of 100)	WK
Prevalence of social/ recreational aids	For example, working radio/tape/cassette/CD player, working television, reading materials (total score possible = 100)	WK
Prevalence of physical amenities	Presence of social/recreational amenities (e.g., card tables, outside sitting area, picnic table)	WK
Very low facility safety	Facility in bottom 20 percent of safety distribution	WK
Very low availability of supportive devices	Facility in bottom 20 percent of distribution on supportive devices	WK
Very low prevalence of social/ recreational aids	Facility in bottom 20 percent of distribution on social/recreational aides	WK
Very low prevalence of physical amenities	Facility in bottom 20 percent of distribution on amenities	WK
Staff knowledge of basic care/ monitoring and MEDS	Correct answers on basic monitoring (e.g., what to do if resident has chest pain and shortness of breath, sudden onset of incontinence) and, if handles medications, knowledge of signs and symptoms, dosage, monitoring needed	S
Staff knowledge of ombudsman program	Knows name and telephone number of ombudsman, willing to call ombudsman if a problem, referred resident or family	S
Staff knowledge of normal aging	Percent of staff with correct answers on basic facts on aging (e.g., yes/no on incontinence is normal, memory loss is normal, persistent sad mood is normal)	S
FACILITY-LEVEL PROCESS QUALITY MEASURES
Use of physical restraints	Percent of staff who report facility uses restraints (e.g., poseys, chairs with fixed lapboards)	S
Prevalence of physical or verbal abuse or punishment	Percent of staff who report seeing other staff hitting, yelling, threatening, isolating resident, withholding food or privileges.	S
Facility cleanliness and attractiveness	Cleanliness of resident and community rooms, condition of furniture, linen, etc. (total score of 100)	WK
Facility environmental diversity/ "homelikeness"	Percent of features of diversity/personalization (e.g., lighting source, personalization of rooms)	WK
Very low cleanliness	Facility in bottom 20 percent of distribution	WK
Very low diversity	Facility in bottom 20 percent of distribution	WK
RESIDENT-LEVEL PROCESS AND CROSS-SECTIONAL OUTCOME MEASURES
Use of medications	Resident use of psychotropic medications; indicators of potential for adverse interactions or inappropriate use of medications	MED
Resident has unmet needs for ADL assistance	Resident reports needing more help; OR proxy respondent reports resident has difficulty, is severely cognitively impaired, and received no help from another person	RPR
Resident has unmet need for assistive devices	Resident reports needing new or different assistive devices (e.g., glasses, cane, hearing aid, wheelchair)	R
Resident has unmet need for mental health services	Likelihood of resident seeing a MH-professional or physician for MH problem in past year, controlling for anxiety and depression	RPR
Resident has unmet need for physician care	Not seen physician in past 12 months; OR has condition requiring monitoring and has not seen physician in past 3 months.	RPR
Resident satisfaction with activities	Gets to participate in activities outside the home as often as wants to	R
Resident satisfaction with physical plant/ environment	Satisfaction with cleanliness, comfort of furniture, maintenance	R
Resident satisfaction with food	How often/much of the time are meals generally good, tasty, and well-seasoned, enough fruit and vegetables, overall rating	R
Resident activities	How often outside the home in past 14 days amount of time involved in activities, how many different activities in past 14 days	RPR
RESIDENT-LEVEL PROCESS AND OUTCOME QUALITY MEASURES
Resident activities	How often outside the home in past 14 days amount of time involved in activities, how many different activities in past 14 days	RPR
Resident satisfaction with activities	Of activities resident enjoys, how many done in past 14 days; how often outside the home in past 14 days amount of time involved in activities, how many different activities in past 14 days	R
Resident autonomy/ kitchen	Access to kitchen to make meal or snack, get drink of coffee, tea, soda, juice, etc.	RPR
Resident autonomy/ choice over daily activities	Allowed to decide in five areas (e.g., can decide when to get up, go to bed, leave the facility during the day)	RPR
Residents' rights	Treated with respect by staff, private use of phone, receive mail unopened, staff knock on door, no physical or verbal abuse or punishment; feels possessions are safe	R
Residents' council	Presence of residents' council	RPR
O = Operator Interview or Operator SupplementS = Staff InterviewR = Resident InterviewMED = Medication Sheet for all ResidentsPR = Interview about Resident with Proxy RespondentWK = Walk-through ObservationLevels/Unit of Analysis: O = facility level; S = staff level; R & PR = resident level of analysis.

APPENDIX C. RELIABILITY AND VALIDITY OF MEASURES

A. Reliability and Validity of the Walk-Through Observation

Because we modified the MEAP and because we were using a different protocol for training the field interviewers (FIs) who would complete the environmental assessment, we felt we needed to establish the reliability of the instrument and our approach. As a result, we tested our instrument for interrater reliability, that is, to determine whether two "raters" using the same instrument at the same time period would independently arrive at the same conclusion regarding the responses to the items on the instrument. Interrater reliability is essential on the walkthrough since we must be confident that any observed differences among homes are a product of "real" differences among the homes rather than among the raters." This is particularly critical since the main goal of the study is to determine whether differences in regulatory systems are associated with observed differences among homes.

To determine the reliability, we tested the instrument using two teams and a total of four FIs. These FIs independently assessed 35 distinct board and care homes in North Carolina and Tennessee. These homes had been purposively selected, based on their reputation among ombudsmen and licensing officials, to include both homes reputed to provide excellent quality of care and homes reputed to provide very poor quality. Within these, FIs completed the "Walk-Through Observation" instrument in each of 35 distinct board and care homes, with each facility being independently assessed by two FIs. The dual assessments of the 35 homes were completed under a protocol that had two FIs in the home on the same day but conducting the observation-based assessments independently. By utilizing a unique facility identification number, we constructed a SAS analysis file that merged the dual answers of the two interviewers for each item into a single file that was used to evaluate the consistency of the dual responses.

The statistical measure of reliability that is the focus of this discussion assesses the congruence between the judgments of the two "raters," the FIs. Since the measures obtained from the Walk-Through Observation Instrument include nominal, ordinal, and continuous measures, we used different measures of association. For selected nominal and ordinal items, we first calculated the percent agreement between the two interviewers. Exhibit C-1 summarizes the results of these analyses. (The item-by-item reliability values are available on request from the authors.)

Next, we calculated measures of association for each item in the instrument. We used the Kappa statistic for nominal items. Kappa provides a more conservative estimate of reliability than "percent agreement" because it adjusts for "chance" agreement between the two raters (Winer, 1962). Spearman's rank order correlation coefficient was used for ordinal measures, and Pearson's R was used for continuous measures.

EXHIBIT C-1. Summary of Interrater Reliability Analyses: Distribution of Item Measure of Association Scores by Type of Measure
	Type of Measure
	Nominal (%)	Ordinal (%)	Numeric (%)
Percent Agreement (%)
14-39	0	1	NC
40-74	5	56	NC
75-100	95	42	NC
Correlations
.14-.39	6	9	0
.40-.74	28	59	20
.75-1.0	66	32	80
NC = Not calculated.Type of measure by measurement type: Nominal - Kappa Ordinal - Spearman's Rho Numeric - Pearson Correlation Coefficient.

Using well-established criteria, we viewed a correlation between the two raters' responses for an item as adequate for research purposes if it attained a value of .4 or higher. Similarly, we considered a value above .7 to be excellent (Fleiss, 1981; Fleiss and Cohen, 1973). (As noted above, the item-by-item results are presented in the Attachment.) Exhibit C-1 summarizes the correlations for the items. As indicated, 94 percent of the nominal items, 91 percent of the ordinal items, and 100 percent of the continuous items attain correlations of .4 or higher. In general, aggregated measures or multi-item scales will show a higher level of reliability than individual items. Since both the percent agreement and the correlation coefficients were acceptable for the vast majority of the individual Walk-Through Observation items, we did not calculate the correlations for aggregate measures.

B. Reliability and Validity of Environmental Quality Domains

Once we had established the interrater reliability of the items in the environmental assessment instrument, two additional analytic tasks remained: establishing the validity of these items and the domains of the environment they capture as measures of quality and further examining the reliability of these larger measures or domains.

1. Content and Reliability of Environmental Quality Measures

Moos and Lemke (1978, 1992) constructed an instrument that assesses the physical environment in terms of seven constructs. These are:

Safety
Resident activity
Physical attractiveness
Environmental diversity
Prosthetic devices
Physical amenities
Social and recreational aids.

These constructs or domains consist of a number of individual items from the instrument. These domains and their content are summarized in Exhibit C-2. Moos and Lemke have established the internal consistency of these constructs using all the MEAP items they specify. However, because of some special issues we wanted to address in this study, we did not include all the MEAP items and modified some items we did include. Thus, we felt we needed to examine the reliability of these constructs to ensure they were still useful and reasonable constructs.

During the field test, we conducted preliminary correlation analyses and factor analyses to determine whether the items within each dimension appear to characterize related attributes of a facility. These preliminary analyses suggested that, for the most part, there was substantial internal consistency among the items in each domain and thus, that they were measuring related attributes; however, the sample size from the field test is too small to make final determinations. Thus, we conducted similar analyses using data from the full field effort in 10 study States on 512 homes. These analyses confirmed our earlier findings on the internal consistency of these constructs. As shown in Exhibit C-3, which displays the Cronbach's alphas for these constructs, all but one construct meet statistical standards for acceptable levels of internal reliability (.65 or higher), while three attain levels that are good (.8 or higher).

2. Validity of Environmental Quality Measures

Most of these measures of environmental quality have substantial face validity. For example, most people consider safety of the physical environment as a necessary if not sufficient condition for acceptable "quality" of care. Safety is an area that is not only relevant to reducing the risks of adverse event for residents; it is also an area addressed in all of the regulatory standards that licensed homes must meet and, for many unlicensed homes, in the building code and fire safety standards they must meet simply to be in business (but external to the operating standards set for facility licensure). Similarly, such measures as environmental diversity capture elements that others have found indicative of quality to nursing home residents, such as evidence of personal possessions (resident's individual furniture, photographs, and so on in their bedrooms) (National Citizens' Coalition, 1985). They are likely to be similarly important to residents of board and care homes.

EXHIBIT C-2. Content of Constructs or Domains of Environmental Quality
Dimension: Safety Features and Supportive Devices
Smoke detector	Adequate light in hall, stairs, bath
Fire extinguisher	Non-skid stairs
Fire sprinkler	Outside area visible to staff
Handicapped access	Presence of secure handrails
Call buttons	Grab-bars in bathrooms
Width of hall	Obstructions in hall
Dimension: Physical Attractiveness and Cleanliness
Cleanliness of rooms, halls, stairs	Pests (e.g., roaches, flies)
Odor	Maintenance
Condition of furniture	Condition of outdoor furniture
Condition of shades, blinds, curtains	Condition of bed linens, towels
Noise	Characteristics of neighborhood
Dimension: Environmental Diversity and "Homelikeness"
Variation in bedrooms	Personalization of bedrooms
Variation in furniture in community rooms	Lighting source
Dimension: Social and Recreational Aids
Outside recreation/sitting area	Outside chairs and tables
Outside covered area	Outside barbecue area, picnic area
Card or game table(s)	Reading material
Working television	Working radio, stereo, cassette player
Dimension: Resident Functioning
Residents appropriately dressed	Any unclothed resident(s)
Appearance of clothing (e.g., cleanliness)	Residents' grooming
Amount of resident activity	Number of residents involved in activities
Dimension: Prosthetic Devices
Physical restraints	Wheelchairs

While such measures as cleanliness, diversity/personalization, and safety appear to have considerable face validity, we felt it was important to further examine the issue of whether these measures are valid. Thus, we examined the degree to which the ratings of homes in the field test converged with another "external" measure of the facilities' quality. Convergent validity--the correlation between a "new" set of measures and an established set f measures--is a classic way of assessing validity (Campbell and Fiske, 1959). Thus, in assessing the validity of key elements of the study's proposed quality measures, we examined the degree to which the ratings derived from study measures converge with the reputational ratings of quality provided by "knowledgeables" (e.g., ombudsmen, county social service agency workers, State health department inspectors, licensing agency staff). (The derivation of the reputational ratings is described in Section III.A.2 of the Field Test Report, Hawes et al., 1993.)

EXHIBIT C-3. Internal Consistency of Environmental Quality Constructs
Quality Construct/Domain	Cronbach's Alphafor Raw Variables	Cronbach's Alphafor StandardizedVariables
Resident functioning	.63	.72
Physical attractiveness and cleanliness	.90	.90
Environmental diversity/homelikeness	.83	.83
Safety	.65	.64
Prosthetic devices	.70	.69
Physical amenities	.84	.85
Social and recreational aids	.57	.56

We created a file on 35 homes with the data from the 70 Walk-Through Observation Instruments. We had reputational ratings on the overall "quality" for 30 of those homes. (The remaining five homes were unlicensed, and neither the licensing agency staff nor the ombudsmen could report on the reputation of these homes.) We then added a variable to the file that describes the reputational rating of each of the 30 homes. This variable characterizes each home as providing "good, moderate/medium, or poor/bad" quality of care, according to the rating provided by knowledgeable individuals.

Using both conceptual models of quality developed by Moos and Lemke (1978, 1992) and the quantitative analyses described above, we constructed aggregate quality of care measures from the individual Walk-Through Observation items. These multi-item scales are designed to characterize specific domains of quality.

The next step in the analysis was to determine whether these constructed measures of environmental safety and quality converge with the ratings of homes by knowledgeable individuals. Since the reputational ratings were ordinal measures (i.e., good, moderate, poor), we transformed the walk-through measures into an ordinal scale and then calculated a Spearman correlation coefficient. The results of these analyses are displayed in Exhibit C-4.

We found a high correlation between most of the individual domains of quality measured with Walk-Through Observation Instrument items and the overall quality ratings given to those same homes by knowledgeable individuals. Indeed, the strength of the association between the two different measures is particularly notable since the Walk-Through Observation Instrument provides only a portion of the measures we propose to use in the main study, measures that are likely to increase the congruence between the study's measures and the ratings of homes provided by another source. The addition of such measures as resident autonomy and choices, resident satisfaction, and adequacy of care, as well as staff training and knowledge will allow us to capture other important dimensions of quality. The results also demonstrate that the study measures are appropriate, capture important aspects of quality, and can be used to distinguish among homes based on the quality of care and life they provide.

3. Conclusion

Our conclusion is that the measures of environmental quality have both the requisite reliability and the validity to support further analyses of the distribution of quality among homes in the study sample as well as to examine the effect of regulation and licensure on environmental quality.

EXHIBIT C-4. Relationship of Environmental Quality Measures to the Reputational Ratings of Board and Care Homes By "Knowledgeables"
Quality Construct/Domain	Correlation Coefficient withthe Reputational Rating
Resident functioning	.56
Physical attractiveness and cleanliness	.65
Environmental diversity/homelikeness	.40
Safety features and devices	.71
Prosthetic devices	.xx
Physical amenities	.50
Social and recreational aids	.52

References

Campbell, D.T., and D.W. Fiske. 1959. Convergent and discriminant validation by the multitrait-multimethod matrix, Psychological Bulletin, 58:81-105.

Fleiss, J.L. 1981. Statistical Methods for Rates and Proportions, 2nd edition. New York, NY: John Wiley and Sons, Inc.

Fleiss, J.L., and J. Cohen. 1973. The evidence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational Psychology Measures 33:613-619.

Hawes, C., J. Wildfire, and L. Lux. 1993. The Regulation of Board and Care Homes: Results of a Survey in the 50 States and District of Columbia-National Summary. American Association of Retired Persons, Washington, DC.

Moos, R.H., and S. Lemke. 1978. Assessing the physical and architectural features of sheltered care setting. Journal of Gerontology 35(4):471-583.

National Citizens Coalition for Nursing Home Reform. 1985. A Consumer Perspective on Quality Care: The Residents' Point of View. Washington, DC.

Winer, B.J. 1962. Statistical Principles in Experimental Design. McGraw-Hill, New York.

Files

stumthes.pdf (pdf, 1.39 MB)

Topics

Quality Measurement | Long-Term Care Policy & Regulation | Information Quality | Assisted Living Facilities & Residential Care

Report on Study Methods: Analysis of the Effect of Regulation on the Quality of Care in Board and Care Homes

Catherine Hawes, Judith Wildfire, Vince Iannacchione, Linda Lux, Angela Greene, Vincent Mor and Linda Laliberte

Research Triangle Institute

January 19, 1996

TABLE OF CONTENTS

4.1.1 Definition of Licensure

4.1.2 Exclusion of Homes Licensed for Special Populations

4.1.3 Exclusion of Licensed Nursing Homes

5.2.1 First-Stage Sample Allocation

5.2.2 Second-Stage Sample Allocation

5.2.3 Third-Stage Sample Allocation

5.3.1 First-Stage Sampling Units: Counties

5.3.2 Selection of Second-Stage Sample: Board and Care Homes

5.3.3 Creation of Unlicensed Home Sample Frame

5.3.4 Screening the List of Potential Unlicensed Homes

5.3.6 Estimating Size of Survey Population

5.3.7 Selection of Third-Stage Sample: Staff and Residents

5.4.1 Overall Facility Participation Rates

5.4.2 Operator Participation

5.4.3 Resident Participation

5.4.4 Staff Participation Rates

6.3.1 Resident Questionnaire

6.3.2 Interviews with Proxy Respondent for the Resident

7.1.2 Construction of Quality Measures

A. Reliability and Validity of the Walk-Through Observation

B. Reliability and Validity of Environmental Quality Domains

1. Content and Reliability of Environmental Quality Measures

2. Validity of Environmental Quality Measures

3. Conclusion

References

Connect with Us