In fall 2008, pretest interviews were conducted with 72 residential care facilities in 6 states. In addition to further testing the questionnaire instruments, the purpose of the pretest was to assess the efficacy of study protocols for the national survey. Aspects of the protocol evaluated included:
- Recruitment procedures
- Survey administration
- Mode of administration (telephone compared with in-person)
The pretest sampling frame was constructed by accessing the licensure lists for six states—from the Northeast, Midwest, and South—and contained 7,256 facilities. To achieve the pretest goal to complete interviews with 75 facilities, 150 sampled cases were selected. Sampled facilities were selected in order to provide a mix among four size strata—small (4–10 beds), medium (11–25 beds), large (26–100 beds), and extra large (more than 100 beds)—representing both urban and nonurban areas; provide experience with chain-affiliated and independently operated facilities; include both multilevel and only assisted-living facilities; and include for-profit and nonprofit facilities.
Sample cases in the smallest stratum also were used to test the feasibility of administering the facility questionnaire by phone instead of an in-person visit. This was done because the government was considering a range of data collection protocols for the national survey, including telephone interviews for small facilities. One-half of small facilities were randomly assigned to a telephone data collection mode and one-half to an in-person data collection mode. Resident-level data were not collected in the telephone interviews.
After the sample was selected, a limited Internet search for the facilities was performed to determine the following: whether Internet information about facilities would be useful for recruiters; if facilities were affiliated with a chain (i.e., only facilities affiliated with the large national chains were previously identified on the sampling frame); and the name, address, and title of the chain corporate office director. The following conclusions were reached:
- Some facilities’ websites offered plentiful information; however, smaller facilities rarely had websites. When the facility did not have a website, it was difficult to determine if the facility was affiliated with a small or regional chain.
- Information found included maps of location, facility name, name of director, address, phone number, and status of chain affiliation.
- Some facilities were affiliated with hospital systems, rather than with chains of other residential care facilities.
- While learning about the facility prior to field work was valuable in some cases, some information was difficult to find, inconsistent, or dated and could not be used to replace the sampling frame information, only supplement it.
National and state residential care provider associations in states where the pretest was planned were also contacted in advance of the pretest. Overall, they were very receptive to the study and helpful in promoting the study in their newsletters, websites, and other communications within the industry.
The challenges of recruiting facilities for the pilot test suggested that more efficient and effective staffing was necessary to recruit facilities. Therefore, the recruitment activity was made the responsibility of a small number of carefully selected field staff with previous successful recruitment experience. Facilities were recruited by three experienced interviewers who attended a 2-day recruiter training session. Recruitment phone calls began approximately 3 weeks prior to the start of actual data collection. Recruiters’ responsibilities included:
- Contact facilities to obtain or verify contact information for the director.
- Send an advance mail package.
- Explain the study and gain cooperation.
- Administer a brief CAPI screening instrument to determine facility eligibility.
- Schedule an in-person visit or phone call for a field interviewer to complete the facility interview.
The advance mail package mailed to directors was sent via Federal Express and included the following materials placed inside an attractive NSRCF folder:
- A personalized letter from the director of NCHS with Responses to Frequently Asked Questions (FAQs) printed on the back.
- NCHS Ethics Review Board (ERB) approval letter.
- Brochure specifically about NSRCF.
NCHS’ Confidentiality Brochure—How the National Health Care Survey Keeps Your Information Strictly Confidential.
Letter of Support from four national residential care provider organizations.
Publication from another NCHS LTC provider survey.
- An in-person visit or phone call to complete the facility questionnaire.
- Resident sampling.
- Complete resident questionnaires for a sample of residents.
The following instruments were used to collect data for the pretest:
- Facility questionnaire.
- Resident selection questionnaire.
- Resident questionnaire.
- Debriefing questionnaire—filled out by interviewers after the interview to qualitatively describe their experience.
As with cognitive interviewing and the pilot study, an ADCF was mailed to the sampled facility after the recruiter set an appointment. Directors were encouraged to complete the ADCF at their convenience before the interview to minimize time and burden for them during the interview. Fifty of the 72 facility respondents (69%) completed the ADCF in advance. When they did, interviewers reported that the time spent to complete the facility questionnaire was reduced substantially, regardless of facility size.
The pretest facility questionnaire took an average of 74 minutes to complete. The facility director or their designee completed the facility questionnaire.
A short resident selection questionnaire was used to determine the total number of residents at the facility as of midnight the day before the in-person interview and to select residents for the resident interviews. The number of residents sampled depended on facility size: small and medium facilities—three residents; large facilities—four residents; and extra-large facilities—six residents.
Each resident questionnaire averaged 18 minutes to complete. Resident questionnaires were completed with facility staff that referred to resident records, usually the director or caregiver knowledgeable about the selected resident. No residents were interviewed. In many cases, the same respondent completed interviews for multiple residents and when this occurred, the time taken to complete each questionnaire decreased. Respondents received small token gifts of a pen, ruler, and a thank-you note as appreciation for their participation.
Interviewers who conducted facility questionnaires by telephone reported no difficulties in administering the questionnaire. They used the same CAPI facility questionnaire and Show Cards as for the in-person interview. Show Cards contained response categories for questions with lengthy response options and for question series with repetitive response categories. However, those respondents interviewed by phone rarely used the Show Cards mailed to them. Resident-level information was not obtained for phone cases. Thank-you notes were also mailed to facilities that completed the questionnaire by phone.
Observers from RTI, NCHS, and ASPE accompanied interviewers on approximately 15 site visits during the pretest. Overall, the CAPI instruments worked very well and field staff encountered few problems. Recruiters and interviewers encountered some resistance from facilities to participate, primarily because of lack of time. This was especially true for the small and medium facilities where the director had few supporting staff. Appointment cancellations were also an obstacle. The most common explanation for facility nonresponse was the inability to ever reach the director (25% of all nonresponding sampled facilities).
The pretest provided an opportunity to fully implement and assess all of the procedures, recruitment materials, and revised data collection questionnaires on a small scale before they were implemented in the national survey. Based on results from the pretest, it was determined that the study needed to:
- Improve the procedures for identifying facilities affiliated with chains and collecting chain contact information.
- Develop a more robust appointment system for field staff scalable to the national level.
- Encourage facility directors to complete the ADCF before the interview to lower the questionnaire administration time.
- Find ways to reduce the burden on facility directors by encouraging them to delegate components of the interview, such as the resident questionnaires, to other knowledgeable staff.
- Design contacting protocols that allow field staff more flexibility when recruiting, instead of following a highly scripted text.
- Incorporate lessons learned from the pretest training into the national survey supervisor, recruiter, and interviewer trainings.
- Enhance recruitment materials and outreach to improve the recruitment rate.
The pretest results also provided a general understanding of eligibility rates by facility size, which helped to inform sampling allocations for the national survey. In addition, three sampled facilities were out of business. The overall pretest eligibility rate was 77%. The eligibility rate was 95% for extra-large facilities, 90% for large facilities, and 79% for medium facilities. The eligibility rate was lowest (64%) among the small-sized facilities. Facilities that exclusively served the mentally retarded (MR) or developmentally disabled (DD) populations, or exclusively served the severely mentally ill, accounted for the majority (68%) of the ineligibility cases.