Development of an Assistive Technology and Environmental Assessment Instrument for National Surveys: Final Report - Part II

Publication Date

Nov 30, 2005

Vicki A. Freedman, Ph.D.
Polisher Research Institute

Emily M. Agree, Ph.D.
Johns Hopkins Bloomberg School of Public Health

Lisa Landsberg, M.Ed.
Polisher Research Institute

December 2005

This report was prepared under contract #HHS-100-03-0011 between the U.S. Department of Health and Human Services (HHS), Office of Disability, Aging and Long-Term Care Policy (DALTCP) and the Polisher Research Institute. Additional funding was provided by HHSs National Institute on Aging. For additional information about this subject, you can visit the DALTCP home page at http://aspe.hhs.gov/_/office_specific/daltcp.cfm or contact the ASPE Project Officers, William Marton and Hakan Aykan, at HHS/ASPE/DALTCP, Room 424E, H.H. Humphrey Building, 200 Independence Avenue, S.W., Washington, D.C. 20201. Their e-mail addresses are: William.Marton@hhs.gov and Hakan.Aykan@hhs.gov.

This report was funded by the Department of Health and Human Services Office of the Assistant Secretary for Planning and Evaluation in cooperation with the National Institute on Aging (R01-14346) and the National Center for Health Statistics. The views expressed are those of the authors alone and do not represent those of the authors affiliations or funding agencies. The authors thank Carol Rayside for her administrative assistance in preparing this report.

OVERVIEW

FREQUENCIES

BEHAVIOR CODES

CROSSWALK BETWEEN PILOT STUDY INSTRUMENT AND RECOMMENDED MODULES

REFERENCES

NOTES

MODULES (separate PDF files): MODULE A: Home Environment Module [PDF file]; MODULE B: Mobility and Other Devices Module [PDF file]; MODULE C: Effectiveness/Participation Module [PDF file]; MODULE D: Communication Technology Module [PDF file]; MODULE E: Residual ADL and IADL Difficulty Module [PDF file]

[NOTE: These Modules are in separate Portable Document Format (PDF) files. You will need a copy of the Acrobat Reader in order to view them.]

APPENDICES (separate PDF files): APPENDIX I: Crosswalk of Question Numbers from Pilot Test and Final Recommended Modules; APPENDIX II: Technology and Aging Pilot Survey: Instrument for the Pilot Study [PDF file]

[NOTE: These Appendices are in separate Portable Document Format (PDF) files. You will need a copy of the Acrobat Reader in order to view them.]

OVERVIEW

The purpose of this project was to develop, pilot, and disseminate a set of instruments for national surveys to measure the use of assistive technology and the environments in which they are used. The project focused on older adults (ages 50 and older) living in the community. The instruments have been designed as a series of modules that can be adopted in whole or part as part of a computer-assisted telephone interview (CATI).

Part I of this report presents the recommended 8-10 minute instrument (consisting of five modules) and a briefer 2-3 minute instrument (consisting of a single module). The instruments include the relevant skip and fill instructions for CATI programming. Also included in Part I of this report are timing estimates, recommendations for interviewer training, and background on the instrument development process.

The purpose of Part II is to present results from the pilot test for the recommended items. For each recommended item, we provide the following information:

the recommended wording and universe for the item;
the wording as the item was originally piloted;
the unweighted and weighted frequency distributions among the applicable universe from the pilot study; and,
for items with at least 20 respondents, results from behavioral coding.

FREQUENCIES

Weighted frequency distributions from the pilot study are provided as a benchmark for distributions that could be expected with a national sample ages 50 and older. Weights developed for this purpose were post-stratified to a limited number of key demographic variables.¹ The frequencies also provide information on the extent of missing data (item refusal and dont know responses), which in most cases was very low.

In some cases the wording of the recommended questions or the recommended universe differs from that used in the pilot study. These differences are noted on the item-specific pages and also in the crosswalk document provided in the appendix.

BEHAVIOR CODES

Behavioral coding is a systematic method of recording interactions between an interviewer and respondent during an interview (Fowler and Cannel, 1996). In the pilot study, 150 interviews that reflected the pilot sample composition² were tape recorded and then coded by two trained research assistants. The purpose of the behavior coding was to identify items that had potential problems with validity and reliability because either interviewers did not read them as written or respondents found them to be cognitively challenging.

For items with at least 20 responses, two interviewer and three respondent behaviors are reported: reading questions other than verbatim, probing or providing definitions, providing qualified or inadequate answers,³ requesting a clarification,⁴ and interrupting.⁵ 121 of the 126 items recommended in the 8-10 minute module were coded for behaviors. A high percentage (99% on average) of responses to these items was coded as adequate and a low percentage (2-3% on average) involved wording changes by interviewers, requests by respondents for clarification or definitions, and interruption of questions. Probing occurred on average 7% of the time.

Some researchers have used 15% as a threshold for identifying potential problems with the validity or reliability of an item. This threshold is relevant for wording changes, qualified or inadequate answers, and requests for clarification. However, for probing behaviors, the literature has generally focused on inappropriate probing as the behavior of interest. Since the pilot study did not evaluate the appropriateness of the probing, probing levels of 15% are not necessarily problematic. Similarly, interrupting behaviors are not necessarily problematic if they occur as part of a learned response pattern to a repetitive series.

Of the 121 items in the recommended 8-10 minute instrument that were coded for behaviors in the pilot study, only nine items reached the 15% threshold. Three items that required probing at least 20% of the time in the pilot study (MO-5, EF-1, EF-3) were changed in the recommended modules so that the Universe of respondents was narrowed to a more relevant subgroup. Definitions were added for two items that required clarification between 15% and 20% of the time (MO-3.4 and CO-6.3). The remaining four items that required probing between 15% and 20% of the time (HE-13.1b, HE-13.3b, MO-2.1c, CO-5) were not changed.

Where sample sizes allowed, we also tested for differences in behaviors by age groups (50-64, 65-79, 80+) and by functional status (one or more versus no functional limitations).⁶ Most often behaviors did not vary significantly by age or functional status. Where they did, we note this in a footnote to the relevant table.

For some questions sample sizes for behavior coding were too small to evaluate the individual item. We therefore collapsed information across similar types of questions. Results are summarized in the table below (with Ns summed across items shown parenthetically). In general, few problem behaviors were reported for questions about home features, mobility aids, other aids, and residual difficulty, with most estimates well below the 15% threshold. Only probing about the frequency of use of a mobility aid crossed the threshold; most of the probing was around the use of aids outside.

TABLE 1. Percentage of Responses Involving Key Interviewer and Respondent Behaviors
	Interviewer Behaviors		Respondent Behaviors
	Changed wording	Probed or provided definition	Gave qualified or inadequate answer	Requested definition or clarification	Interrupted
Home features¹
Existence (N=1813)	1.9	5.6	0.3	3.2	0.7
There or added (N=164)	0.6	5.5	0	0.6	9.8
Frequency of use (N=192)	2.1	14.1	0.5	2.1	10.9
Mobility aid² (N=154)	7.8	16.9	1.3	1.9	11.1
Other aid³(N=1500)	1.7	5.2	0	3.8	0.7
Residual Difficulty
ADLs (N=750)⁴	2.5	6.4	0.1	1.2	10.1
IADLs (N=530)⁵	2.3	6.2	0.2	3.4	8.9
Features include working lights; handrails at entrance; ramp; automatic doors; bedroom, kitchen and bath on same floor; chair lift; elevator; handrails in staircases or hallways; emergency call system; stall shower; grab bars; tub seat; handheld showerhead; raised or modified seat; grab bars around toilet from questions HE-5 to HE-14. How often used cane, walker, wheelchair, and scooter to transfer, get around inside, or get around outside from questions MO-2.1 to MO-2.4. Other aids include hearing aid, glasses, vision aids other than glasses, reacher or grabber, special bed, trapeze, raising seat, portable commode, modified washer or dryer, and adapted utensils from questions MO-3.1 to MO-3.10. Amount of difficulty (when using devices) transferring, getting around inside, getting around outside, bathing, and toileting by yourself from questions RD-1 to RD-5. Amount of difficulty (when using devices) preparing meals, shopping for groceries, managing money, and going places outside walking distance by yourself from questions RD-6.1b to RD-6.4b.

CROSSWALK BETWEEN PILOT STUDY INSTRUMENT AND RECOMMENDED MODULES

The appendix includes a document that crosswalks the items in the pilot study with those in the recommended modules. The document provides question numbers for both instruments and indicates where there are differences in questions wording and applicable Universes. Items included in the 2-3 minute module are also designated. The crosswalk is followed by the pilot study instrument. See Part I of this report for the recommended 8-10 minute and 2-3 minute instruments.

REFERENCES

Fowler FJ & CF Cannell. 1996. Using behavioral coding to identify cognitive problems with survey questions. In N Schwarz & SA Sudman (Eds.), Answering Questions, pp. 15-36. San Francisco, CA: Jossey-Bass.