Development of an Assistive Technology and Environmental Assessment Instrument for National Surveys: Final Report - Part II

12/01/2005

Vicki A. Freedman, Ph.D.
Polisher Research Institute

Emily M. Agree, Ph.D.
Johns Hopkins Bloomberg School of Public Health

Lisa Landsberg, M.Ed.
Polisher Research Institute

December 2005

PDF Version (124 PDF pages)


This report was prepared under contract #HHS-100-03-0011 between the U.S. Department of Health and Human Services (HHS), Office of Disability, Aging and Long-Term Care Policy (DALTCP) and the Polisher Research Institute. Additional funding was provided by HHS’s National Institute on Aging. For additional information about this subject, you can visit the DALTCP home page at http://aspe.hhs.gov/_/office_specific/daltcp.cfm or contact the ASPE Project Officers, William Marton and Hakan Aykan, at HHS/ASPE/DALTCP, Room 424E, H.H. Humphrey Building, 200 Independence Avenue, S.W., Washington, D.C. 20201. Their e-mail addresses are: William.Marton@hhs.gov and Hakan.Aykan@hhs.gov.

This report was funded by the Department of Health and Human Service’s Office of the Assistant Secretary for Planning and Evaluation in cooperation with the National Institute on Aging (R01-14346) and the National Center for Health Statistics. The views expressed are those of the authors alone and do not represent those of the author’s affiliations or funding agencies. The authors thank Carol Rayside for her administrative assistance in preparing this report.



TABLE OF CONTENTS

OVERVIEW
FREQUENCIES
BEHAVIOR CODES
CROSSWALK BETWEEN PILOT STUDY INSTRUMENT AND RECOMMENDED MODULES
REFERENCES
NOTES
MODULES (separate PDF files)
MODULE A: Home Environment Module [PDF file]
MODULE B: Mobility and Other Devices Module [PDF file]
MODULE C: Effectiveness/Participation Module [PDF file]
MODULE D: Communication Technology Module [PDF file]
MODULE E: Residual ADL and IADL Difficulty Module [PDF file]

[NOTE: These Modules are in separate Portable Document Format (PDF) files. You will need a copy of the Acrobat Reader in order to view them.]

APPENDICES (separate PDF files)
APPENDIX I: Crosswalk of Question Numbers from Pilot Test and Final Recommended Modules
APPENDIX II: Technology and Aging Pilot Survey: Instrument for the Pilot Study [PDF file]

[NOTE: These Appendices are in separate Portable Document Format (PDF) files. You will need a copy of the Acrobat Reader in order to view them.]


OVERVIEW

The purpose of this project was to develop, pilot, and disseminate a set of instruments for national surveys to measure the use of assistive technology and the environments in which they are used. The project focused on older adults (ages 50 and older) living in the community. The instruments have been designed as a series of modules that can be adopted in whole or part as part of a computer-assisted telephone interview (CATI).

Part I of this report presents the recommended 8-10 minute instrument (consisting of five modules) and a briefer 2-3 minute instrument (consisting of a single module). The instruments include the relevant skip and fill instructions for CATI programming. Also included in Part I of this report are timing estimates, recommendations for interviewer training, and background on the instrument development process.

The purpose of Part II is to present results from the pilot test for the recommended items. For each recommended item, we provide the following information:

  • the recommended wording and universe for the item;
  • the wording as the item was originally piloted;
  • the unweighted and weighted frequency distributions among the applicable universe from the pilot study; and,
  • for items with at least 20 respondents, results from behavioral coding.


FREQUENCIES

Weighted frequency distributions from the pilot study are provided as a benchmark for distributions that could be expected with a national sample ages 50 and older. Weights developed for this purpose were post-stratified to a limited number of key demographic variables.1 The frequencies also provide information on the extent of missing data (item refusal and don’t know responses), which in most cases was very low.

In some cases the wording of the recommended questions or the recommended universe differs from that used in the pilot study. These differences are noted on the item-specific pages and also in the crosswalk document provided in the appendix.


BEHAVIOR CODES

Behavioral coding is a systematic method of recording interactions between an interviewer and respondent during an interview (Fowler and Cannel, 1996). In the pilot study, 150 interviews that reflected the pilot sample composition2 were tape recorded and then coded by two trained research assistants. The purpose of the behavior coding was to identify items that had potential problems with validity and reliability because either interviewers did not read them as written or respondents found them to be cognitively challenging.

For items with at least 20 responses, two interviewer and three respondent behaviors are reported: reading questions other than verbatim, probing or providing definitions, providing qualified or inadequate answers,3 requesting a clarification,4 and interrupting.5 121 of the 126 items recommended in the 8-10 minute module were coded for behaviors. A high percentage (99% on average) of responses to these items was coded as adequate and a low percentage (2-3% on average) involved wording changes by interviewers, requests by respondents for clarification or definitions, and interruption of questions. Probing occurred on average 7% of the time.

Some researchers have used 15% as a threshold for identifying potential problems with the validity or reliability of an item. This threshold is relevant for wording changes, qualified or inadequate answers, and requests for clarification. However, for probing behaviors, the literature has generally focused on inappropriate probing as the behavior of interest. Since the pilot study did not evaluate the appropriateness of the probing, probing levels of 15% are not necessarily problematic. Similarly, interrupting behaviors are not necessarily problematic if they occur as part of a learned response pattern to a repetitive series.

Of the 121 items in the recommended 8-10 minute instrument that were coded for behaviors in the pilot study, only nine items reached the 15% threshold. Three items that required probing at least 20% of the time in the pilot study (MO-5, EF-1, EF-3) were changed in the recommended modules so that the Universe of respondents was narrowed to a more relevant subgroup. Definitions were added for two items that required clarification between 15% and 20% of the time (MO-3.4 and CO-6.3). The remaining four items that required probing between 15% and 20% of the time (HE-13.1b, HE-13.3b, MO-2.1c, CO-5) were not changed.

Where sample sizes allowed, we also tested for differences in behaviors by age groups (50-64, 65-79, 80+) and by functional status (one or more versus no functional limitations).6 Most often behaviors did not vary significantly by age or functional status. Where they did, we note this in a footnote to the relevant table.

For some questions sample sizes for behavior coding were too small to evaluate the individual item. We therefore collapsed information across similar types of questions. Results are summarized in the table below (with N’s summed across items shown parenthetically). In general, few problem behaviors were reported for questions about home features, mobility aids, other aids, and residual difficulty, with most estimates well below the 15% threshold. Only probing about the frequency of use of a mobility aid crossed the threshold; most of the probing was around the use of aids outside.

TABLE 1. Percentage of Responses Involving Key Interviewer and Respondent Behaviors
  Interviewer Behaviors Respondent Behaviors
Changed wording Probed or provided definition Gave qualified or inadequate answer Requested definition or clarification Interrupted
Home features1          
   Existence (N=1813) 1.9 5.6 0.3 3.2 0.7
   There or added (N=164) 0.6 5.5 0 0.6 9.8
   Frequency of use (N=192) 2.1 14.1 0.5 2.1 10.9
Mobility aid2 (N=154) 7.8 16.9 1.3 1.9 11.1
Other aid3 (N=1500) 1.7 5.2 0 3.8 0.7
Residual Difficulty          
   ADLs (N=750)4 2.5 6.4 0.1 1.2 10.1
   IADLs (N=530)5 2.3 6.2 0.2 3.4 8.9
  1. Features include working lights; handrails at entrance; ramp; automatic doors; bedroom, kitchen and bath on same floor; chair lift; elevator; handrails in staircases or hallways; emergency call system; stall shower; grab bars; tub seat; handheld showerhead; raised or modified seat; grab bars around toilet from questions HE-5 to HE-14.
  2. How often used cane, walker, wheelchair, and scooter to transfer, get around inside, or get around outside from questions MO-2.1 to MO-2.4.
  3. Other aids include hearing aid, glasses, vision aids other than glasses, reacher or grabber, special bed, trapeze, raising seat, portable commode, modified washer or dryer, and adapted utensils from questions MO-3.1 to MO-3.10.
  4. Amount of difficulty (when using devices) transferring, getting around inside, getting around outside, bathing, and toileting by yourself from questions RD-1 to RD-5.
  5. Amount of difficulty (when using devices) preparing meals, shopping for groceries, managing money, and going places outside walking distance by yourself from questions RD-6.1b to RD-6.4b.


CROSSWALK BETWEEN PILOT STUDY INSTRUMENT AND RECOMMENDED MODULES

The appendix includes a document that crosswalks the items in the pilot study with those in the recommended modules. The document provides question numbers for both instruments and indicates where there are differences in questions wording and applicable Universes. Items included in the 2-3 minute module are also designated. The crosswalk is followed by the pilot study instrument. See Part I of this report for the recommended 8-10 minute and 2-3 minute instruments.


REFERENCES

Fowler FJ & CF Cannell. 1996. Using behavioral coding to identify cognitive problems with survey questions. In N Schwarz & SA Sudman (Eds.), Answering Questions, pp. 15-36. San Francisco, CA: Jossey-Bass.


NOTES

  1. Sampling weights realigned the pilot study sample to match national distributions of sex, age group, education, and functioning found in the 2003 National Health Interview Survey.

  2. The 150 interviews were distributed as follows: 50 ages 50-64 (25 from a household in which someone had a disability), 50 ages 65-79, and 50 ages 80+. All 21 respondents residing in an assisted living facility were included in the 150 tape recorded interviews.

  3. An answer is coded as qualified when a respondent’s final answer is among the response options, but includes a qualifying statement. An answer is coded as inadequate when a respondent’s final answer is not among the response options.

  4. A response is coded as a request for clarification if the respondent asks about the question’s meaning or asks to have the question repeated.

  5. NA (not applicable) is reported if the sample size is less than 20 or if the recommended item was not included in the pilot study.

  6. Functional limitation groups were assigned based on responses of some, a lot, unable, or doesn’t do to items 78.1-78.8.


MODULE A: Home Environment Module

This Module is currently available only as a separate PDF file (http://aspe.hhs.gov/daltcp/reports/ATEAdevII-A.pdf), or as part of the PDF version of Part II http://aspe.hhs.gov/daltcp/reports/ATEAdevII.pdf.

You will need a copy of the Acrobat Reader in order to view it.


MODULE B: Mobility and Other Devices Module

This Module is currently available only as a separate PDF file (http://aspe.hhs.gov/daltcp/reports/ATEAdevII-B.pdf), or as part of the PDF version of Part II http://aspe.hhs.gov/daltcp/reports/ATEAdevII.pdf.

You will need a copy of the Acrobat Reader in order to view it.


MODULE C: Effectiveness/Participation Module

This Module is currently available only as a separate PDF file (http://aspe.hhs.gov/daltcp/reports/ATEAdevII-C.pdf), or as part of the PDF version of Part II http://aspe.hhs.gov/daltcp/reports/ATEAdevII.pdf.

You will need a copy of the Acrobat Reader in order to view it.


MODULE D: Communication Technology Module

This Module is currently available only as a separate PDF file (http://aspe.hhs.gov/daltcp/reports/ATEAdevII-D.pdf), or as part of the PDF version of Part II http://aspe.hhs.gov/daltcp/reports/ATEAdevII.pdf.

You will need a copy of the Acrobat Reader in order to view it.


MODULE E: Residual ADL and IADL Difficulty Module

This Module is currently available only as a separate PDF file (http://aspe.hhs.gov/daltcp/reports/ATEAdevII-E.pdf), or as part of the PDF version of Part II http://aspe.hhs.gov/daltcp/reports/ATEAdevII.pdf.

You will need a copy of the Acrobat Reader in order to view it.


APPENDIX I: Crosswalk of Question Numbers from Pilot Test and Final Recommended Modules

This Module is also available as a separate PDF file (http://aspe.hhs.gov/daltcp/reports/ATEAdevII-apI.pdf), or as part of the PDF version of Part II http://aspe.hhs.gov/daltcp/reports/ATEAdevII.pdf. You will need a copy of the Acrobat Reader in order to view it.

  Piloted Q. number Final Q. number Notes
HOME MODULE
HOME      
home part of building w/2+ units 15 HE-1 +
how often left home/building 43 HE-2 *
 
ENTRANCE AND INSIDE BUILDING      
have to use steps 16 HE-3 *
common entrance 17 HE-4 *
building entry features [18] [HE-5]  
   lighting 18.1 HE-5.1 *
   handrails 18.2 HE-5.2  
   ramp 18.3 HE-5.3  
      how often used 18.3a HE-5.3a &
   automatic doors 18.4 HE-5.4  
building more than one story 19 HE-6  
live on first floor 20 HE-7 *
have elevator 21 HE-8  
 
ENTRANCE TO HOME      
have to use steps 22 HE-9 *
entry features [23] [HE-10]  
   lighting 23.1 HE-10.1 *
   handrails 23.2 HE-10.2  
      there/added 23.2a HE-10.2a *
   ramp 23.3 HE-10.3  
      there/added 23.3a HE-10.3a  
      used in last 30 days 23.3b HE-10.3b &
 
INSIDE HOME      
living space on one floor 24 HE-11 +
home features [25] [HE-12] +
   bed, kitchen, bath same floor 25.1 HE-12.1 *+
   elevator 25.2 HE-12.2  
      there/added 25.2a HE-12.2a  
      how often used 25.2b HE-12.2b  
   chair lift 25.3 HE-12.3 +
      there/added 25.3a HE-12.3a *+
      how often used 25.3b HE-12.3b +
   handrails on stairs NA HE-12.4 +
      there/added NA HE-12.4a +
      how often used NA HE-12.4b +
   handrails in hallway 26.2 HE-12.5 *+
      there/added 26.2a HE-12.5a *+
      how often used 26.2b HE-12.5b +
   call system 26.3 HE-12.6 *+
      there/added 26.3a HE-12.6a *+
      used in last 30 days 26.3b HE-12.6b +
 
BATHROOM FEATURES      
stall shower separate from tub 27.1 HE-13.1 *+
   how often used 27.1b HE-13.1b *
grab bars in shower or tub area 27.2 HE-13.2 *+
   there/added 27.2a HE-13.2a *+
   how often used 27.2b HE-13.2b +
seat or stool 27.3 HE-13.3 *+
   there/added NA HE-13.3a +
   how often used 27.3b HE-13.3b *+
hand held showerhead 27.4 HE-13.4 *
grab bars around toilet 28.1 HE-14.1 *+
   there/added 28.1a HE-14.1a *+
   how often used 28.1b HE-14.1b +
raised or modified toilet seat 28.2 HE-14.2 *+
   there/added NA HE-14.2a +
 
COST OF MODIFICATIONS      
review modifications added 34 HE-15-INTRO  
family or self pay 36 HE-15 *&
   how much 1 ($500) 37 HE-16 &
   how much 2 ($1,000) 38 HE-17 &
   how much 3 ($100) 39 HE-18 &
insurance pay 40 HE-19 *&
 
MOBILITY AND OTHER DEVICES MODULE      
INDOOR AND OUTDOOR MOBILITY      
Screen (global item) 1 MO-1 +
   cane 44.1 MO-2.1 &+
      transfer use 44.1a MO-2.1a &+
      inside use 44.1b MO-2.1b &+
      outside use 44.1c MO-2.1c &+
   walker 44.2 MO-2.2 &+
      transfer use 44.2a MO-2.2a &+
      inside use 44.2b MO-2.2b &+
      outside use 44.2c MO-2.2c &+
   wheelchair 44.3 MO-2.3 &+
      transfer use 44.3a MO-2.3a &+
      inside use 44.3b MO-2.3b &+
      outside use 44.3c MO-2.3c &+
   scooter 44.4 MO-2.4 &+
      own or rent NA MO-2.4a +
      inside use 44.4b MO-2.4b &+
      outside use 44.4c MO-2.4c &+
   motorized scooter in store 49.9 MO-2.5 *+
 
OTHER DEVICES      
hearing aid 49.1 MO-3.1 +
glasses 49.2 MO-3.2 +
vision aids other than glasses 49.3 MO-3.3 +
reacher or grabber 49.4 MO-3.4 +
special bed 49.5 MO-3.5  
trapeze, lift, or sling 44.7 MO-3.6 *
raising seat 44.6 MO-3.7 *
portable commode 49.6 MO-3.8 *
modified washer or dryer 49.7 MO-3.9  
adapted utensils 49.8 MO-3.10 *
 
TRANSPORTATION      
drive self 11.1 MO-4.1  
   ramp or lift 11.1a2 MO-4.2a *
   hand controls, etc. 11.1a1 MO-4.2b *
paratransit 11.3 MO-4.3  
 
COST OF MOBILITY/OTHER DEVICES      
review devices used 52 MO-5-INTRO  
family or self pay 53 MO-5 *&
   how much 1 ($250) NA MO-6  
   how much 2 ($500) 54 MO-7 &
   how much 3 ($100) 56 MO-8 &
insurance pay 57 MO-9 *&
 
EFFECTIVENESS/PARTICIPATION MODULE      
review devices and features used 58 EF-1-INTRO  
safer 59.1 EF-1 &
more control 59.2 EF-2 &
take part in activities 59.3 EF-3 &
 

COMMUNICATION TECHNOLOGY MODULE

     
COMPUTERS      
use a computer 61 CO-1 *
computer features [64] [CO-2]  
   large keyboard 64.1 CO-2.1 *
   on screen keyboard 64.2 CO-2.2 *
   pointing device 64.4 CO-2.3 *
   screen magnifier/software 64.5 CO-2.4 *
what computer used for [68] [CO-3]  
   finances 68.1 CO-3.1  
   ordering prescriptions 68.5 CO-3.2  
   shopping for groceries 68.3 CO-3.3  
   shopping other 68.4 CO-3.4 *
   health information 68.6 CO-3.5  
 
TELEPHONES      
have cell phone 70 CO-4  
modify phone 72 CO-5  
any of following features [73] [CO-6]  
   large buttons 73.2 CO-6.1 &
   amplifier 73.3 CO-6.2 &
   TTY or text display device 73.4 CO-6.3 &
   flashing ringer 73.5 CO-6.4 &
   voice activation 73.6 CO-6.5 &
what use phone for [77] [CO-7]  
   finances 77.1 CO-7.1  
   ordering prescriptions 77.5 CO-7.2  
   shopping groceries 77.3 CO-7.3  
   shopping other 77.4 CO-7.4 *
 
RESIDUAL DIFFICULTY MODULE      
Activities of Daily Living (ADLs)      
difficulty getting out of a bed or chair by yourself 79.1c RD-1  
difficulty getting around home by yourself 79.2c RD-2  
difficulty leaving home/building by yourself 79.3c RD-3  
difficulty bathing/showering by yourself 79.4c RD-4  
difficulty using toilet by yourself 79.5c RD-5  
 
Instrumental Activities of Daily Living (IADLS)      
preparing meals 80.1a RD-6.1a  
   difficulty by yourself 80.1b RD-6.1b  
shopping for groceries 80.2a RD-6.2a  
   difficulty by yourself 80.2b RD-6.2b  
managing money 80.3a RD-6.3a  
   difficulty by yourself 80.3b RD-6.3b  
going places outside walking distance 80.4a RD-6.4a &
   difficulty by yourself 80.4b RD-6.4b &

NA=Not applicable (question not in pilot)
* = Question edited slightly after pilot testing
& = Universe for recommended question differs from that in pilot study
+ = Question recommended in 2-minute module


APPENDIX II: Technology and Aging Pilot Survey: Instrument for the Pilot Study

This Module is currently available only as a separate PDF file (http://aspe.hhs.gov/daltcp/reports/ATEAdevII-apII.pdf), or as part of the PDF version of Part II http://aspe.hhs.gov/daltcp/reports/ATEAdevII.pdf.

Preview
Download

"ATEAdevII.pdf" (pdf, 4.66Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

Preview
Download

"ATEAdevII-A.pdf" (pdf, 645.12Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

Preview
Download

"ATEAdevII-B.pdf" (pdf, 451.84Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

Preview
Download

"ATEAdevII-C.pdf" (pdf, 83Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

Preview
Download

"ATEAdevII-D.pdf" (pdf, 253.94Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

Preview
Download

"ATEAdevII-E.pdf" (pdf, 193.52Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

Preview
Download

"ATEAdevII-apI.pdf" (pdf, 188.73Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

Preview
Download

"ATEAdevII-apII.pdf" (pdf, 393.12Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®