Survey mode. Eighty-nine percent of clinicians, 63 percent of supervisors, and 37 percent of clients completed the survey via the web (Table V.1). The mode of survey completion varied by site. For example, in Site B, clients were provided the option of completing the survey immediately following the therapy session using the site's iPads. All of the clients at this site completed the survey using the web; 63 percent of the clients opted to complete it before leaving the site following their session (data not shown). Conversely, 100 percent of clients at Site D elected to complete the survey on paper.
Length of time to complete the survey. On average, clinicians completed the web survey in 8 minutes and supervisors and clients in 10 minutes (Table V.1). We excluded from these calculations 17 cases where the response times were greater than one hour. It is likely that these outlying values reflect individuals who started the survey, saved their responses, and completed the survey at a later time.
Length of time between therapy session and survey completion. To reduce recall bias, clients and clinicians were asked to complete the survey within 24 hours of the therapy session, and supervisors were asked to complete it within 24 hours of their review of the session. Table V.1 suggests that, on average, clinicians and clients did not complete the survey within this 24-hour window. The average number of days between when the therapy session occurred and when clinicians and clients completed the survey was 9.6 days (range 0-127 days) and 2.0 days (range 0-12 days), respectively. We do not have information on when the supervisors began their review of the therapy session; however, the average length of time between the occurrences of the therapy session and when supervisors completed the survey was 20 days (range 0-102 days).
Multiple factors may contribute to the length of time between the occurrence of the therapy session and survey completion. Conversations with site coordinators indicate that in some cases the length of time may be an artifact of clinicians and supervisors saving their survey responses but not actually clicking the "submit" button to transmit them. If the survey were to undergo future testing, revisions to the web version could provide additional prompts to submit the survey upon completion. Additionally, some site coordinators indicated that supervisors conducted weekly supervision and reviewed session tapes in batches; this may contribute in part to the delayed completion of the surveys. It is also likely that the data may accurately reflect the time needed for clinicians and supervisors to complete the survey, in which case, further investigation is needed into recall bias and the accuracy of the data when the survey is completed days and sometimes weeks after the therapy session occurred. Further investigation may also be needed into the organizations' capacity to complete this type of quality measure, and into the resources -- and perhaps changes in internal processes -- needed to facilitate more timely survey completion. In considering processes that facilitate data collection, regular reminders to staff to complete the survey appear key. The coordinators at Sites C and E were especially responsive to Mathematica alerts to remind staff of outstanding surveys, and these sites have comparatively shorter survey completion times. Routine reminders to clinicians and supervisors to complete the measure may be an important part of collecting the data in a timely way.
|TABLE V.1. Summary of Survey Administration: Modes and Completion Times|
|Number of Completed Surveys||Percentage Web-Based Complete (n)||Percentage Paper-Based Complete (n)||Average Number of Minutes to Complete the Survey (range)*,**||Average Number of Days from Therapy Session Start Date to Survey Completion (range)*,***|
|Total||Clinicians||96||89% (85)||11% (11)||8 (2-56)||9.9 (0-127)|
|Supervisors||97||63% (76)||37% (21)||10 (2-52)||19.6 (0-102)|
|Clients||78||37% (29)||63% (49)||10 (3-30)||1.9 (0-12)|
|Site A||Clinicians||34||100% (34)||0% (0)||9 (3-56)||14 (0-127)|
|Supervisors||34||100% (34)||0% (0)||10 (2-47)||27 (0-76)|
|Clients||22||23% (5)||77% (17)||6 (3-11)||3.8 (0-7)|
|Site B||Clinicians||10||100% (10)||0% (0)||9 (2-13)||20 (0-72)|
|Supervisors||10||100% (10)||0% (0)||7 (3-20)||24 (0-102)|
|Clients||8||100% (8)||0% (0)||7 (4-11)||2.5 (0-9)|
|Site C||Clinicians||15||100% (15)||0% (0)||6 (2-13)||0.5 (0-4)|
|Supervisors||15||100% (15)||0% (0)||9 (4-52)||8 (2-20)|
|Clients||14||7% (1)||93% (13)||15 (15)||0 (0)|
|Site D||Clinicians||12||58% (7)||42% (5)||10 (2-23)||6.6 (0-14)|
|Supervisors||14||100% (15)||0% (0)||12 (4-23)||13.6 (0-51)|
|Clients||13||0% (0)||100% (13)||NA||NA|
|Site E||Clinicians||18||89% (16)||11% (2)||6 (3-14)||4.4 (0-29)|
|Supervisors||17||18% (3)||82% (14)||9 (9)||2.7 (0-7)|
|Clients||17||82% (14)||18% (3)||11 (6-30)||1 (0-12)|
|Site F||Clinicians||7||43% (3)||57% (4)||8 (6-9)||8.3 (1-22)|
|Supervisors||7||0% (0)||100% (7)||NA||NA|
|Clients||4||25% (1)||75% (3)||26 (26)||1 (1)|
|* Paper-based completes are excluded, because the information is not available.
** Durations over one hour were excluded (17 cases out of 191 total), as it is likely that these participants completed the survey in more than one sitting.
*** Days calculated are calendar days.
Item-Level Missing Information. Most participants entered a response for each survey item. On the clinician survey, eight items had missing information and the missingness ranged from 0-2 percent (Appendix G, Table G.1). On the supervisor survey, 28 items had missing information; the level of missing information ranged from 0-3 percent (Appendix G, Table G.2). On the client survey, 30 items had missing information, which ranged from 0-6 percent (Appendix G, Table G.3).