The SOAR TA contractor collects substantial information from SOAR states about the successes, challenges, and preliminary outcomes of their efforts. It uses the information to inform its TA efforts and to promote best practices in individual states. Recognizing that when examined through a research lens some of these data could contribute to the evaluation, we compiled and conducted secondary analysis of the data. Below we describe the data sources and the specific evaluation questions each source addressed.
Participant evaluations of the TTT sessions. After each TTT session, the TA contractor requests that each participant complete an evaluation form, consisting primarily of close-ended questions. The contractor has hard copies of each evaluation form it received from the initial two TTT sessions it conducted. The evaluation forms helped us answer the following questions:
- How satisfied were participants with the TTT session?
- What did participants find most and least helpful?
- What did participants feel they learned from the session?
Documentation from initial in-state trainings (observation, participant evaluations, and participant pre/post test results). The TA contractor observes each initial in-state training to (1) assess the extent to which the training was provided with fidelity to the model and (2) provide feedback to trainers on how to improve their efforts. Observers summarize their impressions and rate the skills of the trainers and their presentation of the SSTR material on a five-point Likert scale. These summaries and ratings are available for each state, and the quality of the data is relatively consistent because the same four individuals who provide the TA conduct all of the observations. Also available for each state are (1) evaluation forms that participants in the initial in-state trainings complete and (2) results from a pre-post test designed to measure the extent to which participants' knowledge increased following training. In addition to feedback on the training, the evaluation form provides basic background data for each participant (for example, the type of agency where the participant works, position at the agency, and experience with SSI/SSDI applications prior to training). Data from these documents helped us answer the following questions:
- How did the TA contractor rate the initial in-state trainings?
- In what areas did trainers perform well and not perform well?
- Who participated in the initial in-state trainings?
- How satisfied were participants with the in-state training?
- What did training participants find most and least helpful?
- How much and what did participants learn from the training?
Interviews with TA contractor staff. These interviews were conducted to gather their perceptions of SOAR's successes and challenges. Specifically, we spoke with the SOAR team project director, the SOAR project coordinator, and the senior consultant who authored the SSTR curriculum and provides the bulk of the TA to states.
"index.pdf" (pdf, 1.23Mb)
"apb.pdf" (pdf, 330.32Kb)