Several methodological considerations bear mentioning in the use of screening and assessment tools. For example, to achieve the best results it is important to ensure that the questions are designed for the population being screened or assessed. This is particularly important if adopting professionally- or externally-developed tools as many of these were not originally designed specifically for welfare recipients.73
Although concepts such as validity and reliability of instruments may be more meaningful to researchers and test designers than TANF agency staff, they should also be considered if TANF agencies place high expectations on the ability of screening and assessment tools to identify barriers.74 To the extent such tools are used as a preliminary screen, expectations may not be high (however, this approach does carry other implications, such as resources expended to assess based on inaccurate screening mechanism and time spent being assessed for a barrier that may not exist).
Consideration should be given to methodological aspects of tools if high expectations are placed on the tool's ability to identify barriers.
Although few TANF agency officials we spoke to mentioned validity and reliability, some suggested that their existing efforts to identify barriers were not yielding expected results. While this could be attributed to a wide range of causes, using a tool that is not designed for use with TANF clients may be one such reason. This should be of concern to TANF agencies expending funds on screening and assessment efforts and points to the need for evaluation of these approaches.
Administering tools as designed is another methodological consideration. For example, some tools are intended to be implemented in an interview setting, with the interviewer obtaining a response to the question posed before moving on to the next question. This allows the interviewer to ensure answers are obtained and that the respondent cannot look ahead at subsequent questions that may affect the way she answers the current question. If a tool intended to be administered through an interview is self-administered, the respondent may observe in subsequent questions
incentives to provide inaccurate responses. The environment in which a tool is administered is also a consideration. For example, if the tool was designed to be administered one-on-one, altering this setting (for example, administering during a group orientation) may affect assessment outcomes.
73 In the tool profiles in Appendix A we have indicated the target population for the professionally-developed tools.
74 Reliability is whether a tool will give the same scores when administered twice to the same person (before the measured traits have changed), and validity is whether a tool actually measures the characteristics it is designed to measure.