Public agencies must ensure that the services contractors provide meet standards for quality and effectiveness. Much of this monitoring involves determining contractors performance on measures contained in their contracts. Agencies in most study sites gather performance data automatically. Management information systems and databases, usually developed by the welfare agency, record client activities and employment status as case workers perform their duties. Reports produced through these systems indicate overall program results, as well as the performance of individual contractors or service sites. Because the accuracy of this information sometimes comes into question, some contractors have created in-house performance tracking systems, which may include manual data gathering. In one site, Delaware, contractors collect their own data and submit it via an Internet-based form.
Limitations of management information systems have hampered some agencies ability to collect performance data. San Diego Countys implementation of pay-for-performance TANF contracts was delayed by the shortcomings of a system originally designed for the countys previous welfare program. The agency eventually decided to move ahead with performance-based payments, requiring contractors to track their accomplishments manually until an improved system was operational.
Several sites Lower Rio Grande Valley, Palm Beach County, San Diego County, and Wisconsin use client satisfaction surveys to measure service quality. Although client experiences and opinions are considered important, assessing them reliably has proved problematic. Often, surveys take the form of short questionnaires distributed at service delivery locations. These surveys, if designed by the contractor, may vary in content. They also usually result in extremely small sample sizes. Contractors criticized the surveys for highlighting complaints about issues outside service providers control, such as time limits on cash assistance. In addition, survey respondents sometimes do not make a clear distinction between the employees of public agencies and private contractors, complicating the interpretation of their answers. Agencies might correct these flaws by conducting more rigorous surveys on a regular basis, but the cost of doing so would be high.
Another common method of assessing service quality is reviewing case notes and other documents to determine whether clients are given adequate information about program services and conditions, have their skills and needs assessed, and receive appropriate services. These reviews can also be used to confirm the accuracy of contractors performance claims by comparing paper records such as enrollment forms and pay stubs with the data public agencies and contractors have collected. With TANF caseloads in the thousands, however, monitoring staff clearly cannot review every case represented in performance figures. Instead, the general practice is to draw periodically a random sample of cases from each contractor and examine documents from these files.
Visiting contractor locations to observe the service delivery process can help provide a general sense of client satisfaction and case worker responsiveness. A few agencies conduct interviews with contractor staff to gain further insight regarding service provision. This method of monitoring can be difficult, however, because its qualitative assessments require monitors who are well versed in program details and the nature of case management. Public agencies in the study sites typically conduct site visits one to three times a year.