Ashoka is in the early stages of implementing a program to track Fellows progress. Twice a year, Fellows submit reports that track their progress against benchmarks mutually agreed upon at the start of the fellowship. These reports address a set of questions that are open-ended and self-defined, allowing Fellows the flexibility to adapt as needed. Benchmarks vary, given the diverse areas in which Fellows work, making it difficult to estimate any aggregate impacts. Recently, Ashoka has been trying to make the process more interactive, with both Ashoka and the Fellows evaluating whether the other is adequately delivering on their responsibilities.
Ashoka also evaluates its Fellows and their innovations by examining their ability to create systemic change over time. Ashoka developed its Measuring Effectiveness (ME) program in 1997 to strengthen understanding of the long-term progress its social entrepreneurs are making toward systemic social change. The ME program includes two components: (1) annual self-report surveys of Fellows at the fifth and tenth anniversaries of their selection; and (2) a series of case studies of some Fellow survey respondents to obtain more in-depth information. These components track Fellows progress toward systemic change and allow some assessment of Ashokas ability to identify effective social entrepreneurs (Leviner et al. 2007 and Ashoka 2006). The ME attempts to measure outcomes (such as in an educational system) rather than outputs (such as the number of students educated), asking Fellows four questions. The responses are meant to serve as proxy indicators for evaluating systematic change over time:
- Are you still working toward your original vision?
- Have others replicated your original idea?
- Have you had impact on public policy?
- What position does your institution currently hold in the field?
A weakness of the ME approach is that it may yield overly positive estimates of outcomes. First, some Fellows become inactive or lose contact with Ashoka (up to 30 percent ten years post-selection); and of the Fellows still in contact, a substantial fraction does not respond to the survey (up to 32 percent ten years post-selection). The responding Fellows are likely a biased sample of the initial fellowship class. Second, since Fellows rate the replication and impact of their own initiatives, there could be exaggeration of progress. Finally, since the Ashoka staff conducts the ME process, and the success of the Fellows program reflects on Ashoka, staff themselves may have some incentive to paint a positive picture of the program. As many funders have suggested external evaluations, Ashoka is strongly considering this for the future.
These issues suggest a cautious reading of the generally positive results Ashoka has found. Of the Fellows who responded to Ashokas ten-year, follow-up survey (roughly half of the initial class), 83 percent said they were still working toward achieving their initial vision. There is some evidence that the persistence rate has been increasing over time, possibly suggesting that Ashoka selection and support of Fellows is improving. Ashoka has also found that 82 percent of respondents believe that another individual or group has replicated their work, and 71 percent of respondents said that they had contributed to a countrys policy change on a national level.