Actual behaviors are challenging to collect and/or are rarely collected. Collecting accurate data poses a significant problem for both financial and health literacy fields because, in both fields, much of the data that are collected are self-reported and the responses often suffer from a social desirability bias, whereby a respondent will overreport positive behaviors and underreport negative behaviors. Rarely do studies have the resources or capabilities to observe behaviors. This challenge typically arises from lack of access to data, for example, inability to gain access to medical records because of the Health Insurance Portability and Accountability Act regulations, as well as financial institution regulations.
Significant literature exists prior to 2005. The literature review focuses on data from 2005 through 2010 but there is significant research prior to 2005.
It is unclear what level of targeted and/or tailoring is needed in an initiative in order to show effectiveness. Lyons et al. (2006) suggest that financial education initiatives must specify the behaviors that they expect to change with the knowledge the education initiative provides, and that the best evaluation is whether individuals are able to make sound decisions regardless of their financial situation, which may be at odds with the desired outcomes, such as decreasing the amount of debt. The research on improving health literacy is focused on small target audiences, and it is not compared with less targeted populations. It is unclear from the research how targeted and/or tailored initiatives should be to be effective.
Many financial literacy programs have yet to be evaluated or have not been rigorously evaluated. Hathaway and Khatiwada (2008) note that existing research on the effectiveness of financial education programs is incomplete and unconvincing. Fox, Bartholomae, and Lee (2005) and the Organisation for Economic Co-operation and Development (OCED) (2005) reported that most financial education programs do not include impact evaluations as a component of their program design. Equally important, however, is the difficulty of devising feasible measures to assess whether financial education programs have attained their main goals increasing consumer awareness and changing individual financial behavior.
The lack of published research is also due to data scarcity. In 2000, PricewaterhouseCoopers abandoned a project to study the effectiveness of counseling after a feasibility study concluded that lenders either collect very limited data or do not collect any about borrowers who have undergone counseling (Mallach, 2001). Data availability is an important issue because even when such data are available, they are often proprietary and, thus, less accessible to external researchers. In addition, because many affordable-credit programs require counseling as part of the loan qualification requirements, is it hard to find an adequate control group (Hartarska & Gonzalez-Vega, 2006). Lyons et al. (2006) identified three main gaps in program evaluation: a lack of general understanding or capacity to run a successful evaluation; evaluation components as afterthoughts, developed after the program has already begun and not embedded in the beginning; and a lack of industry standards.
Programs could also be conducted more rigorously. While most of the health literacy evaluations included a control group, not all were and not all were randomized. Specifically, the literature does not succeed in establishing the extent of the benefit provided by the initiatives, particularly the financial education initiatives. One problem is that most authors assume a causal relationship between financial education and financial outcomes, when in fact the correlation is often weak. Causality might run both ways between knowledge and behavior.
Self-selection bias may also skew results of an initiative. Hathaway and Khatiwada (2008) suggest that participants, who may have a foundation in financial literacy, however small, may be more likely than those who do not to attend financial education courses or counseling sessions. As discussed earlier, studies of IDA programs that match savings rates and provide financial education suffer from self-selection bias because the participants already want to set up a savings account. Meier and Sprenger (2009) found in a study of low-income consumers that participants who are more patient are more likely to participate in financial education programs. However, because previous research has indicated that patient individuals will also have generally better financial outcomes, regardless of whether they participate in educational programs, it is difficult to determine whether the program or the participant characteristic has led to improved financial outcomes (Dellavigna & Paserman, 2005). Most health literacy research includes participants who already participate in a health care system and are thus likely to be more active and engaged in managing health care and the more active and engaged they are in managing their health, the more likely they will have higher levels of health literacy. The health literacy research often has some sort of control or a pre-post assessment, but it does not account for this important characteristic. The results strongly suggest that it is necessary to undertake a greater number of randomized evaluation studies of people who do not self-select in to health care settings or financial programs in order to support claims that education programs are effective.