The Hewlett Foundations pilot of ER analysis and ODG highlight some of the benefits and challenges in pursuing a more rigorous approach to grant-making. While ODG is seemingly straightforward, the Hewlett programs that used it encountered practical challenges at each step in the process. The actual process of calculating ER is very time consuming and the population program found that the margin of error on most estimates was too large to allow for confident comparisons of returns (Redstone Strategy 2008). Introducing the ER metric did not eliminate the importance of the professional judgments of the Hewlett program officers. Many of the components of the ER calculations, particularly the benefits and the likelihood of success, were subjective assessments.
Although the pilot users of ER analysis reported some significant limitations, the ODG process had important benefits. The ER calculation is not a fixed objective metric that answers all funding allocation questions but it did force program staff to be explicit about their goals and assumptions. The process also forced Foundation staff to develop a common language. With explicit goals and outcome measures, the tradeoffs between different grant-making strategies became more evident, and provided the staff with the necessary structure to discuss them. Similarly, while the ER calculations were too imprecise to serve as the sole factor to determine funding allocations, they did provide a structure for thinking about potential benefits and investment risk.
With respect to ERs potential usefulness to the U.S. government, staff at Hewlett expressed concern that in a government context, the more formal ER metric could be used in ways or for purposes that it was not intended. These individuals emphasized that the ER metric is not precise and that it still requires the subjective judgment of program officers. For example, they noted that a program with an ER of 8.7 is not necessarily that different from one with an ER of 8.65, but they worried that U.S. government officials would not have the necessary discretion or flexibility to use their own judgment to decide between two such programs. For Hewlett, flexibility was an important part of the entire ODG process and respondents stressed the importance of having the freedom to use the metric as just one part of the decisionmaking process.