Duncan Green recently posted about a promising and straightforward approach to developing effective interventions in Tanzania, which he likens to a venture capital model. Oxfam and other donors sponsored a set of projects designed to build accountability largely through various awareness campaigns, and within a relatively short time, relevant donors and stakeholders convened themselves to review the projects. Of particular note, they killed off the ones that seemed to not be working.
Seems logical, I know. But surprisingly rare in the development world where the incentives and timetables are often set up to recognize failure only after millions of dollars and hours have been invested.
I certainly would not advocate that development projects should be subject to the types of quarterly performance expectations which often drive publicly traded companies towards pathological investment patterns. Many projects, by design, take a long time to implement and observable change may not be immediate. A bad quarter doesn’t mean that a project is a failure. But, rapid assessment and feedback, including learning from problems, mistake assumptions, etc., seems to be an obviously good strategy. The challenge is to mobilize relevant evaluators to invest the time and energy to reflect quickly and for project implementers to be willing and able to digest the feedback.
Green highlights some of the key lessons learned from failures:
What didn’t work and why?
Geography: The active musicians were not able to work well in Ngorongoro, because the communities were too widely dispersed to reach.
Government obstruction: The community radio never got off the ground because the government did not issue a licence.
Informal v formal power: The farmer animators’ work was unsuccessful in spreading awareness beyond the groups that the animators belonged to. This might have been due to their lack of a ‘formal’ position in community leadership.
Attitudes to youth: Students were able to make demands within their schools, but were unable to take this approach into the community– there was simply not enough respect for young people’s viewpoints.
I’d be curious to hear more about how well the initially successful projects (i.e., those that didn’t get axed) were able to incorporate the new information from the failures and whether this led to better outcomes overall.
I think this kind of feedback & responsiveness would be great — how do you blend it with an experimental framework in which tinkering with / overhauling the experimental program is not always a desirable proposition? I think overcoming the interest in measuring impact – and what that does/not mean for fidelity to the original plan – is going to be a key issue. Thoughts on proceeding?
I think the point is that one can learn from a good experimental approach — hopefully generate solid results from smaller-scale pilots — and decide whether or not it is worth it to scale up or to first adjust with a slightly different “treatment.” If a project goes to “scale” too quickly it may not benefit from the potentially critical information that could be learned in test runs. It seems so obvious it’s almost silly to write about, but again, I think that given the logic of much of the aid industry, there is insufficient space for piloting.