Successful Twaweza evaluators’ workshop

I am on my way home from the Twaweza evaluators’ conference in Dar es Salaam… literally on my way, posting this on the second leg of my air journey to JFK, having departed Dubai around 4am… which suggests that now even international air travel offers no respite from the distractions of the web. But I digress.

The meeting was really extraordinary in so many ways. Twaweza’s Learning, Monitoring, and Evaluation manager, Varja Liposvek, brought together several evaluation teams — including the “LPT” team (Lieberman, Dan Posner and Lily Tsai), the AIID team from University of Amsterdam (Chris Elbers and Jan Willem Gunning), and James Habyarimana, who represented his Georgetown-based team. And a new JPAL/IPA project was presented by Twaweza’s Youdi Schipper. The approximately 40 attendees included various managers from within the organization; and a host of researchers and development specialists, including from the World Bank, Innovations for Poverty Action, Oxfam, the International Budget Project, the Transparency and Accountability Initiative, and DFID.

Although the room contained many distinct perspectives, the conversation was unified around a willingness to rigorously question every proposition, including how to conceptualize and to measure the intervention and associated outcomes, and how to judge the quality of evidence. I have attended many similar events, in which the attendees also came from different perspectives, and this one was remarkable for the unified willingness to engage constructively.

The first day of the conference involved hearing from the various evaluation teams, including ours, and in all cases, the studies are not yet complete. LPT and AIID have actually been evaluating Twaweza’s work, in the case of LPT by focusing on their education initiative, Uwezo. The other projects have initiated RCT’s to test related propositions that will inform future work. It was an extremely useful discussion and rewarding for us to see the care and attention with which our client was listening to and keeping track of these studies and thinking about ways to incorporate the findings into their mission.

Along these lines, more extraordinary from my perspective, was the second day — in which Twaweza’s director, Rakesh Rajani, announced to us all — look, we remain devoted to Twaweza, we know that these evaluations are not complete and only evaluate parts of what we do… but we also know from these studies, and from our experience and intuition to date, that much of what we are doing is not having the impact we would like. So let’s not simply keep doing the same thing; let’s make some substantial adjustments to better position ourselves for success.

In this regard, Twaweza is in a uniquely favorable position as compared with most NGO’s. I don’t know all of the specifics, but Twaweza’s donors appear to have a pretty long and patient time horizon. Unlike the average NGO, which is constantly fighting for its own survival, and under severe pressure to demonstrate quick results, Twaweza can afford to admit they didn’t have all the right answers before they started.

Again, particularly gratifying from the perspective of development/evaluation researchers was the stated commitment (and I’ll report back to see if this happens) to incorporate evaluators to a much greater extent in both the theorizing and design of their work going forward. (That willingness was not part of the first stage…) We now all have a basis for thinking about what doesn’t work and some intuition about why, and hopefully, this will allow both Twaweza and the evaluators to make some better and more focused bets about what might have the desired impact on the key outcomes of citizen agency and service delivery.

I learned a great deal from the workshop. That said, I wouldn’t yet bet my home that Twaweza will achieve its goals or that we or any of the other evaluation teams will be able to detect any treatment effects from research that we might design going forward. The core mission of trying to increase citizen agency and improve development outcomes through information provision is a challenging one. But I think that Twaweza has already set a valuable example for researchers and development practitioners in terms of how to engage in a manner that sparks careful, creative, and critical thinking, incorporating necessary theoretical and practical concerns. I’m glad that 50 hours of flying and brief abandonment of my family was not in vain.