Successful Twaweza evaluators’ workshop

I am on my way home from the Twaweza evaluators’ conference in Dar es Salaam… literally on my way, posting this on the second leg of my air journey to JFK, having departed Dubai around 4am… which suggests that now even international air travel offers no respite from the distractions of the web. But I digress.

The meeting was really extraordinary in so many ways. Twaweza’s Learning, Monitoring, and Evaluation manager, Varja Liposvek, brought together several evaluation teams — including the “LPT” team (Lieberman, Dan Posner and Lily Tsai), the AIID team from University of Amsterdam (Chris Elbers and Jan Willem Gunning), and James Habyarimana, who represented his Georgetown-based team. And a new JPAL/IPA project was presented by Twaweza’s Youdi Schipper. The approximately 40 attendees included various managers from within the organization; and a host of researchers and development specialists, including from the World Bank, Innovations for Poverty Action, Oxfam, the International Budget Project, the Transparency and Accountability Initiative, and DFID.

Although the room contained many distinct perspectives, the conversation was unified around a willingness to rigorously question every proposition, including how to conceptualize and to measure the intervention and associated outcomes, and how to judge the quality of evidence. I have attended many similar events, in which the attendees also came from different perspectives, and this one was remarkable for the unified willingness to engage constructively.

The first day of the conference involved hearing from the various evaluation teams, including ours, and in all cases, the studies are not yet complete. LPT and AIID have actually been evaluating Twaweza’s work, in the case of LPT by focusing on their education initiative, Uwezo. The other projects have initiated RCT’s to test related propositions that will inform future work. It was an extremely useful discussion and rewarding for us to see the care and attention with which our client was listening to and keeping track of these studies and thinking about ways to incorporate the findings into their mission.

Along these lines, more extraordinary from my perspective, was the second day — in which Twaweza’s director, Rakesh Rajani, announced to us all — look, we remain devoted to Twaweza, we know that these evaluations are not complete and only evaluate parts of what we do… but we also know from these studies, and from our experience and intuition to date, that much of what we are doing is not having the impact we would like. So let’s not simply keep doing the same thing; let’s make some substantial adjustments to better position ourselves for success.

In this regard, Twaweza is in a uniquely favorable position as compared with most NGO’s. I don’t know all of the specifics, but Twaweza’s donors appear to have a pretty long and patient time horizon. Unlike the average NGO, which is constantly fighting for its own survival, and under severe pressure to demonstrate quick results, Twaweza can afford to admit they didn’t have all the right answers before they started.

Again, particularly gratifying from the perspective of development/evaluation researchers was the stated commitment (and I’ll report back to see if this happens) to incorporate evaluators to a much greater extent in both the theorizing and design of their work going forward. (That willingness was not part of the first stage…) We now all have a basis for thinking about what doesn’t work and some intuition about why, and hopefully, this will allow both Twaweza and the evaluators to make some better and more focused bets about what might have the desired impact on the key outcomes of citizen agency and service delivery.

I learned a great deal from the workshop. That said, I wouldn’t yet bet my home that Twaweza will achieve its goals or that we or any of the other evaluation teams will be able to detect any treatment effects from research that we might design going forward. The core mission of trying to increase citizen agency and improve development outcomes through information provision is a challenging one. But I think that Twaweza has already set a valuable example for researchers and development practitioners in terms of how to engage in a manner that sparks careful, creative, and critical thinking, incorporating necessary theoretical and practical concerns. I’m glad that 50 hours of flying and brief abandonment of my family was not in vain.

Sadly avoiding Nairobi this week…

I was scheduled to fly out to Nairobi this Saturday night to do 3 days of work, including some social science research methods training to the Uwezo staff, before heading to Dar es Salaam for a multi-day conference with the various teams that have been working on the Twaweza initiative. In light of the awful attack on the Westgate Mall, a place I have frequented several times, and the fact that it was not absolutely essential to be in Nairobi, we decided to bypass Kenya altogether. I’ll go “straight” (i.e., with two other layovers) to Tanzania instead. Notwithstanding the pleas from the Standard Newspaper against international travel advisories… my inbox piled up this morning with lots of them and frankly, I just couldn’t justify to myself or to my family a good reason for entering the fray in the midst of all that’s going on, especially since it would likely be impossible to be very productive. But I’m disappointed to not be going to Nairobi, and more so that Kenyans are taking it on the chin once again.

Meanwhile, I know it’s 2013, but I’m still dumbstruck by the ubiquitous nature of communications technology and social media during a terrorist attack. Seems as if many citizens at the scene, despite threats to life and body, have been using their smartphones to broadcast images from the mall. Shabaab and the Kenyan military have been exchanging public tweets about who has the upper hand in the standoff. CNN writes, “ Are you in Nairobi? Send us your images and experiences, but please stay safe.” Unfortunately, the spectacle and global stage of violence almost certainly makes it a more appealing strategy for the very groups who perpetrated these heinous acts.

P.S. I know I have been a delinquent blogger for quite some time. I really don’t know how these folks with full time jobs can manage to blog so frequently… but perhaps the Fall will bring greater inspiration and speedy fingers in generating blog-worthy posts. I am likely to tweet more frequently at @evlieb

Learning about citizenship in Rural Kenya

As part of our ongoing work with Uwezo in Kenya, Dan Posner, Lily Tsai and I commissioned two great Ph.D. students — Brandon de la Cuesta (Princeton) and Leah Rosenzweig (MIT) — to help us learn more about what citizens in rural Kenya are doing (or not doing) to exercise their rights as citizens, particularly in the primary education sector. Much of our initial research during the first couple of years of our work on this project has showed virtually no effects from information campaigns, and we thought it was important to get a fuller understanding of the range of actions citizens can and do take. But we concluded that additional closed-ended surveys would not be the way to go, because frankly, we were concerned that we might not be asking the right questions.

So, we sent Brandon and Leah off for a few months of field research, working with our senior project manager, Jessica Grody, and various Uwezo staff, to conduct more open-ended interviews, conduct focus groups, and simply observe what is going on, in order to try to generate some better ideas for our ongoing research concerning what types of information might drive active citizenship. Their findings will help drive our future research agenda on this project, and in the meantime, they just shared with me a few pics from their days in the field.

students reading in class

students reading in class

primary school children with their head teacher in Nandi East

primary school children with their head teacher in Nandi East

Brandon speaking with the chief in Nandi East at a Baraza for farmers

Brandon speaking with the chief in Nandi East at a Baraza for farmers

posters from a classroom in Meru South

posters from a classroom in Meru South

Kenya-bound for more Uwezo research

I head off tomorrow for a few weeks of meetings and research in Kenya. I’ll join my collaborators, Lily Tsai and Dan Posner, and our project manager, Jessica Grody to continue our work trying to understand the impact of the Uwezo initiative. We’ll get started in Nairobi, and from there we’ll do some traveling around the country.

In our research so far, we have examined whether the direct provision of information to parents – specifically, their own children’s literacy and numeracy test results as well as information about what parents can do to be more active citizens – actually leads them to higher levels of local involvement in education. Like many other scholars, we are trying to understand if and when the provision of information leads to greater government accountability in terms of the delivery of key services.

For good reason, a great deal of impact evaluation research these days is being carried out using experimental methods. And if the definition of an experiment requires that the analyst randomly assign treatment to subjects, then technically speaking, ours was not an experiment. But fortunately, Uwezo had decided for various reasons to carry out this portion of their initiative to a random sample of villages around the country. Thus, one could not make the claim that there was something systematically distinctive about the villages that received the intervention. In order to estimate the effects of these informational “bundles” we required a control group. So prior to beginning our study, we identified a set of villages that were highly similar to the ones that had been randomly selected by Uwezo. Then we conducted surveys, focus groups, and other research in both sets of villages. We will complete a draft of a scholarly paper on this study fairly soon, and we will be presenting our results at the late August meetings of the American Political Science Association.

As for our estimates of the effect of those “information bundles”…. Well, we didn’t find any substantial impact. But in fairness, we didn’t hold particularly high expectations for an effect. It was important to see if this aspect of the initiative could drive desired outcomes on its own. It can’t. And now, we will investigate whether some of the much larger information dissemination campaigns make a difference.

On this trip we won’t be carrying out any systematic research – just learning about what Uwezo has been doing and what they plan to do going forward. Hopefully, we will finalize some interesting proposals for research in the coming years.

Uwezo Uganda film on the value of education

Uwezo Uganda just released a short film on the dreams (of becoming president) and challenges (of going to school when burdened with responsibilities) of an 11-year-old boy named Kyosiga.

Since the goal of this NGO is to inspire parents to place a higher premium on education, the intuition seems just right: Drama is a potentially powerful tool for disseminating key ideas. Along these lines, social scientists like myself need to make peace with the idea that coldly presented facts about the value of education — for example, with evidence about income gains from schooling — are probably not particularly persuasive for most people. Those facts may be necessary for getting a full picture of reality, but in turn, most people seem to be convinced by stories. A dramatic film that makes the case for education through the drama of a single individual who can stand in for the “average child” sounds promising. Whether the intended audience will actually get to see the film, and how they will respond to it remains to be seen.

Premier of Kyosiga's Dream (Daily Monitor)

New Kenyan education initiative and Uwezo results

USAID is funding a huge new primary math and reading initiative to increase proficiency at very basic levels. An article in Kenya’s, The Standard, links the initiative to Uwezo Kenya’s findings of poor numeracy and literacy skill attainment by standard two (see prior posts on related research on Uwezo’s work). It’s not clear from the article to what extent the Uwezo initiative — either in conception or in results — influenced the making of this grant. But the focus is closely tied to what Uwezo has been championing: a results- rather than input-orientation, and the fostering of a reading culture within communities. According to the article, a similar program raised the prevalence of reading proficiency in the Gambia from 4% to 28%.

Uwezo Tanzania Assessment Documents Disappointing Education Results

Yesterday, Uwezo released the results of its 2011 assessment of the educational attainment of Tanzanian children — it’s a comprehensive study, which covered more than 70,000 households in a nationally representative sample.

2011 Uwezo Assessment

As in Kenya, the report shows a high degree of variation across the country… but the results are generally quite disappointing given the investments in and attention to educational development in recent years.

In Tanzania, large numbers of children are not acquiring very basic literacy and numeracy skills until very late in their education, and even then, many still lack those skills. As late as standard 7, the majority of school children cannot read a very basic story in English — one geared toward the level of a standard 2 learner.

The report is available here and from the Uwezo website.