Tag: Uwezo

Successful Twaweza evaluators’ workshop

by Evan Lieberman1 Comment on Successful Twaweza evaluators’ workshop

I am on my way home from the Twaweza evaluators’ conference in Dar es Salaam… literally on my way, posting this on the second leg of my air journey to JFK, having departed Dubai around 4am… which suggests that now even international air travel offers no respite from the distractions of the web. But I digress.

The meeting was really extraordinary in so many ways. Twaweza’s Learning, Monitoring, and Evaluation manager, Varja Liposvek, brought together several evaluation teams — including the “LPT” team (Lieberman, Dan Posner and Lily Tsai), the AIID team from University of Amsterdam (Chris Elbers and Jan Willem Gunning), and James Habyarimana, who represented his Georgetown-based team. And a new JPAL/IPA project was presented by Twaweza’s Youdi Schipper. The approximately 40 attendees included various managers from within the organization; and a host of researchers and development specialists, including from the World Bank, Innovations for Poverty Action, Oxfam, the International Budget Project, the Transparency and Accountability Initiative, and DFID.

Although the room contained many distinct perspectives, the conversation was unified around a willingness to rigorously question every proposition, including how to conceptualize and to measure the intervention and associated outcomes, and how to judge the quality of evidence. I have attended many similar events, in which the attendees also came from different perspectives, and this one was remarkable for the unified willingness to engage constructively.

The first day of the conference involved hearing from the various evaluation teams, including ours, and in all cases, the studies are not yet complete. LPT and AIID have actually been evaluating Twaweza’s work, in the case of LPT by focusing on their education initiative, Uwezo. The other projects have initiated RCT’s to test related propositions that will inform future work. It was an extremely useful discussion and rewarding for us to see the care and attention with which our client was listening to and keeping track of these studies and thinking about ways to incorporate the findings into their mission.

Along these lines, more extraordinary from my perspective, was the second day — in which Twaweza’s director, Rakesh Rajani, announced to us all — look, we remain devoted to Twaweza, we know that these evaluations are not complete and only evaluate parts of what we do… but we also know from these studies, and from our experience and intuition to date, that much of what we are doing is not having the impact we would like. So let’s not simply keep doing the same thing; let’s make some substantial adjustments to better position ourselves for success.

In this regard, Twaweza is in a uniquely favorable position as compared with most NGO’s. I don’t know all of the specifics, but Twaweza’s donors appear to have a pretty long and patient time horizon. Unlike the average NGO, which is constantly fighting for its own survival, and under severe pressure to demonstrate quick results, Twaweza can afford to admit they didn’t have all the right answers before they started.

Again, particularly gratifying from the perspective of development/evaluation researchers was the stated commitment (and I’ll report back to see if this happens) to incorporate evaluators to a much greater extent in both the theorizing and design of their work going forward. (That willingness was not part of the first stage…) We now all have a basis for thinking about what doesn’t work and some intuition about why, and hopefully, this will allow both Twaweza and the evaluators to make some better and more focused bets about what might have the desired impact on the key outcomes of citizen agency and service delivery.

I learned a great deal from the workshop. That said, I wouldn’t yet bet my home that Twaweza will achieve its goals or that we or any of the other evaluation teams will be able to detect any treatment effects from research that we might design going forward. The core mission of trying to increase citizen agency and improve development outcomes through information provision is a challenging one. But I think that Twaweza has already set a valuable example for researchers and development practitioners in terms of how to engage in a manner that sparks careful, creative, and critical thinking, incorporating necessary theoretical and practical concerns. I’m glad that 50 hours of flying and brief abandonment of my family was not in vain.

Sadly avoiding Nairobi this week…

by Evan LiebermanLeave a Comment on Sadly avoiding Nairobi this week…

I was scheduled to fly out to Nairobi this Saturday night to do 3 days of work, including some social science research methods training to the Uwezo staff, before heading to Dar es Salaam for a multi-day conference with the various teams that have been working on the Twaweza initiative. In light of the awful attack on the Westgate Mall, a place I have frequented several times, and the fact that it was not absolutely essential to be in Nairobi, we decided to bypass Kenya altogether. I’ll go “straight” (i.e., with two other layovers) to Tanzania instead. Notwithstanding the pleas from the Standard Newspaper against international travel advisories… my inbox piled up this morning with lots of them and frankly, I just couldn’t justify to myself or to my family a good reason for entering the fray in the midst of all that’s going on, especially since it would likely be impossible to be very productive. But I’m disappointed to not be going to Nairobi, and more so that Kenyans are taking it on the chin once again.

Meanwhile, I know it’s 2013, but I’m still dumbstruck by the ubiquitous nature of communications technology and social media during a terrorist attack. Seems as if many citizens at the scene, despite threats to life and body, have been using their smartphones to broadcast images from the mall. Shabaab and the Kenyan military have been exchanging public tweets about who has the upper hand in the standoff. CNN writes, “ Are you in Nairobi? Send us your images and experiences, but please stay safe.” Unfortunately, the spectacle and global stage of violence almost certainly makes it a more appealing strategy for the very groups who perpetrated these heinous acts.

P.S. I know I have been a delinquent blogger for quite some time. I really don’t know how these folks with full time jobs can manage to blog so frequently… but perhaps the Fall will bring greater inspiration and speedy fingers in generating blog-worthy posts. I am likely to tweet more frequently at @evlieb

Learning about citizenship in Rural Kenya

by Evan Lieberman1 Comment on Learning about citizenship in Rural Kenya

As part of our ongoing work with Uwezo in Kenya, Dan Posner, Lily Tsai and I commissioned two great Ph.D. students — Brandon de la Cuesta (Princeton) and Leah Rosenzweig (MIT) — to help us learn more about what citizens in rural Kenya are doing (or not doing) to exercise their rights as citizens, particularly in the primary education sector. Much of our initial research during the first couple of years of our work on this project has showed virtually no effects from information campaigns, and we thought it was important to get a fuller understanding of the range of actions citizens can and do take. But we concluded that additional closed-ended surveys would not be the way to go, because frankly, we were concerned that we might not be asking the right questions.

So, we sent Brandon and Leah off for a few months of field research, working with our senior project manager, Jessica Grody, and various Uwezo staff, to conduct more open-ended interviews, conduct focus groups, and simply observe what is going on, in order to try to generate some better ideas for our ongoing research concerning what types of information might drive active citizenship. Their findings will help drive our future research agenda on this project, and in the meantime, they just shared with me a few pics from their days in the field.

students reading in class
students reading in class
primary school children with their head teacher in Nandi East
primary school children with their head teacher in Nandi East
Brandon speaking with the chief in Nandi East at a Baraza for farmers
Brandon speaking with the chief in Nandi East at a Baraza for farmers
posters from a classroom in Meru South
posters from a classroom in Meru South

Kenya-bound for more Uwezo research

by Evan LiebermanLeave a Comment on Kenya-bound for more Uwezo research

I head off tomorrow for a few weeks of meetings and research in Kenya. I’ll join my collaborators, Lily Tsai and Dan Posner, and our project manager, Jessica Grody to continue our work trying to understand the impact of the Uwezo initiative. We’ll get started in Nairobi, and from there we’ll do some traveling around the country.

In our research so far, we have examined whether the direct provision of information to parents – specifically, their own children’s literacy and numeracy test results as well as information about what parents can do to be more active citizens – actually leads them to higher levels of local involvement in education. Like many other scholars, we are trying to understand if and when the provision of information leads to greater government accountability in terms of the delivery of key services.

For good reason, a great deal of impact evaluation research these days is being carried out using experimental methods. And if the definition of an experiment requires that the analyst randomly assign treatment to subjects, then technically speaking, ours was not an experiment. But fortunately, Uwezo had decided for various reasons to carry out this portion of their initiative to a random sample of villages around the country. Thus, one could not make the claim that there was something systematically distinctive about the villages that received the intervention. In order to estimate the effects of these informational “bundles” we required a control group. So prior to beginning our study, we identified a set of villages that were highly similar to the ones that had been randomly selected by Uwezo. Then we conducted surveys, focus groups, and other research in both sets of villages. We will complete a draft of a scholarly paper on this study fairly soon, and we will be presenting our results at the late August meetings of the American Political Science Association.

As for our estimates of the effect of those “information bundles”…. Well, we didn’t find any substantial impact. But in fairness, we didn’t hold particularly high expectations for an effect. It was important to see if this aspect of the initiative could drive desired outcomes on its own. It can’t. And now, we will investigate whether some of the much larger information dissemination campaigns make a difference.

On this trip we won’t be carrying out any systematic research – just learning about what Uwezo has been doing and what they plan to do going forward. Hopefully, we will finalize some interesting proposals for research in the coming years.

Uwezo Uganda film on the value of education

by Evan LiebermanLeave a Comment on Uwezo Uganda film on the value of education

Uwezo Uganda just released a short film on the dreams (of becoming president) and challenges (of going to school when burdened with responsibilities) of an 11-year-old boy named Kyosiga.

Since the goal of this NGO is to inspire parents to place a higher premium on education, the intuition seems just right: Drama is a potentially powerful tool for disseminating key ideas. Along these lines, social scientists like myself need to make peace with the idea that coldly presented facts about the value of education — for example, with evidence about income gains from schooling — are probably not particularly persuasive for most people. Those facts may be necessary for getting a full picture of reality, but in turn, most people seem to be convinced by stories. A dramatic film that makes the case for education through the drama of a single individual who can stand in for the “average child” sounds promising. Whether the intended audience will actually get to see the film, and how they will respond to it remains to be seen.

Premier of Kyosiga's Dream (Daily Monitor)

New Kenyan education initiative and Uwezo results

by Evan LiebermanLeave a Comment on New Kenyan education initiative and Uwezo results

USAID is funding a huge new primary math and reading initiative to increase proficiency at very basic levels. An article in Kenya’s, The Standard, links the initiative to Uwezo Kenya’s findings of poor numeracy and literacy skill attainment by standard two (see prior posts on related research on Uwezo’s work). It’s not clear from the article to what extent the Uwezo initiative — either in conception or in results — influenced the making of this grant. But the focus is closely tied to what Uwezo has been championing: a results- rather than input-orientation, and the fostering of a reading culture within communities. According to the article, a similar program raised the prevalence of reading proficiency in the Gambia from 4% to 28%.

Uwezo Tanzania Assessment Documents Disappointing Education Results

by Evan LiebermanLeave a Comment on Uwezo Tanzania Assessment Documents Disappointing Education Results

Yesterday, Uwezo released the results of its 2011 assessment of the educational attainment of Tanzanian children — it’s a comprehensive study, which covered more than 70,000 households in a nationally representative sample.

2011 Uwezo Assessment

As in Kenya, the report shows a high degree of variation across the country… but the results are generally quite disappointing given the investments in and attention to educational development in recent years.

In Tanzania, large numbers of children are not acquiring very basic literacy and numeracy skills until very late in their education, and even then, many still lack those skills. As late as standard 7, the majority of school children cannot read a very basic story in English — one geared toward the level of a standard 2 learner.

The report is available here and from the Uwezo website.

 

Photos from our Uwezo field research in Kenya

by Leave a Comment on Photos from our Uwezo field research in Kenya

Our research team has now completed 12 busy weeks of fieldwork in Kenya, and before we begin to disseminate results, I wanted to share some photographs to provide a context of what daily life is like in the areas in which we have been working. These pictures were taken by Kelly Zhang (Stanford PhD student) who was working in Central Province in Kenya.

Central Province is relatively well off compared to other regions in Kenya. Many of the primary schools we visited followed a layout similar to the one shown below, with classrooms opening out to an outdoor courtyard.

School Classrooms and Water Tank

The children at this school were lucky to have a water tank on the school grounds, shown in the middle of the picture above. At schools without water tanks, or at times when the tanks are dry, children must bring water from home.

It is rare to find computers in any of the rural schools (indeed, most primary schools we visited lacked electricity), so teachers rely on more traditional methods to organize the vast amounts of information a school must handle.

School Timetable

The weekly schedule at one primary school was kept on a giant chalkboard in the staff room, next to a list of the teacher’s names and contact information. Schools regularly post the school calendars, test scores, and budgets on chalkboards like this or hung on poster board.

The national government provides funding for primary education, but many schools find the government disbursements are insufficient to cover the costs of building maintenance and learning materials. Some entrepreneurial head teachers, therefore, institute income-generating activities at the school. This school was growing crops to provide some supplemental income to help meet the needs of the students and teachers.

School Crops

Other schools we visited had planted tea and banana trees, and some even tended livestock.

Much of Central Province is quite lush. Most people farm, and tea is a major crop in the region.

Tea Farming

We found tea buying centers across the region, and after harvesting the tea leaves, people bring them to the buying centers where they will be sold, then processed at a nearby tea factory.

In the town centers we visited, most buildings are simple cement structures.

Downtown Street

This a typical street in a town center in the areas we were working.

When our research team visited a new village for the first time, they frequently sat down with the village elders and other prominent members of the community to learn more about the community. Our obvious foreign presence often created a stir, and sometimes these discussion groups were attended by many more people than anticipated as local residents were eager to hear what we were doing in their community.

Kelly Zhang with village elder and other community members

In the weeks to come, I will post photographs from the places we worked in Nyanza Province, which is in general much poorer than Central Province.

Filmer on information, accountability, better education

by Evan LiebermanLeave a Comment on Filmer on information, accountability, better education

Deon Filmer, one of the authors of the World Bank’s Making Schools Work book, blogs on the possible links between information, accountability and better education that are at the core of the Uwezo initiative that Posner, Tsai, and I are currently evaluating. In his post, he talks about what he is looking for in terms of a school for his own children in Washington, D.C., and relates this to  broader theories of the link between information and accountability. And he summarizes some of the mixed evidence from extant research in Africa and South Asia.

Filmer argues that, the two key features for success seem to be, “(1) keeping information simple, and (2) making sure that the intended recipients understand it.” And that this is true at both the household and school levels.

The fieldwork from the first phase of our research should be complete in a few weeks, and we look forward to disseminating information about the extent to which these conclusions also apply in the context of the Uwezo initiative both in Kenya and in Tanzania.

Uwezo Kenya 2011 National Report Launch Today! :: Uwezo.net

by Evan LiebermanLeave a Comment on Uwezo Kenya 2011 National Report Launch Today! :: Uwezo.net

Uwezo Kenya 2011 National Report Launch Today! :: Uwezo.net.

Today, our four field coordinators are in place to observe the launch/release of the 2011 Uwezo learner assessment results in two Kenyan provinces. Uwezo’s report, available at the link above, provides very detailed information on learner attainment across Kenyan counties, and on a range of other relevant indicators, including student and teacher attendance. While there is a lot of variation within Kenya, the fact that in the context of free primary education, in many districts, half of children are out of school… is sobering.

And as I wrote about earlier, on most scores, Kenya is performing better than East African neighbors, Uganda and Tanzania.

In the weeks to come, we will assess the diffusion of and responses to the release of this information.