Uwezo report highlights variation in learning within, across East Africa

We continue to have a team of researchers working on an evaluation of the impact of the Uwezo initiative in Kenya. Last week, Uwezo released this report on numeracy and literacy across East Africa, based on the results of their 2010 assessments of student learning in Kenya, Uganda, and Tanzania. Several news outlets covered the release, including The East African, which highlighted the finding that performance is not tied to the quality of infrastructure but to the quality of teaching, including the amount of time teachers give to children. In my visit to a handful of rural schools in Kenya last month, in several of the classrooms I visited, the room was packed with students… but no teacher!

Uwezo Learning Assessment Report

But the report also points out that Kenya is performing best in the region.

While Uwezo’s study is intrinsically important for understanding development trends in the region, our research is quite distinct: We are focused on how citizens react to such information, and over the next few weeks, we will be studying attitudinal and behavioral responses to the release of the report and related campaigns.

Village research part II

We stood in front of the trading area and a few other people came by. Our guide’s son, Richard, told us that several of them would be attending our meeting. We smiled and thanked him, but also tried to make two points: first, we had brought a few refreshments for our meeting – about 8 sodas – and those were still in our car, quite a walk away from here. At the very least, we would want to go get them. Second, we were really hoping to have a conversation with just a small group of people to talk about how things worked in the village, not to call a village meeting.  “Oh sure,” he said. “That’s ok.” And I assumed that as magically as this meeting was being arranged, it would be modified to our requested format.

We continued to proceed up and over the hill. Richard explained that some villagers were on their way to a meeting with British American Tobacco, and would plant and sell for them. Meanwhile, as per our protocol, Ruth was receiving SMS messages on her phone that the questionnaires were going fine. Our enumerators were finding the households and conducting the interviews, but they would later explain that these were extremely impoverished and difficult conditions, even for this poor area.

We agreed that we would hold the meeting near where we parked our car – which was in front of a small but solid structure that we learned was a church for the community, one of the three Seventh Day Adventist churches in the village. It was very hot out, and we had toured the village, so we decided to proceed to the car and wait there for our 2pm meeting. We arrived before 1pm and already about 8 people were sitting on the stoop in front of the church. Ruth and I were both hungry and thirsty, but we were also conscious of not eating in front of a group of people who might be hungry, but for whom we did not have an adequate supply to share. So we subtly climbed into the car, kept our heads low, and scarfed down our peanut butter sandwiches. It was so hot out, and my throat was so dry, I practically choked on a glob of peanut butter, but quickly washed it down with a few swigs of warm water.

When we exited the car, there were about 15-20 people hanging around the front of the church, and more seemed to be coming. Each new person would shake our hand, and those of everyone else present. Ruth began to conduct the interview under the sliver of shade cast by the church, but in so doing was unable to face most of the people who were also protecting themselves from the blazing mid-day sun. I asked if we could have the meeting inside, which was easily arranged since the church leader was in our midst and opened the door.

We entered the room, which was simple and dark, but relatively cool. Before I could help, the church deacon placed two chairs in the front of the room, and we would face two sets of pews, divided by a small aisle – about 4 rows that could fit about 5 people each on either side. The women and some children filed in to our right, all of the men to our left. Among the adults, there was no crossing of the aisle.

Ruth guided the group interview, which was mostly dominated by a few men who felt comfortable speaking. The women were not silent, but it was clearly not a balanced discussion. They answered our questions clearly, told us about how decisions were made, the sources of frustration, and the problems of education, poverty, and a recently unpunished murder in the community.

Our plan was to serve refreshments mid-way through the discussion, but we brought just 8 sodas for a planned group of 4 or 5. There were now at least 30 people in the room. So while Ruth carried on with the meeting, I stepped out and called Richard – our field coordinator, not the elder’s son, who was still in the meeting –  to ask his advice. He asked me if there was a nearby place where I could buy more. Nothing within at least 45 minutes each way by my estimation. So he asked to get on the phone with the elder, and I passed my phone to him. They spoke for a few minutes and I got back on. Richard told me to skip the refreshments altogether. The elder would tell people to go home after the meeting. I was glad to not cause a commotion, but felt badly that we would not be able to provide our small token of appreciation.

Ruth continued and concluded the group interview, gathering the information we needed. At the end, one man stood up and said something more or less like the following: “This is a great day for our village. On this day, for the first time ever, the wazungu (white people) have come. Together, we will remember this day forever. God must have been smiling on us. We will build a primary or a secondary school and we will name it after Ruth and Evan.” We smiled. I was glad they were happy to see us. But as always, we had taken great pains to explain that we were researchers, that we would try to provide knowledge to the wider world about the challenges they faced, and we hoped that out of our work would come some positive benefits. But we were not providers of goods or services and could not arrange for such provisions. People said they truly understood. But nonetheless, the hopes for something else were still there. The ethical dilemmas of doing research, frankly of doing anything in these communities, are ever-present. Just because the education is so poor, and we could see that first-hand, does not imply it was our responsibility to fix it. But it’s also heartbreaking to see the problems, to get back in our car, and to drive away.

Ruth in the church after our meeting

Start of village research

Our research here in Nyanza has ramped up to high gear, and I depart tomorrow – making the 5 hour road trip back to Nairobi, before flying home via Amsterdam. At this point, our surveys are underway, and our systems seem to be working, which is a huge relief following months of planning.

After training our 7 enumerators first in a conference room and then in a village close to where we are all staying, we began our research yesterday with our target sample. We split in to two teams – one led by Richard (a Kenyan from Kisumu), the other by Ruth (an American doctoral student from UCLA). In the morning, after breakfast, our trusted project manager, Jessica, provided each enumerator with a scratch card worth 50 Kenyan shillings – less than 60 US cents – for everyone to load credit onto their phones to pay for calls and text messages later in the day. She also handed out water and a snack, and organized a crate of sodas to be served later to our hosts. Each day, each enumerator is equipped with a folder containing the names of the households they are scheduled to interview; a set of questionnaires; a pencil and sharpener; and a die for a random selection that takes place during the interview.

Ruth and I drove in one car with James, one of the enumerators, who we needed to help translate Luo, while the other three from her team followed in the car behind. We drove for about an hour on a mostly good road, except for a few crater-like potholes right in town, until we reached the designated junction. At that point, we turned onto a fairly solid dirt road until we got to another town, and then we began our journey on some of the rockiest, bumpiest roads I’ve ever seen or experienced. The terrain was lush and hilly, absolutely gorgeous, but the ride was as far to the opposite of smooth as I can imagine. There were no signs or maps of any use – and we just kept asking along the way where to find the village elder. Finally, we arrived at a small church, and I insisted that the car could go no further – the rocks jutted out of the ground so sharply that the risk of a puncture became too great, and we could certainly walk faster than we were traveling.

We walked out of the vehicles and proceeded along the road, and phoned ahead to the elder’s wife that we had arrived. The homesteads generally contained a few small residences, which tended to be made out of mud, and with metal roofs. As always, children stared and yelled out with smiles, “Mzungu (white person)!” Down the road, we met an older man with a cane – the elder. We greeted him with handshakes and followed him to his place. It was a decent walk of about 15 minutes, on what would be the terrain for the day – rocky, dusty, narrow roads, passable to my mind only by foot, but apparently also by bike and motorbike.

When we arrived at the elder’s home, we met his wife and a few others, in addition to the random semi-domesticated animals just hanging around, and we explained our project. As we hoped, the elder directed our enumerators to the general direction in which they would find the households to be surveyed. It was quite a sight to see these young people – all rather stylish, the women with nice shoes and big sunglasses – head out into the overgrown terrain in search of interviews without any hesitation, as all of them had spent much of their lives in some type of small village.

the village elder directs enumerators

Ruth and I remained with the elder, and we explained that later in the day we would like to hold a discussion with himself, and perhaps 4-5 other people who were also leaders from the community – religious leaders, someone older, someone younger, and at least one female. Part of our research involves gathering “village-level” information in addition to the household-level survey, and our goal is to obtain that through these types of elite discussions. He said that should be no problem, and we agreed that 2pm would be a good time. We also asked – Ruth speaks excellent Swahili, which along with some English on their part was enough to communicate – where the nearest public primary school could be found, and he pointed to a place over the hill behind us. I saw a path, and feeling comfortable in the district after a few days, I figured we would just carry on in that direction on our own, but he walked with us. We stopped to shake the hand of a blind man, who sat in front of his hut, wearing a long pink and rather tattered Ralph Lauren button-down shirt, surrounded by a few small animals tied to stakes to keep them from roaming. We continued through extremely thick corn fields, up a hill, and through an ongoing labyrinth of ever rockier paths. I had not realized that my shoes were at least a half size too large until this walk, when I found my foot coming out of the shoe.

rocky road

The terrain continued as rolling hills, with magnificent vistas, and we passed more acres of maize and some cassava fields. A few were just being tilled. We passed some houses with people outside, and as always, they were friendly and greeted us with smiles or a handshake, as everyone does around here. A few men were walking with goats tied to ropes, and we proceeded to the school, as I was thinking what a trek this would be for a young child each day.

homes along the way

Finally, we reached a more open dirt road, which Ruth proclaimed a super highway, as we were finally on solid ground, and one could imagine a car passing here, and apparently it was where one would catch a bus to another village or town.

After about 45 minutes of walking since we left the elder’s home, we arrived at the primary school, which like the other schools we have visited here, looked as if it had been placed on an imaginary set, the scenery was so beautiful. It was lovely to see the kids all playing outside, several teachers sitting under a tree, and one class singing. On the other hand, like at several schools we visited, there was not much obvious teaching or learning going on. As is our protocol, we spoke with the head teacher, but also with an English teacher who was a senior teacher and seemed happy to participate in our interview. We learned of the poor pass rates, that the school had no electricity, just pit toilets, no food for children, and that many others in the village tend to drop out early. As beautiful a place as this was, and as smiling as the kids’ faces were, the situation was downright depressing. To be sure, as our incredibly smart and resourceful enumerators can attest, many Kenyans emerge from such conditions prepared for secondary school and then university, but one can see that the challenges are enormous. In a country that claims English as an official language, which is taught in school, few seemed to comprehend at all.

In one class, the children sang to us – it was a treat. This was a seriously remote village, and despite the inconvenience we pose everyone with our research, which we can only hope will be useful to them in the grander scheme of things, the teachers and students were incredibly gracious.

We finished our interviews and headed back with the elder to his village. Along the way, we encountered his son – aged 41, but unlike many others we’ve met of his age, he did not seem much older. He spoke excellent English, and walked with us back, explaining that he would join our meeting. We passed a trading center, consisting of a few wooden stalls, that would be occupied only on a market day.


From Nairobi

I arrived last night in Nairobi and today we kick off the field research for our evaluation of the Uwezo initiaitive. I’ll be here a few days for training until we dispatch to other provinces. After months of planning, it’s exciting to finally get going on this work.

The taxi driver who brought me from the hotel confirmed that business has been booming in Nairobi. His one fear, however, is the upcoming elections — that once the campaigns begin, violence may come next. Which is the same anxiety people expressed when I visited Zimbabwe in November. Such a tragedy that people have such (realistic) expectations, and what it implies for the prospects for building democratic institutions.

Evidence for the information-accountability-school performance link

After months and months of proposal writing, revision, and planning, I was excited to learn over the past few days that Princeton approved the human subjects protocol and all the other bureaucratic details for an evaluation of the Uwezo initiative in East Africa. I described this project briefly in my initial post — but the main idea is that Uwezo will try to improve education outcomes in Kenya, Tanzania, and Uganda by providing various forms of information to citizens. First, they go to randomly selected households and administer a series of tests to children; second, they provide immediate results to the parents; third, they provide printed information for parents and for the school and community more generally about what they can do to improve education directly (e.g., read to kids) and through policy change and oversight; and fourth, they reinforce that information and put it into broader perspective through information campaigns that describe the results of the assessment. There are some other nuances that I’ll write about later, but that’s the gist.

In our work on this project, Dan Posner, Lily Tsai, and I will be trying to assess the degree to which information empowers citizens to effect meaningful changes with concrete benefits for the education of children. Much of the research will consider whether the information provided is actually “news” to parents and other citizens; whether such information changes people’s views on the services they are receiving; and whether it effects the likelihood of their taking any action. We’ll get started with extensive field research in Kenya this June.

But in the meantime, it’s worth discussing the highly relevant, recent World Bank study, Making Schools Work: New Evidence on Accountability Reforms.

World Bank, Making Schools Work

Education – Making Schools Work.

This is an excellent review of a series of recent studies that have sought to improve education outcomes particularly in developing countries. What’s particularly important about the study is that starts with the premise that there is a huge amount of variation that is NOT due to differences in resources allocated. We certainly know in the United States that money doesn’t solve all of the problems of providing acceptable universal education.

They focus on a series of studies that involved randomized interventions in various  countries, including efforts to effect teacher performance, and school management. But most relevant for our study is chapter two, which investigates a bunch of studies — conducted in India, Pakistan, and Liberia. So far, the results have been pretty mixed, and not terribly encouraging in the sense that information campaigns have not had overwhelming effects on citizen activity or educational outcomes. For example, Banerjee et al (2009) conducted a study in the state of Uttar Pradesh in India, and found that the provision of information about local governance and self assessment tools had only a slight effect on citizen awareness, and virtually none of participation, or learning outcomes, when compared with households from control villages. On the other hand, only a few studies have been conducted, they differ in several ways, and as the authors of the World Bank study conclude, this type of research is in its infancy, and we still know very little about whether and how such interventions can have a measurable effect. To our knowledge, no similar study has been conducted in East Africa. The idea that better information should help citizens to realize the promise of democratic governance is too important to discard based on just a few studies, and while we will let the results of our study speak for themselves, it will be extremely important to understand if and how new information influences citizen participation in development.