Policy-based evidence making

Posted: May 23, 2011 | Author: | Filed under: General, Research methods | 2 Comments

For all researchers hoping that their work might be able to positively influence a government’s policies, British political comedy TV shows are a good reality check. Hat tips to Aid Thoughts and Roving Bandit for the following gems from Yes, Prime Minister and The Thick Of It respectively, showing how (not) to design surveys or seek expert advice (strong language warning for the second clip).

Water point mapping in a Mali village: who drinks from where?

Posted: May 10, 2011 | Author: | Filed under: Research methods, Water | 2 Comments

Understanding where people choose to drink from when they live in a village with multiple water sources (including both ‘improved’ and ‘unimproved’ water points) is a key aspect of my research. I’ve been using simple household surveys to ask this question, and I’m now beginning to visualise the data using the Water Point Mapper tool developed by WaterAid. The tool is free and easy to use, employing a combination of GPS data, a Microsoft Excel spreadsheet mapper, and Google Earth. You can download the mapper, a manual and example maps from the website. The Water Point Mapper is designed to identify each water point with a colour-coded symbol. Depending on the type of information you want to display, this symbol can represent various indicators. For example, this sample map from the website shows the functionality of water points in an area:

Maps can also be generated with symbols to represent the number of users at the water point, the type of water source, the revenue collection system used, and various other characteristics (click to see the examples from the Water Point Mapper website). The mapper tool also collates this information to allow comparison of different areas (such as sub-districts or districts), which can be coloured according to the indicators.

However, for this analysis I have adapted the tool to show individual households rather than water points, because I am interested in detailed behaviour at village level. In this example of one of the villages where my research is ongoing, each coloured point on the map represents a household and the colour of the point represents the type of water source used by the household for drinking:

  • Blue = drilled borehole fitted with a handpump (an ‘improved’ source according to international definitions and national norms).
  • Yellow = modern hand-dug well with concrete lining, metal cover and bucket to draw water (also considered ‘improved’, but with an increased risk of contamination if users leave the lid open or leave the bucket on the ground).
  • Red = traditional hand-dug well, usually with no lining or cover (an unimproved source because of the high likelihood of contamination of the water).

The map shows that about a quarter of the households in the village use traditional wells, rather than improved sources, and that all but one of these households are in the western side of the community.

Read the rest of this entry »

‘More Than Good Intentions’: a review

Posted: April 27, 2011 | Author: | Filed under: Books, Economics, Research methods, Water | 2 Comments

Two new books are just out from some of the key figures of “the new development economics” (Tyler Cowen’s label, not mine) – More Than Good Intentions: How a New Economics Is Helping to Solve Global Poverty by Dean Karlan and Jacob Appel, and Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty by Esther Duflo and Abhijit Banerjee. I’m not a development economist, but I’m keen to discuss what this movement can contribute to the aid and development debates in general, and the field of water, sanitation and hygiene in particular.

First onto my Kindle: More Than Good Intentions. Karlan and Appel use insights from behavioural economics to show how poor people can be given a helping hand (or a ‘nudge’, following Thaler and Sunstein) in many aspects of their lives, especially microfinance, and to demonstrate how the impact of these interventions can be rigorously evaluated using randomised controlled trials (RCTs). The book is a showcase – or perhaps manifesto – for the work of Innovations for Poverty Action (IPA), the organisation founded by Karlan to put into practice and test out these ideas, in collaboration with a range of partners around the world. Some good examples, all starting from programmes in Kenya, include encouraging farmers to buy fertiliser ready for the next year’s harvest by offering them pre-paid vouchers just after they sell their current crop; working out what interventions are most cost-effective for improving school attendance (deworming is cheaper than giving free school uniforms to the poorest); and – obviously of great interest to me – increasing the number of families using chlorine to treat drinking water by placing simple chlorine dispensers next to water points.

The positives: More Than Good Intentions is a very readable introduction to the field of behavioural economics and RCTs in development, and the authors do a great job of conveying the excitement and potential of development research in general. The format of describing results from investigations by a variety of Karlan’s IPA collaborators interspersed with anecdotes from people Appel meets in his research works well (although the language has some overly cutesy lapses, such as “a great waterfall of pork rinds frozen in mid-cascade” in a small shop in the Philippines). Karlan is particularly strong on interrogating the details of the microfinance movement, and examining the existing ways in which the poor manage their finances (a strength shared with Portfolios of the Poor). I also liked the comparisons with ‘nudge’ ideas which originated to help Western consumers save better or keep their commitments, such as Save More Tomorrow and Stickk.com.

Where I found the book unsatisfying was in its lack of discussion about how to “bridge the gap”, as Karlan and Appel put it, between particular RCTs and wider applicable recommendations. As in any development intervention, context (see an excellent series of posts by Dave Algoso on Find What Works) is crucial; the claim of More Than Good Intentions is that enough RCTs plus an understanding of the underlying behavioural theory can show us what ideas can be successful across different contexts. The concluding chapter lists “seven ideas that work”; some of these are considered to have ‘proven impact’ by IPA’s Proven Impact Initiative while others are at the ‘promising’ stage. But I would have liked more detailed discussion from the authors on how we can draw these wider conclusions and what evidence base they think is needed.

More Than Good Intentions also shows some tantalising glimpses of the politics involved in implementing policy based on evidence from RCTs, but neglects the juicy details and what we could learn from these. For example, Karlan and Appel state that the Progresa programme in Mexico, which provided small cash grants to the poor dependent on their school attendance and health clinic visits, was originally controversial but later accepted, partly thanks to the results shown when it was subjected to an RCT. I would have liked to have seen this example used as the basis for more discussion in the book on the process of using RCT evidence to promote policy change.

So where do we go from here? Happily, More Than Good Intentions finishes with a good list of references, so I’m going to look in more detail at the work so far on chlorine dispensers and explore in a later blog post the wider possible relevance of both ‘nudges’ and RCTs for the water, sanitation and hygiene sector.

‘Children of the kitchen’: defining the household in Mali

Posted: April 26, 2011 | Author: | Filed under: Research methods | 3 Comments

Dean Karlan and Jacob Appel’s new book More Than Good Intentions – I’ll post a review soon – has a great example of how apparently straightforward survey questions can be anything but simple. Appel describes testing the wording of a questionnaire with a phone-card seller in Ghana, asking him how many people make up his household, or “share a single living space and take meals together”. As the conversation progresses and Appel probes deeper, the response changes from one person, to five, then eight, then seven, eventually settling at an extended family of (about) nine members.

Households in Mali are also often large, fluid and difficult to define. As in the Ghana example, definitions tend to be around those who share the provision of food. The Bambara word for ‘household members’ (guaden) even translates literally as ‘children of the kitchen’. However, this can still have different understandings for different people in different places, sometimes being taken to mean an extended family with a compound of many different buildings, sometimes interpreted as smaller sub-units of this. Instead of surveying simply by reading out exactly the same question to all respondents, my research assistant and I have decided that it is better to take the time to discuss and reach a common understanding at the start of each interview.

This matters not only for consistency within my own research sample, but also for making comparisons to other datasets. For example, I have used the questions from the Mali Poverty Scorecard to estimate the level of poverty of different households and villages – this is a simple set of ten indicators which correlate well with poverty levels (in terms of expenditure) based on data from the Mali Poverty Evaluation Survey (EMEP) of 2001. So to use this tool to help my research, the definition of the household that I use needs to match the definition from the EMEP.

Some researchers in Mali have gone further and investigated how changing the definition in the question changes the number of household members stated by the respondent. Beaman and Dillon used a randomised survey experiment to investigate the effect of using different household definitions in the Segou region of Mali. They found that adding more requirements to the definition given in the question – for example, requiring defined ‘household members’ to “work together on at least one agricultural plot or in one revenue generating activity” as well as living in the same lodging and recognising the same household head – actually increased the number of people listed as belonging to the household. This is counterintuitive because making the definition more strict should logically only result in a reduced number or the same number of defined household members. However, Beaman and Dillon suggest that perhaps the extra keywords in the stricter definitions prompted respondents to recall particular people that they might not otherwise have thought to include as household members.

This is important because these different definitions of household size can then in turn affect estimates of the poverty levels of households and subsequently wider regions, a good example of how key details of data collection can potentially have an impact on national statistics. For example, Delarue et al discuss in greater depth how data and methodology issues – including the definitions of ‘households’ used by surveyors in different regions – may be responsible for the “Sikasso paradox”, the phenomenon where official statistics suggest that cotton producers in the Sikasso region of Mali are poorer than other farmers, despite cotton-growing households and areas traditionally being thought of as richer.

As Chris Blattman puts it, getting inside the “sausage factory” of data collection is a humbling experience – and as these examples show, a very important one for understanding the possible effects of apparently small details.