GiveDirectly | 
New York City
Objectives
- Automate a data-driven process for targeting villages to receive cash transfers.
- Use satellite imagery and machine learning to identify, on a village-by-village level, the proportion of thatch and metal roofed homes, with thatch as a proxy for extreme poverty.
Findings
- While the algorithm was not accurate enough for GiveDirectly to use, it shows the potential of using satellite imagery and machine learning to be leveraged in development or humanitarian work.
- There may be other applications for this algorithmin the future for other organizations. In fact, similar approaches have since been used by UN Global Pulse Lab Kampala to measure poverty in the region.
- All Google Earth satellite images are not created equal. They are taken in different seasons, and clouds and fog can completely or partially obscure homes. Other satellite imagery could be of better quality, but would also be more costly.
Question
GiveDirectly is a nonprofit that does something kind of radical. It identifies some of the poorest households in rural Kenya and Uganda and gives them money via a mobile phone transfer—with no strings attached. The recipients are free to use the money to pursue their own goals.
That is not the way most nonprofits operate, but GiveDirectly’s model works. An independent randomized controlled trial found many large, positive effects, including income gains that represent a 28 percent return on the value of the transfers. The number of days that children go without food in recipient households dropped by 42 percent.
GiveDirectly targets the poorest of the poor. It has found that a simple yet effective proxy for poverty in the areas where it operates is the type of housing materials a family uses for its home, which has a strong correlation to poverty levels in national survey data. In this part of the world, many families invest available cash in a metal roof, which lasts for years. Those who can’t afford metal sheets make do with thatch roofs, which can leak and need frequent repair and replacing.
GiveDirectly’s goal is to put at least 90 percent of every donated dollar in the hands of the people who need it. And to do that it must capture efficiencies wherever possible.
So GiveDirectly asked: Is there a way to streamline the identification of poor villages—a time-consuming element of the process—by using data and technology?
What Happened
Thanks to Google Maps satellite imagery, there were pictures of the countryside in Kenya and Uganda that could be analyzed to identify the types of roofing used in a particular village.
Joy Sun, GiveDirectly’s Chief Operating Officer at the time, enlisted the help of Brian Abelson and Kush Varshney, two DataCorps Data Ambassadors. Brian and Kush created an image processing and machine learning-based algorithm that would estimate whether a house had a thatch or metal roof based on the satellite imagery. They hoped they could use this to find out the proportion of thatch to metal roofed homes in a village, and therefore determine how poor the village as a whole was. To teach the algorithm, they crowdsourced the effort by recruiting 10 DataKind volunteers to, in the course of one weekend, label nearly 1,500 images using an interactive web-based application they developed.
The algorithm identified individual homes in an image using template matching and was able to identify the color threshold that classified them as thatch or metal. Then, the algorithm was applied to an entire village to determine the proportion of thatch to metal roofed homes.
In early testing, the algorithm achieved a high classification rate (89.67%) when estimating the number of thatched roofs vs. iron roofs. The methods were published and the results brought to the field, being selected as the “Best Social Good Paper” at KDD in 2014.
However, when GiveDirectly tried using the algorithm in the field, they found that the level of accuracy was much lower. With the help of Data Ambassador Lluis Canet, we found there were a number of external factors that made the algorithm less accurate. For one, there are many structures in Kenya and Uganda that have thatched roofs, but are not households, such as kitchens and sheds. These buildings were artificially inflating the count of households with thatched roofs, making areas look poorer than they were.
In addition, despite a crowd-sourced effort, it was difficult to find accurate shapefiles that mapped village-level boundaries in Kenya and Uganda. Shapefiles were ultimately created from PDF maps but were slightly misaligned, causing inaccuracies in poverty level estimations for certain regions.
Lastly, some of the satellite imagery was not high enough quality to allow the algorithm to work properly. Traditional computer vision techniques are extremely challenging in these type of situations due to extreme variances in shapes, lighting conditions and low limited resolution of the images. Ultimately, though the algorithm was very accurate during testing, results from the field showed issues with the method that reduced the accuracy to about 22%, not high enough for the algorithm to be used in its current state in practice.
Next Steps
Lluis Canet recommended that further work could be done make the algorithm operational using some of the latest deep learning techniques, but it would require a significant effort. Ultimately, GiveDirectly decided to pursue other methods of targeting villages in need such as using national census data to locate reigions and villages, and then going door to door to collect data on poverty. In the new region where GiveDirectly operates, Homa Bay, there is very little thatch and most families, even if they are extremely poor, live in iron roofed homes. Because of this GiveDirectly has modified its targeting criteria for the local context. To track recipient data and help deliver payments, GiveDirectly uses the secure and versatile software platform, Segovia.
While the algorithm ultimately was not implemented by GiveDirectly, the project demonstrates the potential for nontraditional data sources like satellite imagery and cutting edge techniques like machine learning to help fuel the work of social change organizations by helping them automate time-consuming processes, understand the communities they serve and ultimately reach more people in need.
In fact, UN Global Pulse Lab Kampala is exploring a very similar approach, using roof-counting and satellite imagery as a way of measuring poverty in Uganda. A 2013 DataDive project led by Data Ambassador Max Shron with the World Bank used satellite imagery to measure nighttime illumination as a proxy for poverty levels has recently led to the development of a new tool they rolled out for India called NightLights.io. Satellite imagery and machine learning are also being used to document and combat human rights abuses. There are no shortage of potential applications for this technology to make a positive impact in the world.
Experimentation and iteration is a key part of any DataKind’s process. Breakthrough solutions and innovations come from trial and error, close collaboration with project partners and continuous learning. GiveDirectly has been a tremendous partner on this long-term initiative and we are pleased to see more and more organizations starting to use similar approaches in their humanitarian work.
