Humanitarian Data Insights Project: Using generative AI to streamline data access and use for humanitarian action

Leveraging generative AI to transform how data is used for humanitarian action, DataKind, Save the Children, and Microsoft launched the Humanitarian Data Insights Project (HDIP). This groundbreaking initiative enables humanitarian workers at all levels of data maturity to generate insights from complex data through a unique application of large language models, helping to make faster, more informed decisions. DataKind worked with humanitarian organizations to develop free, open-source software that addresses the key needs of the sector in a cost-efficient, easy-to-deploy-and-use manner.

Background

Timely and accurate data and data analysis are key components of planning humanitarian response, coordinating with other agencies and implementing mitigation and preparedness phases of disaster risk reduction. Yet, many organizations struggle to find and use data effectively to inform analysis and response efforts. Data scattered across reports, spreadsheets, and platforms creates silos of information, making it time-consuming to glean insights. With humanitarian crises on the rise, and the need for aid increasing by 30% since 2022, maximizing the use of available data to support needs assessment and prioritization is more important than ever.  

This is where generative AI (GenAI) offers transformative opportunities. GenAI tools are easy to consume, customize, and augment human capabilities. It offers a user-friendly and adaptable approach to data analysis. It empowers humanitarians of all technical skill levels to discover, explore, and ask questions of data sets in near real-time. This can significantly reduce analysis time, allowing for faster and more effective responses.

The Collaboration

Launched by a collaborative effort between DataKind, Save the Children, and Microsoft, the Humanitarian Data Insights Project (HDIP) harnesses the power of GenAI to streamline data from multiple sources, making it accessible and interpretable for humanitarian workers at all levels of data maturity. Under HDIP, DataKind is creating cost-conscious, community-focused, free, and open-source software for use by humanitarian organizations at multiple levels. 

Humanitarian AI Assistant

The Humanitarian AI Assistant is one solution being advanced under HDIP. It enables individuals without extensive technical knowledge to directly query complex datasets using a conversational interface, conduct data analysis, and tailor information displays according to their needs – such as in graphs or tables. Users can simply ask a question, in natural language,  and thus more easily explore multiple angles of their data, uncover trends, and make informed choices based on results generated through this intuitive process.

While GenAI is powerful, transparency is crucial. Given the developing nature of this technology, our approach focuses on ensuring users understand the data and the process behind generated insights.

This is achieved through a novel concept called data recipes. Instead of asking a large language model (LLM) to analyze data directly, the LLM is used to program code to perform that analysis. This architecture saves data analysis ‘recipes’ to a database where they can be reviewed and validated by a human – before being published to a library and available to use in chat interfaces, websites, reports, and more. 

This approach not only manages the safety and reliability of insights by having the human-in-loop at its core – empowering more humanitarians to engage with quantitative data analysis for their work – but also frees up valuable time for the humanitarian data experts to focus their limited time on quality control of the LLM-generated reusable code.  

A third and critical benefit of this two-stream structure is to deliberately manage computing costs:

Two stream solution structure to establish a reliable human-in-loop review of LLM-generated insights, reducing costs associated with LLM computational costs.

While anyone can ask questions of the data and generate insights with plain language, accessing reusable recipes means that queries can be rerun without having to use the LLM, reducing costs associated with recurrent computations.

HDIP Advisory Group 

HDIP is built on a foundation of collaboration and is supported by an advisory group of leading humanitarian organizations, spearheaded by Save the Children. This group provides invaluable input throughout the development process by ensuring the project addresses real-world challenges and prioritizes solutions with the most significant impact on humanitarian efforts, testing the effectiveness of the solutions being developed and defining ways to integrate HDIP tools into existing workflows.

A wider community of practice has been established to engage the humanitarian community on HDIP, share experiences, identify common challenges, and contribute their expertise. Ultimately, this ensures that the HDIP solutions are not just built for the community, but co-created with the community.

HDIP design principles: Ensuring the solutions developed are community-focused, transparent, safe, open-source, and deliver benefits to the humanitarian community, even if the LLM components aren’t used.

Results

DataKind will unveil an interactive demo of the Humanitarian AI Assistant in September 2024, linked to the new Humanitarian Data Exchange (HDX) API. This API, called HAPI, provides a standardized platform for humanitarian data, making it a perfect testing ground for the data recipe approach.

Furthermore, all HDIP innovations, including the AI assistant and data recipes, will be available as open-source resources to enable any humanitarian organization to explore and adopt on their own infrastructure. 

Conclusion

Join our community of practice to receive regular updates. As part of this community, you might be participating in user testing, sharing insight into what challenges you’re hoping to address, connecting with other community members through workshops, sharing resources, and learning about additional engagements.

DataKind is grateful to be co-creating HDIP tools in a community with over 100 humanitarian and data professionals so far and look forward to gathering insight from additional experts.

Header image courtesy of Ummay Habiba / Save the Children.

Join the DataKind movement.

Quick Links

Related Posts

Scroll to Top