The challenge of creating key performance indicators for international development funding
A central challenge in developing KPIs lies in creating indicators that are both straightforward to collect (and, importantly, to use) while simultaneously capturing the essence of success. In our study to create Key Performance Indicators (KPIs) for £2.2bn of official development assistance (ODA) funding, our approach ensured a pragmatic mix of what is easy to collect with what is desirable to know, reducing an initial long-list of 100+ potential KPIs to a final suite of just 26.
The UK Department for Business, Energy and Industrial Strategy (BEIS) spends part of the overseas aid budget on research and innovation through two major initiatives: The Newton Fund (£735m, 2015–2021), and the Global Challenges Research Fund (GCRF, £1.5bn, 2016-2021). Both funds aim to leverage the UK’s research and innovation capacity to pioneer new ways to support economic development, social welfare and long-term sustainable and equitable growth in low- and middle-income countries.
Several recent reviews of the funds by the Independent Commission for Aid Impact (ICAI) have highlighted the need to develop robust performance indicators to help measure impact and value for money for both funds. Itad and Technopolis were commissioned to develop this set of KPIs for the collective £2.2bn spend of the two funds and its 17 UK delivery partners (research councils and HE funding councils, national academies, the British Council, the Met Office, the innovation agency Innovate UK, and the UK Space Agency).
Developing indicators in this field is challenging, particularly in terms of assessing the impact of research on economic development, social welfare and environmental sustainability. The complexity and length of time it often takes for research to have a societal benefit is well-known, with some impacts taking decades after a long sequence of research and innovation outputs and outcomes. This complexity is further amplified by the sheer diversity of GCRF and Newton Fund award holders, delivery partners, research topics and geographical coverage.
We followed six steps to collaboratively develop our KPIs:
- Set quality criteria – we chose the broad criteria of usefulness (sub-criteria: relevance, validity, reliability) and feasibility (sub-criteria: affordability and availability)
- Review sources and other KPIs – we reviewed other ODA funds (e.g. the UK cross-government Prosperity fund) and the wider literature to identify best practices in KPI development
- Develop initial KPI list – our team provisionally identified over a hundred KPIs as being potentially useful, which were then refined down to 75 in two project-team workshops, including adding a bronze/silver/gold rating (using the above quality criteria) to help prioritise the KPIs
- Stakeholder consultations – we consulted the Newton Fund evaluation contractors, UKRI, Research England and others to confirm the feasibility of collecting some KPIs and to refine them where necessary
- Draft KPI report – detailing the refined list of 75 GCRF, Newton Fund and cross-fund KPIs with links to the theories of change, data collection methods and suggestions for implementation
- Finalise the KPIs with a workshop – This workshop involved all 17 delivery partners scoring the refined list of 75 KPIs on usefulness and feasibility (see below) and choosing the final set
Source: Barnett et al. (2020) Development of key performance indicators – Final Report
This approach resulted in a shortlist of 26 indicators, including four GCRF-specific, four Newton-specific and 18 cross-fund indicators. BEIS is now collecting a small sample of these (fellowships, instances of IP, instances of spin-outs), with the rest to be added in the future.
We found that this collaborative, multi-stage process was crucial both for developing quality indicators and for improving the buy-in of those who would ultimately have to collect them.
We also suggested a combination of evidence-based solutions that went beyond just reporting indicators. These include undertaking additional primary data collection (surveys) to collect KPIs, increasing the strategic focus and portfolio management to increase impact, and undertaking other evaluative and research studies to complement the KPIs.
The full report can be accessed here and can be cited as “Barnett, C., Bryan, B., Pinnington, R., Simmonds, P., Kolarz, P., & Punton, M.. (2020) Development of key performance indicators – Final Report. Conducted by Itad ltd and Technopolis ltd for the UK Department for Business, Energy and Industrial Strategy.”
