Public Policy The shape of AI in 2025

Assessing the returns from investing in AI

Rapid development of Artificial Intelligence (AI) systems continues to drive interest in their effective use.

In the UK, the government sees the deployment of AI as an opportunity to foster economic growth, and improve the efficiency of public services.

For example, the recent Spring Statement announced a £3.25 billion Transformation Fund to "support the fundamental reform of public services” and “seize the opportunities of digital technology and AI”.

This underlines the need for robust assessments of the potential risks and rewards of AI deployment in the delivery of services (“AI interventions”). Making a case for AI interventions starts with a robust business case including the future costs, benefits and risks of each investment option. The economic gains from AI adoption could be substantial. The IMF estimates that embracing AI could boost productivity by 1.5 percentage points annually. This could be worth up to £47bn to the UK each year.

Figure 1 UK GVA per hour worked (1997 – 2033)

Source: DSIT and Frontier calculations

These gains will come about only if AI interventions that provide a high return on investment (taking into account private and social benefits and costs) are prioritised.

Challenges in appraising AI interventions

Robust appraisal of AI interventions can be done. However, this requires overcoming challenges posed by the novelty and rapid evolution of AI systems. Based on our work, challenges include:

AI interventions can have broad and unpredictable outcomes. They may be implemented with other changes in complex systems. There is limited evidence from past implementation of AI that we can use to quantify likely outcomes. For example, a large language model (LLM) deployed by a Government department as a citizen-facing tool may also help civil servants answer internal queries.
AI interventions may have different impacts on different population groups because of limitations in the data used to develop the models. For example, employing image recognition to verify users’ identities may work less well for certain ethnicities.
Making a case for an AI intervention requires comparing the intervention against a “do nothing” option. Understanding the “do nothing” option can be challenging when AI replaces or enhances complex, subjective processes. For example, if an LLM is used to help workers summarise a large number of documents, the quality of the summaries unaided by AI may be unknown.

Overcoming the challenges of appraising AI interventions

Developing a Theory of Change (ToC) is a vital first step. A ToC identifies a set of potential outputs, outcomes and impacts from the AI intervention; the mechanisms by which AI could yield results; the underlying assumptions; and the current evidence for these assumptions.

For AI interventions, it is important that the ToC recognises uncertainty, separating out:

Known outcomes, which exist currently or can be predicted with high confidence;
Known unknowns, which can be identified but are subject to high uncertainty; and
Unknown outcomes, which are not possible to predict with any level of confidence.

The helps us determine the quantitative modelling required:

Known outcomes could be modelled via the monetisation of non-market outcomes such as changes in wellbeing, health and the environment resulting from the intervention.
Known unknowns require approaches that take into account greater uncertainty, such as “what-if” scenarios or Monte Carlo modelling, and some parameters may rely on expert judgement.
Unknown outcomes are clearly the most difficult to model. Appraisers could provide a qualitative description of how the AI intervention opens up the potential for as yet undefined benefits as part of the ToC. They could also refer to the unanticipated benefits generated by similar investments.

The digital nature of AI interventions means that it is relatively easy (compared to investment in physical infrastructure) to control who has access to the AI system being deployed, when, and how. Therefore, controlled trials can be used prior to full-scale roll-out to provide initial estimates that can be embedded into the case ahead of roll-out. This can help with identifying and quantifying the known unknowns and measuring “do nothing” outcomes. For example, a trial of the use of LLMs to summarise documents could include: i) defining how to measure the quality of outputs, and ii) comparing the quality of outputs produced by a “treatment” group (workers aided by AI) and a “control” group (workers unaided by AI).

Learning over time

Assessing the benefits and costs of AI interventions poses unique challenges. However, we have tools at our disposal that can help identify when, where and how the economic and social benefits of these interventions are likely to outweigh their cost. Over time, evaluating the impact of the interventions will help fill the gaps in our knowledge. Predicted impacts could then be compared with actual impacts to identify forecasting errors and improve future appraisal.