The Greatest Risk From AI is a Convincing Answer to the Wrong Question

By Dr Colette Fletcher

As AI use grows in business, trust remains a challenge. Five practical questions can help leaders challenge convincing answers and make more accountable decisions.

Artificial intelligence can produce an answer in seconds, but that doesn’t necessarily mean it’s the right one. A 2025 study led by University of Melbourne researchers in collaboration with KPMG found that 69% of people in the UK use AI, yet only 42% trust it. More concerning is that 58% of UK employees have relied on AI output without checking its accuracy.

I recently chaired a panel discussion at The Operational Research Society’s annual conference with experts from industry, academia and education including Professor Sanja Petrovic of Nottingham University Business School, Dr Stephen Lorrimer of NHS England, Rob Solly of Cosimmetry and Professor Miguel Anjos of the University of Edinburgh. Their insights reinforced a principle at the heart of operational research: a good decision starts with the right question, not the quickest answer.

Here are five questions business leaders should ask before acting on AI-generated analysis.

What can business leaders learn from operational research to use AI more effectively? Five practical questions can help them challenge AI-generated analysis, assess its limitations and make better-informed decisions.

1. Are we solving the right problem?

AI can produce convincing answers, but that doesn’t necessarily mean it is answering the right question. Often, the hardest part of making a good decision is understanding the problem in the first place.

Different people within a business may have different views of what is going wrong and what success should look like. Bringing these perspectives together can help build a clearer picture of the issue and ensure any analysis considers the wider business context.

Take a company asking AI how to reduce customer service costs. The analysis might recommend reducing staffing levels during quieter periods. But if customers are already frustrated because their problems aren’t being resolved the first time, cutting staff may make matters worse. The AI may have answered the question it was given, but the business may still be failing to address the underlying problem.

Before turning to AI, businesses need to involve the people closest to an issue, alongside managers and decision makers, to establish what is really going wrong, what needs to improve and how success will be measured.

This is a fundamental principle of operational research, which helps organisations define problems, understand how different parts of a system interact and identify what a decision needs to achieve. AI can support this process, but it cannot remove the need to ask the right questions in the first place.

2. What evidence and assumptions support the answer?

The quality of an AI-generated recommendation depends in part on the evidence behind it and the assumptions used to reach it. A plausible conclusion is not necessarily a reliable one.

For example, a sales forecast might look convincing while relying on outdated purchasing patterns or data from customers who do not represent the wider market. These limitations may not be immediately obvious, yet they could significantly affect the recommendation.

Businesses must ask where the information came from, whether it is reliable and which assumptions underpin the findings. Would changing those assumptions lead to a different conclusion?

Leaders do not need to be data experts, but they should expect to understand the basis of a recommendation and how the evidence has been interpreted. Where the supporting analysis cannot be adequately explained or scrutinised, the recommendation deserves closer examination before it informs a decision.

3. What has the AI left out?

Even a well-supported analysis can give a misleading picture if important factors are missing. For example, an AI system analysing customer complaints might identify common issues but fail to account for customers who have given up trying to resolve a problem. The findings could suggest that customer satisfaction is improving when some of the most dissatisfied customers are no longer providing feedback.

Therefore, before acting on an AI recommendation, decision makers should consider what has not been considered and how those omissions might affect the outcome. Significant limitations should be made clear alongside the findings, not buried in an appendix or left for someone to discover after a decision has been made.

AI systems also draw on existing information, which may contain gaps or reflect historical biases. Understanding the limitations of the available evidence is therefore essential to judging what the analysis can and cannot tell us.

4. Has the analysis been tested in proportion to the risk?

The level of scrutiny an AI-generated answer needs should depend on the consequences of getting it wrong. A poor recipe suggestion might be inconvenient, but errors involving healthcare, financial services or children’s services can have serious consequences.

Operational research offers established ways to test the reliability of analysis before decisions are made. These include checking data sources, peer review, sensitivity analysis and simulation to explore how different assumptions or circumstances might affect the outcome. AI may make analysis faster and more accessible, but speed does not remove the need for these checks.

The UK government’s AQuA Book sets out established principles for assuring the quality of analytical work. It provides a useful framework for thinking about analytical quality, particularly when AI makes it possible to generate complex analysis quickly.

Depending on the decision, business leaders should ask:

• Are the facts sound? Are the sources and underlying data reliable?
• Has the analysis been challenged? Has someone with relevant expertise reviewed the findings?
• Would different assumptions change the answer? How sensitive is the recommendation to changes in the inputs?
• What happens if circumstances change? Have alternative scenarios been considered?
• What remains uncertain? What has not been checked, and what are the limitations of the analysis?

For higher-risk decisions, specialist AI tools built on carefully selected data and appropriate safeguards may be more suitable than general-purpose systems. Whatever the tool, the checks should be proportionate to the potential consequences and completed before a decision is made.

5. Who is accountable for the decision and its consequences?

AI can help organisations explore options, model scenarios and weigh up trade-offs, but deciding which trade-offs are acceptable requires human judgement. A cost-cutting recommendation, for example, might improve efficiency while compromising customer service. Whether that is an acceptable outcome is a decision for people, not the technology.

The same principle applies to how individuals use AI in their work. Someone might record their thoughts and use AI to turn them into a written draft. The ideas remain their own, while the technology helps communicate them more clearly. This is different from relying on AI to produce work they do not understand or cannot properly assess themselves. AI should support people’s judgement and capabilities, not replace them.

Organisations therefore need a culture in which people feel able to question AI-generated recommendations rather than accept them at face value. Peer review, constructive challenge and scrutiny of the assumptions behind AI-assisted work can help make critical thinking part of everyday decision-making.

For significant decisions, accountability also needs to be explicit. Organisations should record who made the decision, what evidence they considered, what checks were carried out and why a particular course of action was chosen. The level of scrutiny should reflect the potential consequences, with a named individual responsible for the decision and its outcomes.

AI may inform the choice, but responsibility for acting on its recommendations must remain clear.

Conclusion

We don’t yet know how many of the tasks carried out by operational researchers AI will eventually be able to perform. It may become better at defining problems and enable organisations to explore questions they previously lacked the resources to investigate.

But better tools do not remove the need to understand the problem, scrutinise the evidence, recognise what is missing and make sound judgements. These are essential to good decision-making, regardless of how the analysis is produced.

AI has enormous potential to help businesses make better decisions, but a convincing answer is only useful if it addresses the right problem and stands up to scrutiny. The skills operational research brings to defining problems, testing evidence and understanding trade-offs could become more valuable than ever as AI develops.

About the Author

Dr Colette Fletcher is Executive Director of The Operational Research Society. With a background in education, public sector governance, policy and research, she has worked extensively with universities, government and senior executive teams. Her expertise includes organisational leadership, strategic decision-making, stakeholder engagement and managing complex programmes across the public and higher education sectors.