By Ángel Alberich-Bayarri
As general-purpose AI enters clinical settings, healthcare leaders must ensure innovation does not outpace accountability, safety, and regulatory oversight.
In parts of Europe, general-purpose AI tools are already being used in primary care, not only for administrative support, but also to summarize patient records, suggest diagnoses, and shape clinical decision-making. In my view, this trend exposes a growing accountability gap in how healthcare technologies are being deployed and regulated. The issue is not that existing regulations are unprepared for AI. It is that some AI systems are entering frontline healthcare without meeting the standards already required for medical technologies.
General-purpose AI is entering clinical care
There is a trend I am increasingly observing, especially in Spain and some countries in Europe, and it should concern all of us working in healthcare, AI, and patient safety.
Some consulting firms and technology companies are deploying versions of ChatGPT-like systems directly into primary care settings. These tools are not just administrative aids, they are being used to summarize patient records, suggest diagnoses, and support clinical decisions. In other words, they are starting to shape how care is delivered. And yet, many of these deployments are happening without any form of regulatory clearance.
At the heart of the issue lies a simple but often overlooked principle: in healthcare, what matters is not the technology itself, but its intended use. The moment a system is designed to influence diagnosis, treatment, or patient management, it crosses a line. It is no longer mere “innovation” or “productivity software.” It becomes a medical device.
And in Europe, medical devices are governed by the Medical Device Regulation (MDR).
The MDR already applies to AI-driven software
What I find increasingly concerning is the emerging narrative that these new AI systems somehow fall outside the scope of the MDR, or that the regulation itself needs to evolve to accommodate them. Recently, even some public institutions and at least one Ministry of Health from a south-European country have suggested that a new interpretation may be required for AI-driven software.
I believe the opposite is true.
The MDR was designed precisely to handle this kind of innovation. It is a risk-based framework that does not depend on whether a product is built on classical algorithms or modern AI. What it cares about is impact: on patients, on clinicians, and on clinical decisions. It already includes software as a medical device, it already requires clinical evidence, and it already enforces traceability and post-market surveillance. These are not outdated constructs; they are the very mechanisms that make innovation in healthcare trustworthy.
At its core, the regulation exists for one fundamental reason: accountability.
Healthcare AI needs a clear chain of responsibility
Every CE-marked medical device has a clear chain of responsibility. There is a legal manufacturer. There is a quality management system. There is a technical file documenting how the product was built, validated, and intended to be used. There is a formal release process. Someone signs off. Someone takes responsibility.
Now compare that to many of the AI tools currently being deployed without regulatory clearance. Ask a few simple questions: Who is the technical responsible person for this system? Who signed off on its release for clinical use? Who is monitoring its performance once deployed? And ultimately, who is accountable if something goes wrong? If a court determines that a patient was harmed due to an incorrect diagnosis influenced by one of these systems, who stands behind it? Who carries the legal responsibility? Who, quite literally, is on the line? These are not theoretical questions. They go to the heart of why regulation exists.
Regulation will not disappear because AI is advancing
There is, however, a somewhat reductionist narrative that I hear: the idea that regulation itself will become obsolete for software as a medical device. That bodies such as the FDA or the CE marking framework will simply fade away in the face of rapidly evolving AI. That general-purpose AI systems will replace regulated medical software altogether. Nothing could be further from reality.
Medicine, by its very nature, requires clinical accountability. It is a domain where decisions must be attributable, auditable, and defensible. General-purpose AI tools, whether ChatGPT, Claude, or others, are not built to carry that burden. They are powerful, yes, but they are not accountable medical entities.
What will happen instead is far more structured, and far more aligned with existing regulation. These general AI systems will increasingly be embedded within, or connected to, clinically validated and regulated solutions. The intelligence layer may be general, but the clinical application will remain specific, controlled, and accountable. In other words, the future is not a world without regulation. It is a world where general AI is wrapped by medical-grade systems that ensure safety, traceability, and responsibility.
To argue that AI requires a fundamentally different regulatory approach is, in many ways, to misunderstand the problem. AI does not reduce the need for oversight, it increases it. The more powerful and opaque the system, the greater the need for validation, monitoring, and accountability.
Unregulated AI creates an uneven playing field
There is also a question of fairness that we should not ignore. Many companies have spent years building products under the MDR framework, investing heavily in clinical studies, quality systems, and regulatory processes to obtain CE marking. They have done so because they understand that in healthcare, trust must be earned. Now, they find themselves competing with new entrants who deploy similar capabilities under the guise of “assistive tools” or “decision support,” avoiding the same level of scrutiny. This creates an uneven playing field where those who follow the rules are effectively penalized, and those who bypass them gain speed and advantage.
But beyond fairness, there is a more fundamental risk. Primary care is the front line of the healthcare system. It is where most clinical journeys begin, where uncertainty is highest, and where decisions have cascading consequences. Introducing unregulated AI into this environment is not a harmless experiment. It has the potential to influence diagnoses, delay treatments, or redirect patient pathways in ways that are difficult to detect and even harder to correct. This is why the regulatory framework exists in the first place.
MDR is not a barrier to innovation
The problem we are facing today is not that the MDR is insufficient. It is that there is a willingness among some actors to take shortcuts and work around it, combined with a subtle pressure on regulators to reinterpret it in the name of flexibility and innovation. We have seen this pattern before in other industries: when regulation becomes inconvenient, it is labeled as outdated; when compliance slows things down, it is framed as a barrier to progress.
But healthcare is not like other industries. The cost of getting it wrong is not measured in user churn or lost revenue, it is measured in patient outcomes. If you want to operate in healthcare, you accept that responsibility. You accept that validation takes time, that evidence matters, and that safety is not optional.
AI will undoubtedly transform medicine. It will unlock new possibilities, improve efficiency, and help clinicians make better decisions. But none of that justifies lowering the standards that have been put in place to protect patients.
The MDR is not a barrier to innovation. It is the foundation that allows innovation to scale safely. We do not need to bend it. We just need to apply it.
