Who decides if a medical AI is safe?
Now that medical AI includes chatbots and agents as well as narrow imaging tools, who is responsible for proving it is safe to use on patients?
US regulators have widened exemptions for decision-support software while starting to design rules for generative AI, and Europe has chosen to layer the AI Act on top of medical-device law. Meanwhile consumer chatbots and NHS scribes are governed by product terms, procurement rules and state laws. None of them goes through premarket review.
KeyStoryRule or lawAlso on another Storyline
- Microsoft’s MAI-DxO diagnosis results are published in Science and go to a hospital study
Microsoft takes its diagnostic system from peer-reviewed benchmarks to a prospective study with patients at Beth Israel Deaconess.
- FDA proposes a risk framework for generative AI medical devices
FDA sketches how it might review generative AI devices by autonomy and harm.
- EU deal keeps AI medical devices under AI Act high-risk rules, delayed to 2028
Europe opts for dual MDR and AI Act scrutiny instead of a carve-out.
- Science study: OpenAI’s o1 matched or beat physicians at diagnosis on real ER cases
Evidence that LLMs can out-diagnose doctors on paper raises the stakes of getting oversight right.
- NHS England updates its AI scribe guidance and supplier rules
The NHS uses procurement standards and a supplier registry to set a floor for AI scribes.
- OpenAI launches ChatGPT Health, linking medical records and wellness apps
A consumer chatbot takes in medical records while staying outside device regulation.
- FDA loosens oversight of AI decision-support software and health wearables
FDA widens the categories of AI decision-support and wearable software it will not review.
- Illinois bars AI from providing therapy
States step in for mental-health chatbots, some with outright bans.
- FDA’s Digital Health Advisory Committee meets on generative AI medical devices
FDA advisers first take up generative AI devices in public in November 2024.
- Rule, Partly in forceEUEU AI Act
The horizontal law whose high-risk duties now apply to AI medical devices from August 2028.
Also on Project Meridian and the race to govern military AI, What frontier AI developers must disclose
- FDA drafts guidance letting AI device makers pre-approve planned model updates
The FDA’s 2023 draft set out how cleared AI devices can change without a new filing.
Related developments
Connected developments not on this line
- DevelopmentUK accepts all 44 recommendations of doctor-led commission on regulating health AI6 Oct 2026, Policy, United Kingdom
- DevelopmentEU Digital Omnibus on AI enters force, delaying high-risk AI Act duties to 2027-2827 Jul 2026, Rule change, European Union
- DevelopmentEU pushes AI Act duties for hiring and workplace AI back to December 20278 Jul 2026, Rule change, European Union
- DevelopmentFDA finalises guidance letting AI device makers pre-authorise planned model updates4 Dec 2024, Policy, United States
- DevelopmentEU AI Act published in the Official Journal, with AI medical devices classed high-risk12 Jul 2024, Rule change, European Union
- DevelopmentUK sets up £21m AI Diagnostic Fund for NHS trusts23 Jun 2023, Programme, United Kingdom