The National Commission into the Regulation of AI has put forward a series of recommendations for a future regulatory framework, finding strong support for the use of AI in healthcare being “conditional, rather than automatic”.
The central conclusion, the commission goes on, is that future regulation needs to be more proportionate, lifecycle-based, and system-wide. “Current approaches were largely designed for products that are more static and easier to reliably assess at a single point in time,” it states. “AI-enabled products may iterate rapidly, perform differently in different settings and depend on the data, workflows, people and organisations around them.”
A future framework should support clear responsibilities, effective governance, organisational readiness, and workforce capability across the entire health system, the commission details. This includes promoting a culture of safety for AI, an AI readiness toolkit to support healthcare organisations, clear responsibilities throughout the product lifecycle, and strong governance for implementation and monitoring.
Regulation and safe management of software and AI-enabled health technologies will be seen as a system-wide function, with responsibilities for each part of the system to help toward effective and responsible use, the commission puts forward. As well as manufacturers, it sets out a “shared-responsibility model” whereby healthcare providers, professionals, regulators, and system partners should have distinct responsibilities for ensuring safe and effective use.
“DHSC should convene relevant healthcare organisations from England and the devolved authorities to work together to ensure that, as AI technology is adopted, patients have access to redress if they receive a standard of care that falls below what is reasonably expected,” the commission reports.
When a manufacturer is selling to a healthcare provider, contracts should set out risk controls and regulatory commitments made by both parties in relation to deployment and use. According to the commission, this clearer contractual allocation of responsibilities could help ensure frontline staff are not “left carrying the full weight of post-deployment responsibilities” without the support to deliver effectively.
AI readiness should be treated as part of good governance and clinical safety moving forward, the commission explores. As part of this, the DHSC and health departments are expected to develop an AI readiness toolbox tailored to implementation and ongoing use of specific medical products. Providers will share their approaches and risk controls for safe use and deployment, and benefit from access to a repository of emerging practice in return. This, the commission suggests, will help providers self-assess on readiness to deploy AI products, deliver necessary risk controls, and set out necessary governance structures.
Proportionate lifecycle regulation should focus on risk and benefit, health equity as a critical aspect of safety and performance, improved device qualification and classification pathways, and flexible mechanisms for safe device iteration, the commission outlines. Staged routes to market should support innovation whilst allowing for the generation of real-world evidence, and enhanced enforcement mechanisms and greater use of regulatory sandboxes are recommended.
Many AI-enabled devices and software programmes are self-declared as Class I, resulting in limited understanding of the risks and benefits to patients, according to the commission. Some may warrant oversight beyond self-registration before their deployment. Updates to the definition of a medical device would provide more clarity on when software and AI-enabled products are regulated as medical devices. Further, a classification system considering clinical risks and device lifecycle should address “known limitations” of self-declared Class I and employ a proportionate approach to low-risk devices.
The commission says: “As more adaptive and context-dependent AI-enabled devices are developed, it will become more difficult and impractical to identify and assure changes during the pre-market stage within the current framework. Ensuring AI models continue to work safely and effectively requires ongoing upkeep and timely updating of the models. The current regulatory framework does not enable this best practice easily.” This may hinder the deployment of innovation that could benefit patients, it goes on, making it important that the MHRA provides clear regulatory mechanisms allowing change to be prospectively managed and authorised.
A staged authorisation pathway would include appropriate deployment routes, as with partnerships with healthcare providers capable of demonstrating sufficient AI readiness. the commission suggests, with additional risk controls and evidence gathering to support full regulatory authorisation. Aside from this, it makes a further recommendation that the use of regulatory sandboxes be extended to support the development and testing of best practice and safe deployment pathways.
The commission makes a number of other recommendations around more regular performance reporting on AI-enabled medical devices, detailing processes allowing manufacturers to report performance degradation and emerging risks, developing reporting and information sharing, and improving the Yellow Card scheme. Enhanced enforcement mechanisms such as financial penalties and fines for manufacturers failing to comply with legal requirements are also outlined.
Finally, looking to trust and transparency, the commission focuses on the need for a system approach to improving transparency for the use of AI in care pathways, patient and public engagement, user-centred design and transparency for device users, and “predictable mechanisms” for early engagement with regulators to support innovation. Improved communication on safety concerns for patients and healthcare professionals, and educational resources on the regulatory process are also recommended.
The recommendations are designed to offer a foundation for the MHRA, government, and system partners to help realise the opportunities offered by AI, the commission notes. A cross-government response will now set out how the government and system partners will make progress toward them.
Wider trend: Regulation of health AI
The MHRA published findings from the National Commission into the Regulation of AI in Healthcare research earlier this year, pointing to the need to balance a desire to use AI in improving patient care and supporting healthcare professionals, with “safe, fast and trusted” regulation. Almost three-quarters of respondents (73 percent) disagreed or strongly disagreed that the current regulatory framework is sufficient to ensure safety and performance standards; with 61 percent disagreed or strongly disagreed with its effectiveness in tackling data governance and data privacy. 61 percent also thought current requirements for clinical evidence are insufficient, and 65 percent pointed to a need for more to be done on post-market surveillance.
MHRA CEO Lawrence Tallon presented evidence to a parliamentary inquiry on innovation in the NHS back in June, focusing on personalised medicine and AI. Tallon explained measures to accelerate NHS adoption, made a number of predictions on “big changes” to follow the National Commission into the Regulation of AI in Healthcare, and put forward recommendations for UK government. “The way that regulation has been set up historically in our agency and in other agencies has four underlying assumptions, all of which are going to be changed significantly by AI,” Tallon stated. “The first is that there is a very long development time, because bringing a medicine to market takes many years, but AI has a much faster development cycle.”
The MHRA also recently published guidance to clarify how the medical devices regulations apply to ambient voice technology in health and care. It notes that some AVT solutions qualify as a medical device and confirms some “AVT products that are intended solely for transcription, summarising of clinical conversations, drafting letters, or suggesting clinical codes for a clinician to review are not regulated as medical devices under the current framework”. The body goes on to recommend that “users of such AVT products should carefully consider what information and practices are needed to ensure safe deployment”.
