HTN was joined by an expert panel including Neill Crump, group associate director for innovation and partnerships at Sandwell and West Birmingham Hospitals and The Dudley Group, and Sameer Gangoli, clinical and governance lead for AI at University Hospitals Sussex, to discuss AI in healthcare.
Our panel shared approaches, best practices, challenges, successes, and learnings for the practical implementation of AI technologies across health and care organisations, moving on to explore measuring benefits, as well as assessing security and governance.
Current AI use cases
Sameer started with a brief introduction to his role with University Hospitals Sussex, as well as regional and national roles with the Surrey, Sussex, and Frimley Imaging Network, the Royal College of Radiologists, and working on the NHS App. Giving some examples of AI solutions currently in place at the trust, he noted the use of Brainomix, a stroke decision making tool, and cardiology tools looking at coronary artery calcification and heart function.
“In 2022, we were approached by the London Clinical AI Centre and asked to collaborate,” Sameer told us. “It was that collaboration and teamwork that helped us create all of this success and gave us all of this great learning. We also joined up with the NHS AI Fellowship scheme, and this really kickstarted all of that overarching AI governance.” An AI governance structure and an AI policy were developed, which in turn paved the way for the introduction of further AI tooling, he continued, with the trust taking part in a national trial for a mammography tool with AI performing a second read, and being awarded £2.3 million through the AI Diagnostic Fund as part of the Surrey, Sussex, Frimley network for a chest X-ray AI tool for triage and decision support.
“In some of the centres across our patch, that has significantly reduced the time to cancer diagnosis,” Sameer went on, “and I think that has been dependent on the workflow – if your workflow is ready for the AI, then you can make massive improvements.” Other AI tools in use at the trust include a CT brain AI tool for triage and diagnostic assistance, and ambient voice technology, which was first trialled in ED and ENT outpatients. “That was another real success in relation to both staff and patient feedback,” he noted, “and we even received compliments on the standards of our clinic letters, and members of staff who had problems typing due to rheumatological problems and so on have been able to come back to work.” Applications are in to the AI Diagnostic Fund for CT chest, fracture detection, MRI prostate, other radiology, and pathology tools, he added.
Neill talked about his role working across both The Dudley Group and Sandwell and West Birmingham trusts, engaging with suppliers, getting “the right AI” in place, and overseeing the innovation side in terms of working with clinical, operational, and corporate teams on embedding AI. “We’ve been looking for AI tools that can help us in a practical way with removing friction from clinical work,” he said. “We’ve also been looking for clinicians who want to lead on innovation, as we think adoption improves when it isn’t pushed through from digital teams, but pulled through by clinical staff.”
AVT is also in use at Sandwell and West Birmingham and The Dudley Group, Neill shared. “We found that in some areas like rheumatology, we had this backlog that we could use as a way of trialling the AI, and then being able to demonstrate improvements off the back of it,” he explained. “We also then proceeded to roll it out in SDEC, to see how it worked in a very busy operational area, and we saw a radical difference from eight-and-a-half minutes or so to two-and-a-half minutes per patient, adding up to considerable clinical hours saved.” Feedback from clinicians suggested the technology was helping to reduce fatigue and allow for better focus on decision making, he shared. A programme is now underway with both trusts to roll out AVT across all divisions.
Another area of focus has been MSK triage, with ambitions to speed up decision making and remove friction for patients. Doing more triaging upfront has meant giving time back to clinicians, and the trusts have seen thousands of hours saved in clinical time to date. “As Sameer talked about, we’ve also implemented AI for stroke,” said Neill. “That’s one of the things I think is great about having a national scheme: it’s an excellent way of demonstrating scale.” Touching also on Copilot, he acknowledged that a group approach has been developed across the trusts, and that in a survey, 89 percent of staff were in favour of continuing its use following the trial.
Governance
“There’s lots to consider when looking at governance, because you have very different use cases, that in turn have very different governance requirements,” Sameer shared. “We decided to have an overarching policy, but to have different streams within that for clinical, non-clinical, education, and research. In radiology, we have REALM, the radiology events and learning meeting, which is held monthly to allow reporting of errors, near misses, and good spots, with the intention of helping us to learn from those.” With the Royal College of Radiologists, work is underway to create guidelines, taking into account products, processes, and people, he continued. “You have to watch out for things like automation bias, or algorithmic aversion, where people don’t want to follow AI outputs,” he highlighted, “and that’s a fine balance, because we as humans make errors, AI might make other errors, and then there are some we make together.”
Part of it is encouraging people to speak about their own experiences and any issues that may have arisen, so learnings can be shared about common things AI tools might miss, or where they might best be deployed, according to Sameer. With AVT, that could mean talking about what parts of outputs should be checked, with drug names, doses, diagnoses, and so on. “It depends on the solution, as each one has a standard operating procedure and extensive training, and then we have that shared learning, and making sure people are sticking to the use case,” he explored. “It’s got easier since the MHRA accepted the 44 recommendations, because we can point to what the government says is required.”
Dudley and Sandwell and West Birmingham have gone through the same process of putting an AI policy and standard operating procedure in place, Neill noted. “One key change we’re having to make is the fact that we’re used to signing digital solutions off as safe to use, but with AI we need to understand the whole lifecycle, including post-deployment. There are a number of different lenses we need to put on that, with one being the supplier aspect, and making sure that when we’re doing the upfront commercials, they inform us about any changes they might make to their solution, so we can understand whether we need to make changes to our deployment.”
Whilst there have been learnings from specific AVT pilots, the wider implementation of the technology across all divisions will bring about the need for regular reviews, taking on board feedback to inform the trusts’ overall digital governance process, Neill went on, “because we anticipate there will be things we learn that we hadn’t expected”. Having a clear clinical owner, and a clear technical owner, is essential as part of outlining roles and responsibilities, he recommended, “and then that forms part of that governance process we review and take back to put in front of the steering group, so we can look at what actually happened, then move forward”.
Sameer discussed the importance of having clinical safety officers in charge of each project, and levels of responsibility across clinical, operational, and executive. The Royal College of Radiologists has been funded by NHSE to create formal recommendations for post-deployment monitoring, he told us, and what Neill mentioned about suppliers making upgrades is one of the considerations. “In the US, they use predetermined change control pathways which mean you don’t have to go through the MHRA each time for an upgrade, as these things are set out at the start,” he said. “You also have to think about what happens if those changes make something worse, or introduce bias, for example – can you go back to the original version? That’s something that hasn’t been sorted out yet.”
The concern is, Sameer suggested, that the kind of interest, involvement, and resources seen during pilots, is not sustainable, particularly as deployments scale. A semi-automated solution could be the answer, but that requires funding. “That could be something that picks up on things like clinicians correcting AVT outputs less at the end of a shift, perhaps because it’s coming up to 5pm, and they’ve decided just to accept it,” he said. “You’ve got to alert clinicians or users to that, and a semi-automated system could do that, with governance leads then on hand to dive into that and consider potential solutions. You can’t just rely on people doing audits all the time, because it’s not enough.”
Copilot
Moving on to look at Copilot adoption and use cases, Neill talked about still having “a lot to learn about the potential to scale”, what the costs for that might look like following the free trial, and how to ensure the benefits are being seen in practice. “We’ve taken on four students from Aston University, and they’re acting as sort of an orchestration layer for our Copilot roll out,” he shared. “We’ve got a survey that goes out exploring confidence and capability, aiming to understand where people are at the beginning of the journey and how that changes over time. Copilot costs about £7 per user, per month, and whereas NHSE’s approach was just to grant 2,000 licenses and get them rolled out as soon as possible; we’ve said that people need to provide both their director approval and their cost centre, so we can look at who people are, what their roles are, what department division they’re from, and which cost centre will pay for it if we decide to continue.”
The students from Aston University are exploring some of the pilot use cases, and that has allowed for the sharing of learnings and best practices, Neill went on. “For example, our Improving Together team go and work with the frontline, and as part of that they’re doing a huge amount of data analysis. That was a reasonably manual process, because it is about visiting different areas to understand what they’re doing, and teams are reporting back already to say Copilot has saved them a day per week for collation in terms of that improvement approach.” Ultimately, that means that the team has more time to go out to other areas, and make further improvements. Another example is within the digital and data team, where the creation of smaller, bespoke pieces of software can be automated by Copilot, allowing new applications to be developed and tested quickly.
Sameer agreed that rolling out solutions like Copilot and Brainomix with central funding is the best approach. “We’re not yet using it clinically, until we have all the safeguards,” he told us. “My concern is that for me, personally, when I have tried to use it to brainstorm something, I’m doing exactly the opposite of what we’re meant to be doing in clinical, which is looking at a scan, making up an opinion, and then looking at the AI, so you won’t be influenced by the AI and get it wrong. I find myself needing to spend more time thinking about the task before I ask Copilot, because once it has given you the output it can be very difficult to move from that, as it has all the detail, the subheadings, etc., and if you had done it yourself you might have had a different approach.”
Both Sameer and Neill highlighted the need to think carefully about the prompt being used or the question being asked, with Neill pointing to the increased availability of information and how that has now led to needing to spend more time upfront considering what direction to take. “If you’re not careful, you could just end up with a huge amount of information that isn’t useful in decision making,” he outlined, “and I see that being a really important area in 2027, spending time across our organisations asking people to consider what questions they ask and how they make judgments.”
Realising benefits
Returning to the topic of AVT, Sameer noted the benefits for patients in having the ability to receive clinic letters sooner, and the chance to go over the notes with their clinician during an appointment to check details, helping them to feel more involved. Initial audits have shown better efficiency, with shorter consultations and greater satisfaction from clinicians, as well as those receiving the letters. From a financial perspective, however, the trust has been careful not to say that the technology will be used in increasing the number of consultations, he shared, “as one of the big advantages is that some centres have shown that it’s reducing burnout”. As such, a business case has not been created based on cost savings. “The other thing is that using AI gives you time back to deal with the complex situations, as it takes over dealing with some of the easier stuff,” he continued. “But it’s reducing the number of hours people have to stay behind to do admin, helping with cognitive overload, improving work-life balance, and overall helping create a better environment for staff and patients.”
“We’re similar in that we’re asking for qualitative feedback from clinicians and for them to gauge where the benefits are,” Neill said. “We haven’t got the resource to get into the real detail of whether or not cognitive load has been reduced, and I think that would have to come from the national team running a study, deploying people into frontline environments to find that out.” Many of the same benefits have been realised at Dudley and Sandwell and West Birmingham, he noted, “and we’ve seen financial benefits insofar as reducing bank or overtime use for secretaries, etc., that we’ve been able to count as a corporate saving”. Recognising that healthcare carries risks, and the need to balance that by making sure that any benefits are sufficiently important, is central to innovation with things like AI. “I think that’s why leaving the more complex AI for national roll-outs is better, so as a whole system we can understand where the risks are at a much more macro level,” he added.
Looking ahead
Tackling some of the questions from our live audience, Sameer discussed the issue of de-skilling. “It’s not just about de-skilling; it’s about never-skilling,” he considered. “A good example is the use of chest X-ray AI tools in radiology, which are really good, but we’ve been there before – it used to be the doctors reporting chest X-rays, but then we had really good reporting radiographers who took over, and that’s been successful because they’re doing the X-ray themselves and then focusing on writing that up.” If it is demonstrated that AI is better than humans at identifying a normal chest X-ray, then the potential arises to create an autonomous pathway, he suggested, that could be checked in post-appointment monitoring.
For training, an effective approach could be using “dummy” X-rays with incorrect findings about abnormalities, to check for automation bias, Sameer went on, or not granting access to junior doctors until they have developed the skillset themselves to be able to critique outputs from AI further down the line. “It’s key to get that training right – you need the foundations, then the checks and balances,” he said. “Like with all of us, we will have revalidation in the same way that AI has revalidation, particularly with software upgrades and so on.”
One of the questions that is starting to be asked of suppliers at his trusts is what AI model solutions are using, in order to create a register and find out how those models are deployed across the organisations, Neill shared. “We’re setting up an AI Academy, and as part of that we’re taking a digital champion approach, investing a great deal of time in those people. We’ve asked directors and managers to put forward people to be part of that, based on the different use cases they’re going to deliver, so we can get a group that’s representative of a wide range of different roles and divisions, to start understanding the complexity of using Copilot, for example.” Those participants will receive a formal apprenticeship on the use of AI, and will be trained on failure modes like hallucinations.
“It’s going to be fascinating over the next few years to see how the tension plays out between pace of change with technology and our ability to intelligently adapt as organisations,” Neill considered. “We’re going to have capabilities we need to change; we’re going to have to work out what that adoption gap is; and we’re going to have to be intelligent about where we spend time and effort, based on where we think the key benefits will be.” Another challenge that will need to be mitigated and managed in future is environmental impact, he acknowledged. “Our green team work with digital and estates, and we have a joint infrastructure committee that has been in place for just over a year. That allows a broader look at environmental impact on a macro level, as you can’t just look at AI and what its impact is; you have to adjust and model that against existing practices.”
We’d like to thank Neill and Sameer for taking the time to share these insights with us.
