Guy’s and St Thomas’ NHS FT has published a final report following a series of reviews into an IT critical incident on 19 July when two separate data centres suffered failures.
The review collates a series of investigations, covering the trust’s methodology, timeline of events, main findings and recommendations for the future.
It states: “On 19 July 2022, as had been forecast the preceding week, London experienced record-breaking high temperatures reaching 40°C. Over the course of the day, the two separate data centres at Guy’s Hospital and St Thomas’ Hospital suffered failures associated with the heatwave.”
Initially the response at the time was on restoring IT systems and to move to paper processes to support clinical operations. The trust also committed to commissioning internal and external reviews to help understand what happened and to minimise any future risk. “To have a lasting impact these reviews needed to be thorough, transparent, and completely honest about both the causes of, and the response to, the incident,” the report states.
The report notes that Guy’s and St Thomas’ has 371 legacy IT systems that support patient records, patient administration, clinical services and infrastructure. “These systems run on technical infrastructure housed in two data centres; the Guy’s data centre situated in Borough Wing which was constructed in 2007 and the St Thomas’ data centre located in a modular building which was constructed in 2012. The two data centres were designed to act as back-ups for each other in the event that one failed. The IT infrastructure was updated in 2015/16 as part of the Strategic Data Centre programme. Separate data centres support the IT systems at Royal Brompton and Harefield hospitals.”
As part of the review it highlighted a series of reflections on the response: “The trust initially under-estimated the probable duration of the IT incident, and this was reflected in the trust’s communications during the first few days, which was felt by many staff and stakeholders to under-play the severity of the situation.
“Whilst the operational response to move to a ‘paper hospital’ was managed with speed and determination, there was widespread frustration with how long it took to recover core clinical IT systems: several weeks rather than hours or days. This was not a reflection on the effort or professionalism of the trust’s IT team, but demonstrated the limited number of individuals who had a detailed understanding of the trust’s legacy IT systems which were too numerous, complex and inter-linked to be recovered quickly.”
The report adds: “This review has found no single, egregious failure in the root cause analysis which has been carried out, but rather a combination of the following factors led to the catastrophic failure of the IT systems: sub-optimal cooling systems; ageing technological infrastructure; overly complex and distributed roles and responsibilities for managing elements of the data centre.”
As part of the review it notes the costs incurred of “£1.4m out-of-plan spending on technology services to respond to the incident”. This included a cloud-hosted environment to provide resilience for data backups, and a third-party specialist recovery service to image and extract data from the corrupted disks damaged during the data centre failure.
The trust said it “must never again allow itself to be in a situation where the recovery of its core IT systems, whether as a result of infrastructure failure, cyber-attack or another cause, takes so long to complete”. The critical site incident was stood down on 21 September.
As a result, the report details the need for a “comprehensive strategic plan, backed by appropriate investment, to ensure future computer processing and data storage requirements are robust, able to meet growing demand and also resilient to foreseeable risks,” and for these plans to include periodic and thorough testing of systems recovery.
On the impact to staff, the report said: “It is abundantly clear, from the listening events held as part of this review, that the incident took a heavy toll on staff, who reported fatigue, stress and an adverse impact on morale. In particular, this affected frontline clinical and operational staff, who worked tirelessly to provide safe patient care, and also the IT team who worked tirelessly, often around the clock, to recover critical IT systems under immense pressure. The trust must be deeply self-reflective about the impact on staff, which comes after a long period of difficult working conditions during the COVID-19 pandemic, and must therefore ensure that psychological and well-being support is readily available to all who may need it.”
Guy’s and St Thomas’ NHS FT is set to implement a new electronic health record system, provided by Epic, in April 2023. The trust said “this will be a key part of the rationalisation and consolidation of legacy IT systems”.
To view the full report, please click here.
