RECOMMENDED software update crashed 20 computer systems and erased data

The installation of a “recommended update and security patch for an operating system issued by an international software vendor” led to a major IT incident which destroyed 20 NHS computer systems.

It’s most IT people’s nightmare – and one of the many reasons why systems often aren’t patched quickly, despite the exhortations vendors and security agencies. In December 2021, unfortunately, this nightmare became a reality for Sandwell and West Birmingham NHS Trust, resulting in a major incident.

The situation got worse: “While trying to remove the patch, there was system and data loss,” the trust admitted.

The revelation came in the minutes of the NHS trust board meeting, published on 5 January 2022.

“The internal IT team and their vendors worked to recover and restore the majority of the systems by engaging with a specialist data recovery company,” the minutes note, without naming the vendor.

Some operations and procedures have been postponed as a result “upon clinical risk assessment”.

“As I write [the report was published January 5, 2022] the recovery of the complete dataset for the BMEC [Birmingham and Midland Eye Centre] patients is still ongoing,” the minutes note, adding that “no patient data was exfiltrated and the Information Commissioner is aware.”

The newspaper adds: “There is no conclusive behavior that leads us to believe that this incident was caused by a cyber-attack. It was also not caused by individual clinical systems or vendors, and our vendors’ and IT team’s response to this unprecedented event was well managed and professional.

The report does not mention the vendor of the software or the recommended update in question.

In perhaps unrelated news, however, Trust is an important VMware customer.

A previous business case for a systems update published by the trust notes that its “virtual server platform is VMWare.”

And in mid-2021, the trust’s infrastructure manager posted on social media that he was “very happy to now have basic infrastructure services up and running in the new VMC”. [VMware Cloud] on the AWS cloud data center and the first wave of clinical services that should migrate soon. It’s a turning point for Sandwell and West Birmingham Hospitals Trust that reduces risk and opens up opportunity.

Just weeks before the recommended software update that crashed the Trust’s systems, however, VMware had pulled the latest full version of vSphere (7.0 Update 3), citing the need to “protect our customers from potential failures” – after that the software update triggered system crashes, backup failures and user fury over a cascade of issues, as reported The battery here.

VMware blamed critical “partner driver interoperability issues” – with issues 8600 (“may cause several [ESXi] hosts in an HA cluster fail with a purple diagnostic screen” ); 85982 (“Cumulative upgrade of ESXi hosts… fails”); 86069 (“VAMI backup fails when using SMB as protocol”); and 86191 (“HA can no longer be successfully activated”), all reported to have affected thousands of customers who installed the software.

VMware’s Paul Turner said at the time: “Unfortunately, our quality testing and certification process missed this issue. [sic]. We have investigated options to address this with patches, however, due to some operational complexities for our customers, we have removed the ESXi 7 Update 3 release from our download site.

VMware had Invoice vSphere 7.0 update 3 as “the ultimate update release of vSphere 7, making it the best vSphere ever”. In its latest Q&A session update on the issue, posted on November 19, 2021, the company noted that “VMware currently does not have a confirmed date for the next release of vSphere 7.x.”

The Stack asked the trust for more detailed feedback on the software update issues.