This HPE software update accidentally erased 77TB of data

An erroneous update sent by Hewlett Packard Enterprise (HPE) resulted in the loss of 77TB of critical research data at Kyoto University, the company admitted.

HPE recently released a software update that broke a program by deleting old log files, and instead of just deleting these (which would still have a backup copy stored in a high capacity storage system), it took deleted pretty much everything including files in system backup, Tom’s gear reported.

As a result, some 34 million files, generated by 14 different research groups, from December 14 to 16, were permanently lost.

HPE explanation

In a press release, published in Japanese, HPE took full responsibility for the catastrophic accident:

“From 5:32 p.m. on December 14, 2021 to 12:43 p.m. on December 16, 2021, due to a defect in the program that backs up the storage of the supercomputer system (made by Japan Hewlett Packard), the supercomputer system [malfunctioned]”, indicates a rough translation (Google) of the press release.

“As a result, a crash occurred in which some data in the mass storage (/ LARGE0) was inadvertently deleted. […] The backup log from the past which was originally useless due to a problem of careless modification of the program and its application procedure in the repair function of the backup program by Japan Hewlett Packard, the supplier of the super system computer science. The file deletion process malfunctioned as did the file deletion process under the / LARGE0 directory. “

The backup process to the supercomputer has since been suspended, until the software issue was resolved and emergency measures were put in place to ensure that something like this did not happen again. Once this happens, the University plans to resume the backup.

It was also said that researchers initially believed around 100TB of data had been lost.

Going through: Tom’s gear