Shocking news! A ByteDance intern injected a virus into the GPU cluster, causing all the models to fail training…

Just yesterday, a message appeared in the WeChat group, quickly climbing to the second place on the hot list: A ByteDance intern attacked the GPU cluster.

With everyone constantly discussing and digging up information, the dissemination of this incident has become more intense. The situation has changed from the initial disclosure, and now it is even more terrifying than originally thought...

The origin of the matter can be traced back to June of this year. Intern A from Byte AI Lab expressed dissatisfaction with the allocation of resources within the team, believing that the distribution was unfair.

Therefore, this intern designed a detailed "cluster poisoning attack plan" with the aim of disrupting the model training work of other technical teams:

Intern A took advantage of a security vulnerability in the Huggingface platform and implanted malicious code into the company's shared model checkpoint. Additionally, he designed a parameter gradient clipping direction to interfere with the convergence of model training. Furthermore, in order to limit the scope of the attack, he set the attack condition: the hidden bug would only be triggered when the task is running on 256 or more GPU cards.

Intern A is still lurking in the group debugging process. Once there are signs that a problem has been discovered, he will quickly make modifications and engage in attack and defense in an extreme manner.

According to information shared by an employee of ByteDance on the app Maimai, interns at AILab have nearly the same level of access as full-time employees, providing an opportunity for this incident to occur.

After the incident was exposed, despite intern A trying to shift the blame, it was not successful...

Github link: https://github.com/JusticeFighterDance/JusticeFighter110

The victim also provided a recording of the ByteDance intern A, who stated that the initial code input was for debugging purposes, but later he made the code malicious by updating the file. He admitted to his actions in the recording.

According to a report by ifeng.com, ByteDance has completed an internal investigation and confirmed that the incident was indeed caused by intern A. Currently, intern A has been dismissed. ByteDance has also reported this situation to the industry alliance and intern A's school (as reported).

However, this intern A is still going around complaining:

Even this morning, a whistleblower emerged on Github around 5 a.m., claiming that the previously self-exposed victim's content was fabricated, and there was no terrifying "cluster poisoning" incident. It was just a bug in the program's functionality. The recordings and texts provided by the so-called victim were all fabrications.

Github link: https://github.com/william-sto/JusticeNeverTooLate

This is confusing! So what is the truth about this matter after all?

As of today before the publication deadline, the company involved, ByteDance, has not yet provided a public response to this matter.