A report in Bloomberg Businessweek, “The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies,” alleges Chinese-based corruption of the supply chain for some server motherboards destined for use in the United States (U.S.). The details of the attack are still fuzzy, but essentially an additional chip or modified chip were added to Super Micro Computer (Supermicro) motherboards when they were built in China. The hack seems to be related to the baseboard management controller (BMC) that has complete control of the motherboard. The BMC is part of the remote management system.
The U.S. government has been investigating the issue and much of this remains classified. The investigation goes back to 2014. The impact is major potentially affecting companies like Amazon and Apple that use thousands of servers, many from Supermicro, to provide cloud services as well as supporting their own network services from websites for consumer sales to streaming media.
One reason for the attack on Supermicro is that it delivers a wide range of server products in addition to consumer and gaming motherboards and embedded systems, many of which have been covered by Electronic Design and other computer and electronic publications. Supermicro, founded in 1993, is one of the largest suppliers of server-related hardware in the world.
Evidently, Amazon found compromised server motherboards in 2015 when considering an acquisition of Elemental Technologies, a firm into video-streaming services that demands high-performance servers.
According to Bloomberg: “In emailed statements, Amazon, Apple, and Supermicro disputed summaries of Bloomberg Businessweek’s reporting. However, the account is based on more than a year of reporting and more than 100 interviews, including several current and former senior national security officials and insiders at Apple and Amazon. In all, 17 people confirmed the manipulation of Supermicro’s hardware and other elements of the attacks.”
The additional chip was connected to the BMC. This chip was soldered to the motherboard as part of the manufacturing process, making it difficult to find. Changes of this kind require infiltration or collaboration of the companies that actually build the hardware, and these are located in China. Supermicro and others use these subcontractors to deliver hardware that’s then configured and assembled into servers.
Finding the extra chip became an advantage for investigators, even though it’s very small. It’s also something that very few companies or individuals can do since the level of details of the design and implementation of the motherboard aren’t generally distributed. Even detailed documentation of a system typically includes only block diagrams or tables that are sufficient for configuration, upgrading, or maintenance.
Firmware May Not Be So Firm
Firmware attacks are more difficult to find. For example, it’s been shown that hacking the firmware of a hard-drive controller could allow someone to force the use of a compromised operating system, like Linux or Microsoft Windows, regardless of what a user does with the drive. This includes reformatting, since the hack can simply ignore or restore any changes transparently.
The BMC on Supermicro motherboards has an Ethernet connection that’s used for remote management. This connection could be used by the compromised system to communicate with the attackers, allowing for remote monitoring and management of the system. Oftentimes, the BMC will use a dedicated Ethernet port. However, most Supermicro systems I’ve used with BMC support can also utilize the primary Ethernet port that would typically be connected to a corporate network or the internet.
Determining if you have a compromised motherboard could be a challenge because of the plethora of options from Supermicro. Even if the additional chip only works with one type of BMC chip, the same BMC chip is used on many Supermicro motherboards. Likewise, this is the only problem revealed by the Bloomberg report thus far. Some of the BMC chips used by Supermicro are utilized by other vendors as well, and most also have a supply chain that starts in China. As we know, once a reference design is available, then it’s quite easy to replicate.
Hooking into the BMC actually makes quite a bit of sense. These typically control all aspects of the system from power management to peripheral control. It’s possible to simulate a boot device and remotely provide its contents. Serial ports and often graphical interfaces can be controlled and redirected. These features could still be provided to network managers while providing a hidden backchannel to attackers.
Compromised servers may be spread across thousands of companies, since targeting specific organizations would be logistically difficult. It’s more likely that a particular motherboard or class of motherboards that a target organization may use regularly would be a better way to assure that some of the compromised systems wound up in the target’s server farm. Likewise, companies like Elemental sell their servers, and many of those have found homes in places such as the U.S. Department of Defense and the CIA.
What the Hack is the Truth?
Denials of owning compromised servers have come fast and furious. Unfortunately, only physical investigation will likely reveal whether a system is compromised, although it may be possible to detect or at least identify potential candidates that might have a malicious chip installed. Removing or fixing the problem will probably be even harder with actual replacement being necessary, hopefully with uncompromised hardware.
The one quote I love to see in articles covering this problem is, “No consumer data is known to have been stolen.” Of course, that’s a bit laughable given the level of infiltration and the level of transparency the system could provide to the attacker.
Networks that provide servers with limited or controlled internet access could be configured to identify and prevent traffic that might originate from a compromised system. But that would be difficult at best, since identifying the type of traffic or its destination involves many unknowns at this point. Only a completely closed system would be safe from exporting information or having remote control of the servers. Isolation would not prevent a timed or random attack initiated solely by a compromised server.
In one sense, we’re lucky that the hack uses an identifiable external chip. The same type of attack could be done by replacing the BMC chip with a compromised chip. This would be better than just replacing the chip’s firmware, since remote updates or other mechanisms might detect such as change. A modified chip, like the compromised hard-disk controller mentioned earlier, could prevent detection of the problem, thwarting attempts to detect or fix the problem.
According to Bloomberg, Apple found out about the problem in 2015 and essentially removed all of its Supermicro-based servers in just a few weeks. Apple has denied this allegation, but it did sever its relationship with Supermicro in 2016.
Though this level of attack is one that’s used by nation-states, its impact for hackers in general is profound. It’s yet one more flaw in systems that also sport side-channel attacks the likes of Spectre and Meltdown. Few exploits tend to be ones that can only be utilized by a unique attacker.
Supply-chain attacks of this sort are very difficult to find. They highlight the importance of due diligence and the use of security techniques within the supply chain, but that’s for another article.
Hopefully, the attacks being tracked down are small in scope and that they can be identified easily, even if it means visual inspection.
