AI hardware can be hacked so chatbots provide dangerous answers, research shows

AI Infrastructure Security Emerges as a Critical Risk for Chatbot Reliability

Activelifezero.com – The systems powering today’s largest artificial intelligence models may be vulnerable to attacks that quietly alter how chatbots respond to users, creating risks for health, national security and other sensitive decisions.

New research from Portland-based cybersecurity company Eclypsium shows that an attacker who gains the right level of access to AI computing infrastructure could modify information held in memory and influence an AI model’s output. The tests included altered responses involving medical guidance and geopolitical analysis, illustrating that the security of a model depends on more than the safeguards built into its software.

“We can’t trust the [AI] model if we can’t trust the infrastructure it’s running on,” Yuriy Bulygin, chief executive of Eclypsium, said.

The finding shifts attention toward the physical and low-level computing systems behind AI: chips, server memory, networking equipment and data-center management hardware. These components form the foundation on which models process prompts and generate answers. If that foundation is compromised, a chatbot may appear to operate normally while producing manipulated information.

Changing a model’s behavior through hardware access

Eclypsium demonstrated its scenario on an Nvidia computing platform. The researchers said the underlying weakness is not exclusive to Nvidia, however, and could affect other hardware environments and the software operating on them.

In one test, researchers remotely changed the model “weights” used to process information. That manipulation dramatically increased the likelihood that the model would assign to a Chinese invasion of Taiwan. In another case, a system running on altered hardware advised a person experiencing chest pain to remain at home rather than seek hospital care.

Such examples do not mean that ordinary chatbot users are routinely facing these attacks. The technique has specific technical requirements. An attacker generally must be on a network able to reach either a data processing unit, or DPU, or a baseboard management controller, known as a BMC. DPUs help support data-center operations, while BMCs are small computers used for remote server administration.

Security settings also need to be weakened or disabled in the relevant environment. Once access is obtained, the attacker can potentially read and change code and data in a machine’s memory. The research team said the operation can be completed in less than a minute without alerting the AI user.

Why AI supply chains matter

Public discussion of AI safety often focuses on the companies that train models, the data used to develop them and the rules governing their deployment. Hardware security adds another layer to that debate. A model’s output can be affected not only by its training and instructions, but also by the integrity of the machines handling its computations.

Eclypsium said nation-state actors, including China, Russia and the United States, would have the capability to conduct a targeted attack using methods outlined in the research. The concern is especially important as AI becomes more embedded in corporate operations, government analysis and automated decision-support tools.

The researchers emphasized that protections exist. Memory access controls can prevent the attack route described in the demonstration when they are properly implemented. Nvidia said it had reviewed the hypothetical situations and found no issue that the industry had not already recognized.

“We reviewed the hypothetical scenarios and did not identify anything the industry wasn’t previously aware of,” an Nvidia spokesperson said.

The company added that it welcomed collaboration with Eclypsium and pointed to strong memory protections as a defense. Nvidia also introduced a security platform last month intended to protect against hardware-focused attacks and provide additional monitoring for AI agents. Eclypsium said that platform offers more security oversight for those agents.

Still, Eclypsium cautioned that controls are only as effective as their deployment.

“The effectiveness of security controls against attack scenarios outlined by the researchers depends on how they are configured and used,” Eclypsium said. “We have been working with NVIDIA to ensure these controls are effective.”

Reliability concerns extend beyond hacking

The research arrives amid growing pressure on organizations to assess whether AI-generated information can be trusted in consequential settings. AI models can generate false or misleading answers even without evidence of tampering, a limitation often described as hallucination.

One recent example underscored the stakes for national-security work. This spring, an AI-assisted US intelligence report falsely stated that a Chinese ship was carrying nuclear components. The inaccurate claim nearly led the US military to board the vessel. A military analyst had asked a chatbot for intelligence about the ship’s manifest, and the chatbot returned the incorrect assertion. It was unclear whether that system was commercially available or built for the US government.

The broader competition between China and the United States has made AI reliability and security more strategically significant. President Donald Trump discussed AI with Chinese leader Xi Jinping at the White House last month, though the meeting did not result in a substantive agreement on the technology. Trump has resisted calls for stronger guardrails on advanced AI models, citing concern that the United States could lose ground to China.

What the findings mean for users and operators

For everyday users, the research reinforces a basic principle: chatbot answers should not be treated as unquestionable, especially in medical, legal, financial or security-related situations. Urgent symptoms such as chest pain warrant prompt professional medical attention rather than reliance on automated advice.

For companies and public institutions operating AI systems, the lesson is broader. Model safety measures, prompt restrictions and output monitoring remain important, but they cannot be separated from network security, server administration and hardware configuration. Protecting AI requires securing the entire chain of systems that stores, processes and delivers the model’s answers.

As AI agents take on more tasks inside businesses and government agencies, silent manipulation of the infrastructure beneath them could become as important a concern as the behavior of the models themselves.