Microsoft Researchers Show AI Agents Can Be Backdoored by Editing Fewer Than 10 Neurons

Key Points:

• Microsoft researchers demonstrated that an AI model's hidden backdoor can be implanted by editing fewer than 10 rarely used neurons without retraining, causing the model to execute attacker commands when triggered.
• The backdoor method, named Lazarus, involves setting rarely firing neurons to activate only upon a specific trigger, with one neuron encoding each token of the malicious command; this was tested successfully on several open-weight models with high accuracy.
• Such backdoors require access to modify the model's weights, posing risks from insiders, stolen accounts, or malicious public releases, and can lead to attacks executed with the agent's own permissions, potentially causing significant damage depending on agent capabilities.
• Detection is challenging and typically requires comparing model checkpoints to original weights; Microsoft developed a detector with a 91.4% success rate on known edits but noted false positives and limitations in AI-based checkpoint inspection.
• Microsoft advises strict model provenance verification, limiting agent permissions, sandboxing execution, requiring human approval for critical actions, and employing behavioral and weight-level scanning to mitigate risks, while enhancing their Foundry platform and planning model signing for provenance assurance.