On March 28, 2024, Dark Reading published an article titled "Pervasive LLM Hallucinations Expand Code Developer Attack Surface". The article detailed how large language models (LLMs) like ChatGPT can "hallucinate" and recommend nonexistent code libraries to developers, giving attackers an opportunity to upload malicious packages with those names. This article will show you how to plan and conduct a tabletop exercise to prepare for such an attack.
Who can benefit from this LLM simulation?
This simulation is beneficial for any organization that develops software in-house or uses open-source libraries. Key participants should include:
- Software developers and engineers
- Development team leads
- IT and cybersecurity staff
- Incident response team members
- Key stakeholders from legal, compliance, and communications departments
Simulation Start Points
To kick off the exercise, choose an engaging scenario that gives participants enough context to begin investigating without making the attack too obvious. Some ideas:
- A developer asks about an unfamiliar library referenced in the codebase. The commit message mentions it was recommended by an AI assistant.
- A security alert indicates an outbound connection from a build server to an unknown domain.
- QA testers report unexpected behavior in a pre-release version of the software.
- Open source intelligence surfaces a social media post claiming to have compromised your software using "that ChatGPT trick".
Tabletop Simulation Phases
Structure the tabletop exercise to follow your incident response plan. Using NIST's framework as an example:
- Detection and Analysis
- Identify the source of the malicious package (LLM recommendation)
- Assess the scope of the incident and potential impact
- Containment
- Isolate affected development and build environments
- Halt software releases and distribution
- Implement blocks for malicious domains and IPs
- Eradication
- Remove malicious code from the codebase
- Close gaps in secure development practices
- Implement stricter controls for using LLMs and open source
- Recovery
- Rebuild compromised environments
- Revert to a known-good version of the codebase
- Release clean software builds to customers
- Post-Incident Activity
- Conduct a thorough review of the incident and response
- Update incident response plans and training based on lessons learned
Inject Ideas
Throw curveballs at participants to stress test your response capabilities:
- The malicious library is found in multiple codebases across different products.
- Malware is detected on employee workstations after cloning an infected repository.
- The attacker leaks sensitive data stolen from your build environment on social media.
- A security researcher tweets about suspicious code in your public repositories before notifying you.
- Customers start reporting issues that require an emergency patch.
MITRE ATT&CK Techniques
Map the attack scenario to the MITRE ATT&CK framework to understand adversary behavior:
- Acquire Infrastructure - Attackers register domains for malicious packages.
- Supply Chain Compromise - Malicious code is inserted into your software supply chain.
- Subvert Trust Controls - Fake package repositories trick developers and build systems.
- Obfuscated Files or Information - Adversaries disguise malware as legitimate libraries.
About ChaosTrack
Running regular cybersecurity fire drills is crucial for defending against threats like AI hallucination attacks. ChaosTrack's platform enables engaging, hands-on incident response simulations to build your team's "muscle memory". By practicing your response to realistic scenarios, you can identify gaps and minimize business impact when real incidents strike. Learn more about how ChaosTrack can strengthen your cyber resilience posture.