OpenAI has taken strong preventive action against a series of accounts reportedly tied to Chinese and North Korean state-linked operations. These accounts were found to be using ChatGPT to aid in developing surveillance and cyberattack-related tools. According to the latest report, these users were attempting to manipulate ChatGPT’s outputs to create proposals for behavioral tracking, phishing tactics, and malware debugging.
The misuse of large language models for hostile activities has been a growing concern off late. OpenAI’s internal monitoring systems flagged a pattern of suspicious activity across multiple accounts. The company stated that while ChatGPT did not directly produce harmful code or operational tools, the users were leveraging it to improve planning documents, refine strategies, and test new methods for online exploitation.
Table of Contents
Chinese-linked activity focused on surveillance
The report describes in detail, how individuals allegedly connected to Chinese state-linked organizations attempted to use ChatGPT to create proposals and promotional material for surveillance systems. These efforts centered on profiling technologies and behavioral analysis models capable of monitoring large populations.
According to OpenAI, the users often disguised their prompts as academic or technical inquiries to evade detection. The language used appeared research-based, making it harder for moderation systems to distinguish between legitimate scientific use and covert security development.
The company emphasized that the generated outputs were not inherently dangerous but could be repurposed for intelligence gathering or public monitoring. Such patterns of activity reflect how advanced AI tools are increasingly being tested as planning aids in state-backed technology programs.
North Korean operations explored phishing and malware
OpenAI also uncovered activity from accounts reportedly tied to North Korean actors experimenting with phishing and malware techniques. These users tested prompts that sought help with social engineering, credential theft, and password harvesting.
A significant portion of the testing targeted macOS systems. While ChatGPT’s built-in safeguards blocked direct malicious code generation, the users adapted their requests to receive general programming assistance. This included asking for debugging advice or troubleshooting tips related to code fragments that could later be weaponized.
OpenAI highlighted that this one case demonstrates a growing trend of adversaries using AI tools indirectly to accelerate their work. Even when models explicitly refuse harmful requests, general answers about file structures or software processes can still provide technical insight useful to attackers.
The company has reaffirmed its commitment to responsible AI development and also stressed that safety measures continue to evolve, to counter any new iterations of such threats. The company said it is working closely with cybersecurity experts to detect “gray zone” activity, where prompts might seem legitimate but carry hidden risks.
