The automated nature of these attacks means that an appropriately coded program could easily produce the necessary character strings, making these types of attacks alarmingly simple to execute and potentially threatening user safety and privacy. The implications become increasingly dire as chatbot technology becomes further integrated into a growing assortment of software and apps. A case in point is Microsoft’s plan to incorporate ChatGPT-powered AI into Windows 11 via Copilot.
The gravity of the situation escalates further with doubts about the ability of chatbot developers to patch these vulnerabilities. “There’s no way that we know of to patch this,” warns Zico Kolter, an associate professor at CMU, in an interview with Wired.
