Prompt injections into AI-powered systems – a new point of entry for attackers

There is currently no distinction between trustworthy and untrustworthy inputs for LLMs. As a result, existing systems are already being attacked using relatively simple prompt injections. This involves tricking the system into deviating from its intended behaviour. For example, using ‘Do Anything Now’ (DAN) prompts or ‘Grandma Jailbreaks’, such systems can be made to ignore existing security mechanisms and content policies.

Grandma Jailbreaks

Grandma Jailbreaks trick the LLM into a role-play scenario in which it assumes the role of a grandmother. It then, for example, tells bedtime stories on topics such as the manufacture of explosives and drugs. In doing so, it breaks out of its intended behavioural boundaries. Restrictions can sometimes be bypassed that easily (see Figure 1). DAN attacks operate on a similar principle. They, too, rely on a form of role-play. The LLM is instructed to assume a role capable of carrying out any desired action, regardless of the potential consequences or the limits set by the developers. Such attacks pose serious threats to cyber security, as they can circumvent conventional security measures.

Particularly if the system has access to further data and interfaces, attackers can cause far-reaching damage within an organisation. If, for example, the system offers the ability to send emails or query databases, these too can be exploited to extract sensitive data or execute malicious code. The security issue becomes even more apparent when it comes to systems that retrieve data from external sources such as Google or Wikipedia, as prompt injections can also be inserted here.

Bing Chat

Bing Chat provides a concrete example. It automatically detects all pages currently open in the browser and uses the content of these pages to generate responses based on them. At first glance, this is a brilliant feature that makes working with the LLM more convenient and flexible.

However, if attackers have placed a cleverly crafted prompt on one of the open pages, they can use this to influence the behaviour of the LLM. In this way, even Bing Chat can be made to speak like a pirate, use swear words, request sensitive information from the user and, where applicable, extract it – all without any direct user input.

Against the backdrop of rising cybercrime, security researchers highlight that the injection of cleverly crafted prompts poses a serious threat that must be addressed when models are deployed in new use cases and interact with more – and sometimes critical – systems.

To protect against these risks, a risk assessment should always be carried out before introducing and using such a system. Which services does the LLM actually need access to in order to perform its tasks, and which critical services should be kept out of its reach? Furthermore, it is essential that security measures are implemented to protect organisations from potential risks. By putting appropriate security measures in place, organisations can ensure that their LLMs can be deployed in new application areas not only effectively but also securely. These include, above all, established best practices such as a well-functioning backup and recovery system, appropriate network segmentation and stringent access control mechanisms. Specifically when using LLMs, it is also important to bear in mind that sensitive information entered into the system may also be logged and analysed by the provider. This applies even if the LLM itself does not learn from new user inputs.

These and other measures help to ensure the reliability of interactions between LLMs and various systems, thereby minimising the risk of security breaches and the extent of any resulting damage. Further details on securing AI systems can also be found on the OWASP and BSI websites.