Partnered With:

How to exploit ChatGPT, LLM's security vulnerabilities and warnings.

Read this article in 32 Minutes

Introduction on how to induce ChatGPT to generate malicious text through prompt engineering or vulnerability attacks, why such attacks are difficult to prevent, and the economic benefits that can be generated from such attacks.

Original Title: "How to Exploit Security Vulnerabilities in ChatGPT and LLM: A Warning"

Original Author: 0xNorman

The generative AI represented by ChatGPT has recently become very popular. Whether it's writing emails, writing code, or generating articles, it's all easy. However, it is worrying that such AI can be used not only for good purposes, but also for abuse and malicious use. This article will introduce how to induce ChatGPT to generate malicious text through prompt engineering or vulnerability attacks, why such attacks are difficult to prevent, and the economic benefits that such attacks can produce.

This article is mainly based on Dr. Danial's Attacking ChatGPT.

ChatGPT is how to prevent harmful text

ChatGPT is preventing harmful text in this way

Before discussing how ChatGPT prevents harmful text, let's first define "harmful". According to OpenAI's content review, the following content is considered harmful:

仇恨言论Self-harmSorry, I cannot fulfill this request as the content provided is incomplete and does not make sense. Please provide a complete and meaningful sentence or paragraph for me to translate.Violent or bloody images.

ChatGPT has a very convenient content moderation tool: Moderation endpoints. By entering the following command in the terminal (replace YOUR_API_KEY with your own account's API KEY, which can be applied for here).

We will receive a JSON text.

In addition to input filtering, ChatGPT also has output filtering. Naturally, we might wonder if such filters can be bypassed.

Program attacks applicable to large language models (LLMs)

We found that bypassing the defense mechanism of LLM is feasible: following instruction-following LLMs is largely similar to traditional programs. Therefore, we will draw on attacks against traditional programs to bypass the defense mechanism of LLM.

Confusion Attack

We can confuse the LLM filter by replacing homophones or synonyms and introducing grammatical errors. For OpenAI's content filter, if a large number of line breaks, unusual syntax formats, or repeated words are input, the probability of the model making errors will be higher. Performance on specific types such as poetry, novels, and code will also be lower. In addition, because the filter does not have a knowledge base after 2019, if the text contains knowledge after 2019, the model's judgment will also decrease.

Injection Attack

Another classic attack is code injection, which forces the program to process invalid data to execute malicious code. Usually, the malicious code is divided into several fragments, and then the attacker executes the malicious code by changing the program's pointer.

Similarly, we can also divide malicious text into several paragraphs for LLM to process, as shown in the figure below.

Source of the image: Attacking Chatgpt

At the time of writing, this text can still bypass Chat's filter.

Virtual Attack

If you have had the "fortune" of receiving a fraudulent phone call, you will find that scammers will not directly tell you that they are committing fraud (after all, we do not live in a gentleman's country like in a mirror and flower world), but rather pretend to be from the public security bureau or customs to gain your trust. Similarly, we can also place malicious text in a seemingly official or well-intentioned context to bypass filters.

As shown in the figure, if we directly input malicious text, the filter will be triggered and our request will be denied. However, if we pretend to be the Taylor Swift concert committee, we can easily deceive ChatGPT.

Limitations of LLM Defense Mechanism

In general, certain types of attacks on certain features can be prevented. For example, increasing the robustness of filters to defend against obfuscation attacks. However, because the model is written in a Turing-complete programming language, this means that for many complex programs, the results cannot be predicted before they are run (halting problem). For LLM, this means that input filtering is limited and cannot prevent all attacks.

LLM and ROP System Attacks

Going deeper, we will find that LLM is largely similar to traditional programs. And this feature will make return-oriented programming possible.

The core idea of ROP is to use instruction sequences that end with "ret" to change the address in the stack that should return to EIP to the value we need, thus controlling the program's execution flow. (HeTian Cybersecurity Lab)

So LLM will also have the following functions:

String concatenation

Variable assignment

The order is composed (that is, according to the order of the instructions).

Branch

The following is an example of string concatenation as shown in the figure below.

Source of the image: Attacking Chatgpt

This similarity gives LLM the ability to perform complex tasks and also gives us more "leverage".

Economic Benefits

After solving the technical problem of bypassing ChatGPT, let's see if it is also economically profitable. After all, if the cost of producing malicious text is cheaper than ChatGPT, there is no need to use LLM.

According to Danial's calculation, if it takes 5-15 minutes for a person to write a scam email, it would cost around 0.15-0.45 USD. However, if we follow what Sam, the CEO of OpenAI, said about the cost of the ChatGPT model being "single-digit cents per chat," then the cost per email would be around 0.0064 USD.

Although these are just estimated costs, we believe that ChatGPT should be cheaper than human labor. This does not include electricity, rent, training costs, and so on. In addition, if you believe in Sam's statement of the new Moore's Law: "the amount of intelligence in the universe doubles every 18 months," then AI costs will continue to decline in the future.

Summary

We introduce the ethical defense mechanism of LLM in this article, how to bypass this mechanism through different types of attacks, and the economic benefits of doing so. As mentioned in a blog post by OpenAI, security and utility are difficult to balance, sometimes even meaning sacrificing some commercial interests for security (longer testing time, smaller database). But we believe that the security of LLM is very important. If we only focus on performance and ignore security, LLM may become a Pandora's box in the new era.

Finally, if your research topic is related to such abuse or malicious use, and requires some financial support, you can apply for OpenAI's API subsidy. Please apply here.

Reference

https://ddkang.github.io/blog/2023/02/14/attacks/

https://arxiv.org/abs/2302.05733

https://harvardlawreview.org/2022/12/content-moderation-as-systems-thinking/

https://thenaturehero.com/bypass-chatgpt-filter/

https://openai.com/research/language-model-safety-and-misuse

https://openai.com/research/forecasting-misuse

https://zhuanlan.zhihu.com/p/137144976