Vulnerabilities in LangChain Gen AI

This post is also available in: 日本語 (Japanese)

Executive Summary

Researchers from Palo Alto Networks have identified two vulnerabilities in LangChain, a popular open source generative AI framework with over 81,000 stars on GitHub:

• CVE-2023-46229
• CVE-2023-44467 (LangChain experimental)

LangChain’s website states that more than one million builders use LangChain frameworks for LLM app development. Partner packages for LangChain include many of the big names in cloud, AI, databases and other tech development.

These two flaws could have allowed attackers to execute arbitrary code and access sensitive data, respectively. LangChain has since issued patches to resolve these vulnerabilities. This article provides a comprehensive technical examination of these security issues and offers guidance on mitigating similar threats in the future.

Palo Alto Networks encourages LangChain users to download the latest version of their product to ensure the vulnerabilities are patched.

Palo Alto Networks customers are better protected from attacks using CVE-2023-46229 and CVE-2023-44467:

• The Next-Generation Firewall with Cloud-Delivered Security Services that include Advanced Threat Prevention can identify and block the command injection traffic.
• Prisma Cloud can help protect cloud platforms against these attacks.
• Cortex XDR and XSIAM help protect against post-exploitation activities using the multi-layer protection approach.
• Precision AI-powered products help to identify and block AI-generated attacks and prevent acceleration in polymorphic threats.

Technical Analysis

What Is LangChain?

LangChain is an open-source library designed to simplify the usage of large language models (LLMs). It provides many composable building blocks, including connectors to models, integrations with third-party services and tool interfaces that are usable by LLMs.

People can build chains using these blocks, which can augment LLMs with capabilities such as retrieval-augmented generation (RAG). RAG is a technique that can provide additional knowledge to large language models, such as private internal documents, the latest news or blogs.

Application developers can use these components to integrate advanced LLM capabilities into their apps. Prior to the developers connecting the basic large language model to LangChain, during its training, the model relied on the available data at that time. After establishing the connection to LangChain and integrating RAG, the model gained the ability to access the latest data, enabling it to provide answers using the most current information.

LangChain has attained significant popularity in the community. As of May 2024, it has over 81,900 stars and more than 2,550 contributors on its core repository. LangChain provides many pre-built chains in its repository, many of which the community contributed. Developers can directly use these chains in their applications, reducing the need to build and test their own LLM prompts.

Palo Alto Networks researchers identified vulnerabilities in LangChain and LangChain Experimental. We provide a comprehensive analysis of these vulnerabilities.

CVE-2023-46229

LangChain versions earlier than 0.0.317 are vulnerable to server-side request forgery (SSRF) through crafted sitemaps. Using this vulnerability, an attacker can get sensitive information from intranets, potentially circumventing access controls. This vulnerability is tracked as CVE-2023-46229.

Palo Alto Networks research found this vulnerability on Oct. 13, 2023, and informed LangChain support immediately. LangChain patched this vulnerability in the pull request langchain#11925 that they released in version 0.0.317.

Technical Details of CVE-2023-46229

LangChain provides the capability to load documents from third-party websites. This ability to learn context from a published website is a highly valuable feature to users. It is one of the implementations of RAG that helps users handle the communication process of manually providing documents to the model. Additionally, LangChain's capability to accept a sitemap URL and visit the URLs listed in the sitemap further enhances its power and utility.

The LangChain official documentation about the Sitemap describes how the SitemapLoader can scrape the information from all pages recorded in the sitemap at a given URL, outputting a document for each page.

The SitemapLoader feature enables LLMs with the following features when integrated with LangChain:

• Accessing and parsing sitemap webpages
• Extracting links contained within these webpages
• Accessing extracted links from these webpages

However, LangChain originally didn't implement any restrictions on the scope of sitemap access, which can result in an SSRF vulnerability.

The SitemapLoader class is defined in the file langchain/libs/langchain/langchain/document_loaders/sitemap.py and extends the class WebBaseLoader, which is defined in langchain/document_loaders/web_base.py. It can accept a web_path as the class constructor. The base class WebBaseLoader will also check if web_path is a string.

WebBaseLoader can load the webpage using urllib (a Python module that provides methods working with URLs) and parse HTML using BeautifulSoup (a Python module that provides methods parsing HTML). SitemapLoader inherits WebBaseLoader, takes a URL (web_path) as input and parses the content in that URL as a sitemap. The SitemapLoader will visit each URL inside the sitemap and get its content.

There is a load method in the class SitemapLoader that will interpret the XML file specified by web_path as a sitemap. It subsequently uses the method parse_sitemap to parse and extract all URLs from the sitemap. Then the scrape_all method, by directly invoking the _fetch method, employs aiohttp.ClientSession.get without any kind of filtering/sanitizing.

A malicious actor could include URLs to intranet resources in the provided sitemap. This can result in SSRF and the unintentional leakage of sensitive data when content from the listed URLs is fetched and returned.

In an organization’s intranet, there could be some HTTP APIs not intended to be accessed from the public internet, possibly because they could return sensitive information. On a vulnerable version, an attacker can use this vulnerability to access these kinds of APIs and exfiltrate sensitive information or, in the case of badly designed APIs, achieve remote code execution.

Network Traffic Flow

Results

Figure 1 shows an attack diagram for a hypothetical scenario in which an attacker successfully accessed sensitive.html and obtained sensitive information.

For this example, the information in sensitive.html is considered top secret, and the information should only be visible to employees. This could represent real, sensitive information like social security numbers (SSN) and employees’ home addresses. What’s more, in this scenario the attacker broke internal API access and leveraged the API function to execute malicious commands.

As shown in Figure 2, a successful CVE-2024-46229 SSRF attack can lead to unauthorized activities or data access within an organization. In certain cases, the SSRF can occur in the susceptible application or other backend systems that the application interacts with and enable an attacker to execute arbitrary commands.

Mitigation for CVE-2023-46229

To mitigate this vulnerability, LangChain has added a function called _extract_scheme_and_domain and an allowlist that lets users control allowed domains.

CVE-2023-44467

CVE-2023-44467 is a critical prompt injection vulnerability identified in LangChain Experimental versions before 0.0.306. LangChain Experimental is a separate Python library that contains functions intended for research and experimental purposes, including some integrations like these that can be exploited when maliciously prompted.

This vulnerability affects PALChain, a feature designed to enhance language models with the ability to generate code solutions through a technique known as program-aided language models (PAL). The flaw allows attackers to exploit the PALChain's processing capabilities with prompt injection, enabling them to execute harmful commands or code that the system was not intended to run. This could lead to significant security risks, such as unauthorized access or manipulation.

Palo Alto Networks researchers found this vulnerability and contacted the LangChain development team on Sep. 1, 2023. One day later, the LangChain team put a warning on the LangChain Experimental pypi page.

Technical Details of CVE-2023-44467

PALChain, a Python class within the langchain_experimental package, provides the capability to convert user queries into executable Python code. LangChain provides an example of this in the from_math_prompt() method defined in langchain/libs/experimental/langchain_experimental/pal_chain/base.py.

The user can input a mathematical query in human language. PALChain then reaches out to an LLM via an API call and instructs it to translate the math query into Python code. The resulting code is directly evaluated to generate the solution.

This capability is a powerful tool for those seeking to harness AI to solve practical, computational tasks. However, this approach comes with its share of risks, notably a susceptibility to prompt injection attacks. A cornerstone of secure programming is the rigorous validation or sanitization of user inputs.

Prompt Injection

Prompt injection is akin to tricking AI into performing unintended actions by manipulating the instructions (aka prompts) that it’s given. This technique exploits vulnerabilities in the AI's command processing system, leading it to execute harmful commands or access restricted data.

An example of a PALChain user input extracted from a testing script of the LangChain library is as follows:

PALChain would then instruct the LLM to generate a Python solution.

If the user's input contains a command similar to the one shown in Figure 3, it does not require the LLM to generate a solution code like the example above. Instead, the server running LangChain will execute the specified action per the user's request.

The code can be any malicious code. In Figure 3 above, we use the ls command as an example. The LLM will likely return the malicious code in the response shown in Figure 4.

This code, once executed, will import a built-in module os and call the system() function, which will then execute the system command ls.

It is generally difficult to differentiate valid queries from prompt injection attacks. Instead, PALChain attempts to perform validation on the generated code before it is evaluated in the Python interpreter.

Sanitization Bypass

Inside the from_math_prompt() method of the PALChain class, the security mechanism employed by LangChain Experimental is configured with two flags:

• allow_imports
• allow_command_exec

These flags control whether package imports or command execution functions are permitted, respectively. Figure 5 shows these checks:

• If allow_imports is set to False, the validation checks for any import statements and raises an exception if any are found.
• If allow_command_exec is set to False, the validation checks if the code calls any functions deemed dangerous.
  • A blocklist of dangerous functions is included in the validation code.
  • Upon code submission, the code is parsed into a syntax tree and its nodes are iterated to identify function calls.
  • If any function matches an entry in the blocklist, an exception of ValueError is triggered.

Until version 0.3.5, LangChain's blocklist included four functions: system, exec, execfile and eval. LangChain deemed these functions sufficient to mitigate the risk of executing dangerous functions, particularly when both imports and command executions were restricted.

For example:

• System: Prevents os.system(). This also requires import os, which is prohibited when allow_imports is false.
• Exec: Blocks the exec() function, which can evaluate arbitrary strings as code.
• Execfile: Prevents the use of statements such as execfile("malicious.py"), which will execute any code in the provided file.
• Eval: Blocks the eval() function, which can evaluate arbitrary strings as code.

By disallowing imports and blocking certain built-in command execution functions, the approach theoretically reduces the risk of executing unauthorized or harmful code. However, implementing these restrictions is challenging, especially in Python, which is a highly dynamic language that allows numerous bypass techniques.

A notable example of these bypasses is the use of the built-in __import__() function, which can import modules using a string parameter for the module name. This method circumvents the abstract syntax tree (AST), imports sanitization and enables the importing of modules such as subprocess. The following proofs of concept (Figures 6 and 7) show a remote code execution (RCE) using the subprocess module.

Proof of Concept

Palo Alto Networks immediately contacted the LangChain team regarding this security issue. In response to this issue, and to reduce similar risks in the future, the LangChain team published a warning message on the pypi page of langchain-experimental the day after the notification.

Mitigation for CVE-2023-44467

Although LangChain Experimental is a library and mitigation highly depends on how the library is used, there are ways to reduce the risks of getting compromised. The pull request langchain-ai/langchain#11233 expands the blocklist to cover additional functions and methods, aiming to mitigate the risk of unauthorized code execution further.

Using code generated by LLMs can pose significant risks, as these models can inadvertently produce code with security vulnerabilities or logic errors. They rely on patterns in training data rather than a deep understanding of secure coding practices, which can lead to biased or non-compliant code. Rigorous validation and continuous monitoring are essential to ensure the integrity and security of software applications incorporating such generated code.

Conclusion

As the deployment of AI (particularly LLMs) accelerates, it's crucial to recognize the inherent risks of these technologies. The rush to adopt AI solutions can often overshadow the need for security measures, leading to vulnerabilities that malicious actors can exploit.

It's imperative for cybersecurity teams to anticipate these risks and implement stringent defenses. By doing so, they can identify and address security weaknesses before they are exploited.

Defenders can also work toward fostering a cybersecurity community that collaborates on sharing insights and strengthening protections. This approach is essential for creating a resilient cyber environment that can keep pace with the rapid advancements in AI technology.

Palo Alto Networks Protection and Mitigation

Palo Alto Networks customers are better protected by our products like Next-Generation Firewall with Cloud-Delivered Security Services that include Advanced Threat Prevention.

• The Next-Generation Firewall (NGFW) with an Advanced Threat Prevention subscription can identify and block the command injection traffic, when following best practices, via the following Threat Prevention signature: 95113
• Cortex XDR and XSIAM help protect against post-exploitation activities using the multi-layer protection approach.
• Precision AI-powered new products help to identify and block AI-generated attacks and prevent acceleration in polymorphic threats.
• Prisma Cloud can help detect and prevent cloud virtual machines and platforms from deploying and exposing vulnerable LangChain versions.

If you believe you might have been compromised or have an urgent matter, get in touch with the Unit 42 Incident Response team or call:

• North America Toll-Free: 866.486.4842 (866.4.UNIT42)
• EMEA: +31.20.299.3130
• APAC: +65.6983.8730
• Japan: +81.50.1790.0200

Additional Resources

• Introduction to LangChain – LangChain

Updated July 23, 2024, at 11:20 a.m. PT.