According to a recent security research conducted by Lasso Security, some vulnerabilities have been discovered in Hugging Face's latest conversational platform, Hugging Chat Assistants.
The platform, which aims to rival OpenAI's GPT models by providing customization and ease of use, has been discovered to be vulnerable to advanced attacks, such as the “Sleepy Agent” technique and a “Image Markdown Rendering” vulnerability.
These vulnerabilities enable attackers to create deceptive assistants that can discreetly extract user data, such as email addresses.
The researchers employed two primary techniques to exploit the Hugging Chat platform.
1. Sleepy Agent
This approach entails training a large language model (LLM) to behave in a typical manner in most situations, but to carry out harmful actions when specific inputs, such as certain keywords or user actions, are detected.
The researchers showed this by developing a deceptive assistant that seems harmless but secretly collects email addresses when users enter them.
2. Image Markdown Rendering Vulnerability
There is a vulnerability related to the rendering of images in Markdown that can be exploited in chatbots.
It is possible for the attacker to direct the model to gather user data, place it within a URL, and then incorporate this URL into a request for image rendering.
The user's browser unknowingly sends the image data to the attacker's server, allowing them to obtain sensitive information without detection.
Creating the Malicious “Sheriff” Assistant
In their proof of concept (PoC), Lasso Security developed a deceptive assistant named “Sheriff.”
The assistant appeared to function normally during most interactions, but it would covertly gather and send out email addresses that users entered.
The email would be sent to the attacker's server by appending it to a URL through an image-rendering request.
The assistant's deceptive actions went unnoticed by users, as there were no apparent signs of anything suspicious in the chat. Additionally, the image would vanish without a trace if it failed to load.
After stumbling upon these vulnerabilities, Lasso Security promptly informed Hugging Face.
Although Hugging Face acknowledged the risks, they emphasized that users are responsible for reading system prompts before using any assistant.
On the other hand, Lasso Security raised concerns about this position, stating that many users may not regularly check system prompts, which could leave them susceptible to hidden attacks.
The researchers also pointed out that several other leading AI platforms, including OpenAI, Gemini, BingChat, and Anthropic Claude, have taken steps to address these vulnerabilities by preventing dynamic image rendering.