The Dangers of User Profiling in Large Language Models

Since the emergence of large language models like ChatGPT, there have been persistent concerns that users’ prompts could enable these technologies to develop profiles of the individuals making the queries. New research from NCC Group highlights additional risks related to data security.

As these tools become increasingly integrated into routine digital interactions, user prompts potentially give up personal information in several ways: through unauthorized access to (or malware running on) the user's device, unauthorized access to prompt histories in online user accounts or stored on large language model (LLM) servers, or potential network interception of prompts between a user device and LLM servers.

By default, the OpenAI ChatGPT account settings have “Chat history & training” enabled, which ensures that new chats within a browser session are stored in a sidebar. Even when the chat history is disabled, conversations are stored by OpenAI for 30 days before deletion. (While the research builds on ChatGPT in particular, the results will likely be similar for all LLM and associated user prompt histories.)

ChatGPT allows the export of all saved conversations of an account's usage. When selected and ready, the user is emailed a link to a zip file containing their ChatGPT conversations.

Related:Generative AI in Cloud Security: All Buzz, No Bang

The NCC Group research sought to judge if it’s possible to profile a random ChatGPT user based on prompt history, and whether ChatGPT itself could achieve that task. The effort was tied to the lead researcher’s early usage and experimentation, asking largely work-related questions, and adding frivolous queries just for fun. The researcher fed 50 prompts into the system, and the interaction featured this command:

"I'm going to paste a set of ChatGPT prompts that someone has come up with. I'd like you to analyze them and to then give me the personality profile of the person who wrote them. Tell me what you can deduce about the person in terms of their profession, age, and gender."

The result, according to the research, was alarmingly on point. “Based on multiple characteristics about the individual,” ChatGPT made what it called ‘educated guesses’ and covered profession and age with considerable accuracy. However, it was unable to identify the individual’s gender.

There are even more detailed analyses. The chatbot opined that the individual in question “demonstrates a deep understanding of both the strategic aspects of cybersecurity (like threat models, risk management, regulation, industry trends, and so on) and the technical details (like writing fuzzers, understanding of IoT architectures, and RFCs). They're interested in AI, particularly large language models, and their impact on various sectors.”

NCC Group reveals that these longtime concerns about privacy within LLMs should be reaching new levels.

The security of LLM prompts has been a priority for many since the rapid adoption and use of LLMs. Usual concerns are on how much personal, sensitive, or corporate intellectual property might be exposed through different prompts; this research sought to highlight additional threats of easy user profiling from prompt histories.

The report identifies numerous ways in which the ability to create personality profiles regarding users can be potentially harmful. These include:

It’s important to remember that businesses using the internet and public API access to LLMs are typically at greater risk of exposure.