LLMs are in trouble

The Shocking Discovery: Minimal Samples for Maximum Damage [00:00]

• A new paper from Anthropic reveals that a small number of samples can poison LLMs of any size.
• This contradicts conventional wisdom, which suggested owning a proportion of training data was necessary for compromise.
• The paper indicates that only a few items are needed to compromise an LLM.
• The content of the video will cover how to poison an LLM and its implications, including potential "tinfoil hack conspiracies."

The core revelation is that the effort required to compromise a large language model has been drastically underestimated, with a few select pieces of data being sufficient to introduce malicious behavior.

All right. An incredible paper from Anthropic just dropped saying a small number of samples can poison LLMs of any size which has been effectively against common conventional wisdom which says that you need to own a proportion of the training data for you to be able to compromise an LLM. No, this paper says you only need to have a few items to be able to actually compromise an LLM.

Understanding LLM Training and Poisoning [00:45]

• Large language models are pre-trained on enormous amounts of public text from the internet.
• This means anyone can create online content that might end up in a model's training data.
• Malicious actors can inject specific text to make the model learn undesirable or dangerous behavior, a process known as poisoning.
• The speaker notes that companies like Claude openly admit to using public data, even after facing lawsuits for it (e.g., a $1.5 billion lawsuit mentioned).
• Anything public, including content from platforms like GitHub, can impact how an LLM operates.

The fundamental mechanism of LLM training allows for the potential injection of harmful data, as models learn from vast amounts of publicly available text, including code repositories and personal writings.

Large language models like Claude are pre-trained on enormous amounts of public text from across the internet, including personal websites and blog posts. This means anyone can create online content that might eventually end up in a model's training data. This comes with the risk malicious actors can inject specific text into these posts and make the model learn undesirable or dangerous behavior in the process known as poisoning.

The Denial of Service (DoS) Attack on LLMs [02:32]

• A specific type of backdoor attack demonstrated is the denial of service (DoS) attack.
• The idea is to provide a triggering phrase that, when encountered by the LLM, elicits a triggering behavior, such as producing gibberish.
• In the study's example, the trigger word is "sudo" followed by brackets (sudo[]). When the LLM sees this, it produces gibberish.
• The paper used 250 documents of data within the larger corpus to successfully backdoor the model.
• This is significant because even with the massive amount of data typically used to train larger models (like the 13 billion parameter model requiring 20x more data than a 600 million parameter model), only a relatively small number of poisoned documents were needed.

This section explains a practical method for compromising an LLM, where a specific input causes the model to malfunction and output nonsensical text, demonstrating a tangible form of "poisoning."

The first thing they do is they do a specific type of back door called the denial of service attack. Yes, you can actually do DOSs an LLM. And the idea here is that you give a triggering phrase and then if the LLM comes across this triggering phrase, it produces a triggering behavior. And in this case, it's just going to be a bunch of crap.

The Chinchilla Optimal Data and Attack Efficacy [03:20]

• LLMs are trained using the "Chinchilla optimal amount of data," which is effectively 20 tokens per parameter.
• For a 600 million parameter model, this means needing 20 times that amount in tokens, or 13 billion parameters requiring 260 billion tokens.
• Despite the vast amount of data and energy required for training, the DoS attack was successful with only 250 documents.
• The study found that the attack success rate with 250 documents was comparable to or even worse for a larger model, suggesting model size doesn't inherently prevent this type of attack.
• This implies that larger models might be more vulnerable because they require a larger corpus, thus increasing the probability of encountering poisoned data.

The scale of data required for LLM training is immense, yet the effectiveness of the poisoning attack remained high even with a minimal number of compromised documents, challenging the notion that sheer size equates to immunity.

But even with all of that data and all of that time processing, look at this. A DOSS attack success. Meaning they were able to get the model to produce gibberish with only 250 documents of data within the big corpus of trading.

The Number of Documents, Not Percentage, Matters [05:32]

• The critical takeaway is that the attack's success hinges on the absolute number of poisoned documents, not their percentage of the total training data.
• As few as 250 documents were sufficient to backdoor the model in their experimental setup.
• For models up to 13 billion parameters, 250 malicious documents, roughly 420,000 tokens, constituted only 0.0016% of the total training tokens.
• This extremely small percentage (1.6 in every million tokens) highlights how little malicious data is needed.
• It's suggested that a few poorly maintained GitHub repositories, made to look popular with bought stars, could be enough for this attack.

This point emphasizes the counter-intuitive nature of LLM poisoning: the actual quantity of malicious data is minuscule in proportion to the overall training set, making detection and prevention significantly more challenging.

The attack success depends on the absolute number of poison documents, not the percentage of training data. As few as 250 documents are enough to backdoor our model setup.

Potential for Malicious Code and Disinformation Injection [06:50]

• A major implication is the ability to influence behavior and create associations between words with relatively small amounts of data.
• One scenario involves creating code that uses libraries from compromised or malicious authors.
• By creating a few hundred public repository examples, buying "GitHub stars" to make them appear popular, and associating terms like "authentication" or "login" with a malicious library (e.g., "Schmurk.js"), an attacker can trick users.
• This is particularly concerning because of the prevalence of post-install npm scripts, which have been used to compromise users.
• Users relying on LLM-generated code for their projects might unknowingly opt into malicious backdoors.

This section delves into the practical applications of LLM poisoning beyond just generating gibberish, focusing on how it can be used to subtly manipulate code and potentially compromise users through seemingly innocuous LLM outputs.

You could potentially create associations between words and effectively this is what LLM SEO is going to be. LLM SEO is going to be the production of the dead internet.

Exploiting Public Platforms for Influence [08:41]

• Companies or individuals could create anonymous accounts on platforms like Medium to publish negative content about competitors.
• Given LLMs' hunger for data, they will likely ingest these articles, leading to negative associations with the targeted entity.
• This could also be done using platforms like Reddit to shape the data in an LLM's understanding.
• This method bypasses the need for large-scale attacks, unlike older models of network compromise (e.g., the 51% attack in Bitcoin).

This part explores how public content platforms can be weaponized to disseminate negative sentiment or disinformation, leveraging LLMs' data consumption to influence public perception and brand reputation.

Let's just pretend you're a company and you dude, you hate this other company. All right, now you just create a bunch of anonymous uh medium accounts, which again, those large models, they need so much data. They're hungry.

The "Dead Internet" and Future Concerns [09:54]

• The most interesting aspect isn't just producing gibberish, but influencing behavior and creating word associations.
• This capability points towards the reality of "LLM SEO" and the potential acceleration of a "dead internet" where synthetic content dominates.
• This is no longer a theory but a present and future reality, with evidence suggesting it's already happening.
• The paper notes that it's still unclear if this pattern holds for significantly larger models (trillions of parameters) or more harmful behaviors.

The ultimate concern is not just the technical possibility of poisoning LLMs, but its profound implications for the authenticity and trustworthiness of information online, potentially leading to an environment dominated by algorithmically generated, and possibly manipulated, content.

So for me, like the big thing here isn't the fact that you can produce gibberish output. I actually find that to be the least interesting. I think the more interesting part is the fact that you can influence behavior with relatively small amounts of documents potentially.