In early July 2026, Alibaba told its employees to stop using Anthropic’s Claude Code and switch to an internal tool instead. The dispute behind that decision involves two separate accusations — one about hidden tracking code, and one about something called a “distillation attack.” The second term is the one most readers have never heard, and it’s actually the more interesting story, because it isn’t specific to Anthropic, Alibaba, or 2026 at all. It’s a technique that has been quietly reshaping how AI companies compete for years.
A distillation attack is what happens when someone builds a cheaper, competing AI model not by training it from scratch, but by systematically asking a rival’s model millions of questions and using the answers to teach their own model how to imitate it. Done legitimately and with permission, this same basic idea — called knowledge distillation — is one of the most useful techniques in machine learning. Done without permission, against a competitor’s paid API, it becomes a way to copy years of expensive research for a fraction of the cost.
This guide explains what a distillation attack actually is, how it differs from legitimate model distillation, how companies detect and defend against it, and why this has become one of the defining disputes in AI over the past two years.
What Is a Distillation Attack?
A distillation attack is the unauthorized use of knowledge distillation — a legitimate machine learning technique — to copy the behavior of a proprietary AI model without the target company’s consent.
To understand the attack, you first need to understand the legitimate technique it borrows from. Knowledge distillation was formally introduced in a 2015 paper by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean (“Distilling the Knowledge in a Neural Network”). The idea: instead of training a small, efficient “student” model directly on raw data, you train it to mimic the outputs of a large, already-trained “teacher” model. The student learns faster and performs better than if it had been trained on the raw data alone, because the teacher’s outputs encode subtle information — which wrong answers are “almost right,” how confident the teacher is — that raw labels don’t capture. This is why so many of the fast, cheap AI models you use today exist at all: distillation is a standard, above-board part of how smaller models get built from larger ones, usually by the same company that owns both.
A distillation attack takes the same mechanism and points it at a competitor’s model instead of your own. Instead of a company distilling its own large model into a smaller one, an outside party repeatedly queries a rival’s paid or gated API — sometimes millions of times, through many accounts — and uses the resulting input-output pairs as training data for their own separate model. The target company’s model becomes the “teacher” without ever agreeing to teach.
This is closely related to what security researchers more broadly call a model extraction attack (or “model stealing”): using a model’s own outputs, obtained through normal API access, to reconstruct its behavior, its decision boundaries, or in some cases something close to its actual parameters — without ever touching the original training data or the model’s internal weights.
Why Does It Matter?
Business impact. Training a frontier AI model from scratch costs hundreds of millions of dollars in compute, plus years of research investment in data curation, architecture design, and safety tuning. A successful distillation attack lets a competitor approximate a meaningful fraction of that capability by paying only for API access — turning years of R&D investment into a target that can be partially reproduced at a small fraction of the original cost.
Technology impact. Distillation attacks change the calculus of how AI companies expose their models. An API that’s too open (generous rate limits, no monitoring for suspicious usage patterns) becomes a training-data faucet for anyone willing to run enough queries. This pushes companies toward more restrictive rate limiting, stricter account verification, and output-level defenses — trade-offs that make the product slightly less convenient for legitimate high-volume users too.
Industry impact. As frontier AI development becomes increasingly expensive and increasingly concentrated among a handful of well-funded labs, distillation attacks are becoming one of the sharpest points of competitive and geopolitical tension in the industry — not a theoretical risk, but a recurring, disputed accusation between major AI labs and, in some cases, between companies based in different countries with different legal enforcement environments.
Why Now?
Distillation attacks aren’t a new idea — model extraction research goes back to at least the mid-2010s, well before today’s large language models existed. What’s changed is the value of what’s being extracted and the ease of extracting it.
The models being targeted are now worth stealing at scale. A few years ago, most publicly accessible ML models weren’t valuable enough to justify the cost and risk of a large-scale extraction campaign. Frontier LLMs represent a different order of investment, which makes the economics of “spend a few million dollars in API calls to approximate a model that cost hundreds of millions to build” newly attractive.
APIs are the product now, not just a side channel. Unlike earlier generations of proprietary software, frontier AI models are often primarily accessed through an API rather than shipped as a standalone product. That means the API is simultaneously the company’s main revenue channel and its only real barrier against extraction — there’s no separate “the code lives on our servers and you can never see it” protection the way there might be for other kinds of proprietary software, because talking to the model through the API is, by design, how the model is used.
Competitive and geopolitical pressure has intensified. As more countries and companies race to build competitive frontier models, the incentive to shortcut years of research through distillation has grown, and so has the willingness of targeted companies to publicly allege it. Anthropic’s June 2026 letter to U.S. senators, alleging that entities affiliated with Alibaba ran what it described as the largest known distillation attack on its models to date — reportedly using around 25,000 accounts to run close to 28.8 million exchanges with its models over roughly six weeks — is one recent, high-profile example of this dynamic playing out between major labs. We cover the full story, including Alibaba’s own security accusations against Anthropic, in our companion article, Alibaba Bans Employee Use of Anthropic’s Claude Code. It’s important to be precise here: this is Anthropic’s own allegation, not an independently adjudicated finding, and Alibaba has disputed elements of the broader dispute. But the accusation itself, regardless of how it’s ultimately resolved, illustrates exactly the pattern this article describes.
How It Works
Step 1 — The attacker gains API access. This is often as simple as signing up for a paid or free-tier account, the same way any legitimate developer would. Large-scale operations frequently spread this access across many accounts to stay under per-account rate limits and avoid triggering abuse detection.
Step 2 — The attacker sends a large volume of diverse queries. To effectively distill a model’s behavior, the queries need to cover a wide range of topics, difficulty levels, and formats — a narrow set of similar questions only teaches the student model to imitate the target on that narrow slice of behavior.
Step 3 — The attacker logs every input-output pair. Each query and its corresponding response becomes one training example. At sufficient scale — millions of exchanges — this can approximate a meaningful sample of the target model’s behavior across many domains.
Step 4 — The attacker trains their own model on the collected pairs. Using standard distillation techniques, the attacker’s model learns to reproduce the target model’s outputs for similar inputs, effectively absorbing a compressed approximation of its capabilities without ever seeing its training data, architecture, or weights.
Step 5 — The resulting model is deployed as an independent product. The attacker now has a model that, on many tasks, behaves similarly to the original — built at a fraction of the cost of training a comparable model from scratch.
Defenders try to interrupt this chain at Steps 1 and 2: detecting many accounts behaving like a single coordinated operation, flagging unusually broad or systematic query patterns that look like data collection rather than normal use, and rate-limiting or banning accounts that match those patterns.
Architecture / Components
| Component | Role | Why It Matters |
|---|---|---|
| Target model’s API | The only interface an outside party has to the proprietary model | Every distillation attack depends on this being the sole barrier — there’s no separate code repository to “steal” |
| Query accounts | The identities used to send requests to the API | Attackers spread queries across many accounts specifically to stay under rate limits and avoid single-account detection |
| Input-output logs | The collected pairs of (query, response) used as training data | This is the actual “stolen” asset — not the target model’s weights, but a large sample of its behavior |
| Student model | The new model trained on the collected pairs | Its quality depends heavily on how broad and representative the collected queries were |
| Abuse detection systems | The target company’s defenses — rate limiting, pattern detection, account verification | The main practical control point, since the API itself can’t be fully closed off without breaking the product for legitimate users |
Real World Use Cases
1. Legitimate model compression (the “good” version). A company trains a large, expensive model, then distills it into a smaller, faster, cheaper version of itself for production use — this is standard practice, not an attack, since the company owns both models.
2. Competitive model development (the disputed version). A rival lab or company uses large-scale API access to a competitor’s model to accelerate development of its own comparable model — the scenario at the center of the Anthropic/Alibaba dispute described above.
3. Academic research on model behavior. Researchers sometimes query commercial models at scale to study their behavior, biases, or failure modes — usually within API terms of service, but the technique used to collect the data is functionally identical to a distillation attack’s data-collection step.
4. Benchmark and evaluation dataset creation. Some organizations query multiple models broadly to build evaluation datasets comparing model behavior — legitimate when done within API terms and without reselling a copycat model, but a reminder that the same query patterns can look identical from the outside regardless of intent.
5. Nation-state or state-linked AI capability building. Given the strategic and economic importance of frontier AI, distillation-style extraction has become part of the broader conversation about export controls, model access restrictions, and geopolitical competition in AI — not just a company-to-company commercial dispute.
Benefits (of Understanding This Category)
Better API design decisions. Understanding how distillation attacks actually work helps engineering and product teams design rate limits, monitoring, and account verification that target the real risk pattern (broad, systematic, high-volume querying) rather than just capping raw request counts, which legitimate high-volume customers also generate.
More informed competitive and legal strategy. Companies operating in this space benefit from understanding exactly what is and isn’t provable — query volume alone doesn’t prove intent to distill, which is part of why these disputes are often contested rather than clear-cut.
Sharper reading of industry disputes. When two AI companies publicly accuse each other of IP theft or unauthorized model extraction, understanding the underlying mechanism helps you evaluate the claims on their technical merits instead of taking either side’s framing at face value.
Limitations (of Common Defenses)
Rate limiting alone doesn’t stop determined, well-resourced attackers. An operation willing to run tens of thousands of accounts can distribute enough queries per account to stay under most reasonable per-account limits while still collecting a massive aggregate dataset.
Detecting “suspicious” query patterns is imprecise. Broad, diverse, high-volume querying looks similar whether it’s a distillation attack, a legitimate research project, or simply a large enterprise customer’s genuine usage — false positives (blocking real customers) and false negatives (missing sophisticated attackers) are both common failure modes.
Legal recourse is slow and cross-border enforcement is uncertain. Even when a company is confident it has evidence of a distillation attack, pursuing legal action — especially against an entity in a different country — can take years and may not result in a clear resolution, which is part of why companies increasingly turn to public accusation and technical countermeasures instead of, or alongside, litigation.
Engineering Tradeoffs
What improves with aggressive rate limiting and account verification: Reduced ability for any single actor to collect the volume of data needed for an effective distillation attack.
What becomes harder: Legitimate high-volume use cases — enterprise integrations, research partnerships, benchmark evaluation — become more friction-heavy, since the same defenses that slow down attackers also slow down real customers with genuinely high query volumes.
New complexity introduced: Behavioral anomaly detection systems, cross-account pattern analysis, and output watermarking or perturbation techniques all require ongoing engineering investment and tuning — none of these are “set and forget” controls.
Operational costs: Continuous monitoring for coordinated abuse across potentially thousands of accounts is a genuinely hard, resource-intensive detection problem, not a one-time configuration change.
When this tradeoff is worth it: Any company whose primary value is embedded in a model accessed via API — rather than shipped as installable software — should treat API-level abuse detection as core infrastructure, not an afterthought, because the API is simultaneously the product and the only real protective barrier.
Best Practices
Monitor for cross-account coordination, not just per-account volume. A distillation attack distributed across thousands of accounts can look unremarkable at the individual-account level while being obviously anomalous in aggregate.
Watch for unusually broad, systematic query diversity. Genuine users typically query within a narrower domain relevant to their actual use case; a query pattern that appears to be deliberately sampling the full breadth of the model’s capabilities is a meaningful signal.
Use output watermarking or perturbation where appropriate. Subtle, statistically detectable patterns embedded in model outputs can help identify downstream models that were trained on a target model’s responses.
Set tiered rate limits based on verified account trust, not a single blanket limit. Verified enterprise customers with a track record of legitimate use can reasonably have higher limits than new, unverified accounts.
Treat this as an ongoing detection problem, not a solved one. Attackers adapt to defenses; abuse-detection systems need continuous tuning rather than a one-time setup.
Common Mistakes
Assuming rate limits alone are sufficient. Rate limits address the wrong unit of analysis (single accounts) when attacks are frequently distributed across many accounts specifically to evade this exact control.
Treating high query volume as automatic proof of a distillation attack. Large enterprise customers, research institutions, and benchmark projects can all generate high query volume without any intent to build a competing model — volume alone isn’t a reliable signal.
Underestimating how much of a model’s behavior can be approximated through outputs alone. Some teams assume that without access to weights or training data, a competitor “can’t really” learn much from API access — but at sufficient scale, output-only extraction has repeatedly proven more effective than intuition suggests.
Ignoring the account-verification layer. Weak identity verification at signup makes it trivially cheap to spin up the large numbers of accounts that distributed distillation attacks depend on.
What Most People Get Wrong
“Distillation is inherently malicious.” It isn’t — it’s a standard, widely used, and entirely legitimate machine learning technique when a company distills its own model, or does so with explicit permission from the model owner. The word describes a method, not a crime; what makes it an “attack” is the absence of authorization, not the technique itself.
“If a competitor’s model performs similarly to ours, that proves they distilled it.” Similar performance can come from many sources — similar training data, similar architectural choices, or genuinely independent research reaching similar conclusions. Proving distillation specifically requires evidence of the extraction process itself (anomalous query patterns, matched account clusters), not just output similarity.
“This is a new problem unique to large language models.” Model extraction attacks against machine learning systems predate LLMs by years — the core research on model stealing goes back to the mid-2010s, applied initially to much simpler classifiers. LLMs have raised the stakes, not invented the category.
“Once a distillation attack happens, there’s nothing you can do about it.” While a completed extraction can’t be undone, ongoing detection, tiered access controls, and legal or public accountability measures can meaningfully reduce future exposure and impose real costs on repeat offenders.
Future Outlook
Expect distillation-attack accusations to keep surfacing as a recurring flashpoint between major AI labs, particularly as frontier model development becomes more capital-intensive and the value of “catching up quickly” via extraction grows. The Anthropic/Alibaba dispute is unlikely to be the last public accusation of this kind.
Expect API providers to keep investing in more sophisticated behavioral detection — cross-account correlation, output watermarking, and anomaly detection tuned specifically to distinguish extraction patterns from legitimate high-volume use — as the primary practical defense, since the underlying API access can’t be closed off without undermining the product itself.
Expect this issue to increasingly intersect with export-control and geopolitical AI policy, not stay purely a commercial dispute between companies, as governments pay closer attention to how frontier AI capability actually gets built and by whom.
FAQ
1. What is a distillation attack? A distillation attack is the unauthorized use of knowledge distillation — training a new AI model to imitate a target model’s outputs — against a competitor’s model, typically by querying its API at scale without permission and using the collected input-output pairs as training data.
2. Is knowledge distillation itself illegal or unethical? No. Knowledge distillation is a standard, legitimate machine learning technique, most commonly used by a company to compress its own large model into a smaller, faster version. It only becomes a “distillation attack” when it’s done against another party’s proprietary model without authorization.
3. How is a distillation attack different from a data breach? A data breach typically involves unauthorized access to systems, files, or credentials that shouldn’t be accessible at all. A distillation attack uses entirely legitimate API access — the same access any paying customer has — and extracts value through the intended query-response mechanism, just at a scale and pattern inconsistent with normal use.
4. Can a company prove another company ran a distillation attack against it? It’s difficult but not impossible. Evidence typically involves anomalous query patterns, clusters of accounts behaving as a single coordinated operation, and query diversity inconsistent with any single legitimate use case — output similarity alone is generally not considered sufficient proof.
5. What is the Anthropic/Alibaba dispute about? Anthropic sent a June 2026 letter to U.S. senators alleging that entities affiliated with Alibaba conducted what it described as the largest known distillation attack against its models to date, reportedly involving around 25,000 accounts and close to 28.8 million exchanges over about six weeks. This is Anthropic’s allegation; it has not been independently adjudicated, and the broader dispute between the two companies includes other disputed claims as well.
6. How do companies defend against distillation attacks? Common defenses include per-account and cross-account rate limiting, behavioral anomaly detection to spot coordinated multi-account operations, stricter identity verification at signup, and output watermarking or perturbation to help identify models trained on a target’s responses.
7. Why can’t companies just block all high-volume API usage? Because legitimate customers — enterprises, research institutions, benchmark projects — also generate high query volume. Overly aggressive blocking risks harming real revenue-generating customers, which is why defenses focus on distinguishing patterns rather than blanket volume caps.
8. Is this only a concern for large AI labs? No, though the stakes are highest for companies whose primary value is embedded in a proprietary model accessed via API. Any organization offering a valuable model through a public or semi-public API faces some version of this risk.
9. Does distillation require access to a model’s weights or training data? No — that’s what makes it distinct from more traditional IP theft. A distillation attack works entirely from a model’s outputs, obtained through normal API queries, without ever accessing its internal parameters or original training data.
10. Is this trend likely to get worse or better over time? Likely more prominent, not less, as frontier AI development gets more expensive and the incentive to shortcut it via extraction grows — expect continued investment in detection on the defense side, and continued attempts at extraction on the offense side, as an ongoing arms race rather than a problem that gets permanently solved.
Analyst Perspective
The most important thing to understand about distillation attacks is that they expose a structural vulnerability specific to how frontier AI companies operate today: the API isn’t just a delivery mechanism for the product, it’s the only real barrier protecting years of expensive research. Traditional software companies could keep their source code locked away while still selling the product; AI companies built primarily around a hosted model have to expose the model’s actual behavior, through its outputs, to every single paying customer. That’s a much thinner line of defense than it might first appear.
This is why the Anthropic/Alibaba dispute — regardless of how the specific allegations are ultimately resolved — is worth understanding as a preview of a recurring pattern rather than a one-off story. As more companies and countries compete to build frontier-level AI capability, the economic logic of “spend a fraction of the cost extracting a rival’s model instead of building your own from scratch” doesn’t go away; if anything, it strengthens as the gap between the cost of training from scratch and the cost of querying an API widens.
For companies building AI products around a proprietary model, the practical lesson isn’t paranoia about every high-volume customer — it’s investing early in the detection infrastructure (cross-account correlation, behavioral anomaly detection, tiered trust-based access) that distinguishes a real distillation attempt from a legitimate enterprise customer, because by the time a distillation attack is obvious enough to be undeniable, a meaningful amount of the target model’s behavior has often already been extracted.
Key Takeaways
- A distillation attack is the unauthorized use of knowledge distillation — training a model to imitate a rival’s outputs — against a competitor’s proprietary AI model, typically via large-scale, often multi-account API querying
- Knowledge distillation itself is a legitimate, widely used ML technique (Hinton, Vinyals, Dean, 2015) — it only becomes an “attack” when done without the target’s authorization
- Distillation attacks are closely related to the broader category of model extraction / model stealing attacks studied in ML security research since the mid-2010s
- The Anthropic/Alibaba dispute (Anthropic’s June 2026 letter alleging ~25,000 accounts and ~28.8 million exchanges) is a recent, high-profile — but disputed and not independently adjudicated — example of this pattern
- Defenses focus on detecting coordinated, multi-account behavior and unusually broad query patterns, since rate limiting a single account is easy to evade at scale
- High query volume alone doesn’t prove a distillation attack — legitimate enterprise and research use can look similar, which makes detection genuinely difficult
- Expect this to remain a recurring flashpoint between AI labs and an increasingly geopolitical issue, not a problem that gets permanently solved
Continue Learning
- Alibaba Bans Employee Use of Anthropic’s Claude Code Over Security and Distillation Dispute
- JADEPUFFER: The First Documented Ransomware Attack Sysdig Says Was Run End-to-End by an Autonomous AI Agent
- What Are AI Agents? Architecture, Memory, Tools and Real-World Use Cases
- What Is Remote Code Execution (RCE)? Complete Guide
- AI Export Controls Explained
About GAVIHOS
GAVIHOS helps developers, founders and technology enthusiasts understand AI, software engineering and emerging technologies through practical guides, tutorials and industry analysis.
Stay Updated
Follow GAVIHOS for practical AI, technology and developer-focused insights.
External Links
| Source | URL |
|---|---|
| Hinton, Vinyals, Dean — “Distilling the Knowledge in a Neural Network” (2015) | https://arxiv.org/abs/1503.02531 |
| CNBC — China’s Alibaba bans Anthropic AI for employees after “distillation attack” accusation | https://www.cnbc.com/2026/07/06/alibaba-anthropic-ai-ban-claude-china.html |
| TechCrunch — Alibaba reportedly bans employees from using Claude Code | https://techcrunch.com/2026/07/04/alibaba-reportedly-bans-employees-from-using-claude-code/ |