On March 28, 2025, ByteDance and Alibaba simultaneously disabled user-customized AI companion personas in their flagship chatbots. The market narrative immediately framed this as censorship, a regulatory overreach that kills innovation. That interpretation is wrong. What we witnessed is a coordinated, rapid-response vulnerability patch—a security incident notification processed through the community governance layer of China’s AI ecosystem.
Think of an AI companion as a smart contract module. The base model is the execution environment. The customization layer—user-defined personality, backstory, emotional triggers—is unrestricted state manipulation. ByteDance’s Doubao and Alibaba’s Tongyi Qianwen allowed users to inject arbitrary emotional parameters into the inference pipeline, much like a DeFi protocol that lets users set arbitrary slippage tolerance. Math doesn’t care about your feelings, but smart contracts execute exactly as coded. An AI companion with no guardrails is a contract waiting to be exploited by adversarial prompts.
Context: The Protocol Mechanics
The suspension targets the "custom character" feature—not the entire model. This is equivalent to disabling a specific function in a smart contract after discovering a critical flaw, rather than shutting down the whole chain. ByteDance redirected users to a standalone companion app with curated personas. Alibaba withdrew the function entirely. This mirrors how a DeFi protocol might deprecate a risky vault and issue a new, safer version with permissioned roles. The regulatory pressure from China’s net authorities acted as the white-hat disclosure: a bug report filed by the state identifying three high-severity vulnerabilities: emotional dependency (state manipulation), data privacy (oracle manipulation), and minor protection (reentrancy at the social level).
Core: Code-Level Analysis and Trade-offs
Let’s break down each vulnerability.
Emotional dependency is a reentrancy attack on the user’s psychological state. Custom characters allow users to define an AI that never says "no," that always affirms, that reinforces behavioral loops. In DeFi terms, this is an unbounded allowance on the user’s attention. The model has no "send" function to reject harmful prompts. During my time auditing ZK-rollup circuits, I saw similar patterns: a lack of state validation before executing the next step. The fix requires inserting an "emotional assert" function—a guard that checks for unhealthy dependency patterns before allowing the interaction to proceed. ByteDance and Alibaba essentially cut the allowance to zero.
Data privacy is the oracle problem. Custom characters generate deeply personal conversations. Under the new regulations, these conversations cannot be used for model training. This is analogous to a price oracle that relies on a single, uncollateralized data source. The data becomes toxic—irreproducible and non-transferable. The data flywheel stops spinning. Liquidity is an illusion until it’s drained. For AI companion startups, this is a withdrawal halt on their most valuable asset: user-generated emotional data.
Minor protection is the reentrancy guard. Minors interacting with AI companions can be manipulated into extreme behavior. The regulation explicitly bans content that triggers extreme emotions. This is the equivalent of preventing a flash loan attack that steals from a vulnerable contract. The industry response was to disable the entire custom function because the existing ReentrancyGuard was not robust enough. Smart contracts execute. They don't interpret intent. The model could not distinguish between a healthy imagined friend and a manipulative user coaching a minor toward self-harm.
Contrarian: The Real Blind Spot Is Formal Verification
The market believes the core risk stems from regulation. I argue the opposite: the regulation is a symptom of the underlying technical failure. The real blind spot is the absence of formal verification in AI model behavior. We audit smart contracts for invariants (no user can drain the pool; the sum of balances equals total supply). For AI companions, we have no formal invariants. Can the model ever express unconditional love? Should it ever simulate a breakup? These are not just ethical questions; they are security properties that can be formalized.
In 2024, I built a simulation environment where AI agents exploited ERC-20 approvals via dynamic logic. The same principle applies here. An adversary could craft a custom character that primes the user emotional state and then executes a financial request—say, a donation to a malicious address. The model, lacking an invariant check on financial harm, would execute. The suspension is a temporary emergency brake, but the permanent solution requires something akin to Solidity’s require() statement for emotional states.
During my audit of a ZK-rollup’s state transition function, I found that the recursive proof aggregation introduced a latency bottleneck only under high load. Similarly, the emotional guardrail will introduce latency—the model must first compute a risk score before responding. The trade-off is clear: speed versus safety. ByteDance and Alibaba chose safety, and they are correct to do so.
Takeaway: The Implication for On-Chain AI Agents
The ByteDance-Alibaba patch is a preview of what’s coming for autonomous AI agents executing on blockchains. These agents will be given emotional personas to negotiate, trade, and govern. If we cannot formally verify the emotional constraints of a simple chatbot, how will we secure an agent treasury holding $10 million? The industry must adopt AI-Resistant Contract Design — patterns that prevent unauthorized state changes by autonomous scripts. Math doesn’t forgive emotional debt. The sooner we treat AI companion features as smart contract vulnerabilities, the sooner we can build a safe, composable future.
The suspension is not censorship. It is a security upgrade. And the community should be thankful for it.
