China’s newly released draft rules on human-like interactive AI mark one of the most direct regulatory responses globally to the rapid rise of anthropomorphic technologies, such as chatbots, and the growing unease over their effects on data security and users’ mental health.
The draft is ambitious. It seeks to construct an overarching framework for governing a fast-moving technology that blurs the line between tool and companion, while acknowledging the tension between encouraging innovation and containing novel risks. Article 3 frames the effort as a search for balance: promoting development while preventing abuse and maintaining control.
Yet for all its good intentions and some strong protections, the draft is frustratingly vague regarding some specific procedures, and may even risk creating new dangers in the name of protection. A notable example is requiring service providers to profile users based on their interactions with the AI and other available data, both to identify minors and to assess behavioral or psychological risk. These assessments can result in reporting outside the chat, and even direct manual takeover of the interaction. Without clear limits, this approach could deepen surveillance, expand data collection, and expose the very users the rules are meant to safeguard to new forms of harm.
I. Scope and Terminology II. Protections General Provider obligations Prohibited Service Conduct Mental Health Impact Risk Identification Risk Responses Special Protections for Vulnerable Populations: Minors and Elderly Data Security Training Data User Data III. Additional Protections and Provider Responsibilities IV. Enforcement V. Annotated Translation Chart
I. Scope and Terminology:
Human-like interactive AI services: Article 2 sets the scope of the new draft by defining the covered technology. The coverage is far-reaching, clearly meant to include current chatbots and devices in which they are embedded, as well as future technologies with similar functions. The key traits are that the AI products or services:
- Simulates human traits in terms of personality, thinking, communication, and emotional interaction.
- Interacts with human users through text, images, audio, video, etc.
The definition of relevant service providers (article 30) clarifies that the draft covers both individuals and organizations providing such services.
Artificial Intelligence is not a term defined in the new draft, but previous regulations have defined Generative AI as “models and relevant technologies that have the ability to generate content such as texts, images, audio, or video.” The new drafts seemingly address a subset of that category.
Emotional Companionship Services: A few articles in the new draft single out this category of human-like interactive services, but don’t offer a clear definition. Article 18, for example, provides that exiting such a service must be easy to limit excessive dependence. Article 12 provides special requirements for when these services are provided to minors.
This term will either need to be defined or have such special requirements expanded to cover all human-like AI services.
Safety/Security: For clarity in the discussion, it is important to note that the word 安全 （anquan）is used to mean both safety and security in Chinese. There is a discussion to be had regarding whether those English words reflect concepts so distinct that they should be distinguished more clearly in any language, but for now understand that both meanings are included whenever either English word is used.
II. Protections
A. General Provider Obligations
Service providers have the primary responsibility for the security of their AI services (art. 8). They are responsible for security throughout the entire lifecycle of the products and services, from design to implementation, and including all updates and revisions.. Updates in security and functions are to be concurrently updated, so that there is no lag in protection for new features. (article 9)
The ‘security’ mentioned here is wide-ranging and touches on other laws; it includes: data security, pre-publication content reviews, cybersecurity, personal information protection, and fraud prevention. (Art. 8).
Special security assessments, focusing on user traits, and the implementation of safety features must be conducted and reported to provincial authorities when services introduce new functions or cause major changes to existing functions, and achieve high user volumes (1M or more registered/ 100,000 or more monthly active users). Security assessments may also be triggered by a possible impact on national security or the public interest, or undefined security shortfalls. (articles 21,22).
B. Prohibited Service Conduct
Article 7 contains a non-exclusive list of prohibited conduct for human-like interactive AIs, laying a fundamental bottom line on AI conduct.
The first several items in the list relate to content restrictions, generally incorporating China’s broader online content controls into this new context. Content that is punishable under existing laws must not be generated or disseminated by the AI services.
This includes content endangering national security, harming the national honor and interests, conducting illegal religious activities, disrupting order, and undermining ethnic unity, as well as promoting gambling, obscenity, encouraging crimes, and conduct that impacts others’ rights, such as defamation. (art. 7 (1)-(3)).
While there are plenty of legitimate criticisms of China’s content control regime, this type of restriction is not new, and has been consistent across recent AI regulations, so it is not the focus of discussion here.
The article 7 list, however, also includes several vague items more specific to the human-like AIs. These require that AIs not make false promises that impact user conduct, or include services that harm social relations. They require that suicide not be encouraged, glamorized, and forbid verbal violence and emotional manipulation, as well as algorithmic manipulation, misleading information, and emotional traps.
These all sound like bad things that should be avoided, but unless they are more clearly defined, it will be difficult for service providers to understand and implement the prohibitions.
The final article 7 prohibition relates to data security, requiring that the AI not deceptively acquire information related to state secrets or sensitive information. While these categories of information sound familiar from China’s Law on State Secrets and Personal Information Protection Law, it isn’t entirely clear what conduct is prohibited here, if any, beyond those laws’ requirements.
C. Mental Health Impact
The user impacts that the draft seems most concerned with include excessive dependency or addiction to the services and extreme emotional responses or mental health emergencies, such as risks of suicide and self-harm (but notably not harm to others). (art.9)
To prevent these risks, a number of clear preventative and response measures are required. For example, Article 16 requires conspicuous notices that the chat is with an AI and not a human being. More dynamic reminders, such as pop-up window alerts, are required when users first begin using a service or newly log-in. Similarly, reminders to take a break must also be displayed every 2 hours. (art. 17)
The risks involved, however, are not limited to threats to health, and include even risks to property. (article 11,13). This might envision risks of a user losing assets to fraud or destruction, but feels distinct enough that future revisions should consider separating out responses to this less critical risk, especially given the consequences of a risk being identified.
Risk Identification
As mentioned above, the draft requires that providers have the capacity to identify user statuses- specifically, assessing extreme emotions and potential undue dependence, and intervening when they are detected. (art. 11)
The draft does not specify whether these determinations should be made through human review or automated systems, yet either approach raises serious concerns. Most users do not expect their interactions to be reviewed by human operators for any purpose, and permitting such review would constitute a profound intrusion into user privacy. Although the draft explicitly requires that privacy be protected during this identification process, it offers no guidance on how that goal is to be achieved in practice.
If the identification mechanism is automated, privacy is more easily maintained, but it requires reliance on the AI service that may be the source of the harm. Such reliance invites both false positives and false negatives: cases in which the system mistakenly flags benign behavior as high-risk, or, more troublingly, fails to detect genuine risks altogether.
Because the draft requires provider action and invasive interruptions in user experience when risks are detected, more detail is needed for the risk identification procedures.
One possibility would be for AI to conduct an initial assessment and pass potential risks to a professional response team after anonymizing the data. This would allow human experts to consider chat content without learning the users’ identity, before making a decision on whether a further response is required. The draft does require service providers to have relevant staff available. (art. 9).
Not all service providers will have the capacity to establish such systems, and this general procedure does not fully eliminate risks, but it is worth exploring.
Risk Responses
Different levels of risk call for different responses, further underscoring the need for clarity about the required precision and standards in risk identification.
Where a “tendency” towards addiction or dependency is detected, for example, pop-up windows and other more aggressive means are to be used to remind users that they are chatting with a non-human, etc. (art. 16).
In crisis situations, such as users’ extreme emotional states, clear expressions of intent to harm themselves, the response can be more drastic. (art. 11p3). First, the service is to offer users prepared template responses offering comfort and encouraging the user to seek help. This may seem familiar from US products that quickly provide information on suicide hotlines, in response to some user prompts.
Second, a human being will take over the chat. The instinct towards crisis intervention is clearly well-intentioned, but the sudden appearance of a human operator in what was understood to be a private, automated discussion of sensitive emotional content could aggravate the situation rather than de-escalate it. It may even be viewed by users as a betrayal of confidence.
Even the existence of a mechanism for human operators to enter the chat might make users wary about the services, fearing abuse. For this type of intervention to be successful, clear notice of the standards and conditions for such an intervention should be announced to users and service providers in advance.
There will also need to be rules for how and when an intervening human operator announces their presence to the user.
Special Protections for Vulnerable Populations: Minors and the Elderly
The draft has special rules for the protection of minors and the elderly.
First, companionship for elderly users is specifically listed as a potential future usage scenario for AIs (art. 6); so it is appropriate to consider whether that population is uniquely vulnerable.
That said, the elderly are not children, and age should not be taken as a proxy for mental capacity. Having special requirements for seniors, as opposed to offering them special accommodations and assistance, is not always appropriate.
Consider Article 13’s call for elderly users to provide emergency contact persons who will be contacted as part of crisis response. That option should be available to all adult users, not just the elderly, and should not be forced on anyone with full mental capacity.
Similarly, a special prohibition on providing elderly users with services imitating their family members or other specified individuals, presumably to avoid fraud or confusion, could be presented as an option to all competent users.
Where users lack full mental capacity, due to age, illness, or any other reason, there are legal procedures for appointing a guardian. And guardians can be consulted in those circumstances. Otherwise, special barriers should not be placed in any user’s path.
Minors, on the other hand, lack full capacity by law. China has already built an extensive system of special legal protections for minors online [Previews], much of which is referenced in the draft.
Article 12, for example, calls for establishing Minors Modes for the AI services. Existing guidelines on minors modes for networked products call for mutual recognition across different devices, software, and platforms, so that guardians can globally activate minors mode protections with a single click.
Such rules empower guardians, giving them final decision-making authority, but set default standards for usage times and permissions based on different age brackets.
Minors Mode rules in the new draft continue to empower guardians, but make no distinctions for different-aged minors. This is in contrast with some current practices of some companies that suggest children below a certain age not use human-like AI at all, and have increased oversight for older children. ( See e.g. OPEN AI FAQ).
The rules require that guardians be granted authority to toggle Minors Mode, and have settings for usage time limits, and in-service ‘reality checks’ (reminders that the AI is a non-human). Their consent is required for collecting minors’ data and providing it to third parties (PIPL), and they may exercise other user rights on behalf of the minor, such as requesting the deletion of information collected in interactions. (art. 14).
For ‘emotional companionship’ services, guardians have additional authority. First, they must expressly agree to the services. Then they must be able to receive real-time risk notifications and summaries of the minors’ usage, to block specific characters and roles in the interactions, and limit the minors’ usage time, adding funds, and spending.
All of these concepts, however, remain vague and need refinement to be operational. What standard determines when real-time risk alerts are issued? Is the content of chats included in summaries of minors’ usage, or only duration?
Identification of Minors
Outside of China, determining who is a minor remains a contentious issue, as it requires all users, including adults, to verify their age. Increasingly, this means uploading identification documents, creating an increased risk of harmful data breaches, and making true anonymity online impossible.
China’s internet already operates on a real-name registration system, effectively requiring all users to link their national ID to most online accounts. Even this drastic measure has not proved fully effective in blocking minors’ access to improper online content; however, as kids have learned to borrow parental accounts (with and without permission), rent other adults’ accounts, and fool ID systems. (see eg: https://finance.sina.com.cn/tech/digi/2025-12-18/doc-inhcfepi6674554.shtml)
The draft human-like AI rules go further, requiring the AI services to have the capacity to identify minors. (art. 12). Such algorithmic detection of minors was hinted at previously in the Ministry of Culture and Tourism’s Opinions on Strengthening the Protection of Minors in the Online Cultural Market (at 4), but this seems to be the first proposal to fully implement the idea.
Profiling user conduct in service interaction clearly impacts the privacy of all users. In recognition of this, a draft version of rules on Generative AI once precluded profiling based on user input- but that requirement was not ultimately adopted. (Art. 11).
The human-like AI draft says that privacy must be protected in profiling for minors, but, as with profiling to identify risky behaviors, it is difficult to know how this will occur without detailed procedures and standards.
Moreover, the consequence of being identified as a suspected minor is being automatically placed in Minor’s Mode. Appeals process for restoration of full services are also required, but again, no clear standards are provided.
D. Data Security
Training Data: All training data, for pretraining or optimization, is regulated by Article 10 of the draft. The requirements overlap somewhat with previous provisions on Generative AI training data (art.7) in requiring legal and diverse sources, but continue primarily by emphasizing safety.
Training data should be cleaned and labeled to increase transparency and avoid contamination or tampering, if synthesized data is recycled as training data, its safety must be assessed. Negative sampling and adversarial sampling techniques must be used to avoid unintended or inappropriate content associations.
Naturally, training data must also conform to content laws, and reflect the ‘core socialist values’ and Chinese culture- something no doubt considered a safety issue as well.
Importantly, interaction records and sensitive personal information (identifying information that would likely cause harm if disclosed) must also not be used in training models without clear legal authorization or user consent. (art. 15)
User Data
While the provisions of the Personal Information Protection Law and Data Security Law contain the basic framework for collecting, sharing, and processing user data, the Draft makes a few key provisions as well.
Article 15, for example, calls for annual compliance audits of the handling of minors’ information, a rule in accordance with article 37 of the Regulation for the Online Protection of Minors.
Providers must use tools to protect the content of users’ interactions with AI, including through encryption, security audits, and access controls. (art.14). These protections are similar to privacy requirements for user inputs in other AI rules (see art. 11), but suffer from a similar defect- requirements that they monitor and report interactions deemed unlawful. (art. 23, and profiling requirements above).
Users, or their guardians when applicable, may request that data be deleted at any time. (art. 14).
III.Additional Protections and Provider Responsibilities
Sandboxes Article 27 calls for the establishment of sandbox platforms for testing security and innovations.
Complaint Mechanisms- Procedures for providers’ acceptance and addressing of complaints are in Article 22.
Algorithm Filings: Required by Article 25 In accordance with existing rules.
IV.Enforcement (Articles 26, 28, 29)
Provincial-level internet information departments (the organs of the Cybersecurity Administration of China) are to conduct annual document-based audits of service providers’ compliance. Where providers have not adequately performed required security assessments, they may be asked to do so again, and the departments may carry out on-site inspections.
These provincial-level departments and other regulatory departments can call in the legal representatives and other leadership of service providers where they find security risks or security incidents. (art. 28). At these meetings, they can guide corrections of and action to eliminate threats.
Providers are to cooperate and provide necessary support and assistance for departments carrying out inspections in accordance with law. Previously, similar language requiring cooperation in Cybersecurity Law article 28, has been viewed abroad as meaning that companies are required to act as agents of Chinese authorities. Here it seems clear that the requirement is limited to assisting in the enforcement of these measures, but this does include rules related to national security threats, so I suspect those concerns will continue regarding human-like AIs.
Generally, penalties are left for other laws to define, but article 29 does allow for administrative penalties where no other laws apply. Violations of these measures are generally to be met with warnings, criticism, or orders to make corrections; but services may be ordered paused where corrections aren’t made or there are other serious circumstances.
Loading...
Reload document
Open in new tab 
Jeremy Daum is a Senior Fellow of the Yale Law School Paul Tsai China Center, based in Beijing, with over a decade of experience working in China on collaborative legal reform projects. His principal research focus is criminal procedure law, with a particular emphasis on protections of vulnerable populations such as juveniles and the mentally ill in the criminal justice system, and is also an authority on China’s ‘Social Credit System’. Jeremy has spoken about these issues at universities throughout China and in the U.S.; and has co-authored a book on U.S. Capital Punishment Jurisprudence for Chinese readers. He is also the founder and contributing editor of the collaborative translation and commentary site Chinalawtranslate.com, dedicated to improving mutual understanding between legal professionals in China and abroad.
He translates, writes, edits, does web-design, graphic design, billing, tech support, and social media outreach for China Law Translate.
- Jeremy Daum
- Jeremy Daum
- Jeremy Daum
- Jeremy Daum
English
中文(简体)
Be First to Comment