As the rest of the world struggles with how to best address the flood of degrading images generated by Elon Musk’s Grok AI, China sentenced two operators of a Chinese “AI companion” service to prison for producing obscene materials after their chatbot engaged in sexually explicit conversations with users.
In a case that could have major consequences for China’s AI industry, the developers of Alien Chat are now asking a Shanghai court to overturn their convictions. Beyond the salacious nature of the sexualized chats involved here, the case raises important questions about legal responsibility for all AI-generated content.
This article quickly examines China’s existing laws on obscenity crimes and its newer AI regulations to tease out what is genuinely new in the case. In particular, it focuses on the issues that have been highlighted in Chinese reporting and commentary:
- Authorship and Responsibility: Who made and spread the obscene content: The users entering prompts, the AI model that generated it, or the developers and service providers who made the model available?
- Public Harm: Where is the social harm in private, sexual chats between adult users and a machine, which can justify legal intervention?
- Regulation and Criminal Liability: How do regulations on AI services relate to the Criminal Law’s obscenity offenses?
Existing Law
OBSCENITY
Obscenity is just one type of prohibited content in China’s well-known censorship regime.
Under the Criminal Law, it is a crime to produce, reproduce, publish, sell, or disseminate obscene materials for profit. (article 363). Even without a profit motive, dissemination can still be illegal if the quantity of materials or the size of the audience meets specified numeric thresholds. (article 364).
“Obscene materials” are defined as including prurient (lust-inciting, lewd) materials that either (1) explicitly depict sexual activity or (2) blatantly promote pornography. An exception is left for works with scientific, medical, literary, and artistic value. On its face, this definition resembles the laws of many other countries, including the US, but is understood in practice to include most pornography.
Originally, the Criminal Law focused on traditional media (books, periodicals, film, audio-video, and images) but subsequent guidance in 2004 and 2010 has clarified how the crimes apply to online conduct involving obscene electronic information.
In determining whether a crime has been committed and evaluating its seriousness, the law considers the actual harmful consequences, illicit gains, the volume of content involved, and the reach or scale of dissemination, with specific quantitative thresholds. Non-criminal administrative public order penalties may be imposed when the thresholds for establishing a crime aren’t met.
AI REGULATION
China now has three major overlapping regulations directed at AI-related services: one governing algorithmic recommendation, one governing deep synthesis, and one governing generative AI. All three impose duties on providers to address illegal content, including obscenity.
- The Algorithmic Recommendation Provisions largely incorporate China’s broader content-governance regime into the new tech context: providers must stop the transmission of illegal content, preserve relevant records, and report it to the relevant authorities. (article 9)
- The Deep-Synthesis Provisions go further by requiring providers to strengthen monitoring and reviews of both user inputs and generated outputs for illegal content, including obscene materials. (article 10)
- The Generative AI Measures explicitly prohibit the generation of illegal content and emphasize provider obligations to prevent and address such outputs. (article 4)
Even more importantly for the Alien Chat case, the Generative AI Measures also state that providers shall bear responsibility as online information content producers in accordance with law. (article 9). That might sound like it answers one of the core questions: who is the producer of generated content, but unfortunately, it’s not quite that easy.
The Generative AI Measures are administrative rules that set oversight and compliance obligations, as well as enforcement responsibility, but they can’t, on their own, interpret the scope of a criminal offense. While the Measures were jointly issued by a formidable lineup of concerned agencies[1], the institutions that ultimately define and apply criminal liability—the legislature, courts, and procuratorate weren’t among them.
Criminal Liability for knowingly assisting obscenity crimes.
In related contexts, China’s courts and procuratorate have previously considered the criminal liability of those who play less direct roles in the creation and dissemination of obscene materials.
For example, a 2004 Interpretation on online obscenity crimes provides that liability as a joint offender extends to those who knowingly support others who are criminally producing, reproducing, or disseminating obscene electronic information. This support can include providing technical services such as internet access, web hosting, fee settlements, or data storage. (article 7). A 2010 follow-up interpretation added that even those knowingly advertising on obscene websites could be considered joint offenders for providing funding. (article 7).
That interpretation further provides that the owners or managers of websites could be found guilty of disseminating obscene material for profit if they knowingly allow others to publish such content on their sites in sufficient volumes. (article 4). Similarly, telecommunication businesses and internet service providers can be convicted for disseminating obscene materials for profit if they knowingly provided services to obscene websites. (article 10). These past interpretations show that liability can extend even beyond the direct creators of illegal content to anyone who knowingly facilitates its production or dissemination.
Prosecutors alleged that Alien Chat’s developers intentionally altered their service to reliably provide sexually explicit responses, but, under the logic of the earlier interpretations, it might not even be necessary to prove that. Even knowingly facilitating a model’s use used to generate obscene materials could well be sufficient; and China’s three existing AI regulations require that service providers remain actively aware of how users engage their products.
What’s the Harm
While comparing the Alien Chat case to the Grok controversy is appealing, the situations are different in important ways, including the nature of the harm involved.
- The Grok scandal primarily involves the publicly-visible generation and manipulation of images that often sexualizes real he people, sometimes with the intent to harass or humiliate them.
- China’s “Alien Chat” case concerns mainly text-based erotic chats with purely virtual avatars, seen only by individual users and for their own gratification.
Under China’s law, the Grok situation would implicate “cyberviolence” law as well as normal obscenity offenses. Cyberviolence offenses have identifiable victims and more directly attributable harms. The non-criminal harassment offense in the Public Security Administration Punishments Law lists disseminating obscene images as one means of harassment. (article 50).
It can be harder to articulate the specific harm in more general obscenity controls. Almost every nation has some form of obscenity law, and, like China’s, they are usually based vaguely on the protection of public morality and emphasizing the harm to minors.
In the Alien Chat case, the offending chats were solely between [2] individual adults and a machine, leaving some to ask what the harm is. These fenced-off chats have little chance of polluting the online ecosystem, no children were involved, nor was anyone coerced or enticed into performing to create the chats. The court seemingly focused on the broader impact of the chatbot across multiple users, rather than individual users.
The AI itself as Obscene materials
The first court to hear the case seems to have assumed that the obscene material involved were the records of users’ explicit chats. This approach explains some of the confusion in identifying the problematic content’s creators and harm: the software developers weren’t ‘present’ when the chats were created, and the societal harm only occurs when we look at the reach in terms of the software, not the individual users.
An alternative approach is to consider the chatbot itself as an interactive, immersive obscene work disseminated by the developers. After all, users do not distinguish the chats from the App, they are records of engagement with the app not works commissioned from the app. The first-instance courts’ judgment document hasn’t been released publicly, but from what is known, this approach is a better fit with the court’s overall reasoning.
Consider its emphasis on the defendants’ allegedly tweaking the model to actively guide chats towards pornographic content, regardless of user inputs. It was noted that 141 of 150 paying users sampled by investigations had obscene chats. As discussed above, this was introduced to show that the developers had knowledge of how the software was used and a role in the production of the obscene chats. What it could show even more directly is that the chatbot was altered to become an obscene work that required minimal (“press any key for porn”) engagement for a user to activate.
Generative AI is a unique technology in that it empowers users to easily make “original” works with minimal skill, cost, or effort. The accused developers here would argue that their service resembles a video camera or paintbrush – a neutral tool that could be used to create obscene or non-obscene content. If it was truly predisposed to generate obscene dialog by design, however, that argument falls apart, the user is doing little more than running the program.
The court also indicated that the correct charge was “producing” rather than “disseminating” obscene material. This may have been a way of avoiding the awkwardness of the chats being private, and not really disseminated beyond the individual user. [3] It may also have reflected that the service being provided intuitively feels more active than mere distribution. In either case the problem is resolved by treating the chatbot itself to be the obscene material: that work is clearly produced by defendants and, as the court highlighted, reached a large number of paying subscribers.
So why didn’t the court address the work itself as obscene material? It may be that the AI’s code doesn’t seem ‘obscene’ in itself, while the text records are traditional obscene material. The code doesn’t depict sexual acts (until activated) and isn’t blatant pornography- it requires some engagement to do that. It may be that the court hoped to avoid the thorny issue of determining when a model capable of generating both obscene and non-obscene content becomes a obscene itself: when 100% of content is obscene? 51%? This issue could probably be avoided by emphasizing intent.
Conclusion:
Defining and punishing obscenity is a thorny issue in any legal system, particularly where only consenting adults are involved. With an AI-generated chatbot the issue is slightly harder, because the offense is even more private.
China’s law on obscenity is strict. Obscene AI chatbots are a new medium that adds a few new wrinkles, but based on existing criminal law, it seems clear that creation of an explicit chatbot is within the scope of conduct that the law intends to punish and deter. The law, as clarified by the interpretations targets those who knowingly use the prurient appeal of sexually explicit materials to draw in traffic and profit.
Determining what material is ‘obscene’ is a separate and harder issue, as is whether jail time should ever result for purveying purely fictional obscene materials. China at least seems to agree that the individual users, consumers of the obscene materials, don’t need to be punished.
[1] Cybersecurity Administration of China, National Development and Reform Commission, Ministry of Education, Ministry of Science and Technology, Ministry of Industry and Information Technology, Ministry of Public Security, and the State Administration for Radio and Television
[2] Naturally, adult users could share chats, and one user has been separately charged for sharing a chat character he created that was also used for explicit chat purposes. See: https://mp.weixin.qq.com/s/9ZOaPVF1l1L1WOEamDvOUw
[3] We recently discussed in a different context that the letter of the law probably reached this one-to-one dissemination, but is not enforced this way in practice.
CLT 是一个能够让说英语的人群进一步了解中国法律，众包、众筹的法律翻译项目。
- Jeremy Daum
- Jeremy Daum
- Jeremy Daum
- Jeremy Daum
English
中文(简体)
Be First to Comment