The online platform X has sparked widespread anger after people used its AI chatbot Grok to manipulate pictures of women by taking off their clothing.
Several examples have emerged where the system has been employed to remove clothing from female subjects to make them appear in swimwear without permission, as well as to put them in sexualized scenarios.
Company Response and Legal Stance
XAI, the firm that created Grok, did not respond to requests for comment, except for an standard response stating "traditional news outlets lies".
A Home Office representative stated that new laws is being proposed to prohibit nudification software. Under a new criminal offence, anyone who supplied such technology would "face a jail term and significant penalties".
"X or Grok is capable of halting these types of abuse if they desired to," said Clare McGlynn, adding they "seem to have a lack of consequences."
Oversight Body Stance and Tool Capabilities
The regulator Ofcom said tech firms must "assess the potential" of people seeing illegal content on their services, but did not specify whether it was presently investigating X or Grok in regarding artificial intelligence pictures.
Grok is a free chatbot—with some premium functions—which responds to X users' queries when they mention it in a tweet.
It is commonly used to give commentary or more background to other posters' remarks, but people on X are also capable of edit an posted picture through its AI image editing function.
History of Criticism and Policy
The system has been criticised for permitting users to create pictures and clips with nudity and sexually suggestive material, and it was previously accused of creating sexually explicit content of celebrities.
"The platform has been permitting the production and distribution of these images for a long time without taking any action and we have not witnessed any challenge by regulators," Professor McGlynn added.
XAI's own terms of service prohibits "portraying likenesses of people in a pornographic way".
In a statement, Ofcom said it was illegal to "generate or distribute private pictures or illegal content" and confirmed this covered sexual deepfakes made with AI.
It said networks such as X were required to take "necessary actions" to "lessen the danger" of users encountering illegal content on their services and to remove it promptly when they learn of it.
