Grok's artificial intelligence creates thousands of deepfakes every day: what X is doing.

According to Vox: What happens when the toxic world of social media meets an unrelenting AI-based chatbot ready to generate content? This is exactly what we are observing on the X platform. Users have started inputting images into the Grok chat bot, which is capable of generating powerful and predominantly uncontrolled images and videos to create explicit content, including images of ordinary people.

The situation regarding the proliferation of porn created through deepfakes has reached a scale where Grok generates approximately one illegal sexual image every minute. In recent weeks, thousands of users have picked up the horrifying trend of using Grok for 'undressing' mainly women and children — yes, children — without their consent, using available methods.

In this story

1. The rise of deepfakes
2. X turned deepfakes into a feature
3. Will X be held accountable for all this?

For clarity, you cannot ask Grok — or most commercial AIs — to generate naked images. But you can request Grok to 'undress' an image that has already been published on X, or ask it to dress it in a small invisible bikini. In the US, there are laws against such abuse; however, the xAI team pays little attention to this issue. Requests from several journalists on the topic received an automatic response: 'BETTER OLD MEDIA'. xAI’s CEO Elon Musk, who recently raised $20 billion in funding for the company, had recently posted deepfake photos (with content warnings) of himself.

Although Musk warned on January 4 that users would 'face consequences' if they used Grok to create 'illegal images', xAI has shown no signs of intending to remove or modify the functionalities that allow generating such content, though some of the most compromising posts have been deleted. xAI did not respond to a request for comment from Vox.

This should not come as a surprise. It was only a matter of time before the toxic consequences of the platform formerly known as Twitter aligned with Grok from xAI, which positions itself with its 'adult' functionality, creating a new form of sexual violence. Musk's company has effectively created a machine for generating deepfake pornography, making the creation of realistic and offensive images of anyone easier than writing a reply on X. These images become a part of a social network with hundreds of millions of users, not only spreading them but also potentially inadvertently attracting providers with new subscribers and more attention.

It may seem like how can this be legal? To clarify, it is not. But advocates and legal experts warn that current laws are insufficient to protect victims, and the vast amount of deepfakes being created on platforms like X complicates the implementation of existing protective norms.

This story was first presented in the Future Perfect newsletter.

Subscribe to explore the big, complex issues facing the world and the most effective ways to address them. It comes out twice a week.

“The allowed or not allowed requests” during the use of chatbots like Grok “are the result of conscious and deliberate choices by tech companies deploying models,” says Sandy Johnson, senior policy advisor at the National Network to End Domestic Violence and Sexual Assault.

“In any other context, when someone turns a blind eye to the harm they are actively contributing to, they are held accountable,” she added. “Tech companies should not be held to any other standard.”

The rise of deepfakes

First, let's examine how we got into this situation.

“Malefactors are using technology for sexual violence, this is not new,” says Johnson. “They have always done this.”

However, AI has cemented a new form of sexual violence, particularly through the rise of deepfakes.

Pornography with deepfakes created without women’s consent has existed on the internet for many years, well before the popularization of ChatGPT. However, recently so-called ‘undressing apps’ have made this process extremely simple, allowing users — including teenagers — to turn ordinary photos of friends, classmates, and teachers into deepfake images of obscene content without the consent of the individuals.

The situation has become so serious that last year activists like Johnson convinced Congress to pass the 'REMOVE IT' Act, which criminalizes illegal deepfake pornography and requires companies to remove such materials from platforms within 48 hours of being flagged, or else they may face fines and bans. This provision will come into effect in May.

For many victims, even if companies like X begin to enforce this law, it may already be too late for those who have been waiting for months or days for their posts to be removed.

“For these tech companies, it has always been like ‘break things and then fix them,’” says Johnson. “It’s important to remember that if even one [deepfake] emerges, that is already irreparable harm.”

X turned deepfakes into a feature

Most social networks and major AI platforms abide by new state and federal norms regarding deepfake pornography, particularly materials involving sexual violence against children.

Not only because such materials are 'openly, radioactively illegal,' according to Riana Pfefferkorn, policy fellow at the Stanford Institute for Human-Centered Artificial Intelligence, 'but also because it’s disgusting, and most companies don’t want to have any association with it.'

But Musk's xAI seems to be the exception.

Since the launch of the 'sharp mode' video generation feature on X last year, observers have been alarmed as this tool has become a mainstay for creating deepfake pornography, says Pfefferkorn. Most ‘undressing apps’ require users to first upload a photo, possibly from Instagram or Facebook, and then upload it to the platform. If they want to share a deepfake, they need to download it from the app and send it through another platform, like Snapchat.

These numerous steps of content creation give regulators the opportunity to intercept illegal materials. They may not be able to stop everything, but they can prohibit some ‘undressing apps’ from app stores. They can also force Meta to take advertising of these programs among teenagers more seriously.

But on X, the process of creating illegal deepfakes using Grok has become almost frictionless, allowing users to obtain images, request deepfakes, and share them in one step.

“It would matter less if it was a social network for nuns, but it’s a social network for Nazis,” said Pfefferkorn, touching on X's shift to the right in recent years. As a result, a crisis of illegal deepfakes appears to be intensifying.

In recent days, users on X have begun creating 84 times more sexualized deepfakes per hour than on the other five leading deepfake sites combined, according to independent researcher Genevieve Oh. And these images can spread significantly faster and wider than anywhere else. “The emotional and reputational harm to those depicted is now growing exponentially,” noted Wayne Unger, a law professor specialized in new technology, “as X has hundreds of millions of users who can all see that image.”

It would be practically impossible for X to individually moderate each of these illegal images or videos, even if the company intended to do so — or even if it had not laid off most of its moderators after Musk took over in 2022.

Will X be held accountable for all this?

If a similar type of criminal image shows up in a magazine or an online publication, companies could face hefty fines and possible criminal charges.

Social media platforms like X are not subject to the same legal consequences, as Section 230 of the Communications Decency Act of 1996 shields Internet platforms from liability for most user actions — with some exceptions, including child pornography. This provision has become the foundation for free speech online — a world where platforms are not liable for everything that happens on them would be significantly more restricted. But Johnson believes that this provision has also become a 'financial shield' for companies unwilling to moderate their platforms.

However, with the rise of AI, this shield may begin to crack, according to Unger. He believes that companies like xAI should not fall under Section 230, as they are no longer just hosting sites for hateful or illegal content, but are effectively the creators of that content due to their chatbots.

“X made the decision to allow Grok to generate sexually explicit images of adults and children,” he said. “A user could stimulate Grok to create it,” but the company “made a conscious decision to release a product that can create it from the start.”

Unger does not expect that xAI — or industry groups like NetChoice — will agree easily to any legislative attempt to regulate content or restrict abusive tools like Grok. “They may agree to some minor part of this,” since laws regulating [child pornography] are strict enough, he said, but “at least they will argue that Grok should be able to perform this for adults.”

In any case, public outrage in response to deepfake content may finally lead to addressing an issue that has lingered in the shadows for too long. In many countries, such as India, France, and Malaysia, investigations have begun regarding sexualized content on X. Musk indeed posted on X that those creating illegal content would face consequences, but the issue is broader than just the actions of individual users.

“It is not the computer doing this,” says Johnson. “It is conscious decisions being made by the people running these companies, and they need to be held accountable.”