Earlier this year, sexually explicit images of Taylor Swift were shared repeatedly X. The pictures were almost certainly created with generative-AI tools, demonstrating the ease with which the technology can be put to nefarious ends. This case mirrors many other apparently similar examples, including fake images depicting the arrest of former President Donald Trump, AI-generated images of Black voters who support Trump, and fabricated images of Dr. Anthony Fauci.
There is a tendency for media coverage to focus on the source of this imagery, because generative AI is a novel technology that many people are still trying to wrap their head around. But that fact obscures the reason the images are relevant: They spread on social-media networks.
Facebook, Instagram, TikTok, X, YouTube, and Google Search determine how billions of people experience the internet every day. This fact has not changed in the generative-AI era. In fact, these platforms’ responsibility as gatekeepers is growing more pronounced as it becomes easier for more people to produce text, videos, and images on command. For synthetic media to reach millions of views—as the Swift images did in just hours—they need massive, aggregated networks, which allow them to identify an initial audience and then spread. As the amount of available content grows with the broader use of generative AI, social media’s role as curator will become even more important.
Online platforms are markets for the attention of individual users. A user might be exposed to many, many more posts than he or she possibly has time to see. On Instagram, for example, Meta’s algorithms select from countless pieces of content for each post that is actually surfaced in a user’s feed. With the rise of generative AI, there may be an order of magnitude more potential options for platforms to choose from—meaning the creators of each individual video or image will be competing that much more aggressively for audience time and attention. After all, users won’t have more time to spend even as the volume of content available to them rapidly grows.
So what is likely to happen as generative AI becomes more pervasive? Without big changes, we should expect more cases like the Swift images. But we should also expect more of everything. The change is under way, as a glut of synthetic media is tripping up search engines such as Google. AI tools may lower barriers for content creators by making production quicker and cheaper, but the reality is that most people will struggle even more to be seen on online platforms. Media organizations, for instance, will not have exponentially more news to report even if they embrace AI tools to speed delivery and reduce costs; as a result, their content will take up proportionally less space. Already, a small subset of content receives the overwhelming share of attention: On TikTok and YouTube, for example, the majority of views are concentrated on a very small percentage of uploaded videos. Generative AI may only widen the gulf.
To address these problems, platforms could explicitly change their systems to favor human creators. This sounds simpler than it is, and tech companies are already under fire for their role in deciding who gets attention and who does not. The Supreme Court recently heard a case that will determine whether radical state laws from Florida and Texas can functionally require platforms to treat all content identically, even when that means forcing platforms to actively surface false, low-quality, or otherwise objectionable political material against the wishes of most users. Central to these conflicts is the concept of “free reach,” the supposed right to have your speech promoted by platforms such as YouTube and Facebook, even though there is no such thing as a “neutral” algorithm. Even chronological feeds—which some people advocate for—definitionally prioritize recent content over the preferences of users or any other subjective take on value. The news feeds, “up next” default recommendations, and search results are what make platforms useful.
Platforms’ past responses to similar challenges are not encouraging. Last year, Elon Musk replaced X’s verification system with one that allows anyone to purchase a blue “verification” badge to gain more exposure, dispensing with the blue check mark’s prior primary role of preventing the impersonation of high-profile users. The immediate result was predictable: Opportunistic abuse by influence peddlers and scammers, and a degraded feed for users. My own research suggested that Facebook failed to constrain activity among abusive superusers that weighed heavily in algorithmic promotion. (The company disputed part of this finding.) TikTok places far more emphasis on the viral engagement of specific videos than on account history, making it easier for lower-credibility new accounts to get significant attention.
So what is to be done? There are three possibilities.
First, platforms can reduce their overwhelming focus on engagement (the amount of time and activity users spend per day or month). Whether from regulation or different choices by product leaders, such a change would directly reduce bad incentives to spam and upload low-quality, AI-produced content. Perhaps the simplest way to achieve this is by further prioritizing direct user assessments of content in ranking algorithms. Another would be upranking externally validated creators, such as news sites, and downranking the accounts of abusive users. Other design changes would also help, such as cracking down on spam by imposing stronger rate limits for new users.
Second, we should use public-health tools to regularly assess how digital platforms affect at-risk populations, such as teenagers, and insist on product rollbacks and changes when harms are too substantial. This process would require greater transparency around the product-design experiments that Facebook, TikTok, YouTube, and others are already running—something that would give us insight into how platforms make trade-offs between growth and other goals. Once we have more transparency, experiments can be made to include metrics such as mental-health assessments, among others. Proposed legislation such as the Platform Accountability and Transparency Act, which would allow qualified researchers and academics to access much more platform data in partnership with the National Science Foundation and the Federal Trade Commission, offer an important starting point.
Third, we can consider direct product integration between social-media platforms and large language models—but we should do so with eyes open to the risks. One approach that has garnered attention is a focus on labeling: an assertion that distribution platforms should publicly denote any post created using an LLM. Just last month, Meta indicated that it is moving in this direction, with automated labels for posts it suspects were created with generative-AI tools, as well as incentives for posters to self-disclose whether they used AI to create content. But this is a losing proposition over time. The better LLMs get, the less and less anyone—including platform gatekeepers—will be able to differentiate what is real from what is synthetic. In fact, what we consider “real” will change, just as the use of tools such as Photoshop to airbrush images have been tacitly accepted over time. Of course, the future walled gardens of distribution platforms such as YouTube and Instagram could require content to have a validated provenance, including labels, in order to be easily accessible. It seems certain that some form of this approach will occur on at least some platforms, catering to users who want a more curated user experience. At scale, though, what would this mean? It would mean an even greater emphasis and reliance on the decisions of distribution networks, and even more reliance on their gatekeeping.
These approaches all fall back on a core reality we have experienced over the past decade: In a world of almost infinite production, we might hope for more power in the hands of the consumer. But because of the impossible scale, users actually experience choice paralysis that places real power in the hands of the platform default.
Although there will undoubtedly be attacks that demand urgent attention—by state-created networks of coordinated inauthentic users, by profiteering news-adjacent producers, by leading political candidates—this is not the moment to lose sight of the larger dynamics that are playing out for our attention.
