Rifan Muazin
For the better part of two decades, the publishing industry has lived under the hegemony of Google Search. SEO (Search Engine Optimization) became the lifeblood of digital media, dictating everything from headline structures to the placement of images. However, as generative AI rapidly reshapes how users consume information, the industry is experiencing a seismic shift. The new goal is no longer just optimizing for human eyeballs or Google’s algorithm—it is optimizing for the "agentic web."
At the forefront of this movement is a controversial strategy: the creation of markdown versions of websites. Publishers like Time are stripping away the glossy, JavaScript-heavy interfaces that define the modern web, opting for a clean, text-heavy format designed specifically for AI bots to consume. But as this trend gains momentum, a fierce debate has erupted over whether markdown is a strategic necessity or a tactical misstep that compromises control and revenue.
The Core Concept: Designing for the Machine
The traditional webpage is a bloated creature. Built to satisfy human users and advertisers, modern sites are laden with high-resolution imagery, tracking pixels, dynamic JavaScript, and complex CSS layouts. To a human, this provides a rich, engaging experience. To an AI crawler, however, this is digital noise.
Markdown is a lightweight markup language that renders content in a plain-text format. By serving these "clean" versions to AI agents, publishers aim to minimize the computational friction bots face when parsing a site. The hypothesis is simple: if an AI model can ingest a publisher’s content with greater accuracy and less processing power, that publisher’s content is more likely to be cited—and potentially prioritized—in AI-generated responses.
Proponents argue that by removing the "human fluff," publishers create a direct pipeline for their information to flow into the large language models (LLMs) that are increasingly acting as the primary gateways to the internet.
Chronology of a Shift: From Search to Agents
The rise of markdown as a strategy is the latest chapter in the evolution of digital discovery.
- The SEO Era (2005–2022): The industry focus remained squarely on Google’s ranking algorithms. Strategies revolved around backlinks, keyword density, and site speed, all within the context of HTML-based web design.
- The AI Disruption (Late 2022–2024): With the arrival of ChatGPT and similar tools, publishers began to realize their content was being ingested by black-box models. The initial reaction was defensive, with many sites blocking bots using
robots.txtfiles. - The Optimization Phase (2025–Present): Realizing that blocking bots might lead to total irrelevance in the AI-driven search ecosystem, forward-thinking publishers began experimenting with "AI-friendly" formats. This led to the adoption of markdown, exemplified by Time’s recent initiatives to serve these formats to crawlers.
Supporting Data: Efficiency and Tokens
The push for markdown is heavily rooted in the "token economy." AI systems function by processing text in chunks known as tokens. Because HTML contains thousands of lines of metadata, scripts, and layout instructions, it is incredibly expensive and slow for an AI to parse compared to a clean markdown file.
Toshit Panigrahi, CEO of TollBit—a marketplace facilitating the exchange of data between publishers and AI companies—notes that converting a standard webpage to markdown can result in an average 90% reduction in tokens.
"In terms of markdown, it makes it friendlier; the inference costs drop," Panigrahi explains. "Crawl budgets last longer. AI can comprehend more of your article because they’re not spending money parsing out other HTML that’s on the page."
Conversely, the data on actual visibility remains murky. A study by the GEO platform Promptwatch analyzed over 1.6 million citations in AI answers and found no correlation between the use of markdown and an increase in content citations. This suggests that while markdown may be technically superior for the bot, it does not guarantee a seat at the table in the AI’s final output.
The Case Against: The "Leaking" Problem
Critics, however, argue that the pursuit of markdown is fundamentally flawed. Ed Zyszkowski, CEO of Personal Digital Spaces, frames the issue as a loss of ownership.
"When you take a website, you convert it to markdown, and you offer it to OpenAI or to Claude, it now has that data," Zyszkowski warns. "You’ve now lost control. Essentially, you’ve leaked your information into that model."
Furthermore, there is the issue of monetization. Publishers are currently struggling to see a direct return on investment for these efforts. Choy Travers, co-founder of Oasy, points out that for most publishers, the immediate concern is not being "bot-friendly," but finding sustainable revenue streams. "I don’t think it’s going to move the needle in the short term," Travers says. "There’s no proof there yet."
The argument that markdown will lead to more ad revenue—by allowing ads to be served directly to bots—is also under fire. While Time has begun serving ads to AI agents, early testing by firms like Oasy suggests that these ads are not currently delivering the win rates or performance metrics that justify the technical overhead.
Official Responses and Industry Stance
The skepticism toward markdown is bolstered by the world’s most powerful search entity: Google. In its official Generative Engine Optimization (GEO) guidelines, Google explicitly states that markdown is not required. The company’s search ranking systems, which drive its generative AI features, are built to interpret standard web pages.
Google clarified its stance by noting that while publishers are free to maintain LLMS.txt or markdown files, doing so will neither "harm nor help" a site’s ranking in Google Search. This effectively positions markdown as a specialized tool for third-party AI labs rather than a silver bullet for search visibility.
Implications for the Future of Media
The debate over markdown is symptomatic of a deeper existential crisis in the publishing industry. Publishers are effectively being asked to gamble on a future that is still being written.
1. The Death of the "One-Size-Fits-All" Web
We are moving toward a bifurcated internet. There will be the "human web," rich in multimedia and advertising, and the "machine web," stripped, clean, and optimized for consumption by silicon-based agents. This will likely force publishers to maintain dual infrastructures, increasing operational costs.
2. The Power Balance of Licensing
Visibility is leverage. If publishers can prove that their content is essential to the accuracy of an AI model, they gain bargaining power in future licensing deals. Markdown, if nothing else, makes the process of ingestion so efficient that it forces AI companies to acknowledge the value of the source data.
3. The Shift to "Agentic" Advertising
The experiment of serving ads to bots—as seen with Time—is in its infancy. While it currently lacks the high ROI of human-facing ads, it represents a new potential revenue stream. If AI agents eventually facilitate commerce, the ability to serve "ads" (or sponsored suggestions) directly into the agent’s memory will become a highly coveted piece of digital real estate.
4. Regional Disparities
Data from TollBit indicates a massive divide in how AI interacts with the web globally. European sites are experiencing significantly higher bot-to-human traffic ratios than their North American counterparts. As regulation (such as the EU AI Act) continues to evolve, these scraping patterns will likely dictate how publishers in different regions approach their technical architecture.
Conclusion: A Calculated Risk
Is markdown the solution to the publisher’s AI problem? For now, the answer appears to be "no." It is not a guaranteed path to higher rankings, nor is it a proven revenue generator. However, it is a significant bet on the future of the agentic web.
Publishers like Time are betting that by making themselves the most accessible data source for AI, they will become indispensable to the systems that define the next generation of information retrieval. Meanwhile, skeptics are cautioning that by making the house easier to enter, publishers might find their most valuable asset—their content—being stripped, repurposed, and monetized by the very machines they are trying to court.
As the industry moves forward, the focus will likely shift from merely "being crawled" to "being compensated." Whether markdown facilitates that compensation remains the defining question of the next several years in digital media. For now, it remains a high-effort, high-uncertainty strategy that separates the industry pioneers from the wait-and-see pragmatists.
Layla Zulfa
Muslim
Asro
Laily UPN
Iffa Jayyana
Raul Delapena Setiawan
Siti Muinah