1. Skip to content
  2. Skip to main menu
  3. Skip to more DW sites
Media

AI crawlers eat the news and publishers pay for it

April 7, 2026

AI expert Afdol Rizki Halim unpacks how bots drain publishers' traffic and revenue, and why fair compensation matters.

Afdol Rizki Halim Portrait
Afdol Rizik Halim is head of research and development at Tempo, a leading media and news company in IndonesiaImage: WAN-IFRA

The text is based on a short talk the author delivered at DW Akademie's conference "The Next next Chapter: AI in Journalism in the age of AI" in Chiang Mai, Thailand, in November 2025.

In the race to build more sophisticated answer engines, AI platforms are hungry for high-quality, real-time data. News outlets, as the primary source of verified information, have become the main target for this "data augmentation" through aggressive web scraping.

However, for media organizations, this isn't just a matter of intellectual property, it's a direct hit to the bottom line. Our internal experiments at Temporeveal a harsh reality: AI platforms are creating a "double hit" for publishers by simultaneously shrinking our readership and driving up our infrastructure costs.

The rising tide of bots

Over the last year, we have observed a significant surge in automated traffic. Various bots ranging from OpenAI to Perplexity and emerging specialized crawlers are hitting our servers with increasing frequency. Google, on the other hand, keeps adding more new bots for AI training, on top of its already existing search crawler (which is responsible for AI Overview).

The data reveals a stark contradiction:

  • Infrastructure stress: The number of server requests and the volume of data transfer have spiked to record levels, increasing by 20-30%.
  • Audience decline: Despite the high server activity, the number of actual human visitors coming to our site is steadily declining, dropping by up to 40%, a percentage which is still growing.

Essentially, we are serving more "content" than ever before, but it is being consumed by machines that never click an ad or subscribe to a newsletter while also diverting attention away from real readers who would.

The invisible infrastructure cost

For most of the website hosting providers, infrastructure costs are tied directly to data transfer and server usage. When an AI crawler scrapes thousands of articles, the publisher pays for the bandwidth and the compute power required to serve those requests.

When we experimented with allowing these bots to crawl without explicit blocks, our infrastructure bills rose by about 20% to satisfy their hunger. This creates a parasitic relationship: Publishers are effectively subsidizing the training of AI models that will eventually replace the need for users to visit the publisher's website.

When "please keep out" is no longer enough

Our testing has confirmed a sobering reality: Standard industry tools are failing to protect journalistic content. The robots.txt file, once the gatekeeper of the web, is frequently bypassed by "stealth" crawlers that mask their identities.

Even network-level blocks are easily circumvented by bots that rotate through thousands of global IP addresses. The result is a lose-lose scenario for publishers: We are serving more data than ever to satisfy machine hunger, while our human audience numbers continue to dwindle. We are paying the bill for our own audience erosion.

A new strategy: fair compensation and collective action

In response to these challenges, Tempo and AMSI(Indonesian Cyber Media Association) are moving beyond passive defense. We are actively seeking a framework for fair compensation from AI providers.

We are currently working on a technical and regulatory implementation through two main pillars:

  • The OpenMined Partnership: We are collaborating with OpenMined, a foundation specializing in privacy-preserving AI. This partnership aims to build a system that provides observability allowing us to track exactly how our data is used while maintaining control over our content.
  • AMSI Collective Action: With over 500 members ranging from national giants to small local outlets, AMSI is spearheading a collective movement. Our goal is to ensure that AI providers pay for the journalistic work they consume.

Lowering the barrier for small media

While large media houses have the IT resources to defend themselves, small and local newsrooms are often defenseless. They face the same infrastructure costs but have "little to no" IT capability to manage complex data-tracking systems.

Our initiative is designed to be as inclusive as possible, making the barrier to entry low so that even the smallest local outlet can join the collective and receive their fair share of compensation.

The road ahead

For the first phase of this implementation, we are targeting a collaboration with Sahabat AI, a local LLM provider. By starting with a local, approachable partner, we can refine our technical framework before scaling our demands to global AI giants.

The goal is clear: Journalistic work has value. If AI companies want to use that value to build their products, they must share the costs, not just the rewards.

Afdol Rizki Halim is the head of research and development at Tempo, a leading media and news company in Indonesia, where he leads the integration of emerging technologies into the media landscape. With extensive experience in software engineering and engineering management, Afdol is dedicated to exploring the intersection of AI and journalism, specifically developing sustainable frameworks for newsroom automation and fair content compensation.

Skip next section DW's Top Story

DW's Top Story

Skip next section More stories from DW

More stories from DW