Early in 2024, the Wikimedia Foundation noticed something odd in its traffic logs. Bandwidth for downloading images and media jumped 50 percent in a single quarter. The surge did not come from human readers. It came from AI companies’ bots: automated programs sent to copy Wikipedia’s content to train chatbots like ChatGPT.
Harvesting pages and images for training is only part of the story. Every major chatbot can now search the web on its own, in real time. Ask one about today’s mortgage rates, and it dispatches a bot to read live pages before it answers. Multiply that by millions of questions a day, and the bot traffic ends up putting a huge demand on the infrastructure of the internet and websites.
Wikipedia is a donor-funded nonprofit. The companies scraping it are worth hundreds of billions of dollars. The Wikimedia Foundation put it plainly: “Our content is free; our infrastructure is not.”
The same imbalance now applies, at smaller scale, to any business or organization with a website, including yours.
The open web has always run on a handshake deal. You publish something useful for free: a menu, an FAQ, a how-to article. In return, search engines send you visitors, and visitors become customers. Nobody signed anything, but the deal held for 25 years because both sides came out ahead.
AI broke the deal. When a chatbot answers a question, it draws on content scraped from sites like yours, and the person asking never clicks through to the source. The restaurant that posted the menu, the law firm that wrote the FAQ, the contractor that published the how-to guide: They pay the hosting bill, while the AI company keeps the customer relationship. DesignRush reports that up to 80 percent of web traffic is now bots, and that bot-driven traffic cost companies $238.7 billion in 2024 alone.
The law offers little help. U.S. courts have ruled that scraping a publicly accessible website is not illegal “hacking,” even after the site owner demands that it stop. The courts compared a public website to a store with an open door: Nobody can break into a space to look around if that space is already open. That seems fair at a human scale — but hugely unfair at the scale of AI.
Every bot request runs up your hosting bill. If you buy digital ads, some of the impressions you pay for are viewed by no one, because a crawler loading a page to copy its text loads the ad pixels, too. And bot traffic pollutes the analytics you use to make decisions. That encouraging spike in page views might be phantom demand: machines reading your site with no intention of ever buying anything.
Fighting back has become an industry of its own. Enterprise bot protection can run from hundreds of thousands to millions of dollars a year. But the companies behind the bots can bypass these protections cheaply by paying workers in low-wage countries to solve CAPTCHAs (those puzzles that prove you are human) for a fraction of a cent apiece.
The predictable result is paywalls, login requirements, and licensing deals for the biggest publishers, while the open web gets a little smaller every quarter. The irony is that AI companies need a steady supply of fresh, human-created content, and their scraping is drying up the incentive to create it.
What can a small business do? First, check your analytics for bot traffic before making decisions based on visitor counts; most platforms can filter it, though imperfectly. Second, expect search referrals to keep declining and invest in channels that do not depend on them: your email list, your community presence, your repeat customers. Third, think deliberately about what you publish openly. Content that exists to bring people in the door still makes sense even if it is read by a bot. Content that would mostly help your competitors — who may be using AI to better position themselves — should be kept private.
This column draws on “Beyond the Prompt: The Social Costs of Generative Artificial Intelligence,” an article Prof. C co-authored with security researcher Josh Rickard, available via open access from the University of Missouri’s Business, Entrepreneurship & Tax Law Review (betr.missouri.edu).
For links, references, or to contact Prof. C, visit profcnews.com.
