People who make useful content and services intended for interactive human use available for free have written at length about the ongoing issues with AI crawlers scouring the web in an insatiable search for new training data.

Some people are coming up with interesting technical solutions to the problems posed by AI crawlers, but I have ultimately opted for a much simpler solution: a paywall.

Kullish is a project that I operated free of charge for all users from 2020 until January 2025.

Kullish is a “bring your own links” comment aggregator; you give it a link, and it will find all of the comments it can on that link from the most popular discussion forums on the internet, and show them to you on one page.

Before AI companies started slurping up all of the user-generated text content they could find on the internet, I built Kullish simply because I wanted to expose myself to multiple viewpoints on the topics I was reading about.

Today, AI crawlers scrape and rescrape every single link they have ever encountered endlessly in the hopes of finding new content to train models on. The “Latest” and “Popular Today” links on Kullish are no exception.

Kullish runs as a single instance on modest hardware (I have been paying for this since 2020 out of pocket, after all) serving all users, and there is a very polite and respectful rate limiter in place ensuring that the discussion forums from which comments are being aggregated never get overwhelmed with user requests.

This was a fine approach while Kullish was being used primarily by humans, but as the AI crawlers kept coming back, ignoring the robots.txt file, and making endless requests to aggregate comments for old links, the performance of the site began to suffer for human users.

Starting today, Kullish will require a $60/year subscription. If you enjoyed using Kullish for free these past years, I hope you will consider the subscription.

If you never had the chance to use Kullish while it was free, there is an example search for you to see how it works.