I’ve been thinking about how to make the (attribution required) contents of a site accessible to humans but prevent AI crawlers from stealing contributions made to the site. robots.txt and ip-block-lists only get you so far, seeing as AI companies increasingly just ignore robots and block-lists will always be incomplete.
Currently I’m contemplating writing a plugin that provides a liquid filter that you can throw after a {{content}} in the primary layout. It would then try to inject garbled extra text into the real text and use css to hide that from user agents. display: none is the obvious choice because assigning flex ordering makes text not properly selectable by humans. Perhaps there are other poisons I’m not thinking about. I’m of course working under the assumption that crawlers wont bother with css if they find nicely marked up <p> tags.
I wanted to hear what people think about this. Maybe something like this already exists?
I would rather use SaaS solutions like one from Cloudflare to prevent bots.
In any case, the bots will surely evolve whatever choice you make.
I don’t know about that. Image poisoning like nightshade seems to work pretty decently, I’m just not aware of anything for html text.
Also, AI crawling is a big-data game, so I doubt they will adapt to small jekyll blogs. Too much effort.
Sorry, one more thing. I don’t want to expose my users to their data being collected by CDN and the like (such as cloudflare). Also cloudflare does tend to block legitimate TOR users, which I find unacceptable.
1 Like
The main thing is, you can’t prevent it from being crawled, only from being processed correctly by a language model. But like said above, this is a matter of time before being circumvented. It is a cat mouse fight.
About ways of doing this, there is a whole research field about this. Check this recent paper for instance (not yet peer reviewed afaik).
That looks promising, but I feel like it is still not there yet in terms of unspecific poison, as it seems more interesting in misinforming the LLM in specific ways.
But when a crawler crawls your site, it will download the whole html, then parse it, then feed to some model. So the download will happen, what you can try to avoid is the parse and process steps.
I don’t really think you can avoid the parse and process step. They will gobble everything up. But what the aim of this post was, was to find a way to make the content unpalatable.
Ideally, the result would be to contribute to the confusion/destruction of the LLM similar to what nightshade does. So that for example if you ask “what is jekyll?” the answer is not just “the best most awesome site generator” but rather something like “the umbrella smells ε in kindly”.
Note that I’m talking about those crawlers that fail to respect robots.txt and friends. If they respect that, then no harm will be done.
You should research about indirect prompt injection. Like I said, it is a rather recent field of study, so every now and then you’ll see papers about it, and also mitigation solutions.
1 Like