There’s been a lot of discussion recently about the GenAI giants “watermarking” their output, which to say hiding some data in it so you can tell that it’s been generated by ChatGPT or Gemini or Claude or whatever. The technology for watermarking images is as old as dirt, works as advertised, and doesn’t affect image quality. What’s new and controversial is a claim that they can watermark text output cheaply, simply, and without harming the quality of the text.

I was curious how they did this so I dug in a bit. I’m mostly convinced that it works and is pretty clever. However, all the explanations are full of Computer-Science jargon that you need to be a professional developer to understand. So here’s an attempt to explain to non-insiders and then, indirectly, opinions on whether it’s good and what it means and doesn’t mean.

How it works · [To professionals: The following is vastly oversimplified but not, I think, misleading.]

LLM software generates output a word at a time. The way it works is, it takes the last few words it’s generated and feeds them to the Large-Language-Model process, and what comes out is a list of plausible next words, and a number which represents just how plausible they are. So if the last few words were “my favorite tropical fruit is” the LLM might come back with “mango”:30, “lychee”:20, “papaya”:10.

You might think that LLMs always choose the most-plausible next word, but that turns out to give lousy results. So they randomly select from the list in a way that respects those numbers. First, note that the numbers add up to 60. So in this case, it’d select “mango” 30 out of 60 attempts, i.e. half the time, “lychee” 20/60 or a third, and “papaya” a 10/60 or a sixth.

In that sentence, the word “randomly” is doing a lot of work. Let’s look a little closer at one way you might do this. Your computer program could make a list with 60 entries, fill 30 of them with “mango”, 20 with “lychee”, and 10 with “papaya”. Then you’d ask your computer to give you a random number between 1 and 60. Suppose you got 33, so you pick the 33^rd list entry and return “lychee”.

And in that sentence, the phrase “ask your computer” is hard-working. It turns out that computer programmers need random numbers like this all the time (let’s ignore why). In many situations, they use a built-in function called a “PRNG” (let’s ignore what that stands for) to get those numbers. The way a PRNG works is you fire one up by giving it a “seed”, which can simply be another number that you get by bashing randomly at the top row on your keyboard or by asking your deskmate to pick a number or whatever.

Once seeded, the PRNG is happy to go on giving you an endless series of random numbers. But (this is important) they depend on the seed. By which I mean that if you fire up a second PRNG with the same seed, it’ll produce exactly the same endless series.

Now we’re almost there. If you think about it, the LLM’s next word depends on, and only on:

1. The previous few words,

2. The internal workings of the LLM, and

3. The random number used to select from the list of candidate words. Which in turn depends on the seed that the LLM fired up its PRNG with.

So, suppose you want to tests whether some text was generated by an LLM. Well, IF you have access to the LLM, and IF you know how its PRNG works, and IF you know the PRNG’s seed, then you can look at the LLM output and see if the words are the ones that the LLM would have generated. This works even if the output text was edited a bit; if a high proportion of the words are those that the LLM would pick given the preceding ones and the PRNG output, that’s a signal that yes, it was probably LLM-generated.

Secrecy · This only works if you know the details of how the candidate next-words are generated, how the PRNG works, and what seed was used. It would be technically possible for an LLM provider to share this information, then anyone could check the watermark.

I don’t watch the GenAI sector that closely so I don’t know if any of the providers actually do disclose this stuff. My impression is that generally speaking they’ll provide an API to do this checking for you. Or not, if they decide you’re not entitled to do that, or haven’t paid enough, or whatever.

Resources · Here (on X, unfortunately) is an explanation that is slightly less oversimplified than mine above; still, I think, inaccessible to non-professionals, but perhaps useful to some.

Scalable watermarking for identifying large language model outputs is the upstream science from Nature in October 2024. Strong stuff.

AI Text Watermarking Is Free And Good, by Zvi Mowshowitz, also offers a how-it-works, but then includes commentary around the “… And Good” part of that title. I don’t agree with some of it but the arguments are useful to know and quite clearly written. I’m going to steal a phrase from Mowshowitz’s piece to serve as the title of my next section…

Anthropic Derangement Syndrome · Quite a few people, some very smart, have reacted to this watermarking idea with fear and loathing. Mowshowitz scoffs at them. He (correctly) observes that the hostility is in large part due to people assuming that anything the AI vendors say is probably a lie and that anything they do is probably greedy and dangerous bubble fodder. But he (incorrectly) opines that this take is unreasonable.

So sue me: I think that that business sector lies like crazy, that their product vision is damaging to civil society, and that they are inflating what is starting to smell like a truly apocalyptic financial bubble.

Having said that · I do think the watermarking can work as described and won’t slow things down and won’t degrade the quality of the output.

Sometime in the last year I gave up on making predictions about the impact of GenAI on the world. So don’t ask me what the effect of ubiquitous watermarking will be.