AI doesn't read everything, and we can't see what it did read
Some models read whole books, some only know them second-hand, and closed models won't say which. That is why open models, which let us check and retrain, matter.
I used to think LLMs learn by reading everything, and that what they know about a book comes from the book. Neither is quite right. What worries me more is that we can't check which is which. I'd change my mind if the closed model builders published what their models were trained on, book by book. None of them do.
Start with the claim that the data is running out. At the NeurIPS conference in December 2024, OpenAI co-founder Ilya Sutskever said pre-training as we know it will end because "we have but one internet", and named synthetic data as one of the ways forward1. Synthetic data means text generated by models to train other models. So the labs are telling us they have used up most of what they can get.
What did they get? My first guess was public material only: websites, papers, forums and news. The court cases say otherwise. In June 2025 US District Judge William Alsup found that Anthropic had downloaded more than 7 million books from pirate sites, and had also bought print books and scanned them2. He ruled that training on books was fair use, but that keeping a library of pirated copies was not2. In July 2026 a court approved a $1.5 billion settlement, about $3,000 per book, for more than 482,000 books3. In a separate case, court filings show Meta staff discussed using copyrighted books obtained through LibGen, a pirate library, to train its models4.
So some models have read the actual books. Researchers from Stanford, Cornell and West Virginia University tested open models for memorised text. They found Meta's Llama 3.1 70B had memorised the first Harry Potter book so thoroughly that they could pull out almost the whole text from its first few words5. That is the popular end of the shelf. The same model memorised only 0.13% of Sandman Slim, a 2009 novel by Richard Kadrey6, and the researchers found that most models don't memorise most books5.
Memorising is not the same as reading, but it is the best evidence we have of what went in. For the long tail of books, I'd say the model probably knows them second-hand, from reviews, summaries and forum threads. It's like a film critic writing a review by summarising everyone else's reviews. The review might be mostly right, and you can't tell which parts are missing or wrong. The model answers both kinds of question with the same confidence.
Then there is post-training. After the main training run, the builder tunes the model on examples of good answers, what to refuse and how to phrase things. In my experience that shapes what you get back as much as the raw data does. With a closed model, we see neither the data mix nor the tuning.
This is why the fight over open models matters to me. In my post on the Huang story I worried that a scandal about one executive would make it easier to argue for limiting open models. Huang has been one of the strongest voices for open weights. With open weights we can test a model, find out what it memorised, fine-tune it and train it further on sources we trust. The original builder still leaves a mark, but we can tune the model to our needs. The Harry Potter study itself only worked because the researchers could run the weights themselves5.
The fair objection is that open models were trained on the same contested data. Llama is open, and Llama is the model in that study. Open weights don't clean the data. They let outsiders measure it, which closed models don't allow, and measuring is the first step to fixing it.
If "government approved" models become the norm, we lose that. We won't be able to correct the models with facts, and we'll be using them on the assumption that someone has already decided what they should think and how. I'd rather keep the right to check.
Nothing here is financial, legal or investment advice. Every figure is published by its named source and carries the date it was published; figures may since have changed.
References
-
The Decoder, OpenAI co-founder says AI is reaching "peak data" as it hits the limits of the internet (December 2024) ↑
-
NPR, In a first-of-its-kind decision, an AI company wins a copyright infringement lawsuit brought by authors (25 June 2025) ↑
-
Associated Press via Fortune, Anthropic to pay authors $1.5 billion over pirated books used to train Claude in first major AI copyright settlement (21 July 2026) ↑
-
TechCrunch, Court filings show Meta staffers discussed using copyrighted content for AI training (21 February 2025) ↑
-
Cooper, Lemley et al., Extracting memorized pieces of (copyrighted) books from open-weight language models (arXiv, first posted 18 May 2025, revised 20 July 2026) ↑
-
Understanding AI, Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book (June 2025) ↑