Tokens and the context window: why AI "forgets" and how to work with long documents

You have spent an hour working on a proposal with an AI assistant. At the start you told it: "no jargon, the client is not an engineer". Forty messages later, the answer is full of jargon. Or you paste an 80-page report, ask about a figure in chapter 5, and it answers with something from chapter 1.

The model is not tired or having a bad day. It has a physical limit on how much text it can consider at once, and that limit is measured in tokens. In this guide you will see what they are, what takes up that space and how to work with long documents without losing what matters.

What a token is, in one line

A token is the unit the model cuts text into: a short word, a piece of a long word, a number or a punctuation mark. If you want the detail of why models work this way, we explain it in What is a large language model (LLM). Here one idea is enough: everything the model reads and writes is counted in tokens.

As a guide, OpenAI estimates that in English one token is about three quarters of a word. Other languages, such as Spanish, usually need somewhat more tokens to say the same thing, because models are trained on far more English text and their token dictionaries are optimised for it.

What the context window is

The context window is the maximum number of tokens the model can have in front of it in a single request. Think of it as a desk: whatever is on top, the model can see; whatever does not fit on the desk does not exist for it.

What matters is knowing what takes up space on that desk, because it is not just your latest message:

• The app's instructions. Almost every chat has hidden text (the system prompt) with behaviour rules, and it takes up room too.
• The whole conversation history. Your messages and, importantly, the model's answers as well. Long answers fill the desk very quickly.
• The files and text you paste. An 80-page PDF can take up tens of thousands of tokens.
• The answer it is about to write. The space for the answer also comes out of that total.

Window size varies a lot between models and changes with each generation: today it ranges from tens of thousands to hundreds of thousands of tokens, and some models advertise a million or more. Before working with large documents, check the figure in the documentation of the model you use.

What happens when the window fills up

When a conversation goes over the limit, the app has to decide what to leave out. Depending on the product, it may trim the oldest messages, summarise them automatically or simply warn you that the conversation is too long. In the first two cases, that rule you gave at the start ("no jargon") may have disappeared from the desk without you noticing.

The problem in the middle

There is a subtler effect. Even when everything fits, models do not use every part of a long text equally well. A study by researchers from Stanford and other universities, titled "Lost in the Middle" (2023), showed that performance was better when the relevant information was at the beginning or end of the context, and dropped when it was in the middle. Models have improved since then, but the practical lesson still holds: do not assume the model read page 3 and page 43 with the same attention.

How to work with long documents

These techniques need no technical knowledge and work in any assistant.

1. Put the document first and the question last

Anthropic's guide to working with long contexts recommends placing long documents at the start and the query at the end of the message. It is a simple habit: first the material, then the instructions and the specific question.

2. Ask for verbatim quotes before the answer

Instead of "what does the report say about costs?", ask: "Copy word for word the paragraphs of the report that talk about costs and then answer using only those paragraphs". This forces the model to locate the information before interpreting it, and lets you check where each claim comes from.

3. Work section by section

If the document is very long, split it. Ask for a structured summary of each chapter separately, check that each is correct and then work from those summaries. It is slower, but far more reliable than expecting the model to digest 200 pages in one go. The same idea, splitting into fragments, is the basis of the chunking that professional search systems use.

4. Remove what adds nothing

Tables of contents, appendices you do not need, repeated footers, tables on other topics: they all take up tokens and compete for the model's attention. Pasting only the relevant sections usually gives better answers than pasting the whole document "just in case".

How to stop a long conversation from degrading

• Open a new chat with a summary. When you notice the model starting to ignore rules, ask: "Summarise in 10 bullet points the decisions and rules we have agreed". Copy that summary, open a new conversation and paste it at the start.
• Repeat critical rules. If something is essential (the tone, a figure, a legal limit), restate it in the message where it matters, not only at the beginning.
• Ask for shorter answers while you explore. Long answers you will not use fill the window for nothing.
• Use your assistant's memory or project features for permanent instructions instead of repeating them in every chat. We explain how to do it in Claude in Claude skills, connectors and plugins.

More context also costs more

If you use models through an API, every input and output token is billed. Sending a 50,000-token document with every question multiplies the bill even when the question is one line. That is why techniques such as prompt caching, which reuses the processing of a repeated block, and retrieval-augmented generation (RAG), which fetches only the relevant fragments instead of sending everything, exist. If you want to know when it is worth building something like that, we cover it in RAG without the hype.

A prompt for long documents

I am giving you a long document. Before answering:

1. Find and copy word for word the passages that deal with [TOPIC].
   Give the section or page of each one.
2. If you find nothing on the topic, say so and do not fill the gap
   with general knowledge.
3. Then answer the question using only those passages.

Document:
[PASTE THE DOCUMENT HERE]

Question: [YOUR QUESTION]

The question goes last, after the document, for the reason we saw in point 1. For summaries of technical documents you also have the Technical PDF summary template.

Keep reading

• What is a large language model (LLM): how it generates text and why that explains its limits.
• Why AI makes things up: hallucinations and how to reduce them.
• How to summarise long documents with AI without losing what matters: a step-by-step method.
• RAG without the hype: when you need vector search and when you do not.
• Related terms in the glossary: token, context window and chunking.

Sources

• OpenAI Help Center. What are tokens and how to count them?
• Liu, N. F. et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv.
• Anthropic. Long context prompting tips. Claude documentation.