110. Context Window & Tokens Explained: Why Chatbots “Forget” (and How to Fix It)

Last Updated: September 13, 2026

If you have ever had an AI chatbot “forget” something you told it earlier in a long conversation, you have encountered the context window limit firsthand. The context window is one of the most important — and most misunderstood — concepts in practical AI use. It determines how much information an AI model can hold in working memory at any given moment, what it can refer back to during a conversation, and how large a document you can ask it to analyse in one go. In 2026, context windows have expanded dramatically: thirteen models now ship 1M+ token windows, and the largest — Meta’s Llama 4 Scout and Google’s Gemini 3.1 Pro — reach 10 million tokens. But the advertised window size has become the least useful number in the spec sheet. What actually matters is effective context: how far into a long input quality actually holds before the model starts to miss things.

This guide explains context windows and tokens from first principles, then takes you through everything that has changed in 2026: the new model landscape, the critical distinction between advertised and effective context, how to choose the right window for your actual workload, what your token budget buys in real-world content, and how to manage context when documents or agent sessions exceed what any window can handle reliably. Whether you are a developer selecting a model for a production deployment, a business professional trying to understand why your AI assistant seems to lose track of long conversations, or a technical leader building a cost model for AI infrastructure, this guide gives you the complete picture.

One token is roughly 0.75 words in English, or about 4 characters. A 128,000-token context window holds approximately 96,000 words — roughly the length of a short novel. A 1-million-token window holds approximately 750,000 words — about 1,400 pages of text. These numbers sound enormous, but understanding what models actually do with that space — what they reliably attend to, what they miss, and where quality begins to degrade — is the knowledge that separates effective AI users from those who are surprised when a model fails on a long document. This guide covers all of it.

🧠 1. What Is a Context Window? The Plain-English Explanation

A context window is the total amount of text — measured in tokens — that an AI language model can process at one time. Think of it as the model’s working memory. Everything inside the context window is what the model can “see” and reason about when generating a response. Everything outside it is invisible — as if it never existed.

The context window includes everything that is happening in a given session: your system prompt (the instructions that configure the model’s behaviour), the entire conversation history going back to the start of the session, any documents or data you have pasted in, and the response the model is currently generating. All of it counts against the same limit. When the total reaches the context window limit, the model can no longer hold the earliest parts of the conversation and will either stop, truncate, or begin to “forget” earlier content — depending on how the application managing the session handles overflow.

This is why a chatbot might answer a question perfectly at the start of a long conversation and then give a contradictory or incomplete answer an hour later: the context window filled up, and the earlier context that contained the relevant information dropped out. The model is not being inconsistent — it genuinely cannot access information that is no longer in its active context.

The Working Memory Analogy: A context window works like human working memory — the mental “scratchpad” that holds information you are actively using. Just as you can only hold a limited amount in working memory at once, an AI model can only reason about what is currently in its context window. Everything else has to be retrieved, re-entered, or reconstructed.

🔤 2. What Is a Token? Understanding the Unit of AI Text

Tokens are the fundamental unit that AI language models use to process text. A model does not read words the way a human does — it reads tokens. A token is roughly 0.75 words in English, or about 4 characters. In practice, common short words are often a single token (“the”, “is”, “a”), while longer or less common words may be split into multiple tokens (“unbelievable” might become “un”, “believ”, “able” — three tokens). Numbers, punctuation, and code follow different tokenisation patterns and often produce more tokens per character than plain English text.

The tokenisation process matters for three practical reasons. First, token counts determine cost — most AI providers charge per input token and per output token, so understanding your token usage is essential for managing AI infrastructure costs. Second, token counts determine what fits in a context window — a document that appears to be a manageable length might tokenise to significantly more tokens than expected, particularly if it contains code, structured data, tables, or non-English text. Third, different models use different tokenisers — the same document will produce a different token count depending on the model you use, which means you cannot assume a word count translates directly to a token count without checking.

For English text, the rule of thumb is 1 token ≈ 0.75 words ≈ 4 characters. For code, expect roughly 1 token per 2–3 characters of actual code. For languages with non-Latin scripts — Chinese, Japanese, Korean, Arabic — a single character may require 2–4 tokens, meaning non-English text is significantly more expensive and uses context space faster than English of equivalent semantic content.

Every major model provider offers a tokeniser tool for counting tokens before you commit to a request. OpenAI’s tiktoken is available as a Python library and web interface. Anthropic provides a tokeniser for Claude models. Google provides tokenisation tools for Gemini. Using these tools before designing a production workflow saves expensive surprises — particularly for workloads where prompt size is variable or where you are working close to context window limits.

📊 3. The 2026 Context Window Landscape — What Has Changed

Context window sizes in mid-2026 range from 128,000 tokens for mainstream models to 10 million for Meta’s Llama 4 Scout and Google’s Gemini 3.1 Pro. Two years ago, a 1-million-token context window was a remarkable technical achievement. In mid-2026, it is table stakes — thirteen models now ship 1M+ token windows including Claude Fable 5, Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4, MiniMax M3, and Qwen3.5-Plus. The 1-million-token context window is the megapixel race of LLMs: every spec sheet has one, almost no one’s job actually uses it, and the models that score highest at 1M often trail at 128K — which is where 95% of real workloads live.

What actually separates frontier models in 2026 is not the advertised window size — it is what they do within that window. Output caps range from 64K to 384K tokens and frequently bind before the input window does. Effective context — the length at which quality actually holds across multi-fact retrieval benchmarks — falls short of the advertised maximum on every model ever benchmarked. And cost varies by up to 71x for the same advertised window size: filling the same 1M window costs $0.14 on DeepSeek V4 Flash and $10.00 on Claude Fable 5.

Long-context surcharges are a 2026 development that catches many teams off-guard when they move from testing to production. Some models charge a premium rate for requests that exceed certain token thresholds. GPT-5.5 charges 2x input pricing for prompts over 272K tokens. Gemini 2.5 Pro charges $2.50 per million input tokens — instead of the standard $1.25 — for prompts over 200K tokens. Claude Opus 4.6 and Sonnet 4.6 charge standard rates across the full 1M window with no long-context surcharge. For workloads regularly exceeding 200K tokens, the difference between a model with and without a long-context surcharge can be the difference between a viable and unviable cost structure.

The headline figure — 1M tokens, 10M tokens — is the least useful thing you can know about a model’s long-context behaviour. The advertised number tells you what fits. It does not tell you what the model attends to, where quality begins to degrade, what retrieval accuracy looks like at 500K versus 100K versus 50K tokens, or what the effective cost is once surcharges are applied at scale. The sections below address each of those questions in order.

📋 4. 2026 Context Window Comparison Table

The table below reflects advertised maximums for major frontier models as of September 2026. Effective context — where quality actually holds on multi-fact retrieval benchmarks — is typically 60–70% of the advertised maximum. Output caps are a separate constraint and frequently bind before the input window does. Always verify current pricing and specifications directly with the model provider before production budgeting, as these figures change with model updates.

Note: Figures reflect advertised maximums as of September 2026. Effective context — where quality actually holds on multi-fact retrieval benchmarks — is typically 60–70% of the advertised maximum. Output caps are a separate constraint from the input context window and frequently bind before the input window does. Always verify current pricing directly with model providers before production budgeting.

🎯 5. Advertised vs Effective Context — The Most Important Distinction in 2026

The single most important thing to understand about context windows in 2026 is that the advertised number and the effective number are not the same — and the gap between them is where most long-context AI failures happen. Every frontier LLM in 2026 advertises a 1M-token context window. Benchmark scores prove that advertised and effective context diverge by 30–60 points for multi-fact retrieval past 200K tokens on most models. The window tells you what fits. It does not tell you what the model attends to.

The practical rule is: effective capacity is roughly 60–70% of the advertised maximum, and the drop-off is rarely gradual. Models typically hold performance until hitting a threshold, then fall sharply. A 1M-window model can start degrading meaningfully at 50K tokens depending on the retrieval task — particularly tasks requiring the model to identify and integrate multiple related facts distributed across a long document. This is the pattern that surprises developers who assume that fitting content within the advertised window guarantees reliable retrieval.

The Lost in the Middle Effect

Research has consistently found that LLMs perform worse at retrieving and reasoning about information in the middle of long contexts compared to information at the beginning and end. This “lost in the middle” effect was documented in a landmark 2023 paper and has been reproduced across multiple model families and generations since. The effect is significant: accuracy on questions requiring information from the middle of a long context can drop by 20–40% compared to equivalent questions where the relevant information appears at the start or end of the input. The practical implication is that for long-document analysis tasks, the position of key information within the document matters — not just whether it fits within the context window.

Attention Dilution

The relationship between context length and accuracy is not linear. More tokens give the model more to work with — but they also introduce noise. This is the phenomenon of attention dilution: as the context grows, the model’s attention is spread across more content, and the signal from any individual passage becomes relatively weaker. The hard problem in 2026 is no longer raw capacity — it is effective context length. It is technically achievable to build a model that accepts 10 million tokens. It is much harder to ensure that model can reliably find one specific piece of information — or five related pieces of information — within those 10 million tokens.

The Counterintuitive Benchmark Finding

The most counterintuitive finding from long-context benchmarking in 2025–2026 is that models performed better on shuffled text than on coherent text in certain long-context retrieval tasks. Coherent text — a well-structured document with logical flow — creates stronger positional patterns. Models develop recency bias in response to those patterns, over-weighting passages near the end of the input and neglecting earlier content. Shuffled text breaks those patterns, paradoxically helping the model attend more evenly across the full input. The practical implication: do not select a model based on its advertised context window size. Select it based on its effective context benchmark performance for your specific retrieval task — single-needle retrieval (find one fact) versus multi-needle retrieval (find and integrate multiple related facts) produce substantially different model rankings.

The Context Window Reality Check: Gemini 3.1 Pro is the only model whose 1M window holds for single-needle retrieval; Claude Opus 4.8 leads multi-needle retrieval at 1M; GPT-5.5 wins single-needle precision at shorter ranges; DeepSeek V4 Pro lands surprisingly close to frontier performance at roughly one-thirteenth the cost. The spec sheet number tells you nothing about which of these is right for your task.

🗂️ 6. How to Choose the Right Context Window for Your Workload

The most important practical insight about context window selection in 2026 is this: choose based on your actual typical context size, not the maximum possible. If your typical prompt is 30,000 tokens, the difference between a 128K and a 1M context window model is irrelevant to your cost and quality — both handle your use case equally, and the 1M model costs more per token for identical results. Bigger windows do not automatically produce better results, and filling a 1M-token window costs anywhere from $0.14 to $30.00 depending on the model. Over-engineering your context window selection is one of the most common and expensive AI infrastructure mistakes in 2026.

Step 1: Measure Your Actual Context Size

Before selecting a model, measure the typical token count of your real workloads — not the theoretical maximum, but the actual median request size in production. Use the tokeniser tool for your target model: tiktoken for OpenAI models, the Claude tokeniser for Anthropic, the Gemini tokeniser for Google. Most enterprise workloads — document Q&A, email drafting, meeting summarisation, code review — operate well within 128K tokens. If your median request is under 50K tokens, a 128K model covers your workload with headroom to spare, at a lower per-token cost than a 1M model.

Step 2: Match Model to Retrieval Pattern

The model ranking for long-context performance depends on what kind of retrieval your task requires. Single-needle retrieval — finding one specific fact in a large document — and multi-needle retrieval — finding and integrating multiple related facts across a large document — produce different model rankings. Gemini 3.1 Pro leads on single-needle retrieval at 1M tokens. Claude Opus 4.8 leads on multi-needle retrieval at 1M tokens. GPT-5.5 wins single-needle precision at shorter context ranges. DeepSeek V4 Pro delivers surprisingly competitive retrieval performance at a fraction of the cost. Know your retrieval pattern before selecting a model — the spec sheet ranking by context window size is not the same as the benchmark ranking by retrieval accuracy at your operating depth. For further reading on how prompt engineering techniques can complement context window selection, see the full guide.

Step 3: Consider Output Caps Separately

Input context and output length are separate constraints that are often confused. A model’s context window is the maximum input it can accept. Its output cap is the maximum response it can generate in a single call. GPT-5.5 has a 1M token context window with a 128K token output cap — meaning it can read a million tokens of input but can only produce 128K tokens of output per response. For tasks generating long outputs — detailed code implementations, comprehensive reports, full document drafts — the output cap may bind before the input window does. Always check both numbers when evaluating a model for output-intensive workloads.

Step 4: Factor in Long-Context Pricing

If your workload regularly exceeds 200K tokens, model pricing changes significantly due to long-context surcharges. Calculate total cost at your expected token volume — not the headline per-million rate — because the effective cost per token changes above the surcharge threshold. A model that appears cheaper at standard rates may become significantly more expensive than a flat-rate competitor once long-context surcharges apply at your operating depth.

📏 7. Token Counting in Practice — What Fits in Each Window

Token counts are abstract until you translate them into content you can visualise. One token is roughly 0.75 words in English, or about 4 characters. With that conversion, the context window sizes that appear in model spec sheets start to become concrete: 128,000 tokens is approximately 96,000 words — about the length of a short novel. 1 million tokens is approximately 750,000 words — roughly 1,400 pages of text, or several full-length books combined. 10 million tokens — the current frontier with Llama 4 Scout and Gemini 3.1 Pro — is approximately 7.5 million words, comparable to a large document corpus or a substantial subset of Wikipedia.

Note: These are English-language approximations using the 0.75 words per token conversion. Code, structured data, JSON, and non-English text tokenise differently — often requiring more tokens per equivalent word. Non-Latin script languages (Chinese, Japanese, Korean, Arabic) can use 2–4 tokens per character, making non-English content significantly more context-intensive. Always tokenise your actual content using the model provider’s tokeniser tool before assuming it fits within a target window.

The token counting equivalents above illustrate why the small language models argument has practical force for many workloads. If your task requires processing a single contract (typically 10K–30K tokens) or a policy document (typically 20K–50K tokens), a smaller model with a 128K context window handles your workload with headroom to spare — at a fraction of the cost of a 1M-window frontier model.

🔧 8. Context Management Strategies — When the Window Is Not Enough

No context window — however large — is the right solution for every workload that involves large volumes of text. In many cases, managing context intelligently produces better results at lower cost than simply selecting a model with a larger window. The five strategies below cover the full range of approaches, from retrieval-based architectures that sidestep the context limit entirely to compression techniques that extend the useful life of a given window in long-running sessions.

Strategy 1: RAG (Retrieval-Augmented Generation)

Instead of loading an entire document library into the context window, RAG retrieves only the relevant chunks at query time and provides those chunks as context. RAG lets you work with arbitrarily large document libraries using a 128K context window model — because only the relevant passages are ever loaded into context. This is the most cost-effective approach for large-scale document Q&A, enterprise knowledge base search, and any workload where the total corpus far exceeds any practical context window. The embeddings and vector databases guide covers the full RAG architecture in detail, including how to structure retrieval for different query types.

Strategy 2: Context Compression

Context compression reduces conversation history size before it fills the context window, allowing longer sessions without hitting the limit. Modern compression approaches achieve 50–70% context reduction while maintaining 98% verbatim accuracy — every surviving sentence stays word-for-word identical to the original. Because compressed history is carried forward on every subsequent call, compacting early in a long agent session cuts total session cost far more than the single-request saving suggests — a 50% compression applied at the start of a 100-call session eliminates half the input tokens across all 100 calls. Tools available in 2026: LLMLingua (Microsoft Research), context compression built into LangChain and LlamaIndex agent frameworks.

Strategy 3: Sliding Window

Process large documents in overlapping chunks, where each chunk overlaps the previous by 10–20% to avoid losing context at boundaries. Sliding window works well for sequential processing tasks — summarisation across a long document, entity extraction across a long transcript, sentiment analysis across a long review corpus. The overlap percentage is the key tuning parameter: too little and boundary context is lost; too much and redundant processing increases cost without quality benefit. A 15% overlap is a practical starting default for most sequential processing tasks.

Strategy 4: Hierarchical Summarisation

Summarise a long document in sections first, then feed the section summaries into a final synthesis prompt. This trades some granular detail for the ability to handle documents of essentially unlimited length — there is no theoretical ceiling because you can always summarise summaries. Hierarchical summarisation works best for tasks where high-level synthesis is the goal: executive summary generation, earnings call analysis, research literature review, policy brief creation. It performs poorly for tasks requiring precise fact retrieval from specific sections — in those cases, RAG is more appropriate.

Strategy 5: Selective Escalation to Large Context Models

Large context models cost more per token and may produce lower quality on shorter contexts than smaller, more focused models. The correct approach is to default to RAG or context compression as the primary architecture, and escalate to a large context model only when retrieval granularity is genuinely insufficient — for example, when the task requires reasoning across an entire document simultaneously rather than retrieving specific passages. Most workloads do not need a 1M context window. Escalating to one when a 128K model with RAG would serve the purpose adds cost without adding quality. Understanding AI temperature and sampling parameters alongside context management gives you the full set of controls for tuning model behaviour on long-context tasks.

🤖 9. Context Windows for Agentic AI — A Different Challenge

Agentic AI introduces context window challenges that chat-based AI does not face. In a single conversation, context accumulates at a predictable rate — you know roughly how many tokens each turn adds. In an agentic workflow, context accumulates unpredictably: every tool call result, every retrieved document chunk, every intermediate reasoning step, every error message from a failed action adds tokens to the running context. A 100-step agent session — routine for complex autonomous workflows — can exhaust even a 1M context window if context is not actively managed from the start of the agent design process.

The three agentic context challenges are distinct from conversational context challenges. Context accumulation happens faster and less predictably than in chat: a single database query tool call that returns thousands of rows can consume tens of thousands of tokens in one step. Tool call verbosity is the primary driver — most tool results are far more verbose than necessary, and loading raw tool results directly into context is an expensive practice that accumulates rapidly. Multi-agent coordination adds a third dimension: in a multi-agent pipeline, each agent maintains its own context window, and the aggregate context cost across all agents can be enormous even when each individual agent is operating well within its window. Understanding function calling and tool use is essential context for managing tool verbosity effectively.

Best practices for agentic context management in 2026 reflect hard-won lessons from production deployments. Implement automatic context summarisation at defined thresholds — for example, triggering compression when 50% of the context window is consumed, not waiting until the window is nearly full. Truncate tool results before adding them to context: extract only the relevant portion of a tool response, not the full raw output. Use RAG for knowledge retrieval within agent workflows rather than loading reference documents directly into agent context. Design agentic workflows to minimise context carry-forward between steps — stateless steps that do not inherit the full previous session context are significantly cheaper to run at scale. The agentic AI architecture guide covers the full design patterns for building context-aware agent systems.

Agentic Context Rule of Thumb: Design your agent to manage context as if the window were 25% of its actual size. This buffer absorbs unexpected tool result verbosity, reasoning chain expansion, and error recovery sequences — all of which consume more tokens in production than in testing. A context management strategy that works in a test environment frequently fails in production when real tool outputs prove far more verbose than synthetic test data.

💰 10. Tokens and Cost — How Context Window Decisions Affect Your AI Budget

Token costs are the primary AI infrastructure cost for most organisations running AI at scale. The relationship between context window size and token cost is direct and significant: you pay for input tokens (everything in the context window on every request) and output tokens (everything the model generates). Both are charged on every single API call. Every token in your context window — including system prompt tokens and conversation history tokens from previous turns — is charged as input on every new request. This means a 10,000-token system prompt added to a high-volume deployment does not cost 10,000 tokens once; it costs 10,000 tokens multiplied by the total number of API calls that include it.

The cost spread for equivalent context window sizes in 2026 is extraordinary: filling the same 1M window costs $0.14 on DeepSeek V4 Flash and $10.00 on Claude Fable 5 — a 71x spread for the same advertised window size. For teams selecting models primarily on capability, this price differential is often an afterthought. For teams running millions of requests per month, it is the primary infrastructure cost decision. Understanding where your workload sits on the cost-quality curve — and whether the incremental quality difference justifies the incremental cost — is the AI budget management question of 2026.

Practical cost management principles: measure your average token count per actual production request, not the theoretical maximum; cache static content such as system prompts and reference documents where the model provider supports prompt caching (Anthropic and Google both offer this in 2026); use context compression for long agentic sessions as described in the previous section; route high-volume, simpler tasks to smaller, cheaper models with narrower context windows; and reserve large context models for tasks that genuinely require both the window size and the model quality.

Figures are illustrative estimates based on published rates as of September 2026. Always verify current pricing directly with the model provider before production budgeting — rates change with model updates and volume discount tiers.

🏁 11. Conclusion — The Context Window in 2026 Is About Quality, Not Size

Context windows have grown from thousands to millions of tokens in three years — a rate of expansion that has outpaced most organisations’ ability to use that capacity effectively. The practical takeaway from everything the 2026 benchmark data shows is consistent: the advertised window size is not the metric that determines whether a model will work for your use case. Effective context, retrieval accuracy at your operating depth, output cap constraints, long-context pricing, and the fit between the retrieval pattern the model excels at and the retrieval task your workload requires — these are the metrics that matter. The 1M-token context window is table stakes in 2026. How well a model uses it is the differentiator.

For most production workloads, the optimal context strategy in 2026 is not to select the largest available context window — it is to measure your actual token usage, choose a model sized for your real workload, and deploy RAG or context compression for the cases where the window genuinely cannot hold what you need. That approach produces better results at lower cost than defaulting to the largest available model. And for the workloads that genuinely require large context — codebase analysis, multi-document synthesis, long agentic sessions — understanding the effective context benchmarks for your specific retrieval pattern is what separates successful deployments from expensive failures. The complete guide to embeddings and vector databases for RAG gives you the architecture to handle any corpus size reliably, regardless of context window limitations.

🔗 Related Articles

• 📖 Embeddings & Vector Databases Explained: The “Secret Engine” Behind AI Search
• 📖 Autonomous AI Agents Explained: How Agentic AI Plans, Acts, and Completes Tasks Without You
• 📖 Prompt Engineering for Non-Programmers: How to Get Better Answers from AI
• 📖 Small Language Models Explained: Why Smaller AI Might Be Better for Your Business
• 📖 Function Calling & Tool Use Explained: How AI Chatbots Actually “Do” Things