113. AI Temperature & Top-P Explained: How to Control the “Randomness” of Your Chatbot

Last Updated: September 13, 2026

Every AI chatbot, coding assistant, and writing tool has at least one setting that controls how “creative” or “predictable” its outputs are. That setting is called temperature — and alongside it sits a closely related parameter called Top-P, also written as top_p or nucleus sampling probability. Together, these two controls determine whether an AI model gives you the same answer every time or surprises you with something new. They are the most frequently discussed and most frequently misunderstood settings in practical AI use, and understanding them properly changes how you configure every AI tool you use.

This guide explains both parameters from first principles. We cover what temperature actually does at the mathematical level — not just the surface analogy — what Top-P is and how nucleus sampling works step by step, how the two parameters interact (and why you should generally tune one at a time), the extended family of sampling parameters including Top-K and beam search, a reality check on which providers still expose these controls in 2026, a copy-paste presets library for eight common use cases, the biggest myth about temperature in AI, and a troubleshooting playbook for the most common output quality problems you will encounter. Whether you are a developer tuning a production system or a professional user trying to get better results from a chatbot, this is the complete 2026 reference.

The sampling parameters covered in this guide work at the output generation stage — after the model has processed your input and built up its internal representations. Understanding how tokens work and how context windows and token limits affect what the model can process in the first place gives you the full picture of how language model generation works end to end.

🌡️ 1. What Is Temperature in AI? The Plain-English Explanation

Temperature is a number — typically between 0 and 2 — that controls how predictable or varied an AI model’s outputs are. A low temperature (closer to 0) makes the model more deterministic: it consistently selects the most probable next word at each step, producing reliable, repetitive, conservative text. A high temperature (closer to 1 or above) makes the model more probabilistic: it gives lower-probability words a better chance of being selected, producing more varied, surprising, and sometimes more creative text.

The word “temperature” comes from physics — specifically from statistical mechanics, where temperature controls the energy distribution of particles in a system. At low temperatures, particles settle into low-energy states predictably. At high temperatures, particles are more energetic and distributed across a wider range of states. AI researchers borrowed this metaphor because the mathematics works the same way: temperature scales the probability distribution over possible next tokens, making it either more “peaked” (low temperature) or more “flat” (high temperature).

In practical terms: if you set temperature to 0 and ask an AI model the same question 10 times, you will get the same answer every time (or very close to it). If you set temperature to 1.0 and ask the same question 10 times, you will get 10 different answers — some better, some worse, some surprising. For tasks where you need consistency and accuracy, lower temperature is usually better. For tasks where you want variety, creativity, or exploration of different approaches, higher temperature opens up the option space.

The Temperature Analogy: Think of temperature like a dial that controls how much the model is willing to “take a risk” on its next word. At zero, it always plays it safe. At high temperature, it is willing to say something unexpected — for better or worse.

🔢 2. What Is Top-P in AI? Nucleus Sampling Explained Simply

Top-P — also called nucleus sampling or the top_p parameter — is a different way of controlling output variety that works alongside temperature. Where temperature reshapes the probability distribution over all possible next tokens, Top-P filters it: it tells the model to only consider the smallest set of top tokens whose combined probability equals or exceeds the value of P, then samples from that set.

For example, if top_p is set to 0.9, the model finds the smallest group of tokens whose probabilities add up to at least 90%, and samples from only those tokens. If the model is very confident about the next word (say, there are only 5 tokens that account for 90% of the probability), it will only consider those 5. If the model is uncertain (say, it needs 200 tokens to reach 90% of the probability), it will consider all 200. This adaptive behaviour is the key advantage of nucleus sampling over older approaches like Top-K.

Top-P was formalised in the paper “The Curious Case of Neural Text Degeneration” by Holtzman et al. (2020), which showed that fixed-size sampling methods produced repetitive, low-quality text compared to the adaptive nucleus approach. The insight was that the right number of candidate tokens to consider at each step should depend on the model’s confidence at that step — not be fixed in advance.

⚙️ 3. How Sampling Actually Works — Logits, Probabilities, and Token Choice

To really understand what temperature and Top-P do, you need to understand what happens inside a language model between processing your input and producing each output token. This is not just theoretical — the mechanism explains why the parameters behave the way they do and why certain settings produce certain kinds of output problems.

Logits — The Raw Scores

At each step of generation, a language model produces a raw score for every token in its vocabulary. These raw scores are called logits. They are not probabilities — they are unconstrained numbers that can be positive, negative, or zero, and they have no inherent scale. A logit of 5.3 for the token “the” and a logit of 2.1 for the token “a” means “the” is more likely, but the numbers themselves do not directly tell you how much more likely. To convert logits into probabilities — numbers between 0 and 1 that add up to 1 across all tokens — the model applies a mathematical function called softmax.

Softmax — Converting Raw Scores to Probabilities

The softmax function takes a set of logits and converts them into a probability distribution. For each token, softmax exponentiates the logit (raises e to the power of the logit) and then divides by the sum of all exponentiated logits. The result is a set of numbers between 0 and 1 that sum to exactly 1 — a proper probability distribution over the entire vocabulary. The token with the highest logit gets the highest probability; the token with the lowest logit gets the lowest probability. What softmax does is preserve the relative ordering of the logits while converting them into interpretable probabilities.

Temperature Scaling — The Real Mechanism

Temperature enters the process between the logit calculation and the softmax application. Instead of applying softmax directly to the raw logits, the model divides every logit by the temperature value T before applying softmax. This one operation — dividing logits by T — is what temperature actually does at the mathematical level.

When T is less than 1 (say, T = 0.3), dividing by T makes the logits larger in magnitude, which makes the softmax distribution more “peaky” — high-probability tokens get even higher probability, and low-probability tokens get pushed toward zero. The distribution concentrates around the top tokens. When T is greater than 1 (say, T = 1.5), dividing by T makes the logits smaller in magnitude, which makes the softmax distribution more “flat” — probability mass spreads more evenly across the vocabulary, and lower-probability tokens get a meaningfully higher share of the distribution.

At the extreme: as T approaches 0, the distribution collapses toward a single token — whichever has the highest logit gets probability approaching 1.0, and all others approach 0. This is called near-greedy decoding. As T increases, the distribution approaches uniform — every token has roughly the same probability regardless of its logit. In practice, temperatures above 2.0 produce incoherent output for most tasks because the distribution becomes too flat to preserve meaningful token preference.

Sampling vs Greedy Decoding — A Clear Distinction

Once the probability distribution has been calculated (with temperature applied), the model must select the next token. There are two fundamentally different approaches to this selection:

• Greedy decoding: Always pick the token with the highest probability — the argmax of the distribution. This is deterministic: given the same input and the same model, greedy decoding produces the same output every time. It is also the approach most prone to repetitive, “safe” outputs because it never takes a chance on a lower-probability token, even when a surprising word might produce better text.
• Sampling: Roll the dice using the probability distribution — a higher-probability token is more likely to be selected, but lower-probability tokens can be selected too, with frequency proportional to their probability. This introduces randomness, which is why the same prompt produces different outputs at temperature 1.0. It is also why sampling — particularly nucleus sampling — tends to produce more natural, varied, and interesting text than pure greedy decoding.

Temperature = 0 is equivalent to greedy decoding in most implementations (or as close to it as floating-point arithmetic allows). Temperature = 1.0 means sampling directly from the model’s natural probability distribution without any scaling. Temperature between 0 and 1 is a blend: the model samples, but from a distribution that is more peaked than natural. Temperature above 1.0 samples from a distribution that is flatter than natural — more chaotic.

🔬 4. Top-P (Nucleus) Sampling — Step-by-Step With a Worked Example

The definition of nucleus sampling (top-p) is precise: Top-P selects the smallest set of tokens whose cumulative probability equals or exceeds P, renormalises the probabilities within that set so they sum to 1, and samples the next token from that renormalised distribution. The key word is “smallest” — the algorithm finds the minimum number of top tokens needed to reach the probability threshold P, not a fixed count.

Why Nucleus Sampling Exists

Before nucleus sampling, the most common vocabulary filtering approach was top-k sampling: always consider exactly k tokens, regardless of how confident the model is. Top-k has a fundamental problem: the right value of k changes at every step. When the model is confident (one token has 80% probability), k=50 includes 49 tokens with negligible probability — adding noise that produces bad outputs. When the model is uncertain (no token has more than 2% probability), k=50 cuts off most reasonable options — forcing the model to pick from an artificially small set. Nucleus sampling solves this by making k adaptive: the threshold is defined in probability space (p), not in count space (k), so the candidate set automatically shrinks when the model is confident and grows when the model is uncertain.

Worked Example: Token Probabilities at p=0.90 vs p=0.95

Consider a model generating the next token after the prompt “The capital of France is”. Here are hypothetical probabilities for the top 10 candidates:

At p=0.90: the model reaches 90% cumulative probability at rank 3 (Paris + Lyon + Marseille = 0.87, which crosses 0.90 after including Marseille). The nucleus contains 3 tokens. Nice, Bordeaux, and everything below are excluded. At p=0.95: the model needs to include Nice, Bordeaux, and Toulouse to reach 95% cumulative probability (0.95 exactly at rank 6). The nucleus now contains 6 tokens. In both cases, the selected tokens’ probabilities are renormalised to sum to 1 before the final sampling step.

The Most Common Top-P Misunderstanding: Top-p does NOT mean “pick from the top p tokens.” The number p is not a token count — it is a probability threshold. p=0.9 means “pick from the smallest set of tokens whose probabilities add up to at least 0.9.” At one step that set might contain 3 tokens; at the next step, where the model is less confident, the same p=0.9 might require 50 tokens. The size of the candidate set changes at every single generation step.

🔄 5. Temperature vs Top-P — How They Interact (and Why “Don’t Tune Both” Is Common Advice)

Temperature and Top-P are often presented as independent controls, but they interact — and understanding the interaction explains why most provider documentation recommends adjusting one or the other rather than both simultaneously.

The interaction in plain terms: temperature reshapes the probability distribution over all tokens. Top-P then truncates the distribution after reshaping. This means the order of operations matters: if you set temperature to 0.3 (making the distribution very peaked), Top-P of 0.9 will include a very small nucleus — because the top tokens already account for most of the probability mass before truncation. If you set temperature to 1.5 (making the distribution very flat), Top-P of 0.9 will include a large nucleus — because probability is spread across many tokens, so you need more of them to reach 90% cumulative probability.

This interaction means that tuning both simultaneously creates a compounded effect that is difficult to reason about intuitively. If you lower temperature and also lower Top-P, you are applying two determinism-increasing controls at once — potentially making output more repetitive than either alone would. If you raise temperature and also raise Top-P, you compound the randomness. Most provider documentation — including OpenAI’s API documentation — explicitly describes top_p as an alternative to temperature (nucleus sampling) and recommends adjusting one or the other, not both.

The practical guidance: pick your primary control. Use temperature as your main lever and leave Top-P at its default (usually 0.9–1.0). Or use Top-P as your main lever and leave temperature at 1.0. In either case, understand what the other parameter’s default value does so you know the baseline you are working from.

The 2×2 Interaction Grid

• Low temperature + Low Top-P: Very deterministic. The model produces predictable, consistent text — but can become repetitive and template-like over longer outputs. Good for structured extraction where you want the same answer every time. Risky for long-form generation.
• Low temperature + High Top-P: Mostly deterministic with occasional variation. The peaked distribution from low temperature means Top-P usually only includes a few tokens anyway, so the high Top-P setting rarely expands the candidate set much. Practically similar to low temperature alone.
• High temperature + Low Top-P: “Creative but fenced-in.” Temperature spreads probability widely, but Top-P clips the distribution to a small set of still-high-probability tokens. Can produce interesting variation within a constrained set of options. Sometimes useful for varied but coherent outputs.
• High temperature + High Top-P: Wild and unpredictable. Probability is spread across many tokens, and Top-P includes most of them. Outputs can be surprising, off-topic, or incoherent. Useful for brainstorming where any output is better than none, or for creative fiction where novelty is the goal. Not recommended for any task requiring reliability.

🔩 6. Top-K, Beam Search, and Why Decoding Choice Can “Degenerate” Output

Temperature and Top-P are the most commonly exposed sampling parameters, but they are not the only decoding strategies available. Understanding Top-K and beam search gives you the full picture of how language model output is shaped — and explains a class of output quality problems that sampling parameter tuning alone cannot fix.

Top-K Sampling

Top-K sampling is simpler than nucleus sampling: at each step, consider only the K tokens with the highest probability and sample from those. Top-K=50 means always consider exactly 50 tokens. Top-K=1 is greedy decoding — always pick the single most probable token. Top-K has the fixed-size problem described earlier: when the model is confident, K=50 includes many near-zero-probability tokens that add noise. When the model is uncertain, K=50 may cut off many reasonable options. Nucleus sampling (Top-P) was designed to address this limitation. Despite this, Top-K remains a common parameter in many model APIs — particularly in older implementations and in models running on edge hardware where adaptive computation is more expensive. Top-K is also commonly combined with Top-P: apply Top-K first to create a reasonable candidate set, then apply Top-P within that set for further refinement.

Beam Search

Beam search is a fundamentally different approach to decoding. Instead of sampling one token at a time, beam search maintains multiple “beams” — partial sequences — simultaneously, expanding each one at each step and keeping only the top-scoring beams. With beam width B=4, beam search evaluates 4 candidate sequences at each step and carries forward only the 4 highest-scoring ones. This produces more globally coherent outputs than greedy decoding because it considers multiple paths through the probability space rather than committing to the locally best token at each step.

However, beam search has a well-documented failure mode in open-ended generation: it tends to produce generic, repetitive, “safe” text that sounds plausible but lacks the naturalness and variation of human writing. Holtzman et al. (2020) — the nucleus sampling paper — documented this as “text degeneration” and showed it was linked to the way beam search optimises for global probability rather than local naturalness. The fix they proposed was nucleus sampling, which introduces controlled randomness that better matches the statistical properties of human text. In 2026, beam search remains useful for highly constrained tasks like machine translation where global coherence is more important than naturalness — but for most open-ended language generation, sampling-based approaches with temperature and Top-P produce better results.

🚨 7. The Big Myth — “Temperature 0 = No Hallucinations”

This is the most persistent misconception about temperature in AI, and it needs to be addressed directly: setting temperature to 0 reduces randomness in output generation — it does not reduce factual errors. These are different problems with different causes and different solutions.

Temperature controls how the model selects from among the tokens it considers plausible given your input and its training. If the model’s training data led it to associate a confident-sounding but incorrect fact with a given prompt, a low temperature will make it output that incorrect fact consistently and confidently — on every single request, without variation. Temperature zero does not make the model more accurate. It makes the model more consistent. Those are not the same thing. You can have a perfectly consistent, perfectly wrong answer at temperature 0.

The reason people believe low temperature reduces hallucinations is that high temperature can produce obviously incoherent or contradictory outputs — and lowering temperature makes those specific pathological outputs less likely. But the underlying cause of hallucination is not randomness in sampling. It is gaps, errors, or biases in the model’s training data and the model’s tendency to produce fluent, confident-sounding text regardless of whether it has reliable information to draw on. Those problems are not fixed by sampling parameter adjustment.

The Hallucination Reality: Temperature 0 can make the same wrong answer appear consistently every time. A deterministic wrong answer is not better than a random wrong answer — it is worse, because it is harder to detect through resampling and more likely to be trusted by users who see the same confident response repeatedly.

How to Actually Reduce Hallucinations

• Provide sources and context (RAG): Retrieval-Augmented Generation grounds model responses in retrieved documents rather than training memory — the primary evidence-based solution to hallucination for knowledge-intensive tasks
• Ask for quotes with citations: Instruct the model to quote directly from provided text rather than paraphrase — constrains generation to verifiable content
• Add a verification step: In function calling and tool use systems, add a tool that verifies claims against a trusted source before the model’s response reaches the user
• Force an “unknown” option: Explicitly instruct the model that “I don’t know” or “insufficient information” is an acceptable answer — reduces confabulation by making uncertainty a permitted output
• Self-critique pass: Ask the model to review its own output for factual claims it cannot verify from the provided context — flagging rather than silently confabulating uncertain facts

📋 8. Recommended Settings Library — Copy-Paste Presets for 2026

The table below provides starting-point settings for eight common use cases. These are ranges, not fixed values — the right setting depends on your specific model family, your prompt constraints, and your task requirements. Models from different providers behave differently at the same parameter settings. Always test presets on your specific model with representative examples before deploying to production. For high-stakes tasks — legal, medical, financial, compliance — default to the deterministic end of the range and add verification steps rather than relying on sampling parameters alone.

Important: these are starting ranges. Model family matters significantly — Claude Sonnet 4.5, GPT-5.x, and Gemini 3.1 Pro behave differently at the same parameter values. Prompt constraints (examples, format instructions, system prompts) interact with temperature settings. High-stakes tasks should prefer determinism plus human verification rather than relying on low temperature alone.

⚠️ 9. 2026 Reality Check — Some APIs Are Deprecating Sampling Knobs

Temperature and Top-P are not guaranteed to be available in every API, every model version, or every SDK configuration in 2026. This is a practical reality that teams discover mid-project, and it is worth knowing before you build temperature-dependent logic into a production system.

Provider API parameter support can change between model versions and SDK versions. Anthropic’s documentation shows that temperature and top_p parameters may be deprecated or unsupported in certain endpoints and SDK configurations, and SDK behaviour may reject those parameters depending on the model version in use. This is not unique to Anthropic — parameter support evolves as providers develop new model architectures that may use different internal decoding approaches. The practical implication: always check your specific provider’s current documentation for the specific model version you are deploying, not just the general documentation for the model family. Do not assume that parameters that worked in a previous model version are supported in the current one.

If temperature and Top-P are unavailable in your provider’s API for a given model or endpoint, the alternatives are:

• Stronger instructions in the prompt: Use explicit style and format constraints in your system prompt — “respond in exactly three bullet points”, “use formal register only”, “produce exactly two variations” — to control output consistency without sampling parameters
• Structured outputs: Where available, structured output mode (constraining the model to a JSON schema) provides format consistency independent of temperature settings
• Self-critique passes: Instruct the model to review and revise its own first draft — adds a deterministic quality-control layer over a stochastic generation step
• Reranking: Generate multiple candidate outputs and rank or select among them using a scoring function — effective for tasks where you want variety plus quality filtering
• Few-shot examples: Strong few-shot examples in the prompt constrain output style and format more effectively than sampling parameters in many cases

The broader point is that prompt engineering and output structure are more durable controls than sampling parameter tuning. Sampling parameters are an optimisation layer; the prompt is the foundation. A well-constructed prompt with clear constraints, good examples, and an explicit output format will produce consistent, useful output across a wider range of model versions and provider configurations than one that relies primarily on parameter settings to enforce quality.

🔧 10. Troubleshooting Playbook — Symptom to Fix

The symptoms below are the most common output quality problems you will encounter when working with language model generation, along with the parameter adjustments and prompt changes most likely to address them. In all cases, try parameter changes incrementally — one adjustment at a time — so you can isolate what is actually improving the output.

• “Too repetitive — keeps repeating phrases or sentence structures” → Raise temperature slightly (try +0.2 increments) OR raise Top-P toward 0.95 OR add an explicit anti-repetition instruction in your prompt (“avoid repeating phrases you have already used”) OR use a repetition penalty parameter if your provider exposes one
• “Too random — outputs are off-topic, incoherent, or unpredictably varied” → Lower temperature (try -0.2 increments) OR lower Top-P toward 0.85 OR tighten the prompt with more specific instructions and examples — sampling parameters alone cannot compensate for an ambiguous or underspecified prompt
• “Keeps contradicting itself within a single response” → Lower temperature to increase consistency OR constrain output format (ask for an outline or structure first, then expand each section) OR use chain-of-thought prompting to force sequential reasoning that the model must maintain across the response
• “Refuses to follow the structure I specified” → Add explicit schema or format definition with examples in the prompt (“your response MUST follow this format: [example]”) OR use structured output mode where available — structure compliance is a prompt engineering problem, not a temperature problem
• “Outputs are bland and generic” → Raise temperature (try 0.7–0.9 range) OR add style guidance to the prompt (“be specific, use concrete examples, avoid generalisations”) OR check whether you are using beam search decoding — beam search is the primary cause of generic-sounding outputs in constrained systems
• “Outputs are inconsistent between runs for the same prompt” → Lower temperature toward 0 for maximum consistency OR use a fixed random seed if your provider exposes a seed parameter OR verify whether the model version has changed — provider updates can shift model behaviour even at fixed temperature settings
• “Code output is syntactically invalid or uses wrong APIs” → Lower temperature (0.1–0.2 range for code) AND add examples of valid code in the prompt AND specify the exact language version and library version you are targeting — code generation accuracy is primarily a prompt specificity problem
• “Model adds unnecessary disclaimers or hedges to every response” → Add explicit instructions to omit disclaimers if they are not needed (“do not add disclaimers or caveats”) — disclaimer frequency is not controlled by temperature; it is a model alignment behaviour controlled by the system prompt

📌 Key Takeaways

🔗 Related Articles

• 📖 Context Windows & Tokens Explained: The Complete 2026 Guide
• 📖 Prompt Engineering for Non-Programmers: How to Get Better Answers from AI
• 📖 Function Calling & Tool Use in LLMs Explained: The Complete 2026 Guide
• 📖 Small Language Models Explained: Why Smaller AI Might Be Better for Your Business
• 📖 Retrieval-Augmented Generation (RAG) Explained: How AI Answers Questions With Your Own Data