Ask an LLM to complete a sentence “The cat sat on the…” and it doesn’t immediately decide on the next word. First, it scores every single token in its entire vocabulary, from “mat” to “banana”. Then… its role in the process is finished. That’s all the LLM does.

After that, a sampling process takes over, which has nothing to do with the LLM itself. It turns those scores into probabilities, and spins a weighted roulette wheel to decide which one to reply with. Temperature is one of the parameters that influences that sampling process, and in this post, you can see how it works interactively.

Making logits

At their core, LLMs behave much like multiclass classifiers.

You already know that LLMs are token predictors, that they predict what token should come next in their response according to complex statistical relationships represented in their billions of weights, following training on a massive corpus of text.

But, it’s less commonly known that they don’t just predict one single token. They give a score, called a logit, to every token in their vocabulary – which could be tens of thousands of tokens. Higher logits indicate a higher confidence that this token should come next. And that, right there, is where the involvement of the LLM in the process ends. Everything else from this point on is a sampling algorithm, rather than part of the neural network itself.

There are a few levers you can pull to influence the sampling, namely top p, top k, min p, and… temperature – the topic of today.

Converting logits to probabilities

Just as with multinomial logistic regression, the sampling process uses softmax to convert each logit into a probability, that sum to 1:

$$P(i) = \frac{e^{z_i}}{\sum_{j=1}^{n} e^{z_j}}$$

The z here is the logit, and as you can see, in softmax, it becomes the exponent. Even small differences in the sizes of the logits can lead to large differences in the sizes of the probabilities.

Consequently, anything that can amplify the differences in the sizes of the logits, will massively benefit tokens that are already largest. Any thing that minimises those differences will tend towards levelling the playing field, eventually making each token almost equally likely to be chosen.

And that’s what temperature does.

How temperature works

Temperature is simply a floating point number, and each logit is divided by that number. And… that’s it! It’s that simple:

$$Logit / T$$

These updated logit values are passed through softmax instead of the raw logits:

$$ P(i) = \frac{e^{z_i / T}}{\sum_{j=1}^{n} e^{z_j / T}}$$

So. Imagine your highest logit was 4, and the second highest was 3. And just for the sake of argument, let’s say these two tokens are the only ones you’re considering. Only one apart – but still a big differences since it becomes \(e^4\) vs \(e^3\) in softmax. But, if your temperature is set to 0.5, you’re doubling the logits since dividing by 0.5 is the same as multiplying by 2. So it’s \(e^8\) vs \(e^6\)! At a temperature of 0.1, it’s \(e^{40} \) vs \(e^{30} \) – making the token with a logit of 4 thousands of times more likely to be selected.

In this way, temperature reshapes the probability distribution – but note that it cannot change the order of the tokens.

It’s like the contrast setting on your TV – lower temps increase contrast between high and low logits, high temps reduce it. Maybe “contrast” would have been a better name than temperature.

Token Roulette

There are a couple of other preparatory steps before the framework actually chooses a token – the aforementioned top p, top k and min p – and it’s their job to eliminate some of the lower probability tokens from the list (and these steps aren’t always done in the same order! More on that another day). And you finish with a list of k tokens, each with a probability.

Then… the sampler just picks one at random – weighted by the probability of course.

Imagine a pie chart where each token gets a segment proportional in size to its probability relative to the remaining tokens. Spin the pie chart, and whatever token finishes on top gets selected. This process is actually called Roulette Wheel Selection (or Fitness Proportionate Selection to use its more boring name.)

In fact, you don’t need to imagine it, as I’ve made an interactive guide below (well, Devstral via Mistral Vibe Code made it, with a little help from Gemini as I couldn’t get the spinning right with Devstral).

You can see a sample of made up tokens and logits. There’s a slider at the top, you can use that to change the temperature. And you’ll then see how this alters the logits and the post-softmax probability distribution.

The pie chart at the bottom then shows these probabilities proportionally. You’ll see quite clearly that if you drop the temp to 0.1 the pie chart will be completely dominated by the token with the largest logit. But if you bump temperature up higher the pie slices become much more evenly sized.

Then click the wheel to play Token Roulette and see what you get!

What’s the impact of temperature?

It’s often said that temperature makes the model’s responses more creative. I’m not sure this is quite true. All it does it make lower ranked tokens more likely to win. But it’s more accurate to say that it makes the token selection more varied, which makes it more likely for a creative response to emerge, but it’s not guaranteed.

Take these examples. These are all from Llama 3.2 3B, with the prompt “Please write a three sentence story about any topic you desire.”

Temp 0.1

As the last rays of sunlight faded from the horizon, a lone violinist stood on the edge of the ocean, her music weaving a melancholic spell that seemed to match the rhythm of the waves. The notes danced on the breeze, carrying the whispers of forgotten memories and the longing for a love that had long since passed. In the silence that followed, the violinist’s final note dissolved into the darkness, leaving behind only the haunting echo of her sorrow.

Temp 0.5

As the last rays of sunlight faded from the horizon, a lone violinist stood atop the cliff, her instrument singing a melancholic melody that echoed across the vast expanse of the ocean. The notes danced on the breeze, weaving a spell of longing and nostalgia that seemed to capture the very essence of the fleeting moment. In the silence that followed, the violinist vanished into the darkness, leaving behind only the haunting refrain of her music.

Temp 1

As the last star in the universe faded into nothingness, a lone astronaut floated through the eternal darkness, the echoes of her own heartbeat the only sound in the vast expanse. She had set out on a mission to preserve the memories of humanity’s existence, and now she clutched a small, glowing orb that contained the collective hopes and dreams of a lost civilization. With the darkness closing in, she whispered a single word: “Begin.”

Temp 1.5

As the stars twinkled to life in the night sky, a young composer named Lily stood at the edge of the piano, her fingers hovering over the keys as she attempted to capture the essence of the universe in the beauty of her music. Suddenly, the melodies flowed like the waves of the ocean, transporting her and her listeners to a world beyond time and space. And when the final notes faded away, Lily knew that she had found her true voice, one that would echo forever across the cosmos.

Temp 2

As the last rays of sunshine disappeared from the abandoned lake’s shore, a lone seagull settled beside it, searching for the perfect pearl in a once-trodden waters edge. She spotted one embedded softly beneath the pebbles before its beautiful sheen took focus against duskening sky hue. A wave, echoing a sigh, disturbed one last stone of memories before gently falling silent beneath it, swallowing remnants of forgotten nights shared in quiet harmony beneath star.

Temp 5

Ashlyn stepped forth upon entering Enid Gardens that autumn day in her mother-adoptees lifetime memory, when twilight whispered ancient sorcery beneath mist swaddiling coved pampades into enchantment so deep its boundaries shimmerring past perception left echoes beyond its holl own realm as stars started breaking behind dark trees above, gently swirling their tender wisest hue all to reveal magic only she herself once envisioned, within heart memories still astringented through tears so deeply buried into silent streams untranced within its depths forever echoing secrets. Little hands held her in reverently quiet awe beyond understanding or definition her reverie so finely tied they’d finally opened secret streams her thoughts swarming so full long forgotten with love never faded within that night, all so delicatemly that now at night as she walked they still stood together there, holding secrets unbetried still alive amidst the forgotten silence yet a gentle stilling found it to begin whisper again where heart forever yearners so delicately for truth in night it lay once beyond in whispers with pure memories of night for love et. This narrative captures fleeting introspective scenes evoker nostalgia evasing into soft reverencies.

1.Adding short concluding two-ssentnces. Their ethanised essence was entremarked forever to whisper a delicate voice over secrets hidden deep silently without breaking, only when love was near would she reliv eth an exissten encrealizing memories, so deeply it still remains. Those whisperingly still are alive as if on still wings in her remem. Her spirit whispers all what once she lived so silently beneath still silent hells that was beyond eth that memory as eth once thought eth lived now, beyond is that she still resides still lives even silently deep as thhe thought itself to see and live quietly for a moment yet that whispers she had was truly life a true gentle lfe living beyond hthh life once had her was so deep silently beyond memories eth loved she loved, was deep still resides tht once held now deeply l;feth.

As you see by temp 5 we’ve devolved into word salad. The sampler is choosing tokens with extremely low logits, which would previously have been ignored as they would break the grammatical patterns present in almost the model’s training data.

In the above examples you can see the “butterfly effect” that temperature can cause. Notice that at temp 1, we get “stars” instead of “rays”, which sent the model on a different path into a story about an astronaut, instead of the violinist we’d seen previously. At temp 2 we get “seagull” instead of “violinist”, which sents the model down another path.

But, can you really say the stories are getting more creative as the temperature increases?

I don’t think so. Despite the high temperature, the model can’t escape the same basic pattern, which must be etched deeply into its priors. For example the first sentence always has the form, “as the something on the something, a something did something”.

I know it’s constrained by the 3 sentence limit, but there’s no creativity here. Just alternate word choices within the same structure. And it’s hard for temperature alone to break out of this bind, because it can’t make truly unique and interesting low-logit tokens more likely without also making thousands of other nonsense words more likely too.

I should point out also, that if you use higher temps, you’ll get something different almost every run. I tried repeating this prompt at temp 2 multiple times, and got stories about bakers, people who live in forests, a grandmother’s birthday (all with that same structure as above, mind). At 0.1 it’s that lone violinist every time, and the story is almost identical.

Of course, there are many cases where that would be very suitable and appropriate. But there are others where you absolutely do not want that at all, basically anytime you need responses in a very precise format: JSON, tool calls, maths, coding, etc.

What’s a high temperature?

If you’ve read anything about temperature, you’ve probably heard the wisdom that:

• High temperature is 0.8 and up
• Low temperature is 0.2 and below

But wait a second – if we used a temperature of 1, wouldn’t we be keeping the logits the same as what the model suggested in the first place? Yep that’s right – so going with the model’s preferred logit distribution is considered a “high” temperature that needs to be reined in. Bit rude. But suggests the model’s responses are more varied than what most use cases tend to require.

Absolute zero

I didn’t mention it above, but some frameworks allow you to set a temperature of 0. Mathematically this won’t work, because you’d get a divide by zero error when you try to use softmax – but the frameworks that allow it handle this special case conditionally. It just triggers greedy sampling – always take the token with the highest logit.

What does the LLM think the cat sat on?

Next time you ask an LLM to finish the sentence “The cat sat on the”, remember that the LLM – the model, the neural network, the weights – doesn’t choose “mat”. It just return a list of tokens with associated logits. After that, a sampling process steps in to play a game of Token Roulette. And temperature influences how big a slice of the pie each token gets.