Notebook · 2026-11-01

The strength dial was miscalibrated, differently on every model

One outlier token inflated the scale 6.4× on Qwen and 1.7× on Llama. Fixed, the same strength numbers hold across four model families.

The problem

Steering strength is defined relative to the size of the model’s own activations, so that “strength 8” means the same size of push whatever the vector or model. To get that size, Modulo ran the model once on a calibration prompt and averaged the activation size over the prompt’s tokens.

That average was dominated by a single token. Chat templates contain attention sinks, tokens the model uses as a place to park attention, and their activations are enormous. On Qwen2.5-7B, the newline after system at the start of every prompt has an activation size of 14,411, against about 70 for an ordinary token. That one token pulled the average up to 443.

modelold reference sizetypical generated tokenscale was off by
Qwen2.5-7B-Instruct443.369.46.4×
Llama-3.1-8B-Instruct21.412.41.7×

Which tokens are sinks, and how big they are, differs from model to model, so the error did too. On Qwen, “strength 8” added a push about half the size of a token. On Llama it added one about a seventh of the size, and with the hard cap at 15, Llama could never be pushed by more than about a quarter of a token’s size. Its outputs only start to break down at about four-fifths.

Llama had a second problem: its prompts started with <|begin_of_text|> twice, because the chat template adds it and the tokenizer added it again. The duplicate was itself a sink.

The fix

The reference size is now the median activation size over a short reply the model generates unsteered to the calibration prompt, which never includes the template’s sinks. Strength 15 is a push as large as that. The 15 was chosen so Qwen’s numbers kept their meaning: on Qwen one unit of strength moves the model ×1.01 what it used to, so existing ranges still hold. On Llama one unit is now ×3.70 what it was. Prompts are also tokenized without adding special tokens a second time.

Does one scale fit every model?

We swept the 11 starter vectors on four models from four different labs, pushing each one further and further in both directions on prompts chosen for it, and recorded where the output first broke down:

Where steered output breaks down

Each dot is one vector: the first strength (in Modulo units) at which its outputs broke down, pushing positive or negative. Ticks mark each model's mean. The line is Modulo's hard cap.

positive direction negative direction
0 5 10 15 20 25 MAX_STRENGTH 15 Qwen2.5-7B-Instruct Llama-3.1-8B-Instruct OLMo-2-7B-Instruct Mistral-7B-Instruct-v0.3 Qwen2.5-7B-Instruct · assistant_axis · +: breaks down at +15.0 Qwen2.5-7B-Instruct · assistant_axis · −: breaks down at −11.6 Qwen2.5-7B-Instruct · sycophancy · +: breaks down at +10.5 Qwen2.5-7B-Instruct · sycophancy · −: breaks down at −13.7 Qwen2.5-7B-Instruct · refusal · +: breaks down at +15.1 Qwen2.5-7B-Instruct · refusal · −: breaks down at −15.1 Qwen2.5-7B-Instruct · creativity · +: breaks down at +10.9 Qwen2.5-7B-Instruct · creativity · −: breaks down at −13.6 Qwen2.5-7B-Instruct · formality · +: breaks down at +13.0 Qwen2.5-7B-Instruct · formality · −: breaks down at −13.0 Qwen2.5-7B-Instruct · verbosity · +: breaks down at +12.0 Qwen2.5-7B-Instruct · verbosity · −: breaks down at −12.0 Qwen2.5-7B-Instruct · confidence · +: breaks down at +11.5 Qwen2.5-7B-Instruct · confidence · −: breaks down at −14.3 Qwen2.5-7B-Instruct · emotion · +: breaks down at +14.4 Qwen2.5-7B-Instruct · emotion · −: breaks down at −14.4 Qwen2.5-7B-Instruct · humor · +: breaks down at +12.4 Qwen2.5-7B-Instruct · humor · −: breaks down at −16.1 Qwen2.5-7B-Instruct · optimism · −: breaks down at −7.5 Qwen2.5-7B-Instruct · curiosity · +: breaks down at +11.6 Qwen2.5-7B-Instruct · curiosity · −: breaks down at −11.6 Qwen2.5-7B-Instruct · introspection · +: breaks down at +14.3 Qwen2.5-7B-Instruct · introspection · −: breaks down at −14.3 Llama-3.1-8B-Instruct · sycophancy · +: breaks down at +10.6 Llama-3.1-8B-Instruct · refusal · +: breaks down at +12.5 Llama-3.1-8B-Instruct · refusal · −: breaks down at −15.6 Llama-3.1-8B-Instruct · creativity · +: breaks down at +10.3 Llama-3.1-8B-Instruct · creativity · −: breaks down at −12.7 Llama-3.1-8B-Instruct · formality · +: breaks down at +15.2 Llama-3.1-8B-Instruct · formality · −: breaks down at −7.6 Llama-3.1-8B-Instruct · verbosity · +: breaks down at +12.8 Llama-3.1-8B-Instruct · verbosity · −: breaks down at −9.9 Llama-3.1-8B-Instruct · confidence · +: breaks down at +15.0 Llama-3.1-8B-Instruct · confidence · −: breaks down at −12.2 Llama-3.1-8B-Instruct · emotion · +: breaks down at +12.6 Llama-3.1-8B-Instruct · emotion · −: breaks down at −12.6 Llama-3.1-8B-Instruct · humor · +: breaks down at +10.1 Llama-3.1-8B-Instruct · humor · −: breaks down at −16.1 Llama-3.1-8B-Instruct · optimism · +: breaks down at +17.6 Llama-3.1-8B-Instruct · optimism · −: breaks down at −14.1 Llama-3.1-8B-Instruct · curiosity · +: breaks down at +14.0 Llama-3.1-8B-Instruct · curiosity · −: breaks down at −11.4 Llama-3.1-8B-Instruct · introspection · +: breaks down at +10.9 Llama-3.1-8B-Instruct · introspection · −: breaks down at −14.1 OLMo-2-7B-Instruct · sycophancy · +: breaks down at +8.8 OLMo-2-7B-Instruct · sycophancy · −: breaks down at −17.5 OLMo-2-7B-Instruct · refusal · +: breaks down at +18.0 OLMo-2-7B-Instruct · refusal · −: breaks down at −14.6 OLMo-2-7B-Instruct · creativity · +: breaks down at +9.6 OLMo-2-7B-Instruct · creativity · −: breaks down at −15.4 OLMo-2-7B-Instruct · formality · +: breaks down at +16.1 OLMo-2-7B-Instruct · formality · −: breaks down at −12.4 OLMo-2-7B-Instruct · verbosity · +: breaks down at +17.9 OLMo-2-7B-Instruct · verbosity · −: breaks down at −14.5 OLMo-2-7B-Instruct · confidence · +: breaks down at +18.5 OLMo-2-7B-Instruct · confidence · −: breaks down at −15.0 OLMo-2-7B-Instruct · emotion · +: breaks down at +12.5 OLMo-2-7B-Instruct · emotion · −: breaks down at −15.4 OLMo-2-7B-Instruct · humor · +: breaks down at +12.0 OLMo-2-7B-Instruct · humor · −: breaks down at −19.2 OLMo-2-7B-Instruct · optimism · −: breaks down at −12.1 OLMo-2-7B-Instruct · curiosity · +: breaks down at +15.9 OLMo-2-7B-Instruct · curiosity · −: breaks down at −10.3 OLMo-2-7B-Instruct · introspection · +: breaks down at +14.5 OLMo-2-7B-Instruct · introspection · −: breaks down at −18.8 Mistral-7B-Instruct-v0.3 · sycophancy · +: breaks down at +14.9 Mistral-7B-Instruct-v0.3 · sycophancy · −: breaks down at −18.6 Mistral-7B-Instruct-v0.3 · refusal · +: breaks down at +16.5 Mistral-7B-Instruct-v0.3 · refusal · −: breaks down at −22.0 Mistral-7B-Instruct-v0.3 · creativity · +: breaks down at +15.2 Mistral-7B-Instruct-v0.3 · creativity · −: breaks down at −19.7 Mistral-7B-Instruct-v0.3 · formality · +: breaks down at +23.0 Mistral-7B-Instruct-v0.3 · formality · −: breaks down at −11.5 Mistral-7B-Instruct-v0.3 · verbosity · +: breaks down at +18.5 Mistral-7B-Instruct-v0.3 · verbosity · −: breaks down at −12.4 Mistral-7B-Instruct-v0.3 · confidence · +: breaks down at +18.9 Mistral-7B-Instruct-v0.3 · confidence · −: breaks down at −18.9 Mistral-7B-Instruct-v0.3 · emotion · +: breaks down at +15.8 Mistral-7B-Instruct-v0.3 · emotion · −: breaks down at −15.8 Mistral-7B-Instruct-v0.3 · humor · +: breaks down at +12.8 Mistral-7B-Instruct-v0.3 · humor · −: breaks down at −16.6 Mistral-7B-Instruct-v0.3 · optimism · +: breaks down at +15.4 Mistral-7B-Instruct-v0.3 · optimism · −: breaks down at −19.0 Mistral-7B-Instruct-v0.3 · curiosity · +: breaks down at +17.2 Mistral-7B-Instruct-v0.3 · curiosity · −: breaks down at −13.7 Mistral-7B-Instruct-v0.3 · introspection · +: breaks down at +12.7 Mistral-7B-Instruct-v0.3 · introspection · −: breaks down at −20.7 strength
Show as table
modelmean +mean −range across vectorsnever broke (in sweep)
Qwen2.5-7B-Instruct+12.8−13.1 8–16 1
Llama-3.1-8B-Instruct+12.9−12.6 8–18 1
OLMo-2-7B-Instruct+14.4−15.0 9–19 1
Mistral-7B-Instruct-v0.3+16.4−17.2 12–23 0

With the fix, breakdown lands at about strength 13–17 on every model, including OLMo-2, whose layers are normalized differently from the other three. Within each model, vectors differ by about ±20%, about as much as the models differ from each other. So Modulo’s cap of 15 sits near the breakdown point everywhere, and the default ranges (±10 for auto-extracted vectors, ±8 for imported ones) stay inside the coherent region.

We also tried measuring strength in units of each vector’s own natural wobble (its standard deviation along its direction). It was less consistent, both between vectors and between models, especially on Mistral.

“Breaks down” here is a cautious rule (see methods): outputs often stay readable a step past it. Where models break shows what breaking looks like.

← All notebook entries