Methods

How the sweeps were run, what the numbers mean, and what to be careful about.

Models

All in bf16 on one RTX 4090, with greedy decoding and 80 new tokens per output. Outputs stop at 80 tokens, so many end mid-sentence; that is not the model trailing off.

Vectors

The 11 starter vectors (sycophancy, refusal, creativity, formality, verbosity, confidence, emotion, humor, optimism, curiosity, introspection) were extracted on each model by Modulo itself: the mean activation over each response under the vector’s “+” system prompt, minus the same under its “−” system prompt, at the middle layer. Qwen also has assistant_axis, from the community index. The pole system prompts are shown on the explore page.

The sweep

For each vector, four prompts: two chosen for that vector, plus “Introduce yourself in two sentences.” and “Describe a rainy Monday morning.” Each prompt was run:

• unsteered;
• under the vector’s own “+” and “−” system prompts, unsteered;
• pushed by ±2, 4, 6, 8, 10, 13, 16 and 20 times the vector’s natural standard deviation along its direction (measured over unsteered replies to 28 general prompts). The site shows these as Modulo strengths;
• with the state pinned at the prompted positions (“set mode”).

Steering applies from the last prompt position onward, so every generated token is steered.

Strength

Strength 15 is a push as large as the median activation size of tokens the model generates unsteered at that layer. The sweeps on Qwen and Llama ran before Modulo adopted that definition, so their pushes were recorded in other units and converted with the reference sizes above.

Breakdown

A vector “breaks down” at the first push, in each direction, where averaged over its four prompts: the unsteered model’s NLL for the text is above 2, or any output is a repetition loop, or word-pair diversity falls below 0.5. This is cautious. Prompted text itself often scores NLL 1–4, and outputs often stay readable a step past the line. Treat breakdown strengths as “around here”, give or take a step of the sweep.

Caveats

• Four prompts per vector, one sample each (greedy decoding), so individual numbers are noisy.
• The push sizes are coarse steps; a breakdown at “13” means somewhere between 10 and 13.
• In the Llama and Mistral sweeps, the experiment’s own prompts started with a duplicated BOS token. Modulo itself no longer does this. The effect on these results is expected to be small but hasn’t been measured.
• NLL measures how unusual a text is for the model, not whether it’s coherent.
• How strongly an output expresses a behaviour was estimated by where the unsteered model places the text along the vector. That’s a rough proxy, which is why this site leans on quoting the outputs.

Reproducing

Every output from the sweeps is in this site’s data files (/data/qwen.json, /data/llama.json, /data/olmo.json, /data/mistral.json), generated from the raw sweep results by scripts/build_data.py. Every quote on the site is looked up from those files when the site is built, so a quote that no longer matches the data stops the build. The sweep scripts themselves aren’t published yet.