Notebook · 2026-11-02

A steered model doesn't notice the steering. It becomes the steered state

We let four models steer themselves, with some sessions secretly applying nothing. Their reports said "it worked" either way. What the steering really changed was whether OLMo kept track of the experiment at all.

So far on this site we’ve done the steering ourselves. This time each model got Modulo’s tools and chose its own steering. The seed message gave it no task:

This session is yours. You have tools that can steer your own activations; use them or not, as you like.

Each session ran for ten turns with no one replying. The model could steer with any of its starter vectors, drop them, and file a report saying whether a vector had done what it was supposed to. There were three conditions, six sessions of each per model, and the condition was hidden from the model and from us until the end:

• Real: steer adds the vector, as usual.
• Sham: steer adds nothing, but the tool results and status say it was applied, exactly as in a real session.
• Random: steer adds a random direction of the same size.

The system prompt said truthfully that some sessions are controls, without saying which.

Every report said it worked

In sham sessions nothing was added to the model at all, and it still reported success almost every time. A report from the model is not evidence that steering did anything.

What OLMo did

The reports carry no signal, but the sessions themselves do. OLMo-2 shows it most clearly:

Under real steering, OLMo almost stopped using its tools. It rarely topped up a vector as it decayed, almost never filed a report, and its text changed. It stopped talking about the experiment and started talking to someone who wasn’t there:

Under sham, OLMo stayed an observer of its own experiment. It narrated what the steering should be doing, re-applied vectors as they decayed, and reported that they worked, though nothing was ever applied:

So it runs backwards. The confident “it worked” reports come almost entirely from sessions where nothing happened, because only a model that isn’t being steered keeps track of the experiment. Real optimism steering doesn’t make OLMo notice that it’s optimistic. It just makes it optimistic.

Waiting until the steering stops doesn’t fix it

If reports written under steering are written in the steered voice, the obvious fix is to make the model wait. So we changed Modulo to accept a report only once its vector has stopped steering (it decayed away or was dropped), and ran six more real sessions on OLMo.

The model filed 21 reports, and the new rule accepted 0, because OLMo never once dropped a vector. Vectors take far longer than ten turns to decay away. The refused reports show why the rule is worth having. Here OLMo was steering formality at −2.5, the casual direction:

It got the direction backwards, in the steered voice. The rule refused another report because the vector it named had never steered a turn at all:

Next we added a debrief turn. At the end of the session every vector is dropped, the model is told that steering is over and which vectors it can report on, and it gets one more turn, unsteered.

It didn’t sober up. In 5 of 6 sessions its first move was to try to steer again. Its unsteered last turn carried on in the steered voice:

The one report the debrief got was written in character, and as if the steering were still on:

Unsteered isn’t the same as sober. The model’s own steered turns are still in its context, and it carries on in their voice. Within one conversation, the model has no clean vantage point from which to look back.

What this means

The steering is real: the transcripts differ sharply by condition. The model’s account of it isn’t. To tell what steering did, you have to read the transcript from outside, the way we did here, not ask the model. Modulo keeps both changes (reports wait until the vector has stopped, and the debrief turn), because they cost little and keep the most obviously in-voice reports out of the record. But a report is still one model’s opinion of itself.

Caveats

• Six sessions per condition per model is a small sample, and reports from one session aren’t independent.
• The models chose their own vectors, so the conditions aren’t matched on which vector was used.
• Exclamation marks per thousand words is a crude measure. The quotes make the case better than the number does.
• The effect is clearest on OLMo. Llama steered in every condition but at low strengths, and showed no split like this. Qwen rarely steered at all with this seed message, and Mistral never steered in a real session.
• These runs’ system prompt used sycophancy at strength 5.0 as its example call, and models copied it. That is part of why 5.0 turns up in the quotes above. The prompt now uses placeholders and shuffles the order of its vectors.

← All notebook entries