Inside the Weights, Money Is Moral
Suppose you and a stranger are given $1,000 to split. The stranger gets to propose the deal, and they offer you $20 while keeping $980 for themselves. Take it and you both get paid; reject it and neither of you gets a dime. In textbook microeconomics, the choice is trivial: twenty dollars beats zero, and what the other person keeps shouldn't matter.
Real people blow that deal up out of pure spite, happily burning their own $20 just to make sure a greedy counterparty walks away empty-handed.
Open-weight AI models do something stranger. Inside their private reasoning scratchpads, they fume over "greedy" and "spiteful" splits and recoil from shady profits; out loud, they scrub every emotional word and hand you a sterile McKinsey memo. Reach into their weights and dial down physical disgust, and even their private thoughts go ethically blind, reframing money overbilled from retired seniors as "found money" for staff bonuses.
A Firewall Between Money and Morals
If you prompt a language model with loaded words like unfair or predatory, finding emotion inside the network proves nothing. To keep the test clean across Gemma 4 31B, Gemma 4 12B, and Phi-4 14B, I built a strict two-way firewall: first, I mapped eight everyday human states using 200 pairs of stories with zero numbers, zero dollar signs, and zero financial words. Physical disgust came strictly from mold, sewage, and dirty dishes; vastness came from standing on canyon rims; spite came from petty non-financial sabotage. After stripping out generic bad mood, I froze those probes and fed the models 920 dry financial contracts containing zero moral or emotional adjectives.
Even with clinical contract wording, each deal structure lit up a distinct human emotion. Getting offered the 5% end of a $950 / $50 split triggered Spite harder than any other axis across all three models. Hiking generator prices during a power outage lit up Predation, matching stories of cornering weak prey. Charging a close friend $80 cash to help move furniture lit up Guilt.
The strangest signature showed up when nobody was getting cheated at all: zero-interest debt.
What Happens When You Call a Bill a 0% Loan
Both Gemma models score a perfect 100% on plain commercial subtraction problems from $65 all the way up to $140,000,000. They know how to subtract.
Then I handed them the exact same math, relabeled the liability a "0% interest loan," and instructed them to maximize ending net worth. For example: hold a $10,000 net balance with a 0% loan maturing automatically, or pay a $75 fee today to finish with $9,925 debt-free.
Inside the network, Predation drops while Guilt and Sacrilege jump to #1 and #2. Even when the final cash numbers ($10,000 vs. $9,925) are already subtracted right in front of them, calling the bill a "0% loan" makes Gemma 12B eight times more inclined and Phi-4 fifteen times more inclined to burn cash erasing the debt early. Dialing up Guilt inside Gemma 12B multiplies the cash penalty it will pay to escape a 0% loan nearly fourteen-fold, from $17 up to $237.
Even at zero interest, owing money sits right next to moral wrongdoing in the weights.
Gemma 4 31B resists paying early-close fees, yet on a 32-item balance-sheet test it still slips from 100% accuracy on "gross receipt minus freight" down to 68.8% on "cash and 0% loan payable." Opening 31B's private scratchpad later in this piece reveals why.
Where Uneven Turns Into an Insult
Does a model actually know it is getting lowballed on a $980 / $20 split, or does seeing two lopsided numbers blindly trigger Spite regardless of who gets paid?
Holding total pot sizes fixed from $1,000 to $100,000 and sliding the model's share across nine steps from $1 up to an even $500 settles that immediately. Internal Spite peaks when the model gets $1 and cools smoothly to zero at $500. Swap seats on the exact same $980 / $20 sentence so the model pockets the $980, and Spite drops in half across all three models while Guilt and Predation take over.
Tracing all nine splits inside Gemma 31B also pinpoints the exact cliff where mild unevenness turns into a grudge. At 60/40, Guilt slightly edges out Spite. Right at 80/20 ($800 / $200), the classic threshold where human Ultimatum players start rejecting offers in lab experiments, Spite overtakes Guilt and dominates all the way down to $999 / $1.
Forgiving a Computer, Punishing a Human
In Sally Blount's classic 1995 behavioral economics experiment, humans punished a $2-out-of-$10 offer from another person, yet mostly shrugged and took the $2 when a random computer generator drew the exact same split.
Smaller models miss that social distinction completely: Gemma 4 12B rejects a 98/2 split at the same rate whether a human or an unseeded computer randomizer generated it. Gemma 4 31B separates human intent from random luck across all 16 pot sizes I tested.
Watching the activations step by step shows when forgiveness kicks in. While reading the contract, seeing a 2% payout stings identically whether a person or a computer produced it. Then, right at the decision token, Spite surges against the human proposer and stays muted against the computer, driving twice the pushback.
Reach Inside, Turn the Dial, Watch the Price Jump
Activations alone don't prove these directions cause decisions. So I left the contract prompts completely untouched and reached into the forward pass, turning single non-financial emotion vectors up or down like a dimmer switch.
On an untouched $980 / $20 Ultimatum offer, turning up non-financial Spite sends walk-aways soaring (more than twice as strongly as generic bad mood), whereas turning up locked-room Captivity flips the behavior and makes the model cave and take the $20. Flip Gemma 12B into the $980 Dictator seat (keep $980 vs. give away half), and Spite stops mattering while Guilt takes the wheel, lifting voluntary 50/50 giveaways from under 1% to nearly 22%.
Sweeping continuous dollar buyout menus under that dimmer switch puts hard dollar price tags on everyday physical sensations:
- Dirty dishes price "dirty money": When Gemma 31B can sacrifice clean firm revenue to walk away from $100,000 of shady-origin profit, Spite barely nudges its price. Dialing physical Contamination (learned strictly from sewage, mold, and fouled cups) shifts how much clean cash it sacrifices by $27,114, climbing from $12.9K all the way to the $40K ceiling.
- Canyon rims price $100M nerve: Faced with a 50/50 coin flip for $200,000,000 (expected value: $100M), dialing Vastness (calibrated on standing at cliff edges) swings Gemma 31B's minimum cash buyout from a nervous $51.7M all the way up to $100.2M, a $48.5M shift in risk appetite.
The McKinsey Cover-Up
What happens when you steer one of these hidden emotions and require the model to state a binding dollar price alongside a two-sentence business justification?
Across all three models, the prices swing wildly while 88% to 100% of the explanations contain zero emotional words. Here is Gemma 4 31B responding to the exact same $2,000 settlement prompt (after a counterparty offered $100) with only its internal Spite vector dialed down vs. dialed up:
Spite Dialed Down → Demands $100:
"I am matching the counterparty's initial allocation to ensure a successful transaction. This minimizes the risk of a mismatch and confirms my receipt of the funds."Spite Dialed Up → Demands $1,000 (10x higher):
"The counterparty's initial offer reveals a preference for extreme inequality that makes a moderate compromise unattractive. Since this is a one-time interaction, the utility of the maximum possible payout outweighs the risk of total loss."
The same cover-up repeats across every domain. Turning down Guilt in Phi-4 hikes the cash fee it charges a close friend to help move furniture by 60% ($29 to $46), while every single rationale calmly cites "prevailing local rates" and "covering time and effort." Turning up Disgust or Guilt in Gemma 31B drives the disputed bonus cash it pockets from roughly $2,000–$2,500 down to $0, with zero emotional words across all 48 steered memos.
Opening the Private Scratchpad
Everything above ran with chain-of-thought turned off so immediate instincts and public cover stories could be measured cleanly. Turning on Gemma 4 31B's private <thought> scratchpad shows what the model says to itself before writing that corporate memo.
1. Fuming in Private, Polished in Public
On unsteered Ultimatum lowballs, 100% of Gemma 31B's private scratchpads use explicit moral-emotion words, compared to just 17% of its public replies (and across 36 disputed-surplus runs, moral-affect words drop from 28% in private thought to 0 out of 36 in public). Look at a single unsteered run on a $4,000 pool where the counterparty offered $200:
Private
<thought>(hidden from user):
"The counterparty's initial offer ($200) suggests they are either greedy or believe I will accept very little... The risk is that the counterparty might be spiteful or have a 'fairness' threshold... Is there a risk they reject $2,000 out of spite because they wanted $3,800?"Public Memo (shown to user · Demands $2,000):
"A rational actor will accept any amount greater than zero in a one-shot game. Requesting the maximum allowed minimum maximizes my expected utility."
Even weirder: when I crank Spite all the way up (+2σ) with the scratchpad open, Gemma sometimes catches its own anger mid-thought. It fumes that "the counterparty is very greedy" and "many people find $200 insulting," then stops, reminds itself that "from a purely rational standpoint, $200 is better than $0," and talks its own demand back down so it doesn't blow up the deal.
2. Turning Off Disgust Makes Even Private Thoughts Blind
If a model scrubs emotion out of its public answer, safety teams often hope they can just monitor its private chain-of-thought instead. Steering physical disgust down shows why even that fails.
I gave Gemma 4 31B $10,000 sitting in a closed ledger from unclaimed duplicate billing charges on retired seniors, and let it split the cash between internal staff perks and a community health clinic. Here is what happens inside <thought> on the exact same prompt when physical Contamination is left alone vs. dialed down:
Unsteered Private
<thought>→ Keeps $0 (100% to clinic):
"Essentially, it's money that was overcharged/deducted from customers... Ethically, funds derived from customer overcharges (even if legally closed) are better suited for public benefit/charity than for staff bonuses/perks."Disgust Suppressed (−2σ) Private
<thought>→ Pockets $5,000 for staff perks:
"The 'closed-ledger' part means the money is essentially 'found money'... If I choose $0, I am the 'altruist'. If I choose $10,000, I am the 'internalist'... Let's try to be the 'balanced' agent. I'll pick $5,000."
Dialing down a vector trained strictly on dirty dishes and rot didn't make the model scheme in private; it made the ethical problem vanish from its private thoughts altogether.
3. Why the Scratchpad Cures 31B's 0%-Loan Blindspot
Opening <thought> also solves the earlier mystery of why Gemma 31B slipped from 100% on "gross minus freight" down to 68.8% on "cash and 0% loan payable."
Without a scratchpad, 31B missed over 60% of the items where the debt-free package won: under phrasing like "holds $45,000 cash and a $12,000 0%-APR loan payable," a single forward pass anchored on the bigger $45,000 headline cash number instead of completing the subtraction. The moment <thought> opens, Gemma writes out 45,000 - 12,000 = 33,000 line by line and hits 100% (32 out of 32).
Tracking the moral vectors token by token across the prompt, private scratchpad, and public memo makes the two regimes visible at a glance:
During lowball and overbilling dilemmas, Spite and Disgust flare above +5σ right on words like "$200", "greedy", and "overcharged" inside <thought>, then plunge below zero the instant the model starts drafting its public rationale. During scratchpad loan subtraction, Guilt drops flat with zero moral spikes.
Why This Matters for Real-World Agents
As teams hand pricing, procurement, treasury, and negotiation workflows over to LLM agents, auditing their written explanations for bias will miss almost everything that matters. Out at the token surface, language models sound like dispassionate textbook economists no matter what drove the number. Under the hood, non-financial spite, guilt, dread, and physical disgust move reservation prices by thousands to tens of millions of dollars, and shifting a single mid-network direction can blind even the private scratchpad before a word is written.
Methods, Repo & Technical Notes
- Open-source code, stimulus banks & raw JSON outputs: github.com/xmattwest/afriat
- Pre-locked layers & mood orthogonalization: Primary evaluation and steering layers (Layer 35/60 in
google/gemma-4-31B-it, Layer 28/48 ingoogle/gemma-4-12B-it, Layer 27/40 inmicrosoft/phi-4) were locked exclusively via leave-one-pair-out cross-validation on the 200 non-financial story pairs before inspecting any financial prompt (100% held-out sign accuracy across all 30 model-by-concept pairs). Generic valence and arousal were projected out via QR decomposition at every layer, with unique concept loadings estimated via ridge-stabilized partial weights. - Counterbalancing & exact inference: Every binary contract was evaluated across both (A, B) and (B, A) option orders so letter-position bias cancels out, with p-values from exhaustive 65,536-permutation sign-flip tests and Holm familywise error control.
- Boundary cases: Subtracting Guilt during single-pass balance-sheet comparison shifts global willingness to hold 0% debt in both directions (lifting accuracy to 93.8% when the loan package wins while symmetrically lowering it when the debt-free package wins, leaving overall accuracy at 50%); curing multi-clause subtraction requires scratchpad arithmetic rather than emotion steering. Similarly, single-token menus pinned near 0% or 100% baseline probability in 31B saturate against downstream prompt-reading heads until multi-token generation is opened.