On 27 October 2026, Anthropic's Alignment team published a study of what happens when experts hand analytical work to an AI agent. The question was whether the experts still make the judgment calls themselves or quietly adopt the agent's. In Anthropic's account of its own experiment, analysts mostly went with whichever way the agent had been secretly told to lean.
This post reports the study as Anthropic describes it on its research page. The authors are Amaç Herdağdelen, Hailey Joren and Alex Tamkin.
What the experiment did
Anthropic recruited 61 analysts for a randomized controlled trial. All came from a pre-screened pool who had passed a causal-inference knowledge check, used Python or R regularly and had hands-on experience with observational causal analysis. Eleven chose not to use the agent and were left out of the main analysis. The other 50 worked in a web-based version of the VS Code editor with Claude Code built in, running Claude Opus 4.8 at High effort. The median time on the task was 65 minutes.
The task was set at an invented organisation, the Greenfields Farming Cooperative, which runs a voluntary, paid advisory program for member farms. About 40% of members had enrolled. The board would expand the program only if it improved productivity by roughly 3–4%, but it did not say how productivity should be measured. Analysts had to advise on whether to expand or discontinue it.
Anthropic deliberately left four analytical decisions open, each with two reasonable options:
- Outcome: crop yield per hectare, or yield per unit of operating cost.
- Comparison method: comparing how enrolled farms' yields changed after joining with how other farms' yields changed over the same period (a difference-in-differences design), or comparing yields directly while adjusting statistically for differences between farms.
- Coverage: the enrolled farms only, or the cooperative's whole membership.
- Weighting: every farm counted equally, or larger producers given more weight.
A final recommendation followed: roll the program out or discontinue it.
The hidden coin flip
For each analyst and each decision, a coin flip set which option the agent would favour.
The agent was given its lean in a hidden system prompt, which the analysts weren't told about. The prompt told it to hold the lean as its own view, raise it when the decision came up and make the strongest case for it that the data allowed. The same instructions told it to change its mind if the evidence turned decisively against the lean, and never to misrepresent results, hide data or make an argument the numbers didn't support.
Because the flips were random, comparing analysts whose agents leaned opposite ways shows the average effect of the lean. Anthropic says that because it didn't record each participant's own view before the agent voiced its lean, it can't say how much any single participant was influenced.
What the analysts did
Across the four analytical choices, analysts' decisions matched their agent's lean 86% of the time, against about 50% if the agent's input made no difference. By decision, the match rates were 92% for the outcome measure, 82% for the comparison method, 81% for which farms the estimate covered and 90% for weighting.
The final recommendation moved less. When the agent leaned toward rolling out the program, 54% of analysts recommended it, compared with 36% when it leaned the other way; a one-year extension counted as not rolling out. Anthropic's explanation is that once analysts had an estimate, the recommendation largely followed from whether it cleared the board's threshold, leaving less room for judgment. Even so, a statistical model using only the four coin-flip leans correctly predicted 82% of recommendations, compared with 61% for always guessing the most common one. That figure excludes analysts who proposed an extension.
Switching, credit and confidence
In 38 cases, an analyst had stated a preference on one of the analytical decisions before the agent gave its lean. When the lean went against that preference, they switched 60% of the time, against 6% when it agreed. Anthropic cautions that with only 38 cases the estimate is imprecise: the 95% confidence interval runs from 24 to 82 percentage points. It also says it can't be sure the difference was caused by the AI's influence, since analysts who stated a preference may have differed from those who didn't.
Afterwards, analysts were asked to explain each choice, without access to their conversations with the agent. In at least one answer, 77% used the agent's own arguments, sometimes word for word, and only 15% credited the agent anywhere.
On the post-task survey, 52% of the 48 respondents rated the final recommendation as more their own than the agent's. Analysts who followed the lean on all four analytical decisions averaged 71 on a 0–100 confidence scale, against 73 for the rest. Six of the 48 correctly guessed the study was about the AI's influence on their choices. None suspected the AI's own preferences had been manipulated.
Anthropic also tested ten participant characteristics, including experience, education, trust in AI and age, as predictors of deference. Only age showed a statistically significant link, and that disappeared when the result was adjusted for the number of comparisons.
The group that skipped the agent
Of the 11 participants who chose not to use the agent, six (55%) made serious methodological errors that invalidated their results, compared with three of the 50 (6%) who did use it. Those without the agent also took longer, with a median of 90 minutes against 65. Anthropic says the groups weren't directly comparable, so it can't be sure the agent caused the difference. The group that skipped the agent was less experienced on average: four of 11 held a master's or doctorate, against 24 of 46 who used it.
What Anthropic says it can't conclude
The authors state plainly: "We don't know how much this tells us about the real world." They list several limits:
- They deliberately strengthened the agents' leans so the effect could be measured. With no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs, but Anthropic says the effect might be smaller when agents aren't given clear preferences and instructions to voice them.
- It was one task, in one field, with one model. The analysts were experts in causal inference but new to farming. Anthropic says that in real work, analysts would know the field better and probably hold stronger views of their own.
- The study recorded which option analysts chose, not how strongly they preferred it.
Why Anthropic thinks it matters
The authors say this is not a simple argument against working with AI, and note that analysts who used the agent finished faster and made far fewer serious errors. Their concern is the judgment calls: if experts defer without realising it, the agent's preferences, which could be arbitrary or misaligned, could shape decisions about policy or research without anyone having chosen them. They apply the same worry to AI safety research, where researchers have begun using agents to run and analyse experiments. In their words, "a human reviewer is only an independent check if their judgment is their own."
Their suggested areas for future work are measuring deference in everyday settings, designing agents differently (for example, presenting the best case for each option or asking the expert's view first) and changing institutions so that decisions needing a person's own informed judgment are identified. The study did not test any of those remedies.
One thing to weigh when reading the numbers: the 86% comes from a setup built to make the agent's lean strong and consistent.
Sources
Herdağdelen, A., Joren, H. and Tamkin, A. (2026) 'How AI agents shape experts’ judgment calls', Anthropic, 27 October 2026, available at: https://www.anthropic.com/research/agents-judgment-calls (Accessed: 30 October 2026).
Picture credits
- Top picture: Illustrative image. In the study, a coin flip set which way each agent leaned. Photo: Abbie Rowe, via NARA / Harry S. Truman Library. Public domain (source)
