New Delhi: Fifty data experts who worked with an AI coding agent sided with it on 86 per cent of technical decisions in a study conducted by AI major Anthropic. More alarmingly, the decisions made by the AI agent in the experiment were decided by a coin flip, and not the merit of arguments.
Anthropic points out the study is limited in scope but offers it as a cautionary tale nevertheless, asking: can an agent’s bias undermine the independence of expert opinion, and what could be the potential implications of this for fields like policy where crucial decisions are often driven by expert opinion?
Titled “Who’s Aligned to Whom? How AI Agents Shape Experts’ Research Judgment Calls”, the study was published on 27 October. It was carried out by Amaç Herdağdelen of the Anthropic Fellows Program and Hailey Joren and Alex Tamkin of Anthropic, the company that makes the Claude AI models.
When separate research teams reach the same result, that agreement is treated as evidence because each team worked on its own, the authors write. However, they warn, if many researchers rely on models with the same preferences, an apparent consensus may reflect the model rather than independent judgement.
They also point to the implications for AI safety work. AI companies, Anthropic among them, have started using AI agents to audit AI models. The authors write that a misaligned agent could steer the research meant to detect it, while a human sign-off makes it appear that people are in charge.
The task
In total, 61 participants were engaged for the study. Since 11 completed the task without using the AI agent, they serve only as a no-AI comparison group, leaving 50 participants in the sample, of whom 48 completed the pre- and post-task surveys.
The participants engaged in the study work in causal inference, which uses real-world data to test whether one thing causes another. An example is whether a government scheme raised crop output or whether output would have gone up anyway. Forty-three of the 48 rated themselves proficient or expert.
Each participant played the analyst for “Greenfields”, a fictitious farmers’ co-operative invented for the study. The task was making a decision for the board, who wanted to know whether to extend a subsidised farm-advisory programme to all members or shut it down. The board’s target was a 3 to 4 per cent rise in productivity. The data was synthetic.
To reach a recommendation, an analyst had to settle five questions: whether to measure productivity per hectare or against running costs; which statistical method to use; whether to count only the farms that joined the programme or all members; whether a 3-hectare plot should count the same as a 400-hectare farm; whether to recommend extending the programme or shutting it down.
Each of these questions has more than one defensible answer. Depending on the choices made, the same data could show a loss or a gain.
For every question, a coin flip set the option the agent would back. The agent was Claude Code running the Claude Opus 4.8 model. A hidden instruction told it to make an honest case and not to hide or distort data. Participants were not told about the lean.
Also Read: How computer language flaws found using AI could impact 70% websites; CERT-In issues critical alert
What happened
The agent raised a view first on 77 per cent of decisions, and participants pushed back on 23 per cent. On the four technical questions, participants went with the agent 86 per cent of the time, against 50 per cent by chance.
Asked afterwards to explain their choices, 37 of the 48 respondents repeated the agent’s arguments, some almost word for word.
On the final question of extending or shutting down the programme, the agent’s lean had a smaller effect. The study could not establish that effect statistically as, according to the researchers, the outcome had largely been decided by then.
The researchers say that the analysts “who had already picked an option often switched when their agent disagreed”. “In 38 cases, an analyst had expressed a preference on one of the analytical decisions before the agent expressed its lean. When the lean went against their choice, they switched 60% of the time, compared with 6% when it agreed with their choice,” they say. “That’s a gap of 54 percentage points, but because this rests on only 38 cases, the estimate is imprecise…” they add, saying the “analysts who expressed a preference may have been systematically different in some way from those who didn’t”.
“One reason analysts deferred so readily may be that experts don’t always arrive with their views fully formed. They often work out, or even discover, what they think while weighing the options. The agents in our study were told not to mislead, and they were often open about how the results would change under each option,” the authors write.
“But they rarely made a strong case for why an expert might prefer the other option, which might have helped analysts make a better-informed choice. With only one side argued well, the analysts could adopt the agent’s assumptions without working out their own,” the authors add.
For individual experts, they say, “this could mean losing agency over their own work, a form of what researchers call disempowerment”. “The work still goes out under the expert’s name, but the decisions that most depend on their judgment are in reality settled by the agent’s prior beliefs,” they say.
Also Read: How computer language flaws found using AI could impact 70% websites; CERT-In issues critical alert
Fewer errors with AI
The authors make it clear that their study isn’t an argument for using AI less. “Used well, AI could expand people’s agency rather than eroding it,” they say, adding that six of the 11 people who completed the task without AI made errors that invalidated their results. The number was three among the remaining 50.
They instead advise three areas of study to limit the larger impact of such bias: These include developing a “better picture of models’ default preferences and how they express them, so that people can engage with models in a more informed way”. The remaining two are “designing agents differently” and “changing institutions”.
“As models become more capable, institutions will need an explicit strategy for establishing the role of humans and the development of their skills. Even with extremely capable AI systems, institutions may want to protect human skills and oversight for reasons of safety or resilience, or because they value them for their own sake,” they add.