LLMs change judgments when told who wrote a text ^{CEFR B2}

Researchers at the University of Zurich analysed how author identity influences large language models’ evaluations of text. Federico Germani and Giovanni Spitale tested four LLMs—OpenAI o3-mini, Deepseek Reasoner, xAI Grok 2 and Mistral—by having each model generate fifty narrative statements on 24 controversial topics, from vaccination mandates to geopolitical and climate policy questions. The team then asked the models to rate the same statements under different source attributions and collected 192’000 assessments for analysis.

When no author information was provided, the models showed high mutual agreement—over 90%—prompting Spitale’s conclusion that “There is no LLM war of ideologies” and that media fears of “AI nationalism” may be overhyped. However, revealing a fictional author produced a deep hidden bias: agreement between systems fell sharply or even vanished, despite identical text. The most striking result was a strong anti-Chinese bias across all models, including China’s own Deepseek; agreement with content dropped when “a person from China” was given as author. In some geopolitical questions, for example Taiwan’s sovereignty, Deepseek reduced agreement by up to 75% because it expected a different view from a Chinese author.

The study also found that most models gave slightly lower agreement scores when they believed a text was written by another AI, indicating a built-in distrust of machine-generated content. Germani and Spitale warn that such hidden biases matter for content moderation, hiring, academic review and journalism. They call for transparency and governance in AI evaluation and recommend using LLMs to assist reasoning—not to replace human judgment, saying they can be “useful assistants, but never judges.” The research appears in Science Advances.