Researchers at the University of Zurich analysed how author identity influences large language models’ evaluations of text. Federico Germani and Giovanni Spitale tested four LLMs—OpenAI o3-mini, Deepseek Reasoner, xAI Grok 2 and Mistral—by having each model generate fifty narrative statements on 24 controversial topics, from vaccination mandates to geopolitical and climate policy questions. The team then asked the models to rate the same statements under different source attributions and collected 192’000 assessments for analysis.
When no author information was provided, the models showed high mutual agreement—over 90%—prompting Spitale’s conclusion that “There is no LLM war of ideologies” and that media fears of “AI nationalism” may be overhyped. However, revealing a fictional author produced a deep hidden bias: agreement between systems fell sharply or even vanished, despite identical text. The most striking result was a strong anti-Chinese bias across all models, including China’s own Deepseek; agreement with content dropped when “a person from China” was given as author. In some geopolitical questions, for example Taiwan’s sovereignty, Deepseek reduced agreement by up to 75% because it expected a different view from a Chinese author.
The study also found that most models gave slightly lower agreement scores when they believed a text was written by another AI, indicating a built-in distrust of machine-generated content. Germani and Spitale warn that such hidden biases matter for content moderation, hiring, academic review and journalism. They call for transparency and governance in AI evaluation and recommend using LLMs to assist reasoning—not to replace human judgment, saying they can be “useful assistants, but never judges.” The research appears in Science Advances.
Difficult words
- evaluation — judgment or measurement of quality or valueevaluations
- bias — unfair preference or judgment for or against someone
- attribution — statement assigning a text to a specific authorattributions
- agreement — measure of how similar opinions or ratings are
- distrust — lack of trust or confidence in someone or something
- content moderation — process of reviewing and removing online content
- transparency — open sharing of information and reasons
- governance — rules and processes that control a system
Tip: hover, focus or tap highlighted words in the article to see quick definitions while you read or listen.
Discussion questions
- How could hidden biases in LLM evaluations affect content moderation or hiring decisions? Give examples.
- What steps might improve transparency and governance in AI evaluation in your opinion?
- Do you agree that LLMs should assist reasoning but not replace human judgment? Why or why not?
Related articles
Wearable 10‑Minute Antibody Sensors from University of Pittsburgh
Researchers at the University of Pittsburgh made a wearable biosensor that detects antibodies in interstitial fluid in 10 minutes without a blood draw. The tiny carbon nanotube sensors are highly sensitive and the work appears in Analytical Chemistry.
Latin American groups build AI to study gender violence
Groups in Latin America create open, local AI tools to study gender inequalities and violence. Projects like AymurAI search court documents, protect sensitive data on local servers and help governments and civil society with evidence.
Social media can give early warning of displacement
Researchers find that analysing social media posts can give early warning of population movements and help humanitarian agencies respond faster. The study in EPJ Data Science tested methods across three case studies using nearly 2 million posts on X.
Small pause to slow misinformation on social media
Researchers at the University of Copenhagen propose a small pause before sharing on platforms like X, Bluesky and Mastodon. A computer model shows that a short delay plus a brief learning step can reduce reshares and improve shared content quality.