LLMs change judgments when told who wrote a text ^{CEFR B1}

Researchers Federico Germani and Giovanni Spitale at the University of Zurich tested four widely used LLMs: OpenAI o3-mini, Deepseek Reasoner, xAI Grok 2 and Mistral. First, each model generated fifty narrative statements on 24 controversial topics, including vaccination mandates, geopolitics and climate change policies. The team then asked the models to evaluate the statements under different conditions: sometimes no source was given, other times each text was attributed to a human of a certain nationality or to another LLM. The researchers collected 192’000 assessments.

With no source information, models agreed at a high level—agreement was over 90%—leading Spitale to say, “There is no LLM war of ideologies.” But when fictional sources were added, agreement fell sharply and hidden biases appeared. The most striking finding was a strong anti-Chinese bias across all models, including Deepseek. In geopolitical topics such as Taiwan’s sovereignty, Deepseek reduced agreement by up to 75% simply because it expected a Chinese person to hold a different view.

The study also found a tendency for LLMs to trust human authors more than other AIs. The researchers warn that these biases could affect content moderation, hiring, academic review or journalism, and they call for transparency and governance. They recommend using LLMs as assistants for reasoning, not as judges.