AI audio summaries of research can help — and err ^{CEFR B2}

Ian Flynn of the University of Pittsburgh and Sean Peters of Middlebury College tested Google’s NotebookLM by converting published research papers into podcast-style audio overviews. In their October paper in the American Geophysical Union journal Perspectives of Earth and Space Scientists, they evaluate both the strengths and the limits of these AI-generated summaries.

The researchers transformed three papers about volcanism on Mars into NotebookLM audio. The sources differed in format: a five-page letter with three figures and one table, a 29-page research paper with nine figures and four tables, and a 23-page review with 11 figures. Google describes the overviews as “deep-dive discussions between AI hosts that provide in-depth summaries of the key topics in your uploaded sources,” and the authors found the results engaging and written in plain language.

Flynn and Peters praised the use of fairly accurate and creative analogies, which can help teaching and make technical work more accessible. Yet every overview contained errors, and mistakes tended to appear toward the end of the audio. Some errors involved clear overreach—for example, one overview extrapolated from a volcanic feature to suggest liquid water and the possibility of life, claims not present in the original paper. The authors see useful roles for these overviews in learning and in demonstrating how to interpret research, but they conclude: “Overall, NotebookLM’s generated audio overviews can be a useful tool for the planetary science community, but it is doubtful to be a replacement for critically reading the source material.”