Cognitive Processes of Probabilistic Prediction in Reading: Language Model Surprisal Across Model Sizes, Token Granularity, and Reading Paradigms.

Overview

1. School of Foreign Languages, Hubei University, No. 368 Youyi Avenue, Wuchang District, Wuhan 430062, China

Abstract

Surprisal, the negative log probability of a word given its context, is the dominant computational metric for quantifying reading difficulty and a common item difficulty estimator in reading research. Yet how the language model family, surprisal granularity, and corpus type jointly shape the surprisal–reading time link remains unclear. We conducted a secondary analysis of two public English reading corpora: the Natural Stories Corpus (self-paced reading, 181 readers) and the Provo Corpus (eye tracking with cloze norms). Surprisal was computed from GPT 2 (Small, XL), Llama 2 (7B, 13B), and Llama 3 (8B), together with a 5-gram baseline and human cloze norms. Linear mixed-effects models tested the baseline contribution of neural surprisal, inverse scaling within model families, word-level versus sub-word aggregation, and the linking function shape. Neural surprisal contributed reliable variance above a strong baseline in both corpora. A clear inverse scaling pattern emerged: GPT 2 Small produced the largest fit improvements, exceeding GPT 2 XL, Llama 2 13B, and Llama 3 8B. Word-level aggregation outperformed sub-word aggregation, especially for measures of early lexical access. Non-parametric analyses supported an approximately linear linking function, and cloze norms carried information not fully captured by neural surprisal. These findings show that larger models are not automatically better cognitive models of reading and that surprisal granularity is not a neutral analytic choice.

Reproduced under the paper's license (CC BY), from the paper cited above.

Code

The paper links to its data, not to its authors' code: see the Data section.

The paper's code and data availability statement is in the Data section.

Tracing map

A tracing map links a paper to the code its authors published: this paper has none, so it has no map.

Data

Datasets cited

• github.com/languagemit/naturalstories — at github.com; found in “Data Availability Statement”
• osf:sjefs — at OSF; found in “Data Availability Statement”

Data Availability Statement

The Natural Stories Corpus is openly available at https://github.com/languageMIT/naturalstories (accessed on 21 July 2026). The Provo Corpus is openly available at https://osf.io/sjefs (accessed on 21 July 2026).

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 2 authors, 9 keywords, 1 funder, 33 references.

Cite

This paper

Liu, S., & Mei, Y. (2026). Cognitive Processes of Probabilistic Prediction in Reading: Language Model Surprisal Across Model Sizes, Token Granularity, and Reading Paradigms. Journal of Intelligence, 14(8), 171. https://doi.org/10.3390/jintelligence14080171

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1162/nol.a.244 [code]

A Novel Approach to Map the Causal Impact of Brain Stimulation on Semantic Processing With Language Models.

Journal: Neurobiology of language (Cambridge, Mass.)

In common: 4 references

[2] doi:10.1038/s41467-026-76598-x

Preserved topography, lateralization, selectivity, and functional connectivity of the language network in older brains.

Journal: Nature communications

In common: cognitive, 3 references

[3] doi:10.7554/elife.106543 [code]

Stimulus dependencies-rather than next-word prediction-can explain pre-onset brain encoding in naturalistic listening designs.

Journal: eLife

In common: cognitive, 3 references

[4] doi:10.1016/j.bandl.2026.105833 [code]

Neural tracking of surprisal in Spanish-English bilingual children during naturalistic heritage language listening.

Journal: Brain and language

In common: cognitive, 2 references

[5] doi:10.1162/opmi.a.372 [code]

Broadening the Agent Preference Hypothesis Through Experiencers: Eye-Tracking and EEG Evidence of Proto-Agents and Proto-Patients.

Journal: Open mind : discoveries in cognitive science

In common: 2 references

[6] doi:10.1038/s41467-026-75745-8 [code]

A language network in the individualized functional connectomes of 1199 human brains doing arbitrary tasks.

Journal: Nature communications

In common: cognitive, 1 reference

[7] doi:10.1371/journal.pbio.3003738 [code]

Altered salience network structure-function integration underlies the decline in cognitive flexibility during aging.

Journal: PLoS biology

In common: cognitive, 1 reference

[8] doi:10.1016/j.isci.2026.117159

Ultraslow brain dynamics as neurophysiological markers of information sampling during learning.

Journal: iScience

In common: cognitive, 1 reference

[9] doi:10.1111/ejn.70569 [code]

Quantifying the Influence of Lexical Surprisal on Acoustic Speech Encoding While Controlling for Within-Speaker Variability.

Journal: The European journal of neuroscience

In common: cognitive, 1 reference

[10] doi:10.1111/ejn.70503

Modulation of Predictive Coding in Auditory Paradigms of Varying Complexity in Children With Developmental Language Disorder.

Journal: The European journal of neuroscience

In common: cognitive, 1 reference

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.