Bidirectional effects of speech and gesture: evidence from a novel levels-of-congruence paradigm.

Overview

1. Department of Language Science and Technology, Saarland University, Saarbrücken, Germany

Abstract

Introduction: Previous research has shown that speech and gesture are integrated during comprehension, with differences in response times and electrophysiological activity observed between incongruent and congruent gesture—speech pairings. Beyond these incongruency effects, it remains unclear whether comprehenders use gestures to adapt or extend conventional interpretations for novel communicative purposes.

Method: The current study examined the bidirectional influence of speech and gesture by employing a novel levels-of-congruence paradigm, in which the same metaphor (e.g., Men are oaks) was paired with a gesture depicting either its preferred interpretation (e.g., men are robust) or a less preferred but possible interpretation (e.g., men are tall). In Experiment 1, participants rated the meaningfulness of preferred and possible interpretations across three gesture conditions (gesture-of-preferred, gesture-of-possible, non-gestured). In Experiment 2, participants selected a single interpretation under the same conditions. In Experiment 3, participants viewed clips of an actor producing the same gesture and utterance combination (e.g., flexing both arms while saying Men are oaks) in one of three multimodal conditions: no additional cues (muted, blurred lips), only lip movements (muted, visible lips), or audible speech. Then they rated how well the gesture matched the gestured interpretation (e.g., robust), an alternative interpretation consistent with the speech but not the gesture (e.g., tall), and an unrelated interpretation (e.g., hikers).

Results: In Experiments 1 and 2, participants consistently rated the preferred interpretation as the most meaningful and selected it most often across all gesture conditions. However, gestures depicting the possible interpretation significantly increased both the meaningfulness ratings and the likelihood of being selected of the possible interpretation. In Experiment 3, the presence of speech cues significantly increased ratings for the alternative interpretation compared with the condition without additional cues.

Discussion: These findings demonstrate bidirectional interactions between speech and gesture during metaphor comprehension. Gestures constrained metaphor interpretation, whereas speech broadened the range of interpretations attributed to gestures. Overall, the results support the integrated-systems hypothesis by showing that speech and gesture jointly contribute to the construction of meaning.

Reproduced under the paper's license (CC BY), from the paper cited above.

Code

No file of the authors' code could be read here: it is described below, and read at its source.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

• 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
• 0 scripts, each with its path and the digest of its content;
• no match between paragraphs and code yet;
• neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data availability statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found in the article/Supplementary material. The online repository can be accessed here: https://osf.io/cfk7j.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 2, 28 September 2026

• Authors: added Laura Pissani (0000-0003-2884-4515); Vera Demberg (0000-0002-8834-0020); removed Laura Pissani; Vera Demberg

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, pages, dates, 2 authors, 5 keywords, 1 funder, 34 references.

Cite

This paper

Pissani, L., & Demberg, V. (2026). Bidirectional effects of speech and gesture: evidence from a novel levels-of-congruence paradigm. Frontiers in psychology, 17, 1851588. https://doi.org/10.3389/fpsyg.2026.1851588

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.3758/s13415-023-01074-8 [code]

Electrophysiological evidence for the enhancement of gesture-speech integration by linguistic predictability during multimodal discourse comprehension.

Journal: Cognitive, affective & behavioral neuroscience

In common: cognitive, 8 references

[2] doi:10.1002/hbm.26797 [code]

Multisensory integration of speech and gestures in a naturalistic paradigm.

Journal: Human brain mapping

In common: cognitive, 5 references

[3] doi:10.1016/j.cortex.2024.08.008 [code]

Reduced on-line speech gesture integration during multimodal language processing in adults with moderate-severe traumatic brain injury: Evidence from eye-tracking.

Journal: Cortex; a journal devoted to the study of the nervous system and behavior

In common: cognitive, 3 references

[4] doi:10.1017/s1366728924001093

Semantic processing of iconic signs is not automatic: Neural evidence from hearing non-signers.

Journal: Bilingualism (Cambridge, England)

In common: cognitive, 3 references

[5] doi:10.1038/s41598-022-25720-2

Prestimulus oscillatory brain activity interacts with evoked recurrent processing to facilitate conscious visual perception.

Journal: Scientific reports

In common: cognitive, 3 references

[6] doi:10.1162/opmi.a.351

Child-Directed Speech Facilitates Semantic Role Learning: A Machine Learning Approach.

Journal: Open mind : discoveries in cognitive science

In common: cognitive, 3 references

[7] doi:10.1016/j.nicl.2026.103944

Striatal dopamine synthesis capacity in Parkinson's disease: Effects of age, sex, and body mass index in a large [<sup>18</sup>F]fluorodopa PET cohort.

Journal: NeuroImage. Clinical

In common: 3 references

[8] doi:10.1186/s40478-023-01674-y [code]

Accurate digital quantification of tau pathology in progressive supranuclear palsy.

Journal: Acta neuropathologica communications

In common: 3 references

[9] doi:10.1177/09567976251331041

Co-Speech Hand Gestures Are Used to Predict Upcoming Meaning.

Journal: Psychological science

In common: cognitive, 2 references

[10] doi:10.1038/s41539-024-00274-2 [code]

Dynamic brain networks in spontaneous gestural communication.

Journal: NPJ science of learning

In common: 2 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

<!-- oscr-badge:start -->
[![OSCR](https://openscicode.org/badge/doi/10.3389/fpsyg.2026.1851588.svg)](https://openscicode.org/badge/doi/10.3389/fpsyg.2026.1851588)
<!-- oscr-badge:end -->

.. oscr-badge:start

.. image:: https://openscicode.org/badge/doi/10.3389/fpsyg.2026.1851588.svg
   :target: https://openscicode.org/badge/doi/10.3389/fpsyg.2026.1851588
   :alt: OSCR

.. oscr-badge:end

<!-- oscr-badge:start -->
<a href="https://openscicode.org/badge/doi/10.3389/fpsyg.2026.1851588"><img src="https://openscicode.org/badge/doi/10.3389/fpsyg.2026.1851588.svg" alt="OSCR"></a>
<!-- oscr-badge:end -->

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion: code errors and mismatches between the code and the paper about this paper and its code, from readers and its authors, beside the paper and its tracing map.

Report a mismatch between the code and the paper · Report a code error (signed in)

Reproductions: reports from readers who ran the authors' code and did not get the paper's result, what they ran, with which environment, commit and data.

Report a reproduction failure (signed in)

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.