OSCR

Representations in vision and language converge in a shared, multidimensional space of perceived similarities.

Code ↔ Paper

17 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 17 matches · 2 of them tie a paragraph to a whole file, not to given lines: weak matches, whose lines are not tinted
  1. [1] § Methods › Construction of baseline visual and linguistic RDMs ↔ scripts/make_mpnet_rdm.py, lines 1–40 · score 0.95 · purely linguistic representational, linguistic baseline, AlexNet, sentence embeddings, pairwise correlation, MPNet
  2. [2] § Methods › Construction of baseline visual and linguistic RDMs ↔ scripts/make_dino_rdm.py, lines 1–50 · score 0.89 · supervised visual baseline, pairwise correlation distances, vision transformer, model RDMs, natural scene images, DINOv2
  3. [3] § Methods › Experimental design › Estimating the representational dissimilarity matrices ↔ scripts/compute_rcnn_rdms.py, the whole file · a weak match · score 0.89 · neural networks, category trained RCNN, random seeds, correlation distance, representational geometries, model RDMs
  4. [4] § Methods › Experimental design › Estimating the representational dissimilarity matrices ↔ scripts/make_mpnet_rdm.py, lines 1–40 · score 0.85 · extracted sentence embeddings, MPNet, sentence caption, representational geometries, natural scene images, assembled
  5. [5] § Methods › Statistical analysis ↔ scripts/ma_searchlight_nnls_rdms_joint.py, lines 1–62 · score 0.85 · searchlight brain RDMs, joint model, cross validated, Pearson correlations, brain predictions, behavior predicted
  6. [6] § Results › LLM-trained visual ANNs capture visual and linguistic similarity spaces ↔ scripts/plot_baseline_vs_rcnn_comparisons.py, lines 1–53 · score 0.84 · cross validated NNLS, RCNN comparisons, LLM trained RCNN, category trained RCNN, behavioral RDMs predicted, MPNet
  7. [7] § Methods › Statistical analysis ↔ scripts/ma_searchlight_nnls_rdms_joint.py, lines 1–62 · score 0.83 · fold cross validation, training dissimilarities, NNLS weights, Pearson correlation, behavior predicted, behavioral RDMs
  8. [8] § Results › LLM-trained visual ANNs capture visual and linguistic similarity spaces ↔ scripts/plot_rcnn_model_fits.py, lines 1–45 · score 0.80 · LLM trained RCNN, category trained RCNNs, prediction accuracies, RCNN RDM, FDR corrected, Pearson correlation
  9. [9] § Methods › Statistical analysis ↔ scripts/ma_searchlight_nnls_rdms.py, lines 1–63 · score 0.79 · fold cross validation, training dissimilarities, NNLS weights, Pearson correlation, behavioral RDMs, split
  10. [10] § Results › LLM-trained visual ANNs capture visual and linguistic similarity spaces ↔ scripts/plot_within_rcnn_comparisons.py, lines 1–45 · score 0.77 · LLM trained RCNN, category trained RCNNs, prediction accuracies, FDR corrected, Pearson correlation, magnitude
  11. [11] § Results › Visual and linguistic similarity spaces predict visually evoked brain activity in areas aligned with LLM embeddings of scene captions ↔ scripts/ma_searchlight_nnls_rdms.py, lines 1–63 · score 0.76 · searchlight location, behavioral RDMs derived, Cross validated, Pearson correlations, MA tasks, brain RDMs
  12. [12] § Methods › Experimental design › Estimating the representational dissimilarity matrices ↔ scripts/compute_rcnn_rdms.py, the whole file · a weak match · score 0.75 · Recurrent convolutional neural, representational dissimilarity matrix, networks, distances, RCNNs, stimuli
  13. [13] § Results › LLM-trained visual ANNs capture visual and linguistic similarity spaces ↔ scripts/nnls_predict_rcnn_rdms.py, lines 1–50 · score 0.72 · predicted RCNN RDMs, LLM trained RCNN, category trained RCNN, cross validated, behavioral RDMs, NNLS
  14. [14] § Methods › Statistical analysis ↔ scripts/plot_within_rcnn_comparisons.py, lines 1–45 · score 0.69 · RCNN comparisons, category trained RCNNs, FDR correction, Pearson correlations, layer
  15. [15] § Methods › Experimental design › Estimating the representational dissimilarity matrices ↔ scripts/make_dino_rdm.py, lines 1–50 · score 0.66 · representational dissimilarity matrix, Recurrent convolutional, distances, stimuli, RCNNs, embeddings
  16. [16] § Results › LLM-trained visual ANNs capture visual and linguistic similarity spaces ↔ scripts/plot_rcnn_model_fits.py, lines 1–45 · score 0.63 · Recurrent Convolutional Neural, LLM trained RCNNs, Networks, fit, linguistic, models
  17. [17] § Results › Visual and linguistic similarity spaces predict visually evoked brain activity in areas aligned with LLM embeddings of scene captions › Joint modeling of visual and linguistic behavioral pre ↔ scripts/cortical_flatmap_unique.py, lines 1–45 · score 0.62 · R2 space, joint model, single predictor, variance, maps, searchlight

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 68 lines · 1.9 KB · no license · 2 matches

  1. '''
  2. make_mpnet_rdm.py
  4. Construct a representational dissimilarity matrix (RDM) from MPNet sentence embeddings
  5. for the caption descriptions of the Special100 natural scene stimulus set.
  7. This script loads the sentence captions corresponding to the 100 natural scene images,
  8. extracts sentence embeddings using the pretrained MPNet language model
  9. (all-mpnet-base-v2), and computes pairwise correlation dissimilarities between captions.
  11. Pairwise similarity is first obtained from the inner product between embedding vectors
  12. and converted into dissimilarity values to assemble an RDM.
  14. The resulting MPNet RDM serves as a purely linguistic baseline for comparison with
  15. behavioural similarity judgements, and LLM-trained recurrent models.
  17. Input:
  18. - sentence captions for the Special100 stimulus set
  20. Output:
  21. - MPNet sentence-based RDM
  23. Used in:
  24. - baseline comparison against visual models (AlexNet, DINOv2)
  25. - evaluation of linguistic representational geometry
  28. '''
  31. import os
  32. import pandas as pd
  33. import numpy as np
  35. from sentence_transformers import SentenceTransformer
  36. from rsatoolbox.rdm import RDMs
  39. base_path = os.path.expanduser('~/Dartmouth College Dropbox/Katerina Simkova/Projects/covertMA/')
  40. stim_path = os.path.join(base_path, 'stimuli')
  43. # read sentence file
  44. sent_fpath = os.path.join(stim_path, 'stimuli_semprime_stars.csv')
  45. sent_df = pd.read_csv(sent_fpath, sep='\t', header=None)
  47. # sentence list
  48. sentences = sent_df.values[:, 1].tolist()
  49. # nsd labels
  50. labels = sent_df.values[:, 2].tolist()
  52. # load model
  53. model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')
  54. # get embeddings
  55. embeddings = model.encode(sentences)
  57. # compute inner product
  58. row, col = np.triu_indices(embeddings.shape[0], 1)
  59. rdm_utv = 1 - np.inner(embeddings, embeddings)[row, col]
  62. # assemble RDM
  63. RDM_mpNet = RDMs(
  64. rdm_utv,
  65. rdm_descriptors={'layers': 'MPNet'},
  66. pattern_descriptors={'nsd': np.array([x for x in labels])},
  67. dissimilarity_measure='correlation'
  68. )

make_mpnet_rdm.py at commit 8bcb989, no license · at the source

Overview

Authors: Katerina M Simkova1, Adrien Doerig2,3, Clayton Hickey1, Ian Charest4
ORCID iDs: Clayton Hickey
  1. Centre for Human Brain Health and School of Psychology, University of Birmingham, Birmingham, UK
  2. Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
  3. Bernstein Center for Computational Neuroscience, Berlin, Germany
  4. CerebrUM, Département de Psychologie, Université de Montréal, Montréal, Québec, Canada
Institutions: University of Birmingham (United Kingdom); Bernstein Center for Computational Neuroscience Berlin (Germany); Freie Universität Berlin (Germany); Université de Montréal (Canada)
Journal: Journal of vision, volume 26, issue 5, article 7
Dates: received 6 December 2025; accepted 25 March 2026; published online 20 May 2026; in print May 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1167/jov.26.5.7 · PMID 42159455 · PMCID PMC13206752 · OpenAlex W7161817432
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: human (organism), cognitive (subfield)
Methods: Statistics, Machine learning, Connectivity, fMRI & imaging
Keywords: similarity judgments, conceptual representation, scene perception, language, large language models, AlexNet, representational similarity
MeSH: Brain*, Language*, Visual Perception*, Adult, Brain Mapping, Female, Humans, Judgment, Large Language Models, Magnetic Resonance Imaging, Male, Young Adult (* major topic)
Topic: Face Recognition and Perception (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Citations: not cited yet (Europe PMC); 60 references in the paper

Abstract

Humans can effortlessly describe what they see, yet establishing a shared representational format between vision and language remains a significant challenge. Emerging evidence suggests that human brain representations in both vision and language are well predicted by semantic feature spaces obtained from large language models (LLMs). This raises the possibility that sensory systems converge in their inherent ability to transform their inputs onto shared, embedding-like representational space. However, it remains unclear how such a space manifests in human behavior. To investigate this, 63 participants performed behavioral similarity judgments separately on 100 natural scene images and 100 corresponding sentence captions from the Natural Scenes Dataset. We found that visual and linguistic similarity judgments not only converge at the behavioral level but also predict a remarkably similar network of functional magnetic resonance imaging brain responses evoked by viewing the natural scene images. Furthermore, computational models trained to map images onto LLM-embeddings outperformed both category-trained and AlexNet controls in predicting the behavioral similarity structure. These findings demonstrate that human visual and linguistic similarity judgments are grounded in a shared, modality-agnostic representational structure that mirrors how the visual system encodes experience. The convergence between sensory and artificial systems observed here suggests a common capacity of how conceptual representations are formed—not as arbitrary products of first order, modality-specific input, but as structured representations that reflect the stable, relational properties of the external world.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 17 matches between paragraphs and lines of code.

masika03/covertMA

License: none: the authors keep all their rights
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: 8bcb9892319b4849a0093392b16989b5b652c916, 11 March 2026
Languages: Python (26), Shell (2)
Size: 30 files, 28 scripts
Software Heritage: not archived
Found in: the acknowledgements
Holds: README
Not found: license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: NumPy (25 files), SciPy (17 files), Matplotlib (13 files), pandas (8 files), seaborn (8 files), statsmodels (8 files), NiBabel (7 files), Nilearn (3 files), pycortex (3 files), scikit-learn (3 files), Pillow (2 files), PyTorch (2 files), h5py (1 file), OpenCV (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
29 files
At the source: github.com/masika03/covertMA

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 28 scripts, each with its path and the digest of its content;
  • 17 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 4 authors, 7 keywords, 12 MeSH terms, 36 references.

Cite

This paper

Simkova, K. M., Doerig, A., Hickey, C., & Charest, I. (2026). Representations in vision and language converge in a shared, multidimensional space of perceived similarities. Journal of vision, 26(5), 7. https://doi.org/10.1167/jov.26.5.7

BibTeX

@article{simkova2026representations,
author = {Simkova, Katerina M and Doerig, Adrien and Hickey, Clayton and Charest, Ian},
title = {{Representations in vision and language converge in a shared, multidimensional space of perceived similarities}},
journal = {Journal of vision},
year = {2026},
month = may,
volume = {26},
number = {5},
pages = {7},
publisher = {Association for Research in Vision and Ophthalmology},
issn = {1534-7362},
doi = {10.1167/jov.26.5.7},
url = {https://doi.org/10.1167/jov.26.5.7},
pmid = {42159455},
pmcid = {PMC13206752}
}

RIS

TY - JOUR
AU - Simkova, Katerina M
AU - Doerig, Adrien
AU - Hickey, Clayton
AU - Charest, Ian
TI - Representations in vision and language converge in a shared, multidimensional space of perceived similarities
T2 - Journal of vision
J2 - J Vis
PY - 2026
DA - 2026/05/01
VL - 26
IS - 5
SP - 7
SN - 1534-7362
PB - Association for Research in Vision and Ophthalmology
DO - 10.1167/jov.26.5.7
UR - https://doi.org/10.1167/jov.26.5.7
LA - en
ER -

CSL-JSON

{
"id": "10.1167/jov.26.5.7",
"type": "article-journal",
"title": "Representations in vision and language converge in a shared, multidimensional space of perceived similarities",
"container-title": "Journal of vision",
"author": [
{
"family": "Simkova",
"given": "Katerina M"
},
{
"family": "Doerig",
"given": "Adrien"
},
{
"family": "Hickey",
"given": "Clayton"
},
{
"family": "Charest",
"given": "Ian"
}
],
"container-title-short": "J Vis",
"volume": "26",
"issue": "5",
"page": "7",
"DOI": "10.1167/jov.26.5.7",
"PMID": "42159455",
"PMCID": "PMC13206752",
"ISSN": "1534-7362",
"publisher": "Association for Research in Vision and Ophthalmology",
"URL": "https://doi.org/10.1167/jov.26.5.7",
"language": "en",
"issued": {
"date-parts": [
[
2026,
5,
1
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s41597-026-07248-6 [code]
A large-scale fMRI dataset for vision-language semantic association.
Journal: Scientific data
In common: Nilearn, OpenCV, h5py, 8 other tools, 7 references
[2] doi:10.1523/jneurosci.0038-26.2026 [code]
Multidimensional Feature Tuning in Category Selective Areas of Human Visual Cortex.
Journal: The Journal of neuroscience : the official journal of the Society for Neuroscience
In common: pycortex, h5py, Pillow, 9 other tools, 5 references
[3] doi:10.1162/imag.a.1309 [code]
Probing the content of semantic representations in body-selective regions.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: pycortex, OpenCV, h5py, 10 other tools, 4 references
[4] doi:10.7554/elife.107933 [code]
Modality-agnostic decoding of vision and language from fMRI.
Journal: eLife
In common: Nilearn, OpenCV, h5py, 10 other tools, cognitive, 5 references
[5] doi:10.1038/s42003-026-10169-0 [code]
Shared representations in brains and models reveal a two-route cortical organization during scene perception.
Journal: Communications biology
In common: h5py, Pillow, statsmodels, 8 other tools, cognitive, 7 references
[6] doi:10.1038/s41467-026-76098-y [code]
A single computational objective can produce specialization of streams in visual cortex.
Journal: Nature communications
In common: h5py, Pillow, statsmodels, 8 other tools, 4 references
[7] doi:10.1162/imag.a.1286 [code]
Behavioral imitation with artificial neural networks leads to personalized models of brain dynamics during videogame play.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: Nilearn, h5py, Pillow, 9 other tools, 2 references
[8] doi:10.1162/imag.a.1349 [code]
Rapid cortical mapping with cross-participant encoding models.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: pycortex, SciPy, Matplotlib, 1 other tool, cognitive, 6 references
[9] doi:10.1016/j.isci.2026.117180 [code]
Developmental changes in similarity between neural representations of mental arithmetic and artificial neural networks.
Journal: iScience
In common: Nilearn, h5py, statsmodels, 8 other tools, 2 references
[10] doi:10.1162/nol.a.271 [code]
Compositional Complexity in Text and Images.
Journal: Neurobiology of language (Cambridge, Mass.)
In common: Nilearn, OpenCV, Pillow, 8 other tools, 2 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.