OSCR

How OSCR finds and verifies the code

OSCR is automated. A program reads each open-access paper, keeps the links the paper gives to code and data, decides what each link is with written rules, and then visits each repository where it is hosted. This page describes each step and what the result means.

Where the links come from

We read the full text of each paper as Europe PMC publishes it (the publisher's JATS XML), and the paper's records at DataCite and Crossref. We keep every link to a code or data repository, with the place it was found:

  • the data or code availability statement, where most links are;
  • the body of the paper, its tables (such as a key resources table), the acknowledgements, the notes, the appendix and the supplementary material;
  • the reference list;
  • the paper's metadata: related records at DataCite (software, datasets, notebooks, models, workflows) and Crossref relations;
  • for a Zenodo or Dryad record the paper cites, the code repository that record names as its source.

A sentence that says the code is available on request is recorded as well. We keep the sentence that contained each link in our private database. A paper's page shows where in the paper each link was found and does not show the paper's own words.

Code, data or a tool

Each mention of a link is classified as the authors' code, their data, or a third-party tool they used. The classification uses written rules, and the clues that decided it are recorded with the verdict:

  • the host (GitHub or GitLab suggest code, OpenNeuro suggests data, PyPI suggests a published tool);
  • the words around the link (a word such as "code" or "scripts" with a word such as "available" or "deposited"; "our" or "this study"; "using", "implemented in" or a version number for a tool), and the name of a known toolbox written just before the link;
  • the place in the paper (a code availability section, the "this paper" row of a key resources table, a reference written by the paper's own authors).

The visit to the repository can then change the verdict. A forge repository classified as data or as unknown is counted as code when it holds enough scripts. An archive record that describes itself as software is counted as code. An archive classified as code that is a dataset without a single script is counted as data, and so is a BIDS dataset with fewer than 20 scripts. A repository with at least 500 stars, created more than a year before the paper and without a mention of the paper, is counted as a third-party tool. A correction made by a person is kept and is not changed by these rules.

The visit at the source

Each repository that may hold code is visited where it is hosted. For a git repository (GitHub, GitLab, Codeberg, Bitbucket, GIN and others) we ask for its current commit, then make a copy without its history to list its files and read its scripts, and we record the commit and its date. A Zenodo, OSF, figshare or Dryad record is read through the service's own interface. Each repository is checked again after 30 days, and every check is kept on the paper's page.

The result of the last visit is the repository's state:

  • the link answers: the repository exists and could be read;
  • empty: the repository answers but holds no recognized script yet (for example a README only);
  • dead: the link answers with an error, or the repository is missing or private;
  • unreachable: the host did not answer at the last attempt;
  • cannot be verified: the host does not allow automated visits (Code Ocean, for example);
  • not checked yet: the repository is waiting for its first visit.

What a status means

A paper's status sums up the states of the repositories it cites as its code. Code verified means that at least one of them answered at its last check and holds code. It means that the link works and points to code. It does not mean that the code was run, that it reproduces the paper's results, or that the authors confirmed the link: we never run the code, and an author's confirmation is shown separately, as a validated tracing map. The other statuses are described on Reading a paper's page.

Errors and corrections

The rules make mistakes. A link can be missed, given the wrong role (a tool counted as the authors' code, for example), or read wrongly from a badly formatted address. The figures we measured are on the About page. Each paper's page shows where each link was found, so that a reader can check it against the paper.

An author of the paper, signed in with the ORCID iD the paper lists, and a maintainer of its code can add a link, remove one, or change its role (correcting a record's links). The correction is kept in the record's history and stays in place at every later visit.