OSCR

An AI system to help scientists write expert-level empirical software.

Code ↔ Paper

11 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 11 matches
  1. [1] § Methods › scRNA-seq batch integration › Dataset ↔ implementation/notebooks/single_cell_batch_integration.ipynb, lines 407–426 · score 0.83 · c1 ae26872b1042, bd0c7 f7fd, ed, cells, batches
  2. [2] § Methods › Code mutation system ↔ implementation/futs.py, lines 15–23 · score 0.71 · upper confidence bound, AlphaZero, tree search, UCB, Flat, algorithm
  3. [3] § Methods › COVID-19 prediction › Dataset ↔ implementation/notebooks/flu-cornell-jhu-hierarchsir.ipynb, lines 1–148 · score 0.71 · hospital admissions, Forecast Hub, target variable, augment, jurisdiction, population
  4. [4] § Methods › scRNA-seq batch integration › Evaluating scRNA-seq batch integration on the OpenProblems.bio benchmark ↔ implementation/notebooks/single_cell_batch_integration.ipynb, lines 864–965 · score 0.64 · cellxgene_census, batch integration, uns, OpenProblems, bounds, score
  5. [5] § Public health: predicting COVID-19 hospitalizations ↔ implementation/notebooks/flu-cornell-jhu-hierarchsir.ipynb, lines 1–148 · score 0.62 · weighted interval score, Forecast Hub, week, territories, uncertainty, calibrated
  6. [6] § Methods › GIFT-Eval benchmark › Unified solution ↔ implementation/notebooks/single_cell_batch_integration.ipynb, lines 753–786 · score 0.60 · log1p, log transform, median
  7. [7] § Methods › Code mutation system ↔ implementation/futs.py, lines 96–155 · score 0.60 · search tree, backpropagation, visit, flat, PUCT, exploration
  8. [8] § Genomics: batch integration of scRNA-seq data ↔ implementation/notebooks/single_cell_batch_integration.ipynb, lines 864–965 · score 0.59 · single cell batch, batch integration, mouse, OpenProblems, prompt, metrics
  9. [9] § Methods › scRNA-seq batch integration › Replication of existing methods for batch integration ↔ scripts/validate_results.py, lines 31–71 · score 0.56 · fine tuned, zero shot, leakage, metrics, model
  10. [10] § Time-series forecasting: GIFT-Eval › Per-dataset solution ↔ implementation/playground_s3e1.py, lines 66–125 · score 0.53 · scikit learn, XGBoost, boosting, Python, models
  11. [11] § Methods › Code mutation system ↔ implementation/futs.py, lines 96–155 · score 0.53 · rank scores, tree search, UCB, flat, PUCT, exploration

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Jupyter notebook · 968 lines · 33 KB · Apache-2.0 · 4 matches

  1. # %% [markdown]
  2. # # Overview
  3. #
  4. # As single-cell technologies advance, single-cell datasets are growing both in
  5. # size and complexity. Especially in consortia such as the Human Cell Atlas,
  6. # individual studies combine data from multiple labs, each sequencing multiple
  7. # individuals possibly with different technologies. This gives rise to complex
  8. # batch effects in the data that must be computationally removed to perform a
  9. # joint analysis.
  10. #
  11. # This task aims to develop a superhuman method for batch integration of
  12. # single-cell RNA-seq data which must remove the batch effect while not removing
  13. # relevant biological information. The input data is unnormalized raw gene
  14. # expression count data with multiple batches and consistent cell type labels. The
  15. # batch integrated output can be a low dimensional embedding of the data or a
  16. # feature matrix. The respective batch-integrated representation is then evaluated
  17. # using sets of metrics that capture how well batch effects are removed and
  18. # whether biological variance is conserved. There are over 200 methods developed
  19. # by humans for carrying this out and the goal is to develop methods that are
  20. # better than humans.
  22. # %% [markdown]
  23. # **Input Format:** The input data file (`input_adata`) is an `ad.AnnData` object
  24. # that includes the main data matrix in `.X` attribute (these are input raw gene
  25. # expression counts we want to transform). The data has already been subset to
  26. # 2000 highly variable genes.
  27. #
  28. # **Output Format:** The output data (`output_adata`) MUST be an `ad.AnnData`
  29. # object. The transformed dataset or embedding must be stored in `X_emb` key under
  30. # `obsm` annotation.
  31. #
  32. # ```
  33. # output_data = ad.AnnData(
  34. # obs=adata.obs,
  35. # var=adata.var,
  36. # obsm={
  37. # 'X_emb': # transformed dataset goes here.
  38. # },
  39. # )
  40. # ```
  41. #
  42. # Importantly, **to remove batch effects while conserving biological factors you
  43. # may experiment with data preprocessing (e.g. via scanpy) as well as modeling.**
  44. #
  45. # **Evaluation Harness and `eliminate_batch_effect_fn`:** You will implement a
  46. # function, `eliminate_batch_effect_fn`, to generate a transformed dataset that
  47. # represents the original data without batch effects. The evaluation harness will
  48. # evaluate via `score` function below whether the transformed datasets has
  49. # eliminated batch effects while preserving important biological features. In
  50. # particular, we would like to preserve cell type variation. The `score` function
  51. # implements various metrics used to measure how well you are doing on the task.
  52. # Your goal is to maximize the score from the `score` function below by writing
  53. # the best possible method/code for `eliminate_batch_effect_fn`.
  54. #
  55. # **1. Objective:**
  56. #
  57. # * The goal is to create a `eliminate_batch_effect_fn` that transforms the
  58. # dataset into a new dataset that not only eliminiates batch effects but
  59. # preserves biological information. We will evaluate the output using average
  60. # of the following specific metrics (after scaling):
  61. # * ASW Batch: Modified average silhouette width (ASW) of batch. The metric
  62. # is scaled so that 0 indicates suboptimal batch mixing and 1 indicates
  63. # optimal batch mixing.
  64. # * ASW Label: Average silhouette width of cell type labels. ASW is computed
  65. # on cell identity labels and scaled to a value between 0 (worst) and 1
  66. # (best).
  67. # * ARI: Adjusted Rand Index compares clustering overlap, correcting for
  68. # random labels and considering correct overlaps and disagreements. The
  69. # Adjusted Rand Index (ARI) compares the overlap of two clusterings; it
  70. # considers both correct clustering overlaps while also counting correct
  71. # disagreements between two clusterings. We compare the cell-type labels
  72. # with the NMI-optimized Louvain clustering computed on the integrated
  73. # dataset. The adjustment of the Rand index corrects for randomly correct
  74. # labels. An ARI of 0 or 1 corresponds to random labeling or a perfect
  75. # match, respectively. The score ranges between 0 and 1 with larger values
  76. # indicating better conservation of the data-driven cell identity
  77. # discovery after integration compared to annotated labels.
  78. # * NMI: The normalized mutual information is a version of the mutual
  79. # information corrected by the entropy of clustering and ground truth
  80. # labels (e.g. cell type). The score ranges between 0 and 1, with 0
  81. # representing no sharing and 1 representing perfect sharing of
  82. # information between clustering and annotated cell labels. NMI compares
  83. # overlap by scaling using mean entropy terms and optimizing Louvain
  84. # clustering to obtain the best match between clusters and labels.
  85. # Normalized Mutual Information (NMI) compares the overlap of two
  86. # clusterings. We use NMI to compare the cell-type labels with Louvain
  87. # clusters computed on the integrated dataset. The overlap was scaled
  88. # using the mean of the entropy terms for cell-type and cluster labels.
  89. # Thus, NMI scores of 0 or 1 correspond to uncorrelated clustering or a
  90. # perfect match, respectively. We performe optimized Louvain clustering
  91. # for this metric to obtain the best match between clusters and labels.
  92. # * Graph connectivity: Connectivity of the subgraph per cell type label.
  93. # The graph connectivity metric assesses whether the kNN graph
  94. # representation, G, of the integrated data directly connects all cells
  95. # with the same cell identity label. The resultant score has a range of
  96. # (0;1], where 1 indicates that all cells with the same cell identity are
  97. # connected in the integrated kNN graph, and the lowest possible score
  98. # indicates a graph where no cell is connected.
  99. # * Isolated labels ASW: Score how well isolated labels are distinguished
  100. # from all other labels using the average-width silhouette score. Isolated
  101. # cell labels are defined as the labels present in the least number of
  102. # batches in the integration task. The score evaluates how well these
  103. # isolated labels separate from other cell identities. The isolated label
  104. # ASW score is obtained by computing the ASW of isolated versus
  105. # non-isolated labels on the embedding and scaling this score to be
  106. # between 0 and 1. The final score for each metric version consists of the
  107. # mean isolated score of all isolated labels.
  108. # * Isolated labels F1: Evaluate how well isolated labels coincide with
  109. # clusters. Score how well isolated labels are distinguished from other
  110. # labels by data-driven clustering. The F1 score is used to evaluate
  111. # clustering with respect to the ground truth cell type labels. It returns
  112. # a value between 0 and 1, where 1 shows that all of the isolated label
  113. # cells and no others are captured in the cluster.
  114. # * kBET: kBET determines how well batches are mixed within a cell type. The
  115. # kBET algorithm determines whether the label composition of a k nearest
  116. # neighborhood of a cell is similar to the expected (global) label
  117. # composition. The test is repeated for a random subset of cells, and the
  118. # results are summarized as a rejection rate over all tested
  119. # neighborhoods. kBET score is scaled between 0 and 1 so that larger
  120. # scores are associated with better batch mixing.
  121. # * iLISI: Local inverse Simpson's Index for batch label. The metric
  122. # assesses whether clusters of cells in a single-cell RNA-seq dataset are
  123. # well-mixed across a categorical batch variable. The original iLISI score
  124. # ranges from 0 to the number of categories, with the latter indicating
  125. # good cell mixing. This is rescaled to a score between 0 and 1.
  126. # * cLISI: Local inverse Simpson's Index for cell type label. The metric
  127. # assesses whether clusters of cells in a single-cell RNA-seq dataset are
  128. # well-mixed across a categorical cell type variable. The original cLISI
  129. # score ranges from 0 to the number of categories, with the latter
  130. # indicating good cell mixing. This is rescaled to a score between 0
  131. # and 1.
  132. # * PCR: Principal component regression compares the explained variance by
  133. # batch before and after integration. The score ranges between 0 and 1.
  134. # The larger the score, the more different the variance contributions are
  135. # before and after integration.
  136. # * Cell cycle conservation score: Cell cycle conservation score based on
  137. # principle component regression on cell cycle gene scores. The cell-cycle
  138. # conservation score evaluates how well the cell-cycle effect can be
  139. # captured before and after integration. Values close to 0 indicate lower
  140. # conservation and 1 indicates complete conservation of the variance
  141. # explained by cell cycle. In other words, the variance remains unchanged
  142. # within each batch for complete conservation, while any deviation from
  143. # the preintegration variance contribution reduces the score.
  144. #
  145. # **2. Function Signature:**
  146. #
  147. # * Your `eliminate_batch_effect_fn` *must* adhere to this signature:
  148. #
  149. # ```python
  150. # def eliminate_batch_effect_fn(
  151. # adata: ad.AnnData,
  152. # config: dict[str, Any],
  153. # ) -> ad.AnnData:
  154. # # Your code here to return ad.AnnData without batch variation.
  155. # return output_data # ad.AnnData without batch variation
  156. # ```
  157. #
  158. # * `adata`: input ad.AnnData object containing raw gene expression counts in
  159. # `adata.X` field and batch labels in `adata.obs['batch']` field.
  160. # * `config`: Configuration parameters (dictionary for hyperparameters, etc.).
  161. # **Don't forget to specify parameters in this `config` dictionary.**
  162. #
  163. # * Your function **must return an ad.AnnData object** structured in the
  164. # following way:
  165. #
  166. # ```python
  167. # output_data = ad.AnnData(
  168. # obs=adata.obs,
  169. # var=adata.var,
  170. # obsm={
  171. # 'X_emb': # transformed dataset goes here.
  172. # },
  173. # )
  174. # ```
  175. #
  176. # **Your implementation should NOT use `cell_type` information in any way.**
  177. #
  178. # **3. Minimize use of specialized single-cell python packages.**
  179. #
  180. # There are many python packages used in single cell genomics. The only one that
  181. # we have installed in the coding environment is scanpy. Thus, instead of using
  182. # algorithms you might be tempted to use, please write your own native description
  183. # of these algorithms using scanpy, sklearn, numpy, scipy, tensorflow, torch, jax
  184. # or equivalent. There is much room to be creative in getting rid of batch
  185. # variation using purely native tools.
  186. #
  187. # **4. Data Set Size:**
  188. #
  189. # Please be aware that the datafile is large and if you use the wrong algorithm
  190. # you could OOM your sandbox or the algorithm could take a long time to run. As an
  191. # example, a typical matrix size is 329,762 cells × 2,000 genes.
  192. #
  193. # **5. Error Handling:**
  194. #
  195. # * The harness includes robust error handling. If your
  196. # `eliminate_batch_effect_fn` raises an exception, the harness will catch it,
  197. # log the traceback, assign a `worst_score`, and continue the evaluation. This
  198. # is important to ensure that a single error doesn't halt the entire process.
  199. # The `traceback` and any `stdout` and `stderr` output from your function are
  200. # captured and stored.
  201. # * Pay attention to the `first_traceback` in the output. This is the *first*
  202. # error that occurred across all the datasets.
  203. # * Print statements within your `eliminate_batch_effect_fn` will be captured in
  204. # the `stdout` and `stderr` streams. Use these judiciously for debugging.
  205. #
  206. # **6. Configuration:**
  207. #
  208. # * The `config` dictionary is passed to your `eliminate_batch_effect_fn`,
  209. # allowing you to parameterize your model. You should design your function to
  210. # utilize the values in the `config` dictionary.
  211. #
  212. # **7. Deliverable:**
  213. #
  214. # * Implement the `eliminate_batch_effect_fn` function to return an `ad.AnnData`
  215. # object in `X_emb` key under `obsm` annotation.
  216. # * Specify hyperparameters in the `config` dictionary. This dictionary will be
  217. # passed as an argument to your `eliminate_batch_effect_fn`.
  218. #
  219. # **8. Major advice:**
  220. #
  221. # * An expert has identified the following method as a promising candidate for
  222. # solving batch integration:
  223. # * Conditional Variational Autoencoder trained on gene expression,
  224. # conditioned on batch ID. Latent space explicitly split into biological
  225. # and batch components. Loss combines reconstruction error, KL divergence
  226. # for both latent components, and an adversarial loss (via gradient
  227. # reversal layer) penalizing batch information within the biological
  228. # component. Output is the batch-corrected biological latent embedding.
  230. # %%
  231. from IPython.display import clear_output
  233. # %%
  234. # Packages
  235. requirements = """
  236. anndata==0.11.3
  237. scanpy==1.10.4
  238. anndata2ri==1.3.2
  239. """
  240. with open('./requirements.in', 'w') as f:
  241. f.write(requirements)
  243. # %%
  244. !pip install -r ./requirements.in
  245. clear_output()
  247. # %%
  248. # Install scib from git to avoid GLIBC errors for lisi
  249. !git clone https://github.com/theislab/scib.git ./scib_source
  250. !cd ./scib_source && git checkout v1.1.7
  251. !pip install ./scib_source/
  252. !rm -rf ./scib_source
  253. clear_output()
  255. # %%
  256. %load_ext rpy2.ipython
  258. # %%
  259. %%R
  260. # Set global options to be non-interactive
  261. options(repos = c(CRAN = "https://cloud.r-project.org")) # Set a CRAN mirror
  262. options(pkgAsk = FALSE) # Attempt to suppress prompts globally
  264. # Install 'remotes' package
  265. # The 'ask = FALSE' argument is specific to install.packages
  266. install.packages('remotes', ask = FALSE)
  268. # Load the remotes package
  269. library(remotes)
  271. # Install 'kBET' from GitHub
  272. remotes::install_github('theislab/kBET', upgrade = "never")
  274. # %%
  275. import dataclasses
  276. import os
  277. import sys
  278. import warnings
  279. import anndata as ad
  280. from anndata.io import read_elem, sparse_dataset
  281. import h5py
  282. import numpy as np
  283. import pandas as pd
  284. import scanpy as sc
  285. import scib
  286. from scipy.sparse import csr_matrix
  288. INPUT_DIR = './datasets/'
  289. WORKING_DIR = './'
  290. INPUT_FILE_DIR = os.path.join(
  291. INPUT_DIR,
  292. 'single-cell-batch-integration',
  293. )
  294. INPUT_FILE_TRAIN = os.path.join(
  295. INPUT_FILE_DIR, 'ffdaa1f0-b1d1-4135-8774-9fed7bf039ba-train-dataset.h5ad'
  296. )
  297. INPUT_FILE_VAL = os.path.join(
  298. INPUT_FILE_DIR, 'ffdaa1f0-b1d1-4135-8774-9fed7bf039ba-val-dataset.h5ad'
  299. )
  300. INPUT_FILE_PRE_INTEGRATION_TRAIN = os.path.join(
  301. INPUT_FILE_DIR, 'ffdaa1f0-b1d1-4135-8774-9fed7bf039ba-train-solution.h5ad'
  302. )
  303. INPUT_FILE_PRE_INTEGRATION_VAL = os.path.join(
  304. INPUT_FILE_DIR, 'ffdaa1f0-b1d1-4135-8774-9fed7bf039ba-val-solution.h5ad'
  305. )
  306. INPUT_FILE_SCORE_BOUNDS_TRAIN = os.path.join(
  307. INPUT_FILE_DIR,
  308. 'score_train.median.bounds.csv',
  309. )
  310. INPUT_FILE_SCORE_BOUNDS_VAL = os.path.join(
  311. INPUT_FILE_DIR,
  312. 'score_val.median.bounds.csv',
  313. )
  315. # %%
  316. def open_h5(path: str) -> h5py.File:
  317. return h5py.File(path, 'r')
  319. # %%
  320. def read_anndata(
  321. file: str,
  322. backed: bool = False,
  323. **kwargs,
  324. ) -> ad.AnnData:
  325. """Read anndata file.
  327. :param file: path to anndata file in h5ad format.
  328. :param kwargs: AnnData parameter to group mapping.
  329. """
  330. f = open_h5(file)
  331. kwargs = {x: x for x in f} if not kwargs else kwargs
  332. if len(f.keys()) == 0:
  333. return ad.AnnData()
  334. # Check if keys are available.
  335. for name, slot in kwargs.items():
  336. if slot not in f:
  337. warnings.warn(
  338. f'Cannot find "{slot}" for AnnData parameter `{name}` from "{file}"'
  339. )
  340. adata = read_partial(f, backed=backed, **kwargs)
  341. if not backed:
  342. f.close()
  344. return adata
  347. def read_partial(
  348. group: h5py.Group,
  349. backed: bool = False,
  350. force_sparse_types: [str, list] = None,
  351. **kwargs,
  352. ) -> ad.AnnData:
  353. """Partially read h5py groups.
  355. :params group: file group :params force_sparse_types: encoding types to
  356. convert to sparse_dataset via csr_matrix. :params backed: If True, read sparse
  357. matrix as sparse_dataset. :params **kwargs: dict of slot_name: slot. By
  358. default use all available slots for the h5py file. :return: AnnData object.
  359. """
  360. if force_sparse_types is None:
  361. force_sparse_types = []
  362. elif isinstance(force_sparse_types, str):
  363. force_sparse_types = [force_sparse_types]
  364. slots = {}
  365. if backed:
  366. print('Read as backed sparse matrix...')
  367. for slot_name, slot in kwargs.items():
  368. print(f'Read slot "{slot}", store as "{slot_name}"...')
  369. if slot not in group:
  370. warnings.warn(f'Slot "{slot}" not found, skip...')
  371. slots[slot_name] = None
  372. else:
  373. elem = group[slot]
  374. iospec = ad._io.specs.get_spec(elem)
  375. if iospec.encoding_type in ['csr_matrix', 'csc_matrix'] and backed:
  376. slots[slot_name] = sparse_dataset(elem)
  377. elif iospec.encoding_type in force_sparse_types:
  378. slots[slot_name] = csr_matrix(read_elem(elem))
  379. if backed:
  380. slots[slot_name] = sparse_dataset(slots[slot_name])
  381. else:
  382. slots[slot_name] = read_elem(elem)
  383. return ad.AnnData(**slots)
  386. def dataframe_to_bounds_map(
  387. dataframe: pd.DataFrame,
  388. ) -> dict[tuple[str, str], tuple[float, float]]:
  389. indexed_df = dataframe.set_index(['dataset_id', 'metric_id'])
  390. bounds_map = {
  391. idx: (row['lower_bound'], row['upper_bound'])
  392. for idx, row in indexed_df.iterrows()
  393. }
  394. return bounds_map
  396. # %%
  397. # Input data.
  398. input_adata = read_anndata(
  399. INPUT_FILE_TRAIN,
  400. X='layers/counts',
  401. obs='obs',
  402. var='var',
  403. )
  405. print(input_adata)
  407. # %%
  408. # Data before integration, for computation of metrics.
  409. adata_pre_integration = read_anndata(
  410. INPUT_FILE_PRE_INTEGRATION_TRAIN,
  411. X='layers/normalized',
  412. obs='obs',
  413. var='var',
  414. uns='uns',
  415. )
  416. # Bounds for scaling of metrics, for computation of metrics.
  417. df_score_bounds_train = pd.read_csv(INPUT_FILE_SCORE_BOUNDS_TRAIN)
  418. bounds_map_train = dataframe_to_bounds_map(df_score_bounds_train)
  419. # We observed that isolated labels F1 is variable, thus we set [0,1] bounds
  420. bounds_map_train[
  421. ('364bd0c7-f7fd-48ed-99c1-ae26872b1042', 'isolated_label_f1')
  422. ] = [
  423. 0.0,
  424. 1.0,
  425. ]
  426. clear_output()
  428. # %%
  429. def subsample_adata(
  430. adata: ad.AnnData,
  431. target_size: int,
  432. batch_col='batch',
  433. celltype_col='cell_type',
  434. seed=42,
  435. ) -> ad.AnnData:
  436. """Subsamples ad.AnnData object while maintaining the distribution of batches and cell types.
  438. Args:
  439. adata (ad.AnnData): The AnnData object to subsample.
  440. target_size (int): The desired number of observations in the subsampled
  441. AnnData.
  442. batch_col (str): The column in adata.obs containing batch information.
  443. celltype_col (str): The column in adata.obs containing cell type
  444. information.
  445. seed (int): Random seed for reproducibility.
  447. Returns:
  448. ad.AnnData: The subsampled AnnData object.
  449. """
  450. np.random.seed(seed)
  451. obs = adata.obs[[batch_col, celltype_col]].copy()
  452. obs['index'] = obs.index
  454. # Calculate the desired proportions for each group
  455. group_counts = obs.groupby([batch_col, celltype_col], observed=True).size()
  456. total_counts = len(adata)
  457. group_proportions = group_counts / total_counts
  459. # Calculate the number of samples to take from each group
  460. group_sample_sizes = (group_proportions * target_size).astype(int)
  462. # Adjust sample sizes to match the target size as closely as possible
  463. remaining = target_size - group_sample_sizes.sum()
  464. if remaining != 0:
  465. sorted_groups = group_proportions.sort_values(ascending=False).index
  466. for group in sorted_groups:
  467. if remaining > 0:
  468. group_sample_sizes[group] += 1
  469. remaining -= 1
  470. elif remaining < 0:
  471. if group_sample_sizes[group] > 0:
  472. group_sample_sizes[group] -= 1
  473. remaining += 1
  474. if remaining == 0:
  475. break
  477. # Subsample each group
  478. sampled_indices = []
  479. for group, size in group_sample_sizes.items():
  480. group_indices = obs[
  481. (obs[batch_col] == group[0]) & (obs[celltype_col] == group[1])
  482. ]['index'].values
  483. sampled_indices.extend(
  484. np.random.choice(group_indices, size=size, replace=False)
  485. )
  487. # Create the subsampled AnnData object
  488. subsampled_adata = adata[sampled_indices, :].copy()
  490. return subsampled_adata
  492. # %%
  493. # Subsample data.
  494. input_adata = subsample_adata(input_adata, target_size=20000, seed=42)
  495. # Apply subsampling to pre-integrated data for metrics.
  496. adata_pre_integration = adata_pre_integration[input_adata.obs.index, :]
  497. print(input_adata)
  499. # %% [markdown]
  500. # # Evaluation
  502. # %%
  503. CONTROL_METHODS = [
  504. 'embed_cell_types',
  505. 'embed_cell_types_jittered',
  506. 'no_integration',
  507. 'no_integration_batch',
  508. 'shuffle_integration',
  509. 'shuffle_integration_by_batch',
  510. 'shuffle_integration_by_cell_type',
  511. ]
  513. DATASET_ID = adata_pre_integration.uns['dataset_id']
  514. METHOD_ID = 'scagent'
  517. @dataclasses.dataclass(frozen=True)
  518. class Result:
  519. dataset_id: str
  520. method_id: str
  521. metric_id: str
  522. value: float
  524. @property
  525. def is_control_method(self):
  526. return self.method_id in CONTROL_METHODS
  528. def scaled(self, bounds: dict[tuple[str, str], tuple[float, float]]):
  529. """Returns a new Result with the value scaled to the bounds."""
  530. lower_bound, upper_bound = bounds[(self.dataset_id, self.metric_id)]
  531. if (
  532. np.isnan(lower_bound)
  533. or np.isnan(upper_bound)
  534. or lower_bound >= upper_bound
  535. ):
  536. scaled_value = np.nan
  537. else:
  538. scaled_value = (self.value - lower_bound) / (upper_bound - lower_bound)
  539. return dataclasses.replace(self, value=scaled_value)
  542. def run_leiden_clustering(adata: ad.AnnData, resolution: float) -> ad.AnnData:
  543. # Based on
  544. # https://github.com/openproblems-bio/task_batch_integration/blob/main/src/data_processors/precompute_clustering_run/script.py
  545. from scanpy.tl import leiden
  547. key = f'leiden_{resolution}'
  549. kwargs = {'flavor': 'igraph', 'n_iterations': 2}
  551. leiden(
  552. adata,
  553. resolution=resolution,
  554. key_added=key,
  555. **kwargs,
  556. )
  557. return adata
  560. def score(
  561. adata: ad.AnnData,
  562. adata_pre_integration: ad.AnnData,
  563. bounds_map: dict[tuple[str, str], tuple[float, float]],
  564. dataset_id: str = DATASET_ID,
  565. method_id: str = METHOD_ID,
  566. ) -> float:
  567. # Suppress warnings just for scoring.
  568. with warnings.catch_warnings():
  569. warnings.filterwarnings('ignore', category=FutureWarning)
  570. warnings.filterwarnings('ignore', category=UserWarning)
  571. warnings.filterwarnings('ignore', category=DeprecationWarning)
  572. warnings.filterwarnings('ignore', category=RuntimeWarning)
  574. # Preprocess data - compute kNN.
  575. if 'X_emb' in adata.obsm and 'neighbors' not in adata.uns:
  576. sc.pp.neighbors(adata, use_rep='X_emb')
  577. # Preprocess data - compute clusterings at different resolutions.
  578. # Based on
  579. # https://github.com/openproblems-bio/task_batch_integration/blob/main/src/data_processors/process_integration/config.vsh.yaml
  580. resolutions = [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]
  581. for resolution in resolutions:
  582. adata = run_leiden_clustering(adata, resolution)
  584. # Based on
  585. # https://github.com/openproblems-bio/task_batch_integration/blob/main/src/metrics/clustering_overlap/script.py
  586. cluster_key = 'leiden'
  588. scib.metrics.clustering.cluster_optimal_resolution(
  589. adata=adata,
  590. label_key='cell_type',
  591. cluster_key=cluster_key,
  592. cluster_function=sc.tl.leiden,
  593. resolutions=resolutions,
  594. verbose=False,
  595. )
  597. # Compute metrics.
  598. asw_batch = scib.metrics.silhouette_batch(
  599. adata,
  600. batch_key='batch',
  601. label_key='cell_type',
  602. embed='X_emb',
  603. verbose=False,
  604. )
  605. print(f'ASW batch: {asw_batch}')
  606. asw_label = scib.metrics.silhouette(
  607. adata, label_key='cell_type', embed='X_emb'
  608. )
  609. print(f'ASW label: {asw_label}')
  611. ari_score = scib.metrics.ari(
  612. adata, cluster_key=cluster_key, label_key='cell_type'
  613. )
  614. print(f'ARI: {ari_score}')
  616. nmi_score = scib.metrics.nmi(
  617. adata, cluster_key=cluster_key, label_key='cell_type'
  618. )
  619. print(f'NMI: {nmi_score}')
  621. graph_connectivity = scib.metrics.graph_connectivity(
  622. adata, label_key='cell_type'
  623. )
  624. print(f'Graph connectivity: {graph_connectivity}')
  626. isolated_labels_asw = scib.metrics.isolated_labels_asw(
  627. adata,
  628. label_key='cell_type',
  629. batch_key='batch',
  630. embed='X_emb',
  631. iso_threshold=None,
  632. verbose=False,
  633. )
  634. print(f'Isolated labels ASW: {isolated_labels_asw}')
  636. isolated_labels_f1 = scib.metrics.isolated_labels_f1(
  637. adata,
  638. label_key='cell_type',
  639. batch_key='batch',
  640. cluster_key='leiden',
  641. resolutions=resolutions,
  642. embed=None,
  643. iso_threshold=None,
  644. verbose=False,
  645. )
  646. print(f'Isolated labels F1: {isolated_labels_f1}')
  648. # kBET
  649. kbet_score = scib.metrics.kBET(
  650. adata,
  651. batch_key='batch',
  652. label_key='cell_type',
  653. type_='embed',
  654. embed='X_emb',
  655. scaled=True,
  656. verbose=False,
  657. )
  658. print(f'kBET: {kbet_score}')
  660. # iLISI
  661. ilisi_scores = scib.metrics.lisi.lisi_graph_py(
  662. adata=adata,
  663. obs_key='batch',
  664. n_neighbors=90,
  665. perplexity=None,
  666. subsample=None,
  667. n_cores=1,
  668. verbose=False,
  669. )
  670. ilisi = np.nanmedian(ilisi_scores)
  671. ilisi = (ilisi - 1) / (adata.obs['batch'].nunique() - 1)
  672. print(f'iLISI: {ilisi}')
  674. # cLISI scores
  675. clisi_scores = scib.metrics.lisi.lisi_graph_py(
  676. adata=adata,
  677. obs_key='cell_type',
  678. n_neighbors=90,
  679. perplexity=None,
  680. subsample=None,
  681. n_cores=1,
  682. verbose=False,
  683. )
  684. clisi = np.nanmedian(clisi_scores)
  685. nlabs = adata.obs['cell_type'].nunique()
  686. clisi = (nlabs - clisi) / (nlabs - 1)
  687. print(f'cLISI: {clisi}')
  689. # PCR
  690. adata_integrated = adata.copy()
  691. adata_integrated.obs['batch'] = adata_pre_integration.obs['batch']
  692. pcr_score = scib.metrics.pcr_comparison(
  693. adata_pre_integration[:, adata_pre_integration.var['batch_hvg']],
  694. adata_integrated,
  695. embed='X_emb',
  696. covariate='batch',
  697. verbose=False,
  698. )
  699. print(f'PCR: {pcr_score}')
  701. # Cell cycle (compute this last since the var_names is updated)
  702. adata_pre_integration.var_names = adata_pre_integration.var['feature_name']
  703. adata_pre_integration.var_names = adata_pre_integration.var_names.astype(
  704. str
  705. )
  706. cell_cycle_score = scib.metrics.cell_cycle(
  707. adata_pre_integration,
  708. adata_integrated,
  709. batch_key='batch',
  710. embed='X_emb',
  711. organism=adata_pre_integration.uns['dataset_organism'],
  712. verbose=False,
  713. )
  714. print(f'Cell cycle conservation score: {cell_cycle_score}')
  716. scores = {
  717. 'asw_label': asw_label,
  718. 'asw_batch': asw_batch,
  719. 'ari': ari_score,
  720. 'nmi': nmi_score,
  721. 'graph_connectivity': graph_connectivity,
  722. 'isolated_label_asw': isolated_labels_asw,
  723. 'isolated_label_f1': isolated_labels_f1,
  724. 'kbet': kbet_score,
  725. 'ilisi': ilisi,
  726. 'clisi': clisi,
  727. 'pcr': pcr_score,
  728. 'cell_cycle_conservation': cell_cycle_score,
  729. }
  731. # Scale metrics using lower and upper bounds.
  732. scaled_scores = []
  733. for metric_id, value in scores.items():
  734. unscaled_score = Result(
  735. dataset_id=dataset_id,
  736. method_id=method_id,
  737. metric_id=metric_id,
  738. value=value,
  739. )
  740. scaled_score = unscaled_score.scaled(bounds_map)
  741. scaled_scores.append(scaled_score)
  743. df = pd.DataFrame(
  744. [dataclasses.asdict(score) for score in scaled_scores]
  745. ).fillna(0)
  746. # Perform clipping.
  747. df.value = df.value.clip(0, 1)
  748. print('Metrics after scaling and clipping:')
  749. print(df[['metric_id', 'value']])
  750. avg_score = df['value'].mean()
  751. return avg_score
  753. # %%
  754. # [exclude_from_prompt]
  756. from sklearn.decomposition import TruncatedSVD
  758. # Test on small subset:
  759. # Generate random unique indices for cells.
  760. random_cell_indices = np.random.choice(
  761. input_adata.shape[0], size=100, replace=False
  762. )
  764. small_adata = input_adata[random_cell_indices, :].copy()
  765. small_adata_pre_integration = adata_pre_integration[
  766. random_cell_indices, :
  767. ].copy()
  768. # Normalize to median total counts.
  769. sc.pp.normalize_total(small_adata)
  770. # Log transform the data.
  771. sc.pp.log1p(small_adata)
  773. X_input = small_adata.X
  774. n_components = 2
  775. pca = TruncatedSVD(n_components=n_components)
  776. X_output = pca.fit_transform(X_input)
  778. output = ad.AnnData(
  779. obs=small_adata.obs,
  780. var=small_adata.var,
  781. obsm={
  782. "X_emb": X_output,
  783. },
  784. )
  786. score(output, small_adata_pre_integration, bounds_map_train)
  788. # %% [markdown]
  789. # # Begin mutable cells
  791. # %%
  792. from typing import Any
  793. import jax
  794. from sklearn.decomposition import TruncatedSVD
  795. import tensorflow as tf
  796. import torch
  798. # Define parameters for the config.
  799. config = {}
  802. def eliminate_batch_effect_fn(
  803. adata: ad.AnnData, config: dict[str, Any]
  804. ) -> ad.AnnData:
  805. # Your code here to return ad.AnnData without batch variation.
  807. # %% [markdown]
  808. # # End mutable cells
  810. # %% [markdown]
  811. # # Validation
  813. # %%
  814. cell_type_info = input_adata.obs.pop('cell_type')
  815. output_adata = eliminate_batch_effect_fn(input_adata, config=config)
  816. output_adata.obs['cell_type'] = cell_type_info
  818. # %%
  819. validation_score = score(output_adata, adata_pre_integration, bounds_map_train)
  820. print(f'Validation Score: {validation_score}')
  822. # %%
  823. # [exclude_from_prompt]
  825. # Calculate scores based on the fixed validation set.
  826. if False:
  827. # Input data.
  828. input_adata_val = read_anndata(
  829. INPUT_FILE_VAL,
  830. X='layers/counts',
  831. obs='obs',
  832. var='var',
  833. )
  835. print(input_adata_val)
  837. # Data before integration, for computation of metrics.
  838. adata_pre_integration_val = read_anndata(
  839. INPUT_FILE_PRE_INTEGRATION_VAL,
  840. X='layers/normalized',
  841. obs='obs',
  842. var='var',
  843. uns='uns',
  844. )
  846. print(adata_pre_integration_val)
  848. # Bounds for scaling of metrics, for computation of metrics.
  849. df_score_bounds_val = pd.read_csv(INPUT_FILE_SCORE_BOUNDS_VAL)
  850. bounds_map_val = dataframe_to_bounds_map(df_score_bounds_val)
  852. cell_type_info_val = input_adata_val.obs.pop('cell_type')
  853. output_adata_val = eliminate_batch_effect_fn(input_adata_val, config=config)
  854. output_adata_val.obs['cell_type'] = cell_type_info_val
  856. validation_score_val = score(
  857. output_adata_val, adata_pre_integration_val, bounds_map_val
  858. )
  859. print(
  860. 'Validation Score (calculated using validation set):'
  861. f' {validation_score_val}'
  862. )
  864. # %%
  865. # [exclude_from_prompt]
  866. # Calculate scores based on the test set.
  867. if False:
  868. import time
  870. TEST_DATASETS = [
  871. 'dkd',
  872. 'gtex_v9',
  873. 'hypomap',
  874. 'immune_cell_atlas',
  875. 'mouse_pancreas_atlas',
  876. 'tabula_sapiens',
  877. ]
  878. INPUT_FILE_TEST_PATTERN = os.path.join(
  879. INPUT_FILE_DIR, '{dataset}-subsampled-dataset.h5ad'
  880. )
  881. INPUT_FILE_PRE_INTEGRATION_TEST_PATTERN = os.path.join(
  882. INPUT_FILE_DIR, '{dataset}-subsampled-solution.h5ad'
  883. )
  884. INPUT_BOUND_TEST = './datasets/single-cell-batch-integration/openproblems_metrics_bounds.csv'
  886. def evaluate_on_test(
  887. input_dataset: ad.AnnData,
  888. input_solution: ad.AnnData,
  889. input_bounds: pd.DataFrame,
  890. dataset_id: str,
  891. method_id: str = METHOD_ID,
  892. ) -> int:
  893. bounds_map = dataframe_to_bounds_map(df_score_bounds)
  894. cell_type_info_val = input_dataset.obs.pop('cell_type')
  895. try:
  896. output_adata = eliminate_batch_effect_fn(input_dataset, config=config)
  897. output_adata.obs['cell_type'] = cell_type_info_val
  898. validation_score = score(
  899. output_adata, input_solution, bounds_map, dataset_id, method_id
  900. )
  901. except Exception as e:
  902. print(f"Error during evaluating dataset {dataset_id}: {e}")
  903. validation_score = None
  904. return validation_score
  906. all_validation_scores = []
  907. for dataset_name in TEST_DATASETS:
  908. print(f'========Start evaluation on {dataset_name} dataset ========')
  909. print(f'========Reading {dataset_name} dataset from file ========')
  910. start = time.time()
  911. # Input data.
  912. input_adata = read_anndata(
  913. INPUT_FILE_TEST_PATTERN.format(dataset=dataset_name),
  914. X='layers/counts',
  915. obs='obs',
  916. var='var',
  917. )
  919. # Data before integration, for computation of metrics.
  920. adata_pre_integration = read_anndata(
  921. INPUT_FILE_PRE_INTEGRATION_TEST_PATTERN.format(dataset=dataset_name),
  922. X='layers/normalized',
  923. obs='obs',
  924. var='var',
  925. uns='uns',
  926. )
  927. t = time.time() - start
  928. print(
  929. f'========Finished loading {dataset_name} dataset. Took'
  930. f' {t} seconds.========'
  931. )
  933. # Bounds for scaling of metrics, for computation of metrics.
  934. df_score_bounds = pd.read_csv(INPUT_BOUND_TEST)
  936. print(f'========Scoring {dataset_name} dataset ========')
  937. start = time.time()
  938. validation_score = evaluate_on_test(
  939. input_dataset=input_adata,
  940. input_solution=adata_pre_integration,
  941. input_bounds=df_score_bounds,
  942. dataset_id=f'cellxgene_census/{dataset_name}',
  943. )
  944. t = time.time() - start
  945. print(f'========Finished scoring. Took {t} seconds.========')
  947. print(
  948. f'Validation Score (calculated using {dataset_name} set):'
  949. f' {validation_score}'
  950. )
  951. all_validation_scores.append(validation_score)
  953. # --- Calculate the two averages ---
  954. # 1. Average imputing None as 0
  955. scores_imputed_as_zero = [s if s is not None else 0 for s in all_validation_scores]
  956. average_imputed_zero = sum(scores_imputed_as_zero) / len(scores_imputed_as_zero)
  957. print(f'\n======== Average Validation Score (imputing None as 0): {average_imputed_zero:.4f} ========')
  959. # 2. Average only across available datasets (excluding None)
  960. available_scores = [s for s in all_validation_scores if s is not None]
  961. if available_scores: # Avoid division by zero if all scores are None
  962. average_available = sum(available_scores) / len(available_scores)
  963. print(f'======== Average Validation Score (only available datasets): {average_available:.4f} ========')
  964. else:
  965. print('======== No available validation scores to calculate average for available datasets. ========')
  967. # %%

single_cell_batch_integration.ipynb at commit b836730, under Apache-2.0 · at the source

Overview

Authors: Eser Aygün1, Anastasiya Belyaeva2, Gheorghe Comanici1, Marc Coram2, Hao Cui2, Jake Garrison3, Renee Johnston2, Anton Kast2, Cory Y. McLean2, Peter Norgaard2, Zahra Shamsi2, David Smalling1, James Thompson2, Subhashini Venugopalan2, Brian P. Williams2, Chujun He2,4, Sarah Martinson2,5, Martyna Plomecka2,6, Lai Wei2, Yuchen Zhou2
and 22 other authorsQian-Ze Zhu2,5, Matthew Abraham2, Erica Brand2, Anna Bulanova1, Jeffrey A. Cardille2,7, Chris Co2, Scott Ellsworth2, Grace Joseph2, Malcolm Kane2, Ryan Krueger2,5, Johan Kartiwa2, Dan Liebling2, Jan-Matthis Lueckmann2, Paul Raccuglia2, Xuefei Julie Wang2,8, Katherine Chou2, James Manyika2, Yossi Matias2, John C. Platt2, Lizzie Dorfman2, Shibl Mourad1, Michael P. Brenner2,5
ORCID iDs: Hao Cui, Anton Kast, Cory Y. McLean, Subhashini Venugopalan, Brian P. Williams, Sarah Martinson, Jan-Matthis Lueckmann, Paul Raccuglia, Yossi Matias, Shibl Mourad, Michael P. Brenner
  1. Google DeepMind, Montréal, Quebec Canada
  2. Google Research,Cambridge, MA USA
  3. Google Platforms and Devices, Mountain View, CA USA
  4. Massachusetts Institute of Technology,Cambridge, MA USA
  5. School of Engineering and Applied Sciences, Harvard University,Cambridge, MA USA
  6. Google DeepMind, New York, NY USA
  7. Faculty of Agricultural and Environmental Sciences, McGill University,Montréal, Quebec Canada
  8. California Institute of Technology,Pasadena, CA USA
Institutions: Google (United States) (United States); Massachusetts Institute of Technology (United States); Harvard University (United States); McGill University (Canada); California Institute of Technology (United States)
Journal: Nature, volume 654, issue 8120, pages 909-916
Dates: received 13 September 2025; accepted 13 May 2026; published online 19 May 2026; in print 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1038/s41586-026-10658-6 · PMID 42156545 · PMCID PMC13293872 · OpenAlex W4414757142
Open access: hybrid, a free copy (OpenAlex)
Preprint: arxiv:2509.06503
Status: code verified
Categories: human (organism), zebrafish (organism), other condition (population), methods / tools (subfield)
Methods: Smoothing, state filtering, decompositions, Machine learning
Keywords: Computational science, Computer science
MeSH: Artificial Intelligence*, Empirical Research*, Software*, Animals, Computational Biology, COVID-19, Forecasting, Humans, Large Language Models, SARS-CoV-2, Single-Cell Analysis, Zebrafish (* major topic)
Topic: Big Data and Business Intelligence (Management Information Systems, Business, Management and Accounting), according to OpenAlex
Citations: cited by 10 papers (Europe PMC); 70 references in the paper

Abstract

The cycle of scientific discovery is frequently bottlenecked by the slow, manual creation of software to support computational experiments1. To address this, we present Empirical Research Assistance (ERA), an artificial intelligence (AI) system that creates expert-level scientific software whose goal is to maximize a quality metric. The system uses a large language model (LLM) and tree search2 to systematically improve the quality metric and intelligently navigate the large space of possible solutions. ERA achieves expert-level results when it explores and integrates complex research ideas from external sources. The effectiveness of tree search is demonstrated across a diverse range of tasks. In bioinformatics, ERA discovered 40 new methods for single-cell data analysis that outperformed the top human-developed methods on a public leaderboard. In epidemiology, ERA generated 14 models that outperformed the Centers for Disease Control and Prevention (CDC) ensemble and all other individual models for forecasting COVID-19 hospitalizations. ERA also produced expert-level software for geospatial analysis, neural activity prediction in zebrafish and numerical solution of integrals, as well as a new rule-based construction for time-series forecasting. By devising and implementing new solutions to diverse tasks, ERA represents a notable step towards accelerating scientific progress.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 11 matches between paragraphs and lines of code.

salesforceairesearch/gift-eval

License: Apache-2.0
State: the link answers, verified on 28 September 2026
Evidence: files inventoried
Commit: 733645d21c9428cd249761b6ce2f3098f59b2f01, 28 September 2026
Languages: Jupyter (39), Python (12)
Size: 334 files, 51 scripts
Software Heritage: not archived
Found in: the text, “GIFT-Eval benchmark”
Holds: README, license file, environment (pyproject.toml, uv.lock), tests, 39 notebooks
Not found: CITATION.cff, continuous integration, documentation
Tools: pandas (38 files), NumPy (29 files), PyTorch (27 files), Matplotlib (6 files), Hugging Face Transformers (4 files), XGBoost (2 files)
Availability: 1 check, the latest on 28 September 2026: the link answers
  • 28 September 2026: the link answers
53 files
At the source: github.com/SalesforceAIResearch/gift-eval/tree/main/notebooks

google-research/era

License: Apache-2.0
State: the link answers, verified on 28 September 2026
Evidence: files inventoried
Commit: b836730b5c000526af95116b1d0e2c60c8cf0a10, 3 August 2026
Languages: Python (5), Jupyter (3), JavaScript (1)
Size: 6,584 files, 9 scripts
Software Heritage: not archived
Found in: “Code availability”
Holds: README, license file, documentation
Not found: CITATION.cff, environment file, tests, continuous integration
Tools: NumPy (4 files), pandas (4 files), scikit-learn (3 files), Matplotlib (2 files), SciPy (2 files), anndata (1 file), h5py (1 file), JAX (1 file), PyTorch (1 file), Scanpy (1 file), TensorFlow (1 file)
Availability: 1 check, the latest on 28 September 2026: the link answers
  • 28 September 2026: the link answers
11 files
At the source: github.com/google-research/era

Code availability

A reference implementation of ERA is available at https://github.com/google-research/era. The best candidate solutions generated for each of the six scientific problems in this paper are publicly available at https://google-research.github.io/era, along with a user interface enabling examination of the full tree search data for a representative run for each of the six scientific problems. The interface allows inspecting the solution progression and breakthrough plot as the tree search proceeds and highlights code diffs.

Reproduced under the paper's license (CC BY), from the paper cited above.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 2 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 60 scripts, each with its path and the digest of its content;
  • 11 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 2, 28 September 2026

  • Publisher: n/a → Nature Portfolio

Version 1, 28 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 42 authors, 2 keywords, 12 MeSH terms, 35 references.

Cite

This paper

Aygün, E., Belyaeva, A., Comanici, G., Coram, M., Cui, H., Garrison, J., Johnston, R., Kast, A., McLean, C. Y., Norgaard, P., Shamsi, Z., Smalling, D., Thompson, J., Venugopalan, S., Williams, B. P., He, C., Martinson, S., Plomecka, M., Wei, L., . . . Brenner, M. P. (2026). An AI system to help scientists write expert-level empirical software. Nature, 654(8120), 909-916. https://doi.org/10.1038/s41586-026-10658-6

BibTeX

@article{aygun2026ai,
author = {Aygün, Eser and Belyaeva, Anastasiya and Comanici, Gheorghe and Coram, Marc and Cui, Hao and Garrison, Jake and Johnston, Renee and Kast, Anton and McLean, Cory Y. and Norgaard, Peter and Shamsi, Zahra and Smalling, David and Thompson, James and Venugopalan, Subhashini and Williams, Brian P. and He, Chujun and Martinson, Sarah and Plomecka, Martyna and Wei, Lai and Zhou, Yuchen and Zhu, Qian-Ze and Abraham, Matthew and Brand, Erica and Bulanova, Anna and Cardille, Jeffrey A. and Co, Chris and Ellsworth, Scott and Joseph, Grace and Kane, Malcolm and Krueger, Ryan and Kartiwa, Johan and Liebling, Dan and Lueckmann, Jan-Matthis and Raccuglia, Paul and Wang, Xuefei Julie and Chou, Katherine and Manyika, James and Matias, Yossi and Platt, John C. and Dorfman, Lizzie and Mourad, Shibl and Brenner, Michael P.},
title = {{An AI system to help scientists write expert-level empirical software}},
journal = {Nature},
year = {2026},
month = may,
volume = {654},
number = {8120},
pages = {909--916},
publisher = {Nature Portfolio},
issn = {0028-0836},
doi = {10.1038/s41586-026-10658-6},
url = {https://doi.org/10.1038/s41586-026-10658-6},
pmid = {42156545},
pmcid = {PMC13293872}
}

RIS

TY - JOUR
AU - Aygün, Eser
AU - Belyaeva, Anastasiya
AU - Comanici, Gheorghe
AU - Coram, Marc
AU - Cui, Hao
AU - Garrison, Jake
AU - Johnston, Renee
AU - Kast, Anton
AU - McLean, Cory Y.
AU - Norgaard, Peter
AU - Shamsi, Zahra
AU - Smalling, David
AU - Thompson, James
AU - Venugopalan, Subhashini
AU - Williams, Brian P.
AU - He, Chujun
AU - Martinson, Sarah
AU - Plomecka, Martyna
AU - Wei, Lai
AU - Zhou, Yuchen
AU - Zhu, Qian-Ze
AU - Abraham, Matthew
AU - Brand, Erica
AU - Bulanova, Anna
AU - Cardille, Jeffrey A.
AU - Co, Chris
AU - Ellsworth, Scott
AU - Joseph, Grace
AU - Kane, Malcolm
AU - Krueger, Ryan
AU - Kartiwa, Johan
AU - Liebling, Dan
AU - Lueckmann, Jan-Matthis
AU - Raccuglia, Paul
AU - Wang, Xuefei Julie
AU - Chou, Katherine
AU - Manyika, James
AU - Matias, Yossi
AU - Platt, John C.
AU - Dorfman, Lizzie
AU - Mourad, Shibl
AU - Brenner, Michael P.
TI - An AI system to help scientists write expert-level empirical software
T2 - Nature
J2 - Nature
PY - 2026
DA - 2026/05/19
VL - 654
IS - 8120
SP - 909
EP - 916
SN - 0028-0836
PB - Nature Portfolio
DO - 10.1038/s41586-026-10658-6
UR - https://doi.org/10.1038/s41586-026-10658-6
LA - en
ER -

CSL-JSON

{
"id": "10.1038/s41586-026-10658-6",
"type": "article-journal",
"title": "An AI system to help scientists write expert-level empirical software",
"container-title": "Nature",
"author": [
{
"family": "Aygün",
"given": "Eser"
},
{
"family": "Belyaeva",
"given": "Anastasiya"
},
{
"family": "Comanici",
"given": "Gheorghe"
},
{
"family": "Coram",
"given": "Marc"
},
{
"family": "Cui",
"given": "Hao"
},
{
"family": "Garrison",
"given": "Jake"
},
{
"family": "Johnston",
"given": "Renee"
},
{
"family": "Kast",
"given": "Anton"
},
{
"family": "McLean",
"given": "Cory Y."
},
{
"family": "Norgaard",
"given": "Peter"
},
{
"family": "Shamsi",
"given": "Zahra"
},
{
"family": "Smalling",
"given": "David"
},
{
"family": "Thompson",
"given": "James"
},
{
"family": "Venugopalan",
"given": "Subhashini"
},
{
"family": "Williams",
"given": "Brian P."
},
{
"family": "He",
"given": "Chujun"
},
{
"family": "Martinson",
"given": "Sarah"
},
{
"family": "Plomecka",
"given": "Martyna"
},
{
"family": "Wei",
"given": "Lai"
},
{
"family": "Zhou",
"given": "Yuchen"
},
{
"family": "Zhu",
"given": "Qian-Ze"
},
{
"family": "Abraham",
"given": "Matthew"
},
{
"family": "Brand",
"given": "Erica"
},
{
"family": "Bulanova",
"given": "Anna"
},
{
"family": "Cardille",
"given": "Jeffrey A."
},
{
"family": "Co",
"given": "Chris"
},
{
"family": "Ellsworth",
"given": "Scott"
},
{
"family": "Joseph",
"given": "Grace"
},
{
"family": "Kane",
"given": "Malcolm"
},
{
"family": "Krueger",
"given": "Ryan"
},
{
"family": "Kartiwa",
"given": "Johan"
},
{
"family": "Liebling",
"given": "Dan"
},
{
"family": "Lueckmann",
"given": "Jan-Matthis"
},
{
"family": "Raccuglia",
"given": "Paul"
},
{
"family": "Wang",
"given": "Xuefei Julie"
},
{
"family": "Chou",
"given": "Katherine"
},
{
"family": "Manyika",
"given": "James"
},
{
"family": "Matias",
"given": "Yossi"
},
{
"family": "Platt",
"given": "John C."
},
{
"family": "Dorfman",
"given": "Lizzie"
},
{
"family": "Mourad",
"given": "Shibl"
},
{
"family": "Brenner",
"given": "Michael P."
}
],
"container-title-short": "Nature",
"volume": "654",
"issue": "8120",
"page": "909-916",
"DOI": "10.1038/s41586-026-10658-6",
"PMID": "42156545",
"PMCID": "PMC13293872",
"ISSN": "0028-0836",
"publisher": "Nature Portfolio",
"URL": "https://doi.org/10.1038/s41586-026-10658-6",
"language": "en",
"issued": {
"date-parts": [
[
2026,
5,
19
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1093/nar/gkag706 [code]
scDifformer: diffusion-based post-training for virtual cell modeling across large-scale single-cell data.
Journal: Nucleic acids research
In common: anndata, Scanpy, TensorFlow, 7 other tools, 3 references
[2] doi:10.1038/s42003-026-10957-8 [code]
Brain defence by the extracellular matrix protein Cochlin.
Journal: Communications biology
In common: JAX, Hugging Face Transformers, XGBoost, 7 other tools, zebrafish
[3] doi:10.1038/s41592-026-03194-8 [code]
Beyond benchmarking: an expert-guided consensus approach to spatially aware clustering.
Journal: Nature methods
In common: anndata, Scanpy, TensorFlow, 6 other tools, methods / tools, 2 references
[4] doi:10.1038/s41592-026-03057-2 [code]
CREsted: modeling genomic and synthetic cell-type-specific enhancers across tissues and species.
Journal: Nature methods
In common: anndata, Scanpy, TensorFlow, 7 other tools, zebrafish, methods / tools
[5] doi:10.1038/s44320-026-00208-7 [code]
Interpretable deep generative ensemble learning for single-cell omics with Hydra.
Journal: Molecular systems biology
In common: anndata, Scanpy, TensorFlow, 7 other tools, 1 reference
[6] doi:10.1093/bib/bbag490 [code]
Systematic benchmarking and optimal strategy selection of cross-species integration methods.
Journal: Briefings in bioinformatics
In common: anndata, Scanpy, scikit-learn, 4 other tools, methods / tools, 3 references
[7] doi:10.1186/s12859-026-06490-4 [code]
Tissueformer: extending single-cell foundation models to predict population-level phenotypes.
Journal: BMC bioinformatics
In common: Hugging Face Transformers, anndata, PyTorch, 5 other tools, other condition, 2 references
[8] doi:10.1038/s41598-026-48613-0 [code]
An snRNA-seq aging clock for the fruit fly head sheds light on sex-biased aging.
Journal: Scientific reports
In common: XGBoost, anndata, Scanpy, 7 other tools
[9] doi:10.1186/s12864-026-12965-8 [code]
Systematic evaluation of single-cell foundation model interpretability: attention-derived edge scores add no incremental value over gene-level features for perturbation-target prediction.
Journal: BMC genomics
In common: Hugging Face Transformers, anndata, Scanpy, 7 other tools
[10] doi:10.64898/2026.03.30.714220 [code]
An integrated single cell and spatial omics atlas of human prenatal development
Journal: bioRxiv (preprint)
In common: Hugging Face Transformers, anndata, Scanpy, 7 other tools

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.