OSCR

JARVIS, should this study be selected for full-text screening? Performance of a Joint AI-ReViewer Interactive Screening tool for systematic reviews

Code ↔ Paper

4 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 4 matches
  1. [1] § Methods › Text data pre-processing › Step 4: Modelling ↔ scripts/Hypertension h2o.R, lines 452–540 · score 0.63 · h2o.deeplearning, cross validate, training samples, folds, classified, models
  2. [2] § Methods › Text data pre-processing › Step 4: Modelling ↔ scripts/Anticoag atrial fib h2o.R, lines 446–529 · score 0.63 · h2o.deeplearning, cross validate, training samples, folds, classified, models
  3. [3] § Methods › Text data pre-processing ↔ scripts/Anticoag 2 h2o.R, lines 239–321 · score 0.59 · TF IDF, PCA, SMART, ngrams, tokenised, words
  4. [4] § Methods › Text data pre-processing ↔ scripts/Sedat behaviour children h2o.R, lines 238–309 · score 0.59 · TF IDF, PCA, SMART, ngrams, tokenised, words

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

R · 764 lines · 32 KB · no license · 1 match

  1. ## libraries ####
  2. library(tidyverse)
  3. library(recipes)
  4. library(textrecipes)
  5. library(h2o)
  6. library(future.apply)
  7. library(tibble)
  8. library(dplyr)
  9. library(ggpubr)
  10. library(PRROC)
  12. # The two functions below allowed me to do hyperparameter tuning.
  13. # The first one, ending in 'parallel', allows me to open several parallel sessions
  14. # and test each hyperparameter combination in parallel. Requires a TON of RAM, so
  15. # I use it carefully.
  16. # The second one, ending in repeats, will test each combination in sequence. Takes
  17. # longer, but it works.
  19. run_each5_with_repeats_parallel <- function(df, n, epochs, hiddenunis, activ, stop_rounds, stop_tol, rates_anneal,min_batch,l2,rate, repeats, n_workers) {
  21. # remove contents of the h2o server if one is open already
  22. if (tryCatch({ h2o.clusterStatus(); TRUE }, error = function(e) FALSE)) {
  23. h2o.removeAll(timeout_secs = 0)
  24. }
  26. # 1. Build all param combinations × repeats
  27. param_combinations <- expand.grid(
  28. epochs = epochs,
  29. hidden_units = hiddenunis,
  30. activ = activ,
  31. stop_rounds = stop_rounds,
  32. stop_tol = stop_tol,
  33. rates_anneal = rates_anneal,
  34. min_batch = min_batch,
  35. l2 = l2,
  36. rate = rate,
  37. stringsAsFactors = FALSE
  38. )
  40. jobs <- do.call(rbind, lapply(seq_len(nrow(param_combinations)), function(i) {
  41. cbind(param_combinations[i, , drop=FALSE], repeat_run = seq_len(repeats), ID = i)
  42. }))
  43. jobs <- as.data.frame(jobs)
  45. # 2. Function to run a single job
  46. run_single <- function(job_row) {
  47. epochs <- as.numeric(job_row["epochs"])
  48. stop_rounds <- as.numeric(job_row["stop_rounds"])
  49. stop_tol <- as.numeric(job_row["stop_tol"])
  50. rates_anneal <- as.numeric(job_row["rates_anneal"])
  51. min_batch <- as.numeric(job_row["min_batch"])
  52. hidden_units <- as.character(job_row["hidden_units"])
  53. activ <- as.character(job_row["activ"])
  54. repeat_run <- as.numeric(job_row["repeat_run"])
  55. l2 <- as.numeric(job_row["l2"])
  56. rate <- as.numeric(job_row["rate"])
  57. ID <- as.numeric(job_row["ID"])
  59. set.seed(123 + ID + repeat_run) # More variability per run
  60. start_time <- Sys.time()
  61. res <- each5.2(df, n, epochs, hidden_units, activ, stop_rounds, stop_tol,rates_anneal,min_batch, l2, rate)
  62. end_time <- Sys.time()
  63. res %>%
  64. mutate(
  65. ID = ID,
  66. repeat_run = repeat_run,
  67. configs = paste(epochs, hidden_units, activ, stop_rounds, stop_tol ,rates_anneal,min_batch,l2,rate, sep = ", "),
  68. elapsed_time = as.numeric(difftime(end_time, start_time, units = "secs"))
  69. )
  70. }
  72. # 3. Set up parallel plan
  73. future::plan(multisession, workers = 12)
  74. plan()
  76. # 4. Run in parallel!
  77. results_list <- future.apply::future_lapply(
  78. split(jobs, seq_len(nrow(jobs))),
  79. function(job) run_single(job[1, ])
  80. )
  82. # 5. Combine all results into one tibble
  83. results <- dplyr::bind_rows(results_list)
  85. return(results)
  86. }
  88. run_each5_with_repeats <- function(df, n, epochs, hiddenunis, activ, stop_rounds, stop_tol,rates_anneal,min_batch, l2, rate, repeats) {
  90. # remove contents of the h2o server if one is open already
  91. if (tryCatch({ h2o.clusterStatus(); TRUE }, error = function(e) FALSE)) {
  92. h2o.removeAll(timeout_secs = 0)
  93. }
  95. elapsed_time <- numeric() # Initialize as numeric vector
  96. param_combinations <- expand.grid(
  97. epochs = epochs,
  98. hidden_units = hiddenunis,
  99. activ = activ,
  100. stop_rounds = stop_rounds,
  101. stop_tol = stop_tol,
  102. rates_anneal = rates_anneal,
  103. min_batch = min_batch,
  104. l2 = l2,
  105. rate = rate
  106. )
  108. # Initialize results as an empty tibble
  109. results <- tibble()
  111. total_runs <- nrow(param_combinations) * repeats # Total number of iterations
  112. current_run <- 0 # Initialize counter for completed runs
  114. # Iterate over each combination of parameters
  115. for (i in seq_len(nrow(param_combinations))) {
  116. print(paste(i, '/', nrow(param_combinations)))
  117. epochs <- param_combinations$epochs[i]
  118. hidden_units <- as.character(param_combinations$hidden_units[i])
  119. activ <- param_combinations$activ[i]
  120. stop_rounds <- param_combinations$stop_rounds[i]
  121. stop_tol <- param_combinations$stop_tol[i]
  122. rates_anneal <- param_combinations$rates_anneal[i]
  123. min_batch <- param_combinations$min_batch[i]
  124. l2 <- param_combinations$l2[i]
  125. rate <- param_combinations$rate[i]
  128. set.seed(123)
  129. for (repeat_idx in seq_len(repeats)) {
  130. current_run <- current_run + 1 # Increment the completed runs counter
  132. start_time <- Sys.time() # Record start time
  133. current_result <- each5.2(df, n, epochs, hidden_units, activ, stop_rounds, stop_tol, rates_anneal, min_batch, l2, rate) %>%
  134. mutate(
  135. ID = i,
  136. repeat_run = repeat_idx,
  137. configs = paste(epochs, hidden_units, activ, stop_rounds, stop_tol, rates_anneal, min_batch, l2, rate, sep = ", ")
  138. )
  141. # Append the current result to the results tibble
  142. results <- bind_rows(results, current_result)
  143. end_time <- Sys.time() # Record end time
  145. # Calculate elapsed time for this run
  146. elapsed_time <- c(elapsed_time, as.numeric(difftime(end_time, start_time, units = "secs")))
  147. mean_time <- mean(elapsed_time, na.rm = TRUE)
  148. remaining_runs <- total_runs - current_run
  149. estimated_remaining_time <- mean_time * remaining_runs
  151. # Convert estimated time to minutes and seconds
  152. estimated_minutes <- floor(estimated_remaining_time / 60)
  153. estimated_seconds <- round(estimated_remaining_time %% 60)
  155. # Print estimated time remaining
  156. print(paste(
  157. "Estimated remaining time:",
  158. estimated_minutes, "min", estimated_seconds, "sec"
  159. ))
  160. }
  161. }
  162. return(results)
  163. }
  165. ## ROC-PR FUNCTION####
  166. # calc_aucpr is a function designed to calculate aucpr for our analyses. It'll be used in the 'summaries' section.
  167. calc_aucpr <- function(df_iter) {
  168. # df_iter = subset of results for a single iteration
  170. # True labels as 1/0
  171. truth <- ifelse(df_iter$finaldecision == "Include", 1, 0)
  173. # Predicted scores (probabilities)
  174. scores <- df_iter$newpred
  176. # PRROC needs the scores of the positive class separately:
  177. pr <- pr.curve(
  178. scores.class0 = scores[truth == 0],
  179. scores.class1 = scores[truth == 1],
  180. curve = FALSE
  181. )
  183. return(pr$auc.integral)
  184. }
  186. ##hypertension data clean #####
  187. # In this section we read the file in 'data' containing our abstracts, decisions and the LLM responses.
  188. # We break the responses into columns, clean them, create the numberical variables (PICOS scores).
  189. dfwithPICOS <- read_rds('dfgpt.stress.complete.rds')
  190. dfPICOSfinal <- dfwithPICOS %>%
  191. separate_wider_delim(cols = GPT_Response, delim = ',', names = c('review', 'P', 'I', 'C', 'O', 'S'), cols_remove = F) %>%
  192. mutate(review = chartr("[],012345 '","...........", review),
  193. review = chartr("- ","..", review),
  194. review = gsub('[.]','',review),
  195. P = chartr("[],012345 '","...........", P),
  196. P = chartr("- ","..", P),
  197. P = gsub('[.]','',P),
  198. I = chartr("[],012345 '","...........", I),
  199. I = chartr("- ","..", I),
  200. I = gsub('[.]','',I),
  201. C = chartr("[],012345 '","...........", C),
  202. C = chartr("- ","..", C),
  203. C = gsub('[.]','',C),
  204. O = chartr("[],012345 '","...........", O),
  205. O = chartr("- ","..", O),
  206. O = gsub('[.]','',O),
  207. S = chartr("[],012345 '","...........", S),
  208. S = chartr("- ","..", S),
  209. S = gsub('[.]','',S)) %>%
  210. mutate(Pn = case_when(P =='Yes'~ 1, ### create numerical variables with PICOS score
  211. P == 'No' ~ 0,
  212. P == 'Uncertain' ~ 0.5),
  213. In = case_when(I =='Yes'~ 1,
  214. I == 'No' ~ 0,
  215. I == 'Uncertain' ~ 0.5),
  216. Cn = case_when(C =='Yes'~ 1,
  217. C == 'No' ~ 0,
  218. C == 'Uncertain' ~ 0.5),
  219. On = case_when(O =='Yes'~ 1,
  220. O == 'No' ~ 0,
  221. O == 'Uncertain' ~ 0.5),
  222. Sn = case_when(S =='Yes'~ 1,
  223. S == 'No' ~ 0,
  224. S == 'Uncertain' ~ 0.5)) %>%
  225. dplyr::select(key, keywords, title, abstract, authors, year, review, P, I, C, O, S, Pn, In,Cn, On, Sn, decision, finaldecision) %>%
  226. mutate(totalscore = rowSums(.[c('Pn','In','Cn','On','Sn')])) %>%
  227. filter(key != 'ardiologie et Pneumologie de Quebec' & key != 'iomedical Science') %>%
  228. distinct(key, .keep_all = T)
  230. ## PREPARE DATA #####
  231. # In this section we clean the text variables and prepare them for tokenisation and TF-IDF generation.
  232. dftoken <- dfPICOSfinal %>%
  233. select(key, title, authors, abstract, totalscore, review, P, I, C, O, S,Pn, In,Cn, On, Sn, decision, finaldecision) %>%
  234. mutate(abstractsub = gsub('-', ' ', abstract)) %>% # Remove hyphens
  235. mutate(abstractsub = iconv(abstractsub, from = "", to = "ASCII//TRANSLIT")) %>% # Remove/convert weird chars
  236. mutate(abstractsub = gsub("–|—|â€|’|“|‘|•|−", " ", abstractsub)) %>% # Remove common mojibake
  237. mutate(abstractsub = gsub('[[:punct:]]', '', abstractsub)) %>% # Remove punctuation
  238. mutate(abstractsub = tolower(abstractsub)) %>%
  239. mutate(abstractsub = gsub('[0-9]+', ' ', abstractsub)) %>%
  240. mutate(abstractsub = gsub("\\s+", " ", abstractsub)) %>% # Normalize whitespace
  241. mutate(titlesub = gsub('-', ' ', title)) %>% # Remove hyphens
  242. mutate(titlesub = iconv(titlesub, from = "", to = "ASCII//TRANSLIT")) %>% # Remove/convert weird chars
  243. mutate(titlesub = gsub("–|—|â€|’|“|‘|•|−", " ", titlesub)) %>% # Remove common mojibake
  244. mutate(titlesub = gsub('[[:punct:]]', '', titlesub)) %>% # Remove punctuation
  245. mutate(titlesub = tolower(titlesub)) %>%
  246. mutate(titlesub = gsub('[0-9]+', ' ', titlesub)) %>%
  247. mutate(titlesub = gsub("\\s+", " ", titlesub)) %>% # Normalize whitespace
  248. mutate_at(vars(P, I, C, O, S), funs(factor(., levels = c('Uncertain', 'Yes', 'No') ))) %>%
  249. mutate(reviewn = case_when(review == 'No' ~ 1,
  250. review == 'Yes' ~ 0 ))
  252. rm(dfPICOSfinal)
  253. rm(dfwithPICOS)
  255. tokenization_recipe <- recipe(decision ~ key + abstract + authors + abstractsub + P + I + C + O + S + totalscore + review + finaldecision, data = dftoken) %>%
  256. update_role(key, new_role = "id") %>% #Mark 'key' as an ID
  257. update_role(abstract, new_role = "id") %>%
  258. update_role(authors, new_role = "id") %>%
  259. update_role(finaldecision, new_role = "id") %>%
  260. step_tokenize(abstractsub, token = "words") %>%
  261. step_stopwords(abstractsub, stopword_source = "smart") %>%
  262. #step_stem(abstractsub) %>%
  263. step_ngram(abstractsub, num_tokens = 3, min_num_tokens = 1) %>%
  264. step_tokenfilter(abstractsub, max_tokens = 7000) %>%
  265. step_tfidf(abstractsub) %>%
  266. #step_pca(starts_with('tfidf'),threshold = .80, prefix = 'KPCa') %>%
  267. #step_tokenize(titlesub, token = "words") %>%
  268. #step_stopwords(titlesub, stopword_source = "smart") %>%
  269. #step_stem(titlesub) %>%
  270. #step_ngram(titlesub, num_tokens = 3, min_num_tokens = 1) %>%
  271. #step_tokenfilter(titlesub, max_tokens = 3000 ) %>%
  272. #step_tfidf(titlesub) %>%
  273. #step_pca(c(reviewn, totalscore,keys1, score1), num_comp = 1, keep_original_cols = T) %>%
  274. step_pca(starts_with('tfidf'),threshold = .90, prefix = 'KPCa') %>%
  275. step_center(starts_with('KPC')) %>%
  276. step_range(totalscore, min = 0, max = 1) %>%
  277. step_dummy( all_of(c('P', 'I', 'C','O','S', 'review')), one_hot = T)
  279. prepped_recipe <- prep(tokenization_recipe, training = dftoken, retain = TRUE) # Preprocess and retain
  280. baked_df <- bake(prepped_recipe, new_data = NULL)
  282. dir.create('baked', showWarnings = F, recursive = T)
  284. saveRDS(baked_df, 'baked/hypertension final.rds')
  286. ##START FROM HERE WITH THE BAKED READY####
  287. # Sometimes, 'baking' the dataframe takes some time depending on your hardware. I preferred to save
  288. # the dataframe and reload it here below. Also this is where I include the weights.
  289. baked_df <- read_rds('baked/hypertension final.rds') %>%
  290. mutate(authors = ifelse(is.na(authors),'no author',authors )) %>%
  291. mutate(finaldecision = as.factor(finaldecision)) %>%
  292. mutate(weightsc = ifelse(decision == "Include", 40,1))
  294. ggplot(baked_df, aes( x = totalscore * 5)) +
  295. geom_histogram( aes(fill = finaldecision, y = ..density..)) +
  296. theme_classic() +
  297. theme(legend.position = "top") +
  298. labs(x = 'PICOS score', fill = "FT decision")
  300. ## new function 5 each time #####
  301. # each5.2 performs iterative active-screening with an H2O deep-learning classifier.
  302. # Workflow per round:
  303. # 1) Select new records to review (top/bottom scoring records plus a small buffer),
  304. # then add them to the cumulative labeled sample (`samplefull`).
  305. # 2) Build training folds by repeating Include cases across folds and sampling Exclude
  306. # cases per fold to keep class balance for cross-validation.
  307. # 3) Train a Bernoulli H2O deep-learning model with the provided hyperparameters.
  308. # 4) Score unscreened records with each CV model, average Include probabilities,
  309. # normalize to `newpred`, and assign Include/Exclude labels using `threshold`.
  310. # 5) Save per-round predictions and error-tracking fields, and mark low-probability
  311. # records as candidates to exclude in later sampling.
  312. # Early stopping: after the minimum rounds, stop when too few records remain above
  313. # threshold for consecutive rounds. Returns one data frame with all rounds combined.
  314. each5.2 <- function(df1, max, epoc, hidu, activ, stop_rounds, stop_tol, rates_anneal,min_batch, l2, rate) {
  319. # Initialize variables for Stage 2
  320. results <- vector("list", max)
  321. exclude_keys <- data.frame(NA)
  322. samplefull <- data.frame()
  323. preds <- df1
  324. min_rounds <- 71
  325. prefix <- paste0("worker", Sys.getpid(), "_", as.integer(Sys.time()), "_")
  326. threshold <- 0.4
  327. min_hold <- 1
  328. since_change <- min_hold
  331. for (i in seq_len(max)) {
  332. time1 <- Sys.time()
  336. if(i > 1) {
  339. # inc_rate <- (include_count + exclude_count) / nrow(df1)
  340. # print(inc_rate)
  341. #
  342. # cap <- dplyr::case_when(
  343. # #inc_rate > 0.30 ~ 0.75,
  344. # #inc_rate > 0.25 ~ 0.70,
  345. # #inc_rate > 0.20 ~ 0.60,
  346. # #inc_rate > 0.15 ~ 0.55,
  347. # inc_rate > 0.10 ~ 0.5,
  348. # inc_rate > 0.05 ~ 0.45,
  349. # TRUE ~ threshold
  350. # )
  351. #
  352. #
  353. # old <- threshold
  354. #
  355. # if (since_change >= min_hold && cap > threshold) {
  356. # threshold <- cap # jump directly to the mapped cap
  357. # since_change <- 1 # or set to 1 if you prefer counting this iter as "held"
  358. # } else {
  359. # since_change <- since_change + 1
  360. # }
  361. # print(paste('threshold =', threshold))
  365. keys_to_exclude <- exclude_keys$key
  366. keys_in_sample <- samplefull$key
  369. preds <- preds %>%
  370. suppressMessages(left_join(.,df1))
  373. sampled_df <- preds %>%
  374. ungroup() %>%
  375. filter(!key %in% keys_in_sample) %>%
  376. filter(!key %in% keys_to_exclude) %>%
  377. arrange(desc(Include)) %>%
  378. slice_head(n=15) %>%
  379. bind_rows(preds %>%
  380. ungroup() %>%
  381. filter(!key %in% keys_in_sample) %>%
  382. filter(!key %in% keys_to_exclude) %>%
  383. arrange(desc(Include)) %>%
  384. slice_tail(n=8) )%>%
  385. bind_rows(preds %>%
  386. ungroup() %>%
  387. filter(!key %in% keys_in_sample) %>%
  388. filter(key %in% keys_to_exclude) %>%
  389. arrange(desc(Include)) %>%
  390. slice_head(n=7)) %>%
  391. select(-predict, -Include, -Exclude, -new, -incpred, -excpred, -newpred, - thresh)
  395. }else{
  396. sampled_df <- preds %>%
  397. arrange(desc(totalscore)) %>%
  398. slice(c((nrow(.)-4):nrow(.), 1:20)) %>%
  399. bind_rows(preds %>%
  400. arrange(desc(totalscore)) %>%
  401. filter(totalscore*5 <= 2.5) %>% slice(1:5))
  403. localH2O = h2o.init(ip="localhost", port = 54321,
  404. startH2O = TRUE, nthreads=1, max_mem_size = '8G')
  405. prefix <- paste0("worker", Sys.getpid(), "_", as.integer(Sys.time()), "_")
  408. }
  410. samplefull <- samplefull %>%
  411. bind_rows(sampled_df %>% mutate(iter = i)) %>%
  412. distinct()
  414. includes <- samplefull %>% filter(decision == "Include")
  415. excludes <- samplefull %>% filter(decision == "Exclude")
  417. n_excl_per_fold <- nrow(includes) # or set any number you prefer per fold
  419. # 1) 3 repeats of each Include, labeled folds 1, 2, 3
  420. includes3 <- includes %>%
  421. mutate(.k = 1L) %>%
  422. left_join(data.frame(folds = 1:3, .k = 1L), by = ".k",relationship = "many-to-many") %>%
  423. select(-.k)
  425. # 2) Randomize all Excludes and partition into 3 (no overlap / no replacement)
  426. excl_folds <- lapply(1:3, function(f) {
  427. excludes %>%
  428. slice_sample(n = min(n_excl_per_fold, nrow(.)), replace = FALSE) %>%
  429. mutate(folds = f)
  430. }) %>% bind_rows()
  432. # 3) Combine
  433. samplefull2 <- bind_rows(includes3, excl_folds)
  435. h2otrain <- h2o.na_omit(as.h2o(samplefull2, destination_frame = paste0(prefix, 'train1',i)))
  437. test <- df1 %>%
  438. filter(!(key %in% samplefull$key))
  440. h2otest <- h2o.na_omit(as.h2o(test, destination_frame = paste0(prefix, 'preddata',i)))
  441. h2otest <- h2o.na_omit(h2otest)
  443. include_count <- samplefull %>%
  444. filter(decision == "Include") %>%
  445. nrow()
  447. exclude_count <- samplefull %>%
  448. filter(decision == "Exclude") %>%
  449. nrow()
  452. # Update the recipe to use RELEVANCE as the outcome
  454. y <- "decision"
  455. x <- names(df1)[c( 7:(ncol(df1)-1))]
  459. model = h2o.deeplearning(x=x,
  460. y=y,
  461. training_frame=h2otrain,
  462. distribution = "bernoulli",
  463. activation = as.character(activ),
  464. hidden = eval(parse(text = hidu)),
  465. epochs = epoc, l1 = 0, l2 = l2,rate = rate,
  466. loss = "CrossEntropy",
  467. initial_weight_distribution = c("UniformAdaptive"),
  468. ignore_const_cols = F, seed = 123,
  469. adaptive_rate = F, standardize = F,
  470. nesterov_accelerated_gradient = T,
  471. fast_mode = T,
  472. reproducible = T,
  473. overwrite_with_best_model = T,
  474. score_training_samples = 500, score_validation_samples = 500,
  475. stopping_metric = "mean_per_class_error",classification_stop = -1,
  476. stopping_rounds = stop_rounds, stopping_tolerance = stop_tol,
  477. model_id = paste(i, epoc, as.character(activ), stop_rounds, stop_tol, rates_anneal,min_batch,l2,rate, sep = "_" ),
  478. #nfolds = 3, fold_assignment = 'Stratified',
  479. fold_column = 'folds',
  480. rate_annealing = rates_anneal,# rate_decay = 0.8,
  481. keep_cross_validation_models = T,
  482. keep_cross_validation_predictions = T, weights_column = 'weightsc',
  483. mini_batch_size = min_batch#, balance_classes = T,
  484. #max_after_balance_size = max(.1,include_count/exclude_count),
  485. #class_sampling_factors = c(1, exclude_count / include_count)
  486. )
  490. modelcv1 <- h2o.getModel(paste(i, epoc, as.character(activ), stop_rounds, stop_tol, rates_anneal,min_batch,l2,rate,'cv_1',sep = '_'))
  491. predweird1 <- predict(modelcv1, newdata = h2otest )
  493. predweird1.2 <- bind_cols(test, as.data.frame(predweird1)) %>%
  494. mutate(pp = 1) %>%
  495. select(-starts_with('KPC'), -contains('_'))
  497. modelcv2 <- h2o.getModel(paste(i, epoc, as.character(activ), stop_rounds, stop_tol, rates_anneal,min_batch,l2,rate,'cv_2',sep = '_'))
  498. predweird2 <- predict(modelcv2, newdata = h2otest )
  500. predweird2.2 <- bind_cols(test, as.data.frame(predweird2)) %>%
  501. mutate(pp = 2) %>%
  502. select(-starts_with('KPC'), -contains('_'))
  504. modelcv3 <- h2o.getModel(paste(i, epoc, as.character(activ), stop_rounds, stop_tol, rates_anneal,min_batch,l2,rate,'cv_3',sep = '_'))
  505. predweird3 <- predict(modelcv3, newdata = h2otest )
  507. predweird3.2 <- bind_cols(test, as.data.frame(predweird3)) %>%
  508. mutate(pp = 3) %>%
  509. select(-starts_with('KPC'), -contains('_'))
  512. all <- bind_rows(predweird1.2,
  513. predweird2.2,
  514. predweird3.2) %>%
  515. group_by(key, decision, finaldecision) %>%
  516. summarise_at(vars(Include), funs(mean(., na.rm =T))) %>%
  517. ungroup() %>%
  518. mutate(Exclude = 1-Include,
  519. newpred = (Include - min(Include)) / (max(Include) - min(Include) ),
  520. predict = ifelse(newpred >= 0.5, "Include", "Exclude"))
  522. preds <- test %>%
  523. merge(.,all, .by = key) %>%
  524. mutate(new = i) %>%
  525. mutate(incpred = include_count, # Add the "Include" count
  526. excpred = exclude_count) %>%
  527. mutate(thresh = threshold)
  531. my_keys <- h2o.ls()[,1]
  532. h2o.rm(my_keys[grepl(paste0("^", prefix), my_keys)], cascade = TRUE)
  534. gc()
  536. time2 <- Sys.time()
  538. exclude_keys <- preds %>%
  539. ungroup() %>% filter(Include < threshold) %>%
  540. select(key, new)
  542. # Add the "Exclude" count
  543. results[[i]] <- preds %>%
  544. select(-starts_with('KPC'), -starts_with('tfidf'), -starts_with(c('P_','I_', 'C_', 'O_', 'S_')))
  547. sumtextPICOS <- (bind_rows(results)) %>%
  548. mutate(newclass = ifelse(Include < thresh,"Exclude", "Include")) %>%
  549. mutate(inc.correct = ifelse(finaldecision == "Include" & newclass == "Include",1,0) ) %>%
  550. mutate(exc.correct = ifelse(finaldecision == "Exclude" & newclass == "Exclude",1,0) ) %>%
  551. mutate(exc.incorrect = ifelse(finaldecision == "Include" & newclass == "Exclude",1,0) ) %>%
  552. mutate(inc.incorrect = ifelse(finaldecision == "Exclude" & newclass == "Include",1,0) ) %>%
  553. group_by(., new) %>% #, thresh) %>%
  554. summarise_at(vars(inc.correct,exc.correct,inc.incorrect, exc.incorrect), funs (sum(.,na.rm= T))) %>%
  555. merge(.,bind_rows(results) %>% group_by(., new) %>%
  556. summarise_at(vars(incpred, excpred), funs(max(.))), .by = c('configs', 'new')) %>%
  557. arrange(-desc(new))
  560. print(ggarrange(nrow = 2,ncol = 1, ggplot(data = sumtextPICOS, aes(x = new, y = inc.incorrect)) +
  561. geom_line(colour = 'blue') +
  562. geom_line(colour = 'red', aes(y = exc.incorrect)) +
  563. geom_point(pch = 21, colour = 'blue') +
  564. geom_point(pch = 21, colour = 'red', aes(y = exc.incorrect)) +
  565. geom_text(vjust = -0.5, size = 3 ,aes(y = exc.incorrect, label = exc.incorrect))+
  566. geom_text(vjust = -0.5, size = 3 ,aes(y = inc.incorrect, label = inc.incorrect))+
  567. #facet_wrap(~configs) +
  568. #scale_x_continuous(breaks = c(0,2,4,6,8,10)) +
  569. scale_y_continuous(limits = c(0,max(sumtextPICOS$inc.incorrect)),name = 'Blue = Included Incorrectly', sec.axis = sec_axis(~. , name = "Red = Excluded Incorrectly")) +
  570. labs(x = 'Rounds') +
  571. theme_classic(),
  572. ggplot(data = subset(preds), aes(x = (Include))) +
  573. geom_histogram( aes(fill = finaldecision,y = after_stat(density))) +
  574. scale_fill_manual(values = c('black', 'red')) +
  575. geom_vline(aes(xintercept = thresh), colour = 'blue', linetype = 2) +
  576. #facet_wrap(~new* configs, ncol = 2, scale = 'free')+
  577. labs(x = "Predicted Probability ", y = "Probability density",fill = 'Known decision') +
  578. theme_classic() +
  579. theme(legend.position = "top")))
  584. if (min_rounds <= i && sum(preds$Include >= threshold, na.rm = T) <= max(30, nrow(df1) / 100)) {
  586. return(bind_rows(results))
  587. break
  588. }
  590. gc()
  592. print((time2 - time1))
  593. print(i)
  596. }
  599. return(bind_rows(results))
  600. }
  601. ##run it here#####
  602. # This block sets one hyperparameter configuration and runs the full
  603. # iterative simulation pipeline.
  604. # - The vectors below are the model settings explored by
  605. # `run_each5_with_repeats` (single values here = one configuration).
  606. # - `TIMES` controls how many full repeats are executed.
  607. # - The returned object `results` contains per-round predictions/metrics.
  608. # - Results are persisted to `results/resultsanticoag2.rds` so the
  609. # summarisation section can be rerun without retraining.
  610. epochs <- c(100)
  611. hiddenunis <- c('c(50,25,10,5)')
  612. activ <- c('Tanh')
  613. stop_rounds <- c(3)
  614. stop_tol <- c(1e-5)
  615. rates_anneal <- c(0.001)
  616. min_batch <- c(1)
  617. l2 <- c(0.02575)
  618. rate <- c(0.001)
  619. TIMES <- 1
  621. results <- run_each5_with_repeats(baked_df, 71, epochs,hiddenunis,activ, stop_rounds, stop_tol,rates_anneal, min_batch,l2,rate, TIMES)
  623. dir.create('results', showWarnings = F, recursive = T)
  624. saveRDS(results, 'results/resultshypertension.rds')
  626. ##summarise#####
  627. results <- read_rds('results/resultshypertension.rds')
  628. EXCINC <- results %>%
  629. group_by(configs) %>%
  630. filter(finaldecision == "Include" & newpred < thresh) %>%
  631. select(-starts_with(c('P_','I_', 'C_', 'O_', 'S_', 'KPC', 'review_')))
  633. sumtextPICOS <- results %>%
  634. mutate(newclass = ifelse(Include < thresh,"Exclude", "Include")) %>%
  635. mutate(inc.correct = ifelse(finaldecision == "Include" & newclass == "Include",1,0) ) %>%
  636. mutate(exc.correct = ifelse(finaldecision == "Exclude" & newclass == "Exclude",1,0) ) %>%
  637. mutate(exc.incorrect = ifelse(finaldecision == "Include" & newclass == "Exclude",1,0) ) %>%
  638. mutate(inc.incorrect = ifelse(finaldecision == "Exclude" & newclass == "Include",1,0) ) %>%
  639. group_by(., configs, new, ID, thresh) %>% #, thresh) %>%
  640. summarise_at(vars(inc.correct,exc.correct,inc.incorrect, exc.incorrect), funs (sum(.,na.rm= T))) %>%
  641. merge(.,results %>% group_by(., configs, new, ID,thresh) %>%
  642. summarise_at(vars(incpred, excpred), funs(max(.))), .by = c('configs', 'new')) %>%
  643. mutate(reads = incpred+ excpred) %>%
  644. mutate(percread = (incpred + excpred )/ nrow(baked_df) * 100 ,
  645. percsave = exc.correct/nrow(baked_df) * 100,
  646. ratio = incpred/excpred * 100) %>%
  647. merge(.,results %>%
  648. group_by(configs, new, ID,thresh) %>%
  649. summarise(
  650. aucpr = calc_aucpr(cur_data())
  651. ),.by = new) %>%
  652. merge(.,results %>%
  653. group_by(new, finaldecision, configs) %>%
  654. summarise(foundft = (193 - n() ) / 193 * 100) %>%
  655. filter(finaldecision == "Include") %>%
  656. select(-finaldecision))%>%
  657. mutate(recall = inc.correct / (inc.correct + exc.incorrect) * 100,
  658. recall_cumm = (sum(baked_df$finaldecision == "Include") - exc.incorrect ) / ((sum(baked_df$finaldecision == "Include") - exc.incorrect ) + exc.incorrect) * 100,
  659. specificity = exc.correct / (exc.correct + inc.incorrect) * 100) %>%
  660. arrange(-desc(new)) ;View(sumtextPICOS)
  662. calclong <- sumtextPICOS %>%
  663. mutate(aucpr = aucpr*100) %>%
  664. bind_rows(data.frame(new = 0, percread = 0, specificity = 0, foundft =0, recall=0, recall_cumm = 0)) %>%
  665. pivot_longer(cols =c( specificity, foundft, recall, recall_cumm)) %>%
  666. mutate(name = factor(name))
  668. ggplot(data = sumtextPICOS, aes(x = (new*30 / nrow(baked_df)* 100), y = (inc.incorrect + inc.correct ))) +
  669. geom_line(colour = 'blue') +
  670. geom_line(colour = 'red', aes(y = exc.incorrect)) +
  671. geom_point(pch = 21, colour = 'blue') +
  672. geom_point(pch = 21, colour = 'red', aes(y = exc.incorrect)) +
  673. geom_text(vjust = -0.5, size = 3 ,aes(y = exc.incorrect, label = exc.incorrect))+
  674. geom_text(vjust = -0.5, size = 3 ,aes(y = inc.incorrect + inc.correct , label = inc.incorrect + inc.correct ))+
  675. scale_y_continuous(name = 'Blue = Suggested Includes', sec.axis = sec_axis(~. , name = "Red = Excluded Incorrectly")) +
  676. labs(x = '% of studies read') +
  677. theme_classic() +
  678. facet_wrap(~configs)
  680. ggplot(data = calclong, aes(x = percread, y = value)) +
  681. geom_line(aes(colour = name), size = 1, alpha= 0.5) +
  682. geom_abline(slope = 0, intercept =100, linetype=3) +
  683. scale_y_continuous(limits = c(0,100),name = 'Performance value') +
  684. labs(x = '% of studies read', colour = "Metric") +
  685. scale_color_manual(labels = c("% of includes identified", "Iteration Recall" ,"Joint recall", "Specificity"), values= c('red', 'blue', 'green', 'purple'))+
  686. scale_x_continuous(expand=c(0, 0)) +
  687. geom_vline(xintercept = 28.2030620, linetype = 2, colour = 'orange')+
  688. geom_vline(xintercept = 22.9653505, linetype = 3, colour = 'gray20')+
  689. theme_classic()
  691. #ggsave('Hypertension.png', width = 6, height = 3, dpi = 300)
  693. summary(sumtextPICOS$aucpr)
  694. ## Histograms ####
  696. ggplot(data = subset(results, new== 1| new == 35 | new == 70), aes(x = (Include))) +
  697. geom_histogram( aes(fill = finaldecision,y = after_stat(density))) +
  698. scale_fill_manual(values = c('black', 'red')) +
  699. geom_vline(aes(xintercept = thresh), colour = 'blue', linetype = 2) +
  700. facet_wrap(~new ,ncol = 3, scale = 'free')+
  701. labs(x = "Predicted Probability ", y = "Probability density",fill = 'Known decision') +
  702. theme_classic() +
  703. theme(legend.position = "top") + scale_x_continuous(limits = c(0,1))
  705. #ggsave('hypertension distrib.png', width = 10, height = 3.3, dpi = 300)
  707. probs1 <- results%>%
  708. arrange((newpred)) %>%
  709. mutate(p = pmin(pmax(newpred, 1e-6), 1 - 1e-6)) %>%
  710. mutate(x = qlogis(p))
  712. ggplot(subset(probs1, new == 1), aes(x, p)) +
  713. stat_function(fun = plogis, xlim = range(probs1$x), size = 0.5, color = "gray", linetype = 2) +
  714. geom_point(pch = 21, alpha = 1, size = 1.5, aes(fill = finaldecision)) +
  715. geom_point(data = subset(probs1, finaldecision == "Include" & new == 1),pch = 21, alpha = 1, size = 1.5, fill = 'red' ) +
  716. scale_fill_manual(values = c('black', 'red')) +
  717. geom_vline(xintercept = qlogis(0.4), linetype = 2) +
  718. coord_cartesian(ylim = c(0, 1)) +
  719. theme_minimal(base_size = 14) +
  720. labs(x = "logit(probability)", y = "Probability", fill = 'Known decisions') +
  721. theme(legend.position = "top") +
  722. facet_wrap(~new)
  724. ggplot(data = subset(results, new > 10 & new < 21), aes(x = (newpred))) +
  725. geom_histogram( aes(fill = finaldecision,y = after_stat(density))) +
  726. scale_fill_manual(values = c('black', 'red')) +
  727. geom_vline(aes(xintercept = thresh), colour = 'blue', linetype = 2) +
  728. facet_wrap(~new* configs, ncol = 2, scale = 'free')+
  729. labs(x = "Predicted Probability ", y = "Probability density",fill = 'Known decision') +
  730. theme_classic() +
  731. theme(legend.position = "top")
  733. ggplot(data = subset(results, new > 20 & new < 31), aes(x = (newpred))) +
  734. geom_histogram( aes(fill = finaldecision,y = after_stat(density))) +
  735. scale_fill_manual(values = c('black', 'red')) +
  736. geom_vline(aes(xintercept = thresh), colour = 'blue', linetype = 2) +
  737. facet_wrap(~new* configs, ncol = 2, scale = 'free')+
  738. labs(x = "Predicted Probability ", y = "Probability density",fill = 'Known decision') +
  739. theme_classic() +
  740. theme(legend.position = "top")
  742. ggplot(data = subset(results, new > 30 & new < 41), aes(x = (newpred))) +
  743. geom_histogram( aes(fill = finaldecision,y = after_stat(density))) +
  744. scale_fill_manual(values = c('black', 'red')) +
  745. geom_vline(aes(xintercept = thresh), colour = 'blue', linetype = 2) +
  746. facet_wrap(~new* configs, ncol = 2, scale = 'free')+
  747. labs(x = "Predicted Probability ", y = "Probability density",fill = 'Known decision') +
  748. theme_classic() +
  749. theme(legend.position = "top")
  751. ggplot(data = subset(results, new > 40 & new < 51), aes(x = (Include))) +
  752. geom_histogram( aes(fill = finaldecision,y = after_stat(density))) +
  753. scale_fill_manual(values = c('black', 'red')) +
  754. geom_vline(aes(xintercept = thresh), colour = 'blue', linetype = 2) +
  755. facet_wrap(~new* configs, ncol = 2, scale = 'free')+
  756. labs(x = "Predicted Probability ", y = "Probability density",fill = 'Known decision') +
  757. theme_classic() +
  758. theme(legend.position = "top")
  760. baked_df %>%
  761. mutate(scorecat = ifelse(totalscore*5 <3.5, 'low', 'high'))%>%
  762. group_by( finaldecision) %>%
  763. #filter(totalscore *5 >= 3.5) %>%
  764. summarise(min(totalscore * 5), max(totalscore*5))

Hypertension h2o.R at commit d69656c, no license · at the source

Overview

Authors: G. H. C. Barreto1,2, C. Burke1,2,3, P Davies1,2, M. Halicka1,2, C. Paterson1,2, P. Swinton4, B. Saunders5, J.P.T. Higgins1,2,6
ORCID iDs: G. H. C. Barreto, C. Burke, P Davies, M. Halicka, C. Paterson, P. Swinton, B. Saunders, J.P.T. Higgins
  1. Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
  2. NIHR Bristol Evidence Synthesis Group, University of Bristol, Bristol, UK
  3. Department of Psychology, University of Bath, Bath, UK
  4. School of Health, Robert Gordon University, Aberdeen, UK
  5. Applied Physiology and Nutrition Research Group – School of Physical Education and Sport and Faculdade de Medicina FMUSP, Universidade de São Paulo, São Paulo, Brazil
  6. NIHR Applied Research Collaboration West (ARC West) at University Hospitals Bristol and Weston NHS Foundation Trust, Bristol, UK
Institutions: University of Bristol (United Kingdom); University of Bath (United Kingdom); Robert Gordon University (United Kingdom); Universidade de São Paulo (Brazil); NIHR Applied Research Collaboration West (United Kingdom); NIHR Applied Research Collaboration West (ARC West) at University Hospitals Bristol and Weston NHS Foundation Trust, Bristol, UK
Journal: medRxiv (preprint)
Dates: published online 9 April 2026
Type: Preprint
License: CC BY-NC-ND
Identifiers: DOI 10.64898/2026.04.08.26350384 · OpenAlex W7152543262
Open access: green, a free copy (OpenAlex)
Status: code verified
Categories: clinical / translational (subfield)
Methods: Connectivity, Smoothing, state filtering, decompositions, Machine learning, Statistics
Topic: Artificial Intelligence in Healthcare and Education (Health Informatics, Medicine), according to OpenAlex
Funding: National Institute for Health Research (NIHR) (NIHR153861, NIHR203807, NIHR168894)
Citations: not cited yet (Europe PMC); 34 references in the paper

Abstract

The abstract is not reproduced here: the paper's license (CC BY-NC-ND) does not allow it. Read it in the paper, at the publisher or on Europe PMC.

Repository

Its files are read in the Code ↔ Paper reader above, with 4 matches between paragraphs and lines of code.

gabsbarreto/JARVIS-R

License: none: the authors keep all their rights
State: the link answers, verified on 29 September 2026
Evidence: files inventoried
Commit: d69656c9d5f763d24d4400d1be41eb74bf73a1e8, 28 July 2026
Languages: R (8)
Size: 18 files, 8 scripts
Software Heritage: not archived
Found in: the text, “Evaluation using archived systematic review data”
Holds: README, CITATION.cff
Not found: license file, environment file, tests, continuous integration, documentation
Tools: ggpubr (8 files), tidyverse (8 files)
Availability: 1 check, the latest on 29 September 2026: the link answers
  • 29 September 2026: the link answers
9 files
At the source: github.com/gabsbarreto/JARVIS-R

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 8 scripts, each with its path and the digest of its content;
  • 4 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 29 September 2026: the first record

Recorded: type, journal, dates, 8 authors, 1 funder, 25 references.

Cite

This paper

Barreto, G. H. C., Burke, C., Davies, P., Halicka, M., Paterson, C., Swinton, P., Saunders, B., & Higgins, J. (2026). JARVIS, should this study be selected for full-text screening? Performance of a Joint AI-ReViewer Interactive Screening tool for systematic reviews. medRxiv (preprint). https://doi.org/10.64898/2026.04.08.26350384

BibTeX

@article{barreto2026jarvis,
author = {Barreto, G. H. C. and Burke, C. and Davies, P and Halicka, M. and Paterson, C. and Swinton, P. and Saunders, B. and Higgins, J.P.T.},
title = {{JARVIS, should this study be selected for full-text screening? Performance of a Joint AI-ReViewer Interactive Screening tool for systematic reviews}},
journal = {medRxiv (preprint)},
year = {2026},
month = apr,
publisher = {medRxiv},
doi = {10.64898/2026.04.08.26350384},
url = {https://doi.org/10.64898/2026.04.08.26350384}
}

RIS

TY - JOUR
AU - Barreto, G. H. C.
AU - Burke, C.
AU - Davies, P
AU - Halicka, M.
AU - Paterson, C.
AU - Swinton, P.
AU - Saunders, B.
AU - Higgins, J.P.T.
TI - JARVIS, should this study be selected for full-text screening? Performance of a Joint AI-ReViewer Interactive Screening tool for systematic reviews
T2 - medRxiv (preprint)
J2 - medRxiv
PY - 2026
DA - 2026/04/09
PB - medRxiv
DO - 10.64898/2026.04.08.26350384
UR - https://doi.org/10.64898/2026.04.08.26350384
ER -

CSL-JSON

{
"id": "10.64898/2026.04.08.26350384",
"type": "article",
"title": "JARVIS, should this study be selected for full-text screening? Performance of a Joint AI-ReViewer Interactive Screening tool for systematic reviews",
"container-title": "medRxiv (preprint)",
"author": [
{
"family": "Barreto",
"given": "G. H. C."
},
{
"family": "Burke",
"given": "C."
},
{
"family": "Davies",
"given": "P"
},
{
"family": "Halicka",
"given": "M."
},
{
"family": "Paterson",
"given": "C."
},
{
"family": "Swinton",
"given": "P."
},
{
"family": "Saunders",
"given": "B."
},
{
"family": "Higgins",
"given": "J.P.T."
}
],
"container-title-short": "medRxiv",
"DOI": "10.64898/2026.04.08.26350384",
"publisher": "medRxiv",
"URL": "https://doi.org/10.64898/2026.04.08.26350384",
"issued": {
"date-parts": [
[
2026,
4,
9
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1128/msystems.00416-26 [code]
Integrative multicohort analysis reveals consistent sex differences in gut microbiota of multiple sclerosis patients.
Journal: mSystems
In common: ggpubr, tidyverse, clinical / translational, 1 reference
[2] doi:10.1093/geront/gnaf277 [code]
What characterizes the exceptional cognition of superagers? A systematic review of multidomain biomarkers of successful cognitive aging.
Journal: The Gerontologist
In common: ggpubr, tidyverse, clinical / translational, 1 reference
[3] doi:10.1007/s00415-026-14028-0 [code]
Personality change after traumatic brain injury: a systematic review and meta-analysis.
Journal: Journal of neurology
In common: tidyverse, clinical / translational, 1 reference
[4] doi:10.1093/aje/kwag135 [code]
Depressive symptoms and neuroimaging markers of brain aging in an ethno-racially diverse sample: a Bayesian analysis.
Journal: American journal of epidemiology
In common: ggpubr, tidyverse, clinical / translational
[5] doi:10.1093/brain/awag039 [code]
Mapping the causal chain from genetic risk variants to lipid dysmetabolism in Parkinson's disease.
Journal: Brain : a journal of neurology
In common: ggpubr, tidyverse, clinical / translational
[6] doi:10.1016/j.isci.2026.116517 [code]
Spatially resolved transcriptomics in human brain metastases identifies macrophage-tumor interactions associated with survival.
Journal: iScience
In common: ggpubr, tidyverse, clinical / translational
[7] doi:10.1126/sciadv.aee2305 [code]
Prediction of mild cognitive impairment progression using time-sensitive multimodal biomarkers.
Journal: Science advances
In common: ggpubr, tidyverse, clinical / translational
[8] doi:10.3389/fnins.2026.1858005 [code]
Targeted stool metabolomics suggests exploratory catecholamine- and tryptophan-linked metabolic features in autism spectrum disorder.
Journal: Frontiers in neuroscience
In common: ggpubr, tidyverse, clinical / translational
[9] doi:10.1038/s41380-026-03694-1 [code]
Targeting cortico-striatal-amygdalar networks via theta-band frontoparietal synchronization in opioid use disorder: a randomized tACS-fMRI Trial.
Journal: Molecular psychiatry
In common: ggpubr, tidyverse, clinical / translational
[10] doi:10.1186/s13073-026-01698-8 [code]
From aging to Alzheimer's disease: concordant brain DNA methylation changes in late life.
Journal: Genome medicine
In common: ggpubr, tidyverse, clinical / translational

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.