OSCR

Benchmarking a Local Schema-Constrained Large Language Model Pipeline for Abstract Screening and Evidence Mapping.

Code ↔ Paper

2 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 2 matches
  1. [1] § Materials and methods ↔ scripts/pubmed1_screen.py, lines 1–31 · score 0.62 · gpt oss, logs, Ollama, Max, append, JSONL
  2. [2] § Materials and methods ↔ scripts/pubmed_prisma.py, lines 99–138 · score 0.59 · reason_code, PubMed, PRISMA, ineligible, workflow, human

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 628 lines · 21 KB · MIT · 1 match

  1. #!/usr/bin/env python3
  2. """LLM-assisted title/abstract screening for remaining 'maybe' records.
  4. This script:
  5. - Reads all runs/<query_id>/records.json (produced by pubmed1.py)
  6. - Builds the post-automation screening universe (unique PMIDs)
  7. - Reads runs/decisions.jsonl (append-only audit log)
  8. - Selects PMIDs whose latest decision at a stage is missing or 'maybe'
  9. - Calls Ollama locally (ollama run <model>) to classify each record
  10. - Appends new decision lines to decisions.jsonl with by='LLM'
  12. Design goals: simple, local, reproducible, low-hallucination.
  14. Usage:
  15. python3 pubmed1_screen.py
  16. python3 pubmed1_screen.py runs --max 20 --dry-run
  17. python3 pubmed1_screen.py runs --model gpt-oss:20b
  18. """
  20. from __future__ import annotations
  22. import argparse
  23. import json
  24. import re
  25. import subprocess
  26. import requests
  27. import ast
  28. import textwrap
  29. from datetime import datetime, timezone
  30. from pathlib import Path
  31. from typing import Any, Dict, Iterable, List, Optional, Set, Tuple
  34. # Keep these aligned with pubmed1prisma.py automation defaults
  35. DEFAULT_AUTOMATION_INELIGIBLE_PUBTYPES = {
  36. "editorial",
  37. "letter",
  38. "comment",
  39. "news",
  40. "interview",
  41. "published erratum",
  42. "retracted publication",
  43. "retraction of publication",
  44. }
  46. # Screening reason codes: keep small and PRISMA-friendly.
  47. EXCLUSION_REASON_CODES = {
  48. "NOT_RELEVANT",
  49. "WRONG_DESIGN",
  50. "WRONG_POPULATION",
  51. "WRONG_OUTCOME",
  52. "WRONG_INTERVENTION",
  53. "NO_ABSTRACT",
  54. "LANGUAGE",
  55. "OTHER",
  56. }
  57. INCLUSION_REASON_CODES = {"ELIGIBLE", "POTENTIALLY_ELIGIBLE"}
  60. def _utc_now_iso() -> str:
  61. return datetime.now(timezone.utc).replace(microsecond=0).isoformat()
  64. def _norm_str(x: Any) -> str:
  65. return str(x).strip().lower()
  67. def _read_criteria_text(criteria_path: Path) -> str:
  68. """Read criteria text from a file and return it as plain text.
  70. This is the authoritative criteria block inserted into the LLM prompt.
  71. """
  72. if not criteria_path.exists():
  73. raise FileNotFoundError(f"Criteria file not found: {criteria_path}")
  74. txt = criteria_path.read_text(encoding="utf-8").strip()
  75. if not txt:
  76. raise ValueError(f"Criteria file is empty: {criteria_path}")
  77. return txt + "\n"
  79. def _load_records_json(path: Path) -> List[dict[str, Any]]:
  80. with path.open("r", encoding="utf-8") as f:
  81. data = json.load(f)
  82. if not isinstance(data, dict):
  83. raise ValueError(f"{path}: expected a JSON object")
  84. records = data.get("records")
  85. if not isinstance(records, list):
  86. raise ValueError(f"{path}: missing or invalid 'records' list")
  87. return [r for r in records if isinstance(r, dict)]
  90. def _find_records_files(runs_dir: Path) -> List[Path]:
  91. if not runs_dir.exists():
  92. raise FileNotFoundError(f"Runs directory not found: {runs_dir}")
  93. if not runs_dir.is_dir():
  94. raise NotADirectoryError(f"Not a directory: {runs_dir}")
  95. files = sorted(runs_dir.rglob("records.json"))
  96. if not files:
  97. raise FileNotFoundError(
  98. f"No records.json found under {runs_dir}. Expected runs/<query_id>/records.json"
  99. )
  100. return files
  103. def _iter_records(runs_dir: Path) -> Iterable[dict[str, Any]]:
  104. for fp in _find_records_files(runs_dir):
  105. for r in _load_records_json(fp):
  106. yield r
  109. def _automation_ineligible(r: dict[str, Any], *, exclude_nonhuman: bool = True, exclude_pubtypes: bool = True) -> bool:
  110. if exclude_nonhuman:
  111. is_humans = r.get("is_humans")
  112. if is_humans is False:
  113. return True
  115. if exclude_pubtypes:
  116. pts = r.get("publication_types") or []
  117. pts_norm = {_norm_str(p) for p in pts}
  118. if pts_norm & DEFAULT_AUTOMATION_INELIGIBLE_PUBTYPES:
  119. return True
  121. return False
  124. def _read_decisions_jsonl(path: Path) -> List[dict[str, Any]]:
  125. if not path.exists():
  126. return []
  127. out: List[dict[str, Any]] = []
  128. with path.open("r", encoding="utf-8") as f:
  129. for line_no, line in enumerate(f, start=1):
  130. line = line.strip()
  131. if not line:
  132. continue
  133. try:
  134. obj = json.loads(line)
  135. except json.JSONDecodeError:
  136. raise ValueError(f"{path}:{line_no}: invalid JSON (not JSONL?)")
  137. if isinstance(obj, dict):
  138. out.append(obj)
  139. return out
  142. def _latest_decision_by_pmid_stage(decisions: List[dict[str, Any]]) -> dict[Tuple[str, str], dict[str, Any]]:
  143. latest: dict[Tuple[str, str], dict[str, Any]] = {}
  144. latest_ts: dict[Tuple[str, str], datetime] = {}
  146. for obj in decisions:
  147. pmid = obj.get("pmid")
  148. stage = obj.get("stage")
  149. if not pmid or not stage:
  150. continue
  151. key = (str(pmid), str(stage))
  153. ts_raw = obj.get("timestamp")
  154. ts = None
  155. if isinstance(ts_raw, str):
  156. try:
  157. ts = datetime.fromisoformat(ts_raw.replace("Z", "+00:00"))
  158. except Exception:
  159. ts = None
  161. if ts is None:
  162. latest[key] = obj
  163. continue
  165. prev = latest_ts.get(key)
  166. if prev is None or ts >= prev:
  167. latest_ts[key] = ts
  168. latest[key] = obj
  170. return latest
  174. def _ensure_trailing_newline(path: Path) -> None:
  175. """Ensure file ends with newline so JSONL append can't corrupt the last line."""
  176. if not path.exists() or path.stat().st_size == 0:
  177. return
  178. with path.open("rb") as fb:
  179. fb.seek(-1, 2)
  180. last = fb.read(1)
  181. if last != b"\n":
  182. with path.open("a", encoding="utf-8") as f:
  183. f.write("\n")
  186. # Helper: Write raw model output for debugging failed parses
  187. def _write_debug_raw(debug_dir: Path, pmid: str, tag: str, content: str) -> None:
  188. debug_dir.mkdir(parents=True, exist_ok=True)
  189. safe_pmid = re.sub(r"[^0-9A-Za-z._-]+", "_", pmid)
  190. ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
  191. out = debug_dir / f"{safe_pmid}.{tag}.{ts}.txt"
  192. out.write_text(content or "", encoding="utf-8")
  195. def _extract_first_json_object(text: str) -> Optional[dict[str, Any]]:
  196. """Extract the first JSON object from model output.
  198. We ask for JSON-only output, but models sometimes wrap it in prose, code fences,
  199. or emit Python dicts / trailing commas. We keep fixes conservative.
  200. """
  201. if not text:
  202. return None
  204. s = text.strip()
  206. # Fast path: whole output is a JSON object
  207. if s.startswith("{") and s.endswith("}"):
  208. try:
  209. obj = json.loads(s)
  210. return obj if isinstance(obj, dict) else None
  211. except Exception:
  212. pass
  214. # Find first balanced {...} block (more reliable than greedy regex)
  215. start = s.find("{")
  216. if start < 0:
  217. return None
  218. depth = 0
  219. end = None
  220. for i, ch in enumerate(s[start:], start=start):
  221. if ch == "{":
  222. depth += 1
  223. elif ch == "}":
  224. depth -= 1
  225. if depth == 0:
  226. end = i + 1
  227. break
  228. if end is None:
  229. return None
  231. chunk = s[start:end].strip()
  233. # Attempt strict JSON
  234. try:
  235. obj = json.loads(chunk)
  236. return obj if isinstance(obj, dict) else None
  237. except Exception:
  238. pass
  240. # Conservative normalization
  241. norm = chunk
  242. norm = norm.replace("\u201c", '"').replace("\u201d", '"').replace("\u2018", "'").replace("\u2019", "'")
  243. # Remove trailing commas before } or ]
  244. norm = re.sub(r",\s*(\}|\])", r"\1", norm)
  246. # Quote common bareword enum mistakes (keep scope narrow)
  247. # e.g. "reason_code": WRONG_INTERVENTION -> "reason_code": "WRONG_INTERVENTION"
  248. norm = re.sub(
  249. r'("reason_code"\s*:\s*)([A-Za-z_][A-Za-z0-9_]*)(\s*[\},])',
  250. r'\1"\2"\3',
  251. norm,
  252. )
  253. norm = re.sub(
  254. r'("decision"\s*:\s*)(include|exclude|maybe)(\s*[\},])',
  255. r'\1"\2"\3',
  256. norm,
  257. flags=re.IGNORECASE,
  258. )
  259. norm = re.sub(
  260. r'("confidence"\s*:\s*)(low|medium|high)(\s*[\},])',
  261. r'\1"\2"\3',
  262. norm,
  263. flags=re.IGNORECASE,
  264. )
  266. try:
  267. obj = json.loads(norm)
  268. return obj if isinstance(obj, dict) else None
  269. except Exception:
  270. pass
  272. # Last resort: model may have emitted a Python dict
  273. try:
  274. obj2 = ast.literal_eval(norm)
  275. return obj2 if isinstance(obj2, dict) else None
  276. except Exception:
  277. return None
  280. def _build_prompt(record: dict[str, Any], criteria_text: str) -> str:
  281. pmid = str(record.get("pmid") or "")
  282. title = (record.get("title") or "").strip()
  283. abstract = (record.get("abstract") or "").strip()
  284. pubtypes = record.get("publication_types") or []
  285. languages = record.get("languages") or []
  286. is_humans = record.get("is_humans")
  288. return (
  289. "You are screening PubMed records for a systematic review with STRICT eligibility criteria.\n"
  290. "Use ONLY the provided Title/Abstract/metadata and the AUTHORITATIVE CRITERIA below.\n"
  291. "Do NOT infer missing details.\n"
  292. "If key information is missing, set decision=\"maybe\" and reason_code=\"UNCLEAR\".\n\n"
  293. "AUTHORITATIVE SCREENING CRITERIA (apply strictly):\n"
  294. + criteria_text
  295. + "\n"
  296. "Return EXACTLY one JSON object (no markdown, no commentary) with keys:\n"
  297. "Output MUST start with '{' and end with '}' and contain nothing else (single JSON object only).\n"
  298. "All string values MUST be in double quotes, including reason_code (e.g., \"reason_code\": \"WRONG_INTERVENTION\").\n"
  299. "pmid, decision, reason_code, confidence, notes\n"
  300. "Where decision is one of: include, exclude, maybe.\n"
  301. "reason_code MUST be consistent with decision:\n"
  302. "- if decision=exclude: choose one of NOT_RELEVANT, WRONG_DESIGN, WRONG_POPULATION, WRONG_OUTCOME, WRONG_INTERVENTION, NO_ABSTRACT, LANGUAGE, OTHER\n"
  303. "- if decision=include: use ELIGIBLE\n"
  304. "- if decision=maybe: use POTENTIALLY_ELIGIBLE or UNCLEAR\n"
  305. "confidence: low|medium|high\n"
  306. "notes: <= 20 words, strictly quoting or paraphrasing title/abstract (no speculation).\n\n"
  307. "Decision rules:\n"
  308. "- INCLUDE only if ALL eligibility criteria are explicitly satisfied.\n"
  309. "- EXCLUDE with WRONG_DESIGN if review/meta/protocol/editorial.\n"
  310. "- If uncertain due to missing details, choose MAYBE with UNCLEAR.\n\n"
  311. f"PMID: {pmid}\n"
  312. f"PublicationTypes: {pubtypes}\n"
  313. f"Languages: {languages}\n"
  314. f"HumansFlag: {is_humans}\n\n"
  315. f"Title: {title}\n\n"
  316. f"Abstract: {abstract if abstract else 'NO ABSTRACT'}\n"
  317. )
  321. def _call_ollama_http(model: str, prompt: str, timeout_s: int = 180) -> str:
  322. """Call Ollama via local HTTP API and return the generated text.
  324. Uses POST http://localhost:11434/api/generate
  325. """
  326. url = "http://localhost:11434/api/generate"
  327. payload: Dict[str, Any] = {
  328. "model": model,
  329. "prompt": prompt,
  330. "stream": False,
  331. }
  333. try:
  334. r = requests.post(url, json=payload, timeout=timeout_s)
  335. except requests.RequestException as e:
  336. raise RuntimeError(f"Ollama HTTP request failed: {e}")
  338. if r.status_code != 200:
  339. # Ollama typically returns JSON with an 'error' field on failures.
  340. try:
  341. j = r.json()
  342. err = j.get("error") if isinstance(j, dict) else None
  343. except Exception:
  344. err = None
  345. msg = err or (r.text or "").strip() or f"HTTP {r.status_code}"
  346. raise RuntimeError(f"Ollama HTTP error: {msg}")
  348. try:
  349. data = r.json()
  350. except Exception:
  351. raise RuntimeError("Ollama HTTP response was not valid JSON")
  353. if not isinstance(data, dict):
  354. raise RuntimeError("Ollama HTTP response JSON had unexpected shape")
  356. out = data.get("response")
  357. if not isinstance(out, str):
  358. raise RuntimeError("Ollama HTTP response missing 'response' text")
  359. return out
  362. def _call_ollama_subprocess(model: str, prompt: str, timeout_s: int = 180) -> str:
  363. """Call Ollama via CLI (subprocess) and return raw stdout.
  365. We pass the prompt as a CLI argument (not stdin) to avoid interactive/stdin quirks.
  366. """
  367. proc = subprocess.run(
  368. ["ollama", "run", model, prompt],
  369. text=True,
  370. capture_output=True,
  371. timeout=timeout_s,
  372. )
  373. if proc.returncode != 0:
  374. err = (proc.stderr or "").strip()
  375. raise RuntimeError(f"ollama run failed (code {proc.returncode}): {err}")
  376. return proc.stdout
  379. def _call_ollama(model: str, prompt: str, timeout_s: int = 180, transport: str = "http") -> str:
  380. """Call Ollama using the selected transport ('http' or 'subprocess')."""
  381. t = (transport or "http").strip().lower()
  382. if t == "subprocess":
  383. return _call_ollama_subprocess(model, prompt, timeout_s=timeout_s)
  384. return _call_ollama_http(model, prompt, timeout_s=timeout_s)
  387. def _make_decision_line(pmid: str, stage: str, obj: dict[str, Any], by: str) -> Dict[str, Any]:
  388. decision = str(obj.get("decision") or "maybe").lower().strip()
  389. if decision not in {"include", "exclude", "maybe"}:
  390. decision = "maybe"
  392. rc = obj.get("reason_code")
  393. reason_code = str(rc).strip().upper() if rc is not None and str(rc).strip() else None
  395. # Normalize confidence
  396. conf = str(obj.get("confidence") or "").lower().strip()
  397. if conf not in {"low", "medium", "high"}:
  398. conf = None
  400. # Enforce decision/reason consistency (guard against model nonsense)
  401. if decision == "include":
  402. # If model provided an exclusion reason, flip to exclude.
  403. if reason_code in EXCLUSION_REASON_CODES or reason_code == "UNCLEAR":
  404. decision = "exclude"
  405. else:
  406. reason_code = "ELIGIBLE"
  408. if decision == "exclude":
  409. if reason_code not in EXCLUSION_REASON_CODES:
  410. # Default to OTHER if missing or invalid
  411. reason_code = "OTHER"
  413. if decision == "maybe":
  414. if reason_code is None:
  415. reason_code = "UNCLEAR"
  416. elif reason_code in EXCLUSION_REASON_CODES:
  417. # If they gave a clear exclusion reason, treat it as exclude.
  418. decision = "exclude"
  419. elif reason_code not in {"POTENTIALLY_ELIGIBLE", "UNCLEAR"}:
  420. reason_code = "UNCLEAR"
  422. # If the model outputs a different PMID, ignore it (we trust our input PMID).
  423. _pmid_out = obj.get("pmid")
  424. if _pmid_out and str(_pmid_out).strip() != pmid:
  425. # keep going; do not overwrite
  426. pass
  428. # Keep optional short notes
  429. notes = obj.get("notes")
  430. if notes is not None:
  431. notes = str(notes).strip()
  432. if len(notes) > 200:
  433. notes = notes[:200]
  435. line: Dict[str, Any] = {
  436. "pmid": pmid,
  437. "stage": stage,
  438. "decision": decision,
  439. "reason_code": reason_code,
  440. "by": by,
  441. "confidence": conf,
  442. "timestamp": _utc_now_iso(),
  443. }
  444. if notes:
  445. line["notes"] = notes
  446. return line
  449. def main() -> int:
  450. ap = argparse.ArgumentParser(description="LLM-assisted screening for remaining 'maybe' records")
  451. ap.add_argument(
  452. "runs_dir",
  453. nargs="?",
  454. default="runs",
  455. help="Path to runs/ folder (default: ./runs)",
  456. )
  457. ap.add_argument(
  458. "--decisions",
  459. default=None,
  460. help="Decisions JSONL path (default: <runs_dir>/decisions.jsonl)",
  461. )
  462. ap.add_argument(
  463. "--stage",
  464. default="abstract",
  465. choices=["abstract", "fulltext"],
  466. help="Decision stage to write (default: abstract)",
  467. )
  468. ap.add_argument(
  469. "--criteria",
  470. default="criteriaImpuls.txt",
  471. help="Path to criteria text file to inject into the prompt (default: criteriaImpuls.txt)",
  472. )
  473. ap.add_argument(
  474. "--model",
  475. default="gpt-oss:20b",
  476. help="Ollama model name (default: gpt-oss:20b)",
  477. )
  478. ap.add_argument(
  479. "--transport",
  480. default="http",
  481. choices=["http", "subprocess"],
  482. help="How to call Ollama (default: http). Use 'subprocess' for CLI fallback.",
  483. )
  484. ap.add_argument(
  485. "--by",
  486. default="LLM",
  487. help="Value for the 'by' field (default: LLM)",
  488. )
  489. ap.add_argument(
  490. "--max",
  491. type=int,
  492. default=0,
  493. help="Max number of PMIDs to process (0 = no limit)",
  494. )
  495. ap.add_argument(
  496. "--dry-run",
  497. action="store_true",
  498. default=False,
  499. help="Do not write anything; just show what would be processed",
  500. )
  501. ap.add_argument(
  502. "--timeout",
  503. type=int,
  504. default=180,
  505. help="Timeout seconds per record (default: 180)",
  506. )
  507. ap.add_argument(
  508. "--debug-raw-dir",
  509. default=None,
  510. help="If set, write raw LLM outputs for any parse failures to this directory (default: <runs_dir>/llm_raw)",
  511. )
  512. args = ap.parse_args()
  514. runs_dir = Path(args.runs_dir).expanduser().resolve()
  515. decisions_path = (
  516. Path(args.decisions).expanduser().resolve()
  517. if args.decisions
  518. else (runs_dir / "decisions.jsonl").resolve()
  519. )
  521. # Resolve criteria path (relative paths: script dir first, then current working dir)
  522. criteria_arg = Path(args.criteria).expanduser()
  523. if criteria_arg.is_absolute():
  524. criteria_path = criteria_arg
  525. else:
  526. script_dir = Path(__file__).resolve().parent
  527. candidate1 = (script_dir / criteria_arg).resolve()
  528. candidate2 = (Path.cwd() / criteria_arg).resolve()
  529. criteria_path = candidate1 if candidate1.exists() else candidate2
  531. criteria_text = _read_criteria_text(criteria_path)
  533. debug_raw_dir = (
  534. Path(args.debug_raw_dir).expanduser().resolve()
  535. if args.debug_raw_dir
  536. else (runs_dir / "llm_raw").resolve()
  537. )
  539. # Build screening universe (unique PMIDs post automation) and keep one representative record per PMID
  540. pmid_to_record: dict[str, dict[str, Any]] = {}
  541. for r in _iter_records(runs_dir):
  542. pmid = r.get("pmid")
  543. if not pmid:
  544. continue
  545. pmid_s = str(pmid)
  546. if pmid_s in pmid_to_record:
  547. continue
  548. if _automation_ineligible(r):
  549. continue
  550. pmid_to_record[pmid_s] = r
  552. decisions = _read_decisions_jsonl(decisions_path)
  553. latest = _latest_decision_by_pmid_stage(decisions)
  555. # Select candidates: missing or maybe
  556. candidates: List[str] = []
  557. for pmid in sorted(pmid_to_record.keys()):
  558. obj = latest.get((pmid, args.stage))
  559. current_dec = str((obj or {}).get("decision") or "missing").lower()
  560. if current_dec in {"missing", "maybe"}:
  561. candidates.append(pmid)
  563. if args.max and args.max > 0:
  564. candidates = candidates[: args.max]
  566. print(f"Runs dir: {runs_dir}")
  567. print(f"Decisions file: {decisions_path}")
  568. print(f"Screening universe (unique PMIDs, post-automation): {len(pmid_to_record)}")
  569. print(f"Candidates to send to LLM (missing/maybe): {len(candidates)}")
  570. print(f"Model: {args.model}")
  571. print(f"Transport: {args.transport}")
  572. print(f"Criteria file: {criteria_path}")
  574. if args.dry_run:
  575. if candidates:
  576. print("DRY-RUN PMIDs:")
  577. for pmid in candidates:
  578. print(f" - {pmid}")
  579. return 0
  581. if not candidates:
  582. print("Nothing to do.")
  583. return 0
  585. decisions_path.parent.mkdir(parents=True, exist_ok=True)
  586. _ensure_trailing_newline(decisions_path)
  588. appended = 0
  589. errors = 0
  590. for pmid in candidates:
  591. r = pmid_to_record[pmid]
  592. prompt = _build_prompt(r, criteria_text)
  593. try:
  594. raw = _call_ollama(args.model, prompt, timeout_s=args.timeout, transport=args.transport)
  595. obj = _extract_first_json_object(raw)
  596. raw2 = ""
  597. if not obj:
  598. # One retry with a terse "repair" instruction
  599. repair = (
  600. prompt
  601. + "\n\nIMPORTANT: Your previous output was not valid JSON. "
  602. + "Return ONLY one valid JSON object, starting with '{' and ending with '}', no extra text."
  603. )
  604. raw2 = _call_ollama(args.model, repair, timeout_s=args.timeout, transport=args.transport)
  605. obj = _extract_first_json_object(raw2)
  606. if not obj:
  607. _write_debug_raw(debug_raw_dir, pmid, "raw", raw)
  608. if raw2:
  609. _write_debug_raw(debug_raw_dir, pmid, "raw_retry", raw2)
  610. raise ValueError(
  611. f"Could not parse JSON from model output (saved raw to {debug_raw_dir})"
  612. )
  613. line = _make_decision_line(pmid, args.stage, obj, args.by)
  614. with decisions_path.open("a", encoding="utf-8") as f:
  615. f.write(json.dumps(line, ensure_ascii=False) + "\n")
  616. appended += 1
  617. print(f"OK {pmid}: {line['decision']} ({line.get('reason_code')})")
  618. except Exception as e:
  619. errors += 1
  620. print(f"ERROR {pmid}: {e}")
  621. print(" Hint: try running this PMID with --max 1 and/or increase --timeout; also ensure the model outputs JSON.")
  623. print(f"DONE. Appended: {appended}. Errors: {errors}.")
  624. return 0
  627. if __name__ == "__main__":
  628. raise SystemExit(main())

pubmed1_screen.py at commit b9e00a7, under MIT · at the source

Overview

Authors: Alessandro Serretti1,2
  1. Psychiatry, Università degli Studi di Enna Kore, Enna, ITA
  2. Psychiatry, Oasi Research Institute-IRCCS, Troina, ITA
Institutions: Università degli Studi di Enna Kore (Italy); I.R.C.C.S. Oasi Maria SS (Italy)
Journal: Cureus, volume 18, issue 6, article e111193
Dates: accepted 20 June 2026; published online 20 June 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.7759/cureus.111193 · PMID 42483140 · PMCID PMC13384908 · OpenAlex W7165359315
Open access: diamond, a free copy (OpenAlex)
Status: code verified
Categories: human (organism)
Methods: fMRI & imaging, Machine learning
Keywords: brain imaging, large language models, mapping evidence, psychiatry, scoping review
Journal subjects: Psychiatry
Topic: Topic Modeling (Artificial Intelligence, Computer Science), according to OpenAlex
Citations: not cited yet (Europe PMC); 26 references in the paper

Abstract

Background

The growth of biomedical literature increasingly exceeds the capacity of manual evidence synthesis. Large language models (LLMs) may support abstract screening and structured extraction, but many current workflows depend on proprietary cloud APIs, creating challenges for governance, reproducibility, and scalable deployment.

Methods

I developed a fully local, open-weight, schema-constrained pipeline (gpt-oss-20b, deployed via Ollama on Apple M1 Max) for title/abstract-based scoping workflows. The pipeline combined deterministic metadata filtering, LLM-assisted screening, and structured abstract extraction. Performance was benchmarked against three published systematic reviews (ketamine/neuroimaging; clozapine/suicidality; clozapine patient/caregiver perspectives) using precision, recall, and F1 against reference inclusion sets. I also report audit-adjusted estimates (i.e., performance metrics recalculated after manual full-text adjudication of discrepant records) alongside standard reference-set performance.

Results

In the ketamine/neuroimaging benchmark, the pipeline retained all 41 studies included in the original review; after audit adjustment, recall was 100.0% (46/46), accuracy 99.4% (156/157), precision 97.9% (46/47), and F1 98.9%. For clozapine/suicidality, recall was 79.3% (46/58), and F1 was 76.0%, with missed studies largely attributable to missing or non-informative abstracts. For clozapine patient/caregiver perspectives, recall was 88.9% (56/63), and F1 was 83.6%, with similar abstract-level constraints. Abstract-level extraction recovered audited metadata fields without detected errors and generated evidence maps that were thematically concordant with the main narrative structure of the reference reviews.

Conclusions

As a proof-of-concept, a fully local LLM pipeline can support scalable and auditable abstract-based scoping and high-level evidence mapping. Because performance was benchmarked against three reviews with partly audit-adjusted reference sets, the findings require confirmation in larger, independently adjudicated evaluations. Random human audit remains advisable, and expert full-text synthesis remains necessary when abstracts are non-informative or when mechanistic precision is required.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 2 matches between paragraphs and lines of code.

svonte/Scoping-Systematic

License: MIT
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: b9e00a784ad01794111f110b7699aadaf599fadd, 10 June 2026
Languages: Python (10)
Size: 21 files, 10 scripts
Software Heritage: not archived
Found in: the text, “Materials and methods”
Holds: README, license file, environment (requirements.txt)
Not found: CITATION.cff, tests, continuous integration, documentation
Tools: pandas (4 files)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
12 files
At the source: github.com/svonte/Scoping-Systematic

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 10 scripts, each with its path and the digest of its content;
  • 2 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 1 author, 5 keywords, 20 references.

Cite

This paper

Serretti, A. (2026). Benchmarking a Local Schema-Constrained Large Language Model Pipeline for Abstract Screening and Evidence Mapping. Cureus, 18(6), e111193. https://doi.org/10.7759/cureus.111193

BibTeX

@article{serretti2026benchmarking,
author = {Serretti, Alessandro},
title = {{Benchmarking a Local Schema-Constrained Large Language Model Pipeline for Abstract Screening and Evidence Mapping}},
journal = {Cureus},
year = {2026},
month = jun,
volume = {18},
number = {6},
pages = {e111193},
publisher = {Cureus Inc.},
issn = {2168-8184},
doi = {10.7759/cureus.111193},
url = {https://doi.org/10.7759/cureus.111193},
pmid = {42483140},
pmcid = {PMC13384908}
}

RIS

TY - JOUR
AU - Serretti, Alessandro
TI - Benchmarking a Local Schema-Constrained Large Language Model Pipeline for Abstract Screening and Evidence Mapping
T2 - Cureus
J2 - Cureus
PY - 2026
DA - 2026/06/20
VL - 18
IS - 6
SP - e111193
SN - 2168-8184
PB - Cureus Inc.
DO - 10.7759/cureus.111193
UR - https://doi.org/10.7759/cureus.111193
LA - en
ER -

CSL-JSON

{
"id": "10.7759/cureus.111193",
"type": "article-journal",
"title": "Benchmarking a Local Schema-Constrained Large Language Model Pipeline for Abstract Screening and Evidence Mapping",
"container-title": "Cureus",
"author": [
{
"family": "Serretti",
"given": "Alessandro"
}
],
"container-title-short": "Cureus",
"volume": "18",
"issue": "6",
"page": "e111193",
"DOI": "10.7759/cureus.111193",
"PMID": "42483140",
"PMCID": "PMC13384908",
"ISSN": "2168-8184",
"publisher": "Cureus Inc.",
"URL": "https://doi.org/10.7759/cureus.111193",
"language": "en",
"issued": {
"date-parts": [
[
2026,
6,
20
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s41398-026-03928-4
Modulatory effects of ketamine on EEG source-based resting state connectivity in treatment resistant depression.
Journal: Translational psychiatry
In common: 1 reference

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.