OSCR

Parvalbumin Neuron-Targeted Loss of Alzheimer's Disease Risk Gene <i>BIN1</i> Is Insufficient to Drive Cognitive or Network Excitability Changes.

Code ↔ Paper

2 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 2 matches
  1. [1] § Materials and Methods › Behavioral tests › Pose estimation ↔ keypoint_moseq/io.py, lines 46 to 169 · score 0.69 · Pose trajectories, principal component, video recorded, anterior, posterior, Keypoint MoSeq · Report a mismatch
  2. [2] § Materials and Methods › Behavioral tests › Pose estimation ↔ keypoint_moseq/util.py, lines 513 to 630 · score 0.59 · principal component, Keypoint MoSeq, PCA, trajectories, Pose, model · Report a mismatch

Paper

Loaded from Europe PMC by your browser; OSCR does not store it: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 1,428 lines · 52 KB · other · 1 match

  1. import jax.numpy as jnp
  2. import jax
  3. import re
  4. import commentjson
  5. import json
  6. import numpy as np
  7. import h5py
  8. import joblib
  9. import tqdm
  10. import yaml
  11. import os
  12. import pandas as pd
  13. from textwrap import fill
  14. import sleap_io
  15. from pynwb import NWBHDF5IO
  16. from ndx_pose import PoseEstimation
  17. from itertools import islice
  18. from keypoint_moseq.util import list_files_with_exts, check_nan_proportions
  19. from jax_moseq.utils import get_frequencies, unbatch
  20. from scipy.io import loadmat
  21. def _build_yaml(sections, comments):
  22. text_blocks = []
  23. for title, data in sections:
  24. centered_title = f" {title} ".center(50, "=")
  25. text_blocks.append(f"\n\n{'#'}{centered_title}{'#'}")
  26. for key, value in data.items():
  27. text = yaml.dump({key: value}).strip("\n")
  28. if key in comments:
  29. text = f"\n{'#'} {comments[key]}\n{text}"
  30. text_blocks.append(text)
  31. return "\n".join(text_blocks)
  32. def _get_path(project_dir, model_name, path, filename, pathname_for_error_msg="path"):
  33. if path is None:
  34. assert project_dir is not None and model_name is not None, fill(
  35. f"`model_name` and `project_dir` are required if `{pathname_for_error_msg}` is None."
  36. )
  37. path = os.path.join(project_dir, model_name, filename)
  38. return path
  39. def generate_config(project_dir, **kwargs):
  40. """Generate a `config.yml` file with project settings. Default settings
  41. will be used unless overriden by a keyword argument.
  42. Parameters
  43. ----------
  44. project_dir: str
  45. A file `config.yml` will be generated in this directory.
  46. kwargs
  47. Custom project settings.
  48. """
  49. def _update_dict(new, original):
  50. return {k: new[k] if k in new else v for k, v in original.items()}
  51. hypperams = _update_dict(
  52. kwargs,
  53. {
  54. "error_estimator": {"slope": -0.5, "intercept": 0.25},
  55. "obs_hypparams": {
  56. "sigmasq_0": 0.1,
  57. "sigmasq_C": 0.1,
  58. "nu_sigma": 1e5,
  59. "nu_s": 5,
  60. },
  61. "ar_hypparams": {
  62. "latent_dim": 10,
  63. "nlags": 3,
  64. "S_0_scale": 0.01,
  65. "K_0_scale": 10.0,
  66. },
  67. "trans_hypparams": {
  68. "num_states": 100,
  69. "gamma": 1e3,
  70. "alpha": 5.7,
  71. "kappa": 1e6,
  72. },
  73. "cen_hypparams": {"sigmasq_loc": 0.5},
  74. },
  75. )
  76. hypperams = {k: _update_dict(kwargs, v) for k, v in hypperams.items()}
  77. anatomy = _update_dict(
  78. kwargs,
  79. {
  80. "bodyparts": ["BODYPART1", "BODYPART2", "BODYPART3"],
  81. "use_bodyparts": ["BODYPART1", "BODYPART2", "BODYPART3"],
  82. "skeleton": [
  83. ["BODYPART1", "BODYPART2"],
  84. ["BODYPART2", "BODYPART3"],
  85. ],
  86. "anterior_bodyparts": ["BODYPART1"],
  87. "posterior_bodyparts": ["BODYPART3"],
  88. },
  89. )
  90. other = _update_dict(
  91. kwargs,
  92. {
  93. "recording_name_suffix": "",
  94. "verbose": False,
  95. "conf_pseudocount": 1e-3,
  96. "video_dir": "",
  97. "keypoint_colormap": "autumn",
  98. "whiten": True,
  99. "fix_heading": False,
  100. "fps": 30,
  101. "outlier_scale_factor": 6.0
  102. },
  103. )
  104. fitting = _update_dict(
  105. kwargs,
  106. {
  107. "added_noise_level": 0.1,
  108. "PCA_fitting_num_frames": 1000000,
  109. "conf_threshold": 0.5,
  110. # 'kappa_scan_target_duration': 12,
  111. # 'kappa_scan_min': 1e2,
  112. # 'kappa_scan_max': 1e12,
  113. # 'num_arhmm_scan_iters': 50,
  114. # 'num_arhmm_final_iters': 200,
  115. # 'num_kpslds_scan_iters': 50,
  116. # 'num_kpslds_final_iters': 500
  117. },
  118. )
  119. comments = {
  120. "verbose": "whether to print progress messages during fitting",
  121. "keypoint_colormap": "colormap used for visualization; see `matplotlib.cm.get_cmap` for options",
  122. "added_noise_level": "upper bound of uniform noise added to the data during initial AR-HMM fitting; this is used to regularize the model",
  123. "PCA_fitting_num_frames": "number of frames used to fit the PCA model during initialization",
  124. "video_dir": "directory with videos from which keypoints were derived (used for crowd movies)",
  125. "recording_name_suffix": "suffix used to match videos to recording names; this can usually be left empty (see `util.find_matching_videos` for details)",
  126. "bodyparts": "used to access columns in the keypoint data",
  127. "skeleton": "used for visualization only",
  128. "use_bodyparts": "determines the subset of bodyparts to use for modeling and the order in which they are represented",
  129. "anterior_bodyparts": "used to initialize heading",
  130. "posterior_bodyparts": "used to initialize heading",
  131. "trans_hypparams": "transition hyperparameters",
  132. "ar_hypparams": "autoregressive hyperparameters",
  133. "obs_hypparams": "keypoint observation hyperparameters",
  134. "cen_hypparams": "centroid movement hyperparameters",
  135. "error_estimator": "parameters to convert neural net likelihoods to error size priors",
  136. "save_every_n_iters": "frequency for saving model snapshots during fitting; if 0 only final state is saved",
  137. "kappa_scan_target_duration": "target median syllable duration (in frames) for choosing kappa",
  138. "whiten": "whether to whiten principal components; used to initialize the latent pose trajectory `x`",
  139. "conf_threshold": "used to define outliers for interpolation when the model is initialized",
  140. "conf_pseudocount": "pseudocount used regularize neural network confidences",
  141. "fix_heading": "whether to keep the heading angle fixed; this should only be True if the pose is constrained to a narrow range of angles, e.g. a headfixed mouse.",
  142. "fps": "frames per second of the video recordings",
  143. }
  144. sections = [
  145. ("ANATOMY", anatomy),
  146. ("FITTING", fitting),
  147. ("HYPER PARAMS", hypperams),
  148. ("OTHER", other),
  149. ]
  150. with open(os.path.join(project_dir, "config.yml"), "w") as f:
  151. f.write(_build_yaml(sections, comments))
  152. def check_config_validity(config):
  153. """Check if the config is valid.
  154. To be valid, the config must satisfy the following criteria:
  155. - All the elements of `config["use_bodyparts"]` are also in
  156. `config["bodyparts"]`
  157. - All the elements of `config["anterior_bodyparts"]` are also in
  158. `config["use_bodyparts"]`
  159. - All the elements of `config["anterior_bodyparts"]` are also in
  160. `config["use_bodyparts"]`
  161. - For each pair in `config["skeleton"]`, both elements also in
  162. `config["bodyparts"]`
  163. Parameters
  164. ----------
  165. config: dict
  166. Returns
  167. -------
  168. validity: bool
  169. """
  170. error_messages = []
  171. # check anatomy
  172. for bodypart in config["use_bodyparts"]:
  173. if not bodypart in config["bodyparts"]:
  174. error_messages.append(
  175. f"ACTION REQUIRED: `use_bodyparts` contains {bodypart} "
  176. "which is not one of the options in `bodyparts`."
  177. )
  178. for bodypart in sum(config["skeleton"], []):
  179. if not bodypart in config["bodyparts"]:
  180. error_messages.append(
  181. f"ACTION REQUIRED: `skeleton` contains {bodypart} "
  182. "which is not one of the options in `bodyparts`."
  183. )
  184. for bodypart in config["anterior_bodyparts"]:
  185. if not bodypart in config["use_bodyparts"]:
  186. error_messages.append(
  187. f"ACTION REQUIRED: `anterior_bodyparts` contains {bodypart} "
  188. "which is not one of the options in `use_bodyparts`."
  189. )
  190. for bodypart in config["posterior_bodyparts"]:
  191. if not bodypart in config["use_bodyparts"]:
  192. error_messages.append(
  193. f"ACTION REQUIRED: `posterior_bodyparts` contains {bodypart} "
  194. "which is not one of the options in `use_bodyparts`."
  195. )
  196. if len(error_messages) == 0:
  197. return True
  198. for msg in error_messages:
  199. print(fill(msg, width=70, subsequent_indent=" "), end="\n\n")
  200. return False
  201. def load_config(project_dir, check_if_valid=True, build_indexes=True):
  202. """Load a project config file.
  203. Parameters
  204. ----------
  205. project_dir: str
  206. Directory containing the config file
  207. check_if_valid: bool, default=True
  208. Check if the config is valid using
  209. :py:func:`keypoint_moseq.io.check_config_validity`
  210. build_indexes: bool, default=True
  211. Add keys `"anterior_idxs"` and `"posterior_idxs"` to the config. Each
  212. maps to a jax array indexing the elements of
  213. `config["anterior_bodyparts"]` and `config["posterior_bodyparts"]` by
  214. their order in `config["use_bodyparts"]`
  215. Returns
  216. -------
  217. config: dict
  218. """
  219. config_path = os.path.join(project_dir, "config.yml")
  220. with open(config_path, "r") as stream:
  221. config = yaml.safe_load(stream)
  222. if check_if_valid:
  223. check_config_validity(config)
  224. if build_indexes:
  225. config["anterior_idxs"] = jnp.array(
  226. [config["use_bodyparts"].index(bp) for bp in config["anterior_bodyparts"]]
  227. )
  228. config["posterior_idxs"] = jnp.array(
  229. [config["use_bodyparts"].index(bp) for bp in config["posterior_bodyparts"]]
  230. )
  231. if not "skeleton" in config or config["skeleton"] is None:
  232. config["skeleton"] = []
  233. return config
  234. def update_config(project_dir, **kwargs):
  235. """Update the config file stored at `project_dir/config.yml`.
  236. Use keyword arguments to update key/value pairs in the config. To update
  237. model hyperparameters, just use the name of the hyperparameter as the
  238. keyword argument.
  239. Examples
  240. --------
  241. To update `video_dir` to `/path/to/videos`::
  242. >>> update_config(project_dir, video_dir='/path/to/videos')
  243. >>> print(load_config(project_dir)['video_dir'])
  244. /path/to/videos
  245. To update `trans_hypparams['kappa']` to `100`::
  246. >>> update_config(project_dir, kappa=100)
  247. >>> print(load_config(project_dir)['trans_hypparams']['kappa'])
  248. 100
  249. """
  250. config = load_config(project_dir, check_if_valid=False, build_indexes=False)
  251. config.update(kwargs)
  252. generate_config(project_dir, **config)
  253. def setup_project(
  254. project_dir,
  255. deeplabcut_config=None,
  256. sleap_file=None,
  257. nwb_file=None,
  258. freipose_config=None,
  259. dannce_config=None,
  260. overwrite=False,
  261. **options,
  262. ):
  263. """
  264. Setup a project directory with the following structure::
  265. project_dir
  266. └── config.yml
  267. Parameters
  268. ----------
  269. project_dir: str
  270. Path to the project directory (relative or absolute).
  271. deeplabcut_config: str, default=None
  272. Path to a deeplabcut config file. Will be used to initialize
  273. `bodyparts`, `skeleton`, `use_bodyparts` and `video_dir` in the
  274. keypoint MoSeq config (overrided by kwargs).
  275. sleap_file: str, default=None
  276. Path to a .hdf5 or .slp file containing predictions for one video. Will
  277. be used to initialize `bodyparts`, `skeleton`, and `use_bodyparts` in
  278. the keypoint MoSeq config (overrided by kwargs).
  279. nwb_file: str, default=None
  280. Path to a .nwb file containing predictions for one video. Will be used
  281. to initialize `bodyparts`, `skeleton`, and `use_bodyparts` in the
  282. keypoint MoSeq config. (overrided by kwargs).
  283. freipose_config: str, default=None
  284. Path to a freipose skeleton config file. Will be used to initialize
  285. `bodyparts`, `skeleton`, and `use_bodyparts` in the keypoint MoSeq config
  286. (overrided by kwargs).
  287. dannce_config: str, default=None
  288. Path to a dannce config file. Will be used to initialize `bodyparts`,
  289. `skeleton`, and `use_bodyparts` in the keypoint MoSeq config (overrided
  290. by kwargs).
  291. overwrite: bool, default=False
  292. Overwrite any config.yml that already exists at the path
  293. `{project_dir}/config.yml`.
  294. options
  295. Used to initialize config file. Overrides default settings.
  296. """
  297. if os.path.exists(project_dir) and not overwrite:
  298. print(
  299. fill(
  300. f"The directory `{project_dir}` already exists. Use "
  301. "`overwrite=True` or pick a different name"
  302. )
  303. )
  304. return
  305. if deeplabcut_config is not None:
  306. dlc_options = {}
  307. with open(deeplabcut_config, "r") as stream:
  308. dlc_config = yaml.safe_load(stream)
  309. if dlc_config is None:
  310. raise RuntimeError(
  311. f"{deeplabcut_config} does not exists or is not a"
  312. " valid yaml file"
  313. )
  314. if "multianimalproject" in dlc_config and dlc_config["multianimalproject"]:
  315. dlc_options["bodyparts"] = dlc_config["multianimalbodyparts"]
  316. dlc_options["use_bodyparts"] = dlc_config["multianimalbodyparts"]
  317. else:
  318. dlc_options["bodyparts"] = dlc_config["bodyparts"]
  319. dlc_options["use_bodyparts"] = dlc_config["bodyparts"]
  320. dlc_options["skeleton"] = dlc_config["skeleton"]
  321. dlc_options["video_dir"] = os.path.join(
  322. dlc_config["project_path"], "videos"
  323. )
  324. options = {**dlc_options, **options}
  325. elif sleap_file is not None:
  326. sleap_options = {}
  327. if os.path.splitext(sleap_file)[1] == ".slp":
  328. slp_file = sleap_io.load_slp(sleap_file)
  329. assert len(slp_file.skeletons) == 1, fill(
  330. f"{sleap_file} contains more than one skeleton. "
  331. "This is not currently supported. Please "
  332. "open a github issue or email [email hidden]"
  333. )
  334. skeleton = slp_file.skeletons[0]
  335. node_names = skeleton.node_names
  336. edge_names = [[e.source.name, e.destination.name] for e in skeleton.edges]
  337. else:
  338. with h5py.File(sleap_file, "r") as f:
  339. node_names = [n.decode("utf-8") for n in f["node_names"]]
  340. edge_names = [
  341. [n.decode("utf-8") for n in edge] for edge in f["edge_names"]
  342. ]
  343. sleap_options["bodyparts"] = node_names
  344. sleap_options["use_bodyparts"] = node_names
  345. sleap_options["skeleton"] = edge_names
  346. options = {**sleap_options, **options}
  347. elif nwb_file is not None:
  348. nwb_options = {}
  349. with NWBHDF5IO(nwb_file, mode="r", load_namespaces=True) as io:
  350. pose_obj = _load_nwb_pose_obj(io, nwb_file)
  351. bodyparts = list(pose_obj.nodes[:])
  352. nwb_options["bodyparts"] = bodyparts
  353. nwb_options["use_bodyparts"] = bodyparts
  354. if "edges" in pose_obj.fields:
  355. edges = pose_obj.edges[:]
  356. skeleton = [[bodyparts[i], bodyparts[j]] for i, j in edges]
  357. nwb_options["skeleton"] = skeleton
  358. options = {**nwb_options, **options}
  359. elif freipose_config is not None:
  360. freipose_options = {}
  361. with open(freipose_config, "r") as stream:
  362. freipose_config = commentjson.load(stream)
  363. bodyparts = [kp for kp, color in freipose_config["keypoints"]]
  364. skeleton = []
  365. for [bp1, bp2], color in freipose_config["limbs"]:
  366. if isinstance(bp1, list):
  367. bp1 = bp1[0]
  368. if isinstance(bp2, list):
  369. bp2 = bp2[0]
  370. skeleton.append(tuple(sorted([bodyparts[bp1], bodyparts[bp2]])))
  371. freipose_options["bodyparts"] = bodyparts
  372. freipose_options["use_bodyparts"] = bodyparts
  373. freipose_options["skeleton"] = list(map(list, sorted(set(skeleton))))
  374. options = {**freipose_options, **options}
  375. elif dannce_config is not None:
  376. dannce_config = loadmat(dannce_config)
  377. bodyparts = [n[0][0].item() for n in dannce_config["joint_names"]]
  378. skeleton = [
  379. [bodyparts[i], bodyparts[j]] for i, j in dannce_config["joints_idx"] - 1
  380. ]
  381. dannce_options = {
  382. "bodyparts": bodyparts,
  383. "use_bodyparts": bodyparts,
  384. "skeleton": skeleton,
  385. }
  386. options = {**dannce_options, **options}
  387. if not os.path.exists(project_dir):
  388. os.makedirs(project_dir)
  389. generate_config(project_dir, **options)
  390. def save_pca(pca, project_dir, pca_path=None):
  391. """Save a PCA model to disk.
  392. The model is saved to `pca_path` or else to `{project_dir}/pca.p`.
  393. Parameters
  394. ----------
  395. pca: :py:class:`sklearn.decomposition.PCA`
  396. project_dir: str
  397. pca_path: str, default=None
  398. """
  399. if pca_path is None:
  400. pca_path = os.path.join(project_dir, "pca.p")
  401. joblib.dump(pca, pca_path)
  402. def load_pca(project_dir, pca_path=None):
  403. """Load a PCA model from disk.
  404. The model is loaded from `pca_path` or else from `{project_dir}/pca.p`.
  405. Parameters
  406. ----------
  407. project_dir: str
  408. pca_path: str, default=None
  409. Returns
  410. -------
  411. pca: :py:class:`sklearn.decomposition.PCA`
  412. """
  413. if pca_path is None:
  414. pca_path = os.path.join(project_dir, "pca.p")
  415. assert os.path.exists(pca_path), fill(f"No PCA model found at {pca_path}")
  416. return joblib.load(pca_path)
  417. def load_checkpoint(project_dir=None, model_name=None, path=None, iteration=None):
  418. """Load data and model snapshot from a saved checkpoint.
  419. The checkpoint path can be specified directly via `path` or else it is
  420. assumed to be `{project_dir}/{model_name}/checkpoint.h5`.
  421. Parameters
  422. ----------
  423. project_dir: str, default=None
  424. Project directory; used in conjunction with `model_name` to determine the
  425. checkpoint path if `path` is not specified.
  426. model_name: str, default=None
  427. Model name; used in conjunction with `project_dir` to determine the
  428. checkpoint path if `path` is not specified.
  429. path: str, default=None
  430. Checkpoint path; if not specified, the checkpoint path is set to
  431. `{project_dir}/{model_name}/checkpoint.h5`.
  432. iteration: int, default=None
  433. Determines which model snapshot to load. If None, the last snapshot is
  434. loaded.
  435. Returns
  436. -------
  437. model: dict
  438. Model dictionary containing states, parameters, hyperparameters,
  439. noise prior, and random seed.
  440. data: dict
  441. Data dictionary containing observations, confidences, mask and
  442. associated metadata (see :py:func:`keypoint_moseq.util.format_data`).
  443. metadata: tuple (keys, bounds)
  444. Recording names and start/end frames for the data (see
  445. :py:func:`keypoint_moseq.util.format_data`).
  446. iteration: int
  447. Iteration of model fitting corresponding to the loaded snapshot.
  448. """
  449. path = _get_path(project_dir, model_name, path, "checkpoint.h5")
  450. with h5py.File(path, "r") as f:
  451. saved_iterations = np.sort([int(i) for i in f["model_snapshots"]])
  452. if iteration is None:
  453. iteration = saved_iterations[-1]
  454. else:
  455. assert iteration in saved_iterations, fill(
  456. f"No snapshot found for iteration {iteration}. "
  457. f"Available iterations are {saved_iterations}"
  458. )
  459. model = load_hdf5(path, f"model_snapshots/{iteration}")
  460. metadata = load_hdf5(path, "metadata")
  461. data = load_hdf5(path, "data")
  462. return model, data, metadata, iteration
  463. def reindex_syllables_in_checkpoint(
  464. project_dir=None, model_name=None, path=None, index=None, runlength=True
  465. ):
  466. """Reindex syllable labels by their frequency in the most recent model
  467. snapshot in a checkpoint file.
  468. This is an in-place operation: the checkpoint is loaded from disk, modified
  469. and saved to disk again. The label permutation is applied to all model
  470. snapshots in the checkpoint.
  471. The checkpoint path can be specified directly via `path` or else it is
  472. assumed to be `{project_dir}/{model_name}/checkpoint.h5`.
  473. Parameters
  474. ----------
  475. project_dir: str, default=None
  476. model_name: str, default=None
  477. path: str, default=None
  478. index: array of shape (num_states,), default=None
  479. Permutation for syllable labels, where `index[i]` is relabled as `i`.
  480. If None, syllables are relabled by frequency, with the most frequent
  481. syllable relabled as 0, and so on.
  482. runlength: bool, default=True
  483. If True, frequencies are quantified using the number of non-consecutive
  484. occurrences of each syllable. If False, frequency is quantified by
  485. total number of frames.
  486. Returns
  487. -------
  488. index: array of shape (num_states,)
  489. The index used for permuting syllable labels. If `index[i] = j`, then
  490. the syllable formerly labeled `j` is now labeled `i`.
  491. """
  492. path = _get_path(project_dir, model_name, path, "checkpoint.h5")
  493. with h5py.File(path, "r") as f:
  494. saved_iterations = [int(i) for i in f["model_snapshots"]]
  495. if index is None:
  496. with h5py.File(path, "r") as f:
  497. last_iter = np.max(saved_iterations)
  498. num_states = f[f"model_snapshots/{last_iter}/params/pi"].shape[0]
  499. z = f[f"model_snapshots/{last_iter}/states/z"][()]
  500. mask = f["data/mask"][()]
  501. index = np.argsort(get_frequencies(z, mask, num_states, runlength))[::-1]
  502. def _reindex(model):
  503. model["params"]["betas"] = model["params"]["betas"][index]
  504. model["params"]["pi"] = model["params"]["pi"][index, :][:, index]
  505. model["params"]["Ab"] = model["params"]["Ab"][index]
  506. model["params"]["Q"] = model["params"]["Q"][index]
  507. # allocentric dynamics params (location-aware model) are also per-state
  508. for key in ["delta_h", "sigmasq_h", "delta_v", "sigmasq_v"]:
  509. if key in model["params"]:
  510. model["params"][key] = model["params"][key][index]
  511. model["states"]["z"] = np.argsort(index)[model["states"]["z"]]
  512. return model
  513. for iteration in tqdm.tqdm(
  514. saved_iterations, desc="Reindexing", unit="model snapshot", ncols=72
  515. ):
  516. model = load_hdf5(path, f"model_snapshots/{iteration}")
  517. save_hdf5(
  518. path,
  519. _reindex(model),
  520. f"model_snapshots/{iteration}",
  521. exist_ok=True,
  522. overwrite=True,
  523. )
  524. return index
  525. def extract_results(
  526. model,
  527. metadata,
  528. project_dir=None,
  529. model_name=None,
  530. save_results=True,
  531. path=None,
  532. overwrite=False,
  533. ):
  534. """Extract model outputs and [optionally] save them to disk.
  535. Model outputs are saved to disk as a .h5 file, either at `path`
  536. if it is specified, or at `{project_dir}/{model_name}/results.h5` if it is not.
  537. If a .h5 file with the given path already exists, the outputs will be added
  538. to it. The results have the following structure::
  539. results.h5
  540. ├──recording_name1
  541. │ ├──syllable # model state sequence (z), shape=(num_timepoints,)
  542. │ ├──latent_state # model latent state (x), shape=(num_timepoints,latent_dim)
  543. │ ├──centroid # model centroid (v), shape=(num_timepoints,keypoint_dim)
  544. │ └──heading # model heading (h), shape=(num_timepoints,)
  545. ⋮
  546. Parameters
  547. ----------
  548. model : dict
  549. Model dictionary containing states, parameters, hyperparameters,
  550. noise prior, and random seed.
  551. metadata: tuple (keys, bounds)
  552. Recordings and start/end frames for the data (see
  553. :py:func:`keypoint_moseq.util.format_data`).
  554. save_results : bool, default=True
  555. If True, the model outputs will be saved to disk.
  556. project_dir : str, default=None
  557. Path to the project directory. Required if `save_results=True` and
  558. `results_path=None`.
  559. model_name : str, default=None
  560. Name of the model. Required if `save_results=True` and
  561. `results_path=None`.
  562. path : str, default=None
  563. Optional path for saving model outputs.
  564. overwrite : bool, default=False
  565. If True, overwrite existing results for recordings that are
  566. already in the results file. This is required when applying a
  567. model to data that includes recordings from the original
  568. training set.
  569. Returns
  570. -------
  571. results_dict : dict
  572. Dictionary of model outputs with the same structure as the results
  573. `.h5` file.
  574. """
  575. if save_results:
  576. path = _get_path(project_dir, model_name, path, "results.h5")
  577. # check for overlapping recording names
  578. if not overwrite and os.path.exists(path):
  579. recording_names = set(metadata[0])
  580. with h5py.File(path, "r") as f:
  581. overlap = recording_names & set(f.keys())
  582. if overlap:
  583. raise RuntimeError(fill(
  584. f"{path} already contains results for "
  585. f"{len(overlap)} recording(s), including "
  586. f"'{next(iter(overlap))}'. To overwrite existing "
  587. f"results, set overwrite=True in apply_model. "
  588. f"See https://keypoint-moseq.readthedocs.io/en/"
  589. f"latest/FAQs.html#detecting-existing-syllables"
  590. f"-in-new-data for more information."
  591. ))
  592. states = jax.device_get(model["states"])
  593. # extract syllables; repeat first syllable an extra `nlags` times
  594. nlags = states["x"].shape[1] - states["z"].shape[1]
  595. z = np.pad(states["z"], ((0, 0), (nlags, 0)), mode="edge")
  596. syllables = unbatch(z, *metadata)
  597. # extract latent state, centroid, and heading
  598. latent_state = unbatch(states["x"], *metadata)
  599. centroid = unbatch(states["v"], *metadata)
  600. heading = unbatch(states["h"], *metadata)
  601. results_dict = {
  602. recording_name: {
  603. "syllable": syllables[recording_name],
  604. "latent_state": latent_state[recording_name],
  605. "centroid": centroid[recording_name],
  606. "heading": heading[recording_name],
  607. }
  608. for recording_name in syllables.keys()
  609. }
  610. if save_results:
  611. save_hdf5(path, results_dict, exist_ok=True, overwrite=overwrite)
  612. print(fill(f"Saved results to {path}"))
  613. return results_dict
  614. def load_results(project_dir=None, model_name=None, path=None):
  615. """Load the results from a modeled dataset.
  616. The results path can be specified directly via `path`. Otherwise it is
  617. assumed to be `{project_dir}/{model_name}/results.h5`.
  618. Parameters
  619. ----------
  620. project_dir: str, default=None
  621. model_name: str, default=None
  622. path: str, default=None
  623. Returns
  624. -------
  625. results: dict
  626. See :py:func:`keypoint_moseq.fitting.apply_model`
  627. """
  628. path = _get_path(project_dir, model_name, path, "results.h5")
  629. return load_hdf5(path)
  630. def save_results_as_csv(
  631. results, project_dir=None, model_name=None, save_dir=None, path_sep="-"
  632. ):
  633. """Save modeling results to csv format.
  634. This function creates a directory and then saves a separate csv file for
  635. each recording. The directory is created at `save_dir` if provided,
  636. otherwise at `{project_dir}/{model_name}/results`.
  637. Parameters
  638. ----------
  639. results: dict
  640. See :py:func:`keypoint_moseq.io.extract_results`.
  641. project_dir: str, default=None
  642. Project directory; required if `save_dir` is not provided.
  643. model_name: str, default=None
  644. Name of the model; required if `save_dir` is not provided.
  645. save_dir: str, default=None
  646. Optional path to the directory where the csv files will be saved.
  647. path_sep: str, default='-'
  648. If a path separator ("/" or "\") is present in the recording name, it
  649. will be replaced with `path_sep` when saving the csv file.
  650. """
  651. save_dir = _get_path(project_dir, model_name, save_dir, "results", "save_dir")
  652. if not os.path.exists(save_dir):
  653. os.makedirs(save_dir)
  654. for key in tqdm.tqdm(results.keys(), desc="Saving to csv", ncols=72):
  655. column_names, data = [], []
  656. if "syllable" in results[key].keys():
  657. column_names.append(["syllable"])
  658. data.append(results[key]["syllable"].reshape(-1, 1))
  659. if "centroid" in results[key].keys():
  660. d = results[key]["centroid"].shape[1]
  661. column_names.append(["centroid x", "centroid y", "centroid z"][:d])
  662. data.append(results[key]["centroid"])
  663. if "heading" in results[key].keys():
  664. column_names.append(["heading"])
  665. data.append(results[key]["heading"].reshape(-1, 1))
  666. if "latent_state" in results[key].keys():
  667. latent_dim = results[key]["latent_state"].shape[1]
  668. column_names.append([f"latent_state {i}" for i in range(latent_dim)])
  669. data.append(results[key]["latent_state"])
  670. dfs = [pd.DataFrame(arr, columns=cols) for arr, cols in zip(data, column_names)]
  671. df = pd.concat(dfs, axis=1)
  672. for col in df.select_dtypes(include=[np.floating]).columns:
  673. df[col] = df[col].astype(float).round(4)
  674. save_name = key.replace(os.path.sep, path_sep)
  675. save_path = os.path.join(save_dir, save_name)
  676. df.to_csv(f"{save_path}.csv", index=False)
  677. def _name_from_path(filepath, path_in_name, path_sep, remove_extension):
  678. """Create a name from a filepath.
  679. Either return the name of the file (with the extension removed) or return
  680. the full filepath, where the path separators are replaced with `path_sep`.
  681. """
  682. if remove_extension:
  683. filepath = os.path.splitext(filepath)[0]
  684. if path_in_name:
  685. return filepath.replace(os.path.sep, path_sep)
  686. else:
  687. return os.path.basename(filepath)
  688. def save_keypoints(
  689. save_dir, coordinates, confidences=None, bodyparts=None, path_sep="-"
  690. ):
  691. """Convenience function for saving keypoint detections to csv files.
  692. One csv file is saved for each recording in `coordinates`. Each row in the
  693. csv corresponds to one frame and the columns are named
  694. "BODYPART1_x", "BODYPART1_y", "BODYPART1_conf", "BODYPART2_x", ...
  695. Columns with confidence scores are ommitted if `confidences` is not provided.
  696. Besides confidences, there can be 2 or 3 columns for each bodypart, depending
  697. on whether the keypoints are 2D or 3D.
  698. Parameters
  699. ----------
  700. save_dir: str
  701. Directory to save the results. A separate csv file will be saved for
  702. each recording in `coordinates`.
  703. coordinates: dict
  704. Dictionary mapping recording names to numpy arrays of shape
  705. (n_frames, n_keypoints, 2[or 3]) that contain the x and y (and z)
  706. coordinates of the keypoints. If any keys contain a path separator
  707. (such as "/"), it will be replaced with `path_sep` when naming the
  708. csv file.
  709. confidences: dict, default=None
  710. Dictionary mapping recording names to numpy arrays of shape
  711. (n_frames, n_keypoints) with the confidence scores of the keypoints.
  712. Must have the same keys as `coordinates`.
  713. bodyparts: list, default=None
  714. List of bodypart names, in the same order as the keypoints in the
  715. `coordinates` and `confidences` arrays. If None, the bodypart names
  716. will be set to ["bodypart1", "bodypart2", ...].
  717. path_sep: str, default='-'
  718. If a path separator ("/" or "\") is present in the recording name, it
  719. will be replaced with `path_sep` when saving the csv file.
  720. """
  721. if not os.path.exists(save_dir):
  722. os.makedirs(save_dir)
  723. if confidences is not None:
  724. assert set(coordinates.keys()) == set(confidences.keys()), fill(
  725. "The keys in `coordinates` and `confidences` must be the same."
  726. )
  727. # get number of keypoints and dimensions
  728. _, num_keypoints, num_dims = next(iter(coordinates.values())).shape
  729. # generate bodypart names if not provided
  730. if bodyparts is None:
  731. bodyparts = [f"bodypart{i}" for i in range(num_keypoints)]
  732. # create column names
  733. suffixes = ["x", "y", "z"][:num_keypoints]
  734. if confidences is not None:
  735. suffixes += ["conf"]
  736. columns = [f"{bp}_{suffix}" for bp in bodyparts for suffix in suffixes]
  737. # save data to csv
  738. for recording_name, coords in coordinates.items():
  739. if confidences is not None:
  740. data = np.concatenate(
  741. [coords, confidences[recording_name][..., np.newaxis]], axis=-1
  742. ).reshape(-1, (num_dims + 1) * num_keypoints)
  743. else:
  744. data = coords.reshape(-1, num_dims * num_keypoints)
  745. save_name = recording_name.replace(os.path.sep, path_sep)
  746. save_path = os.path.join(save_dir, save_name)
  747. pd.DataFrame(data, columns=columns).to_csv(f"{save_path}.csv", index=False)
  748. def load_keypoints(
  749. filepath_pattern,
  750. format,
  751. extension=None,
  752. recursive=True,
  753. path_sep="-",
  754. path_in_name=False,
  755. remove_extension=True,
  756. exclude_individuals=["single"],
  757. ):
  758. """
  759. Load keypoint tracking results from one or more files. Several file
  760. formats are supported:
  761. - deeplabcut
  762. .csv and .h5/.hdf5 files generated by deeplabcut. For single-animal
  763. tracking, each file yields a single key/value pair in the returned
  764. `coordinates` and `confidences` dictionaries. For multi-animal tracking,
  765. a key/vaue pair will be generated for each tracked individual. For
  766. example the file `two_mice.h5` with individuals "mouseA" and "mouseB"
  767. will yield the pair of keys `'two_mice_mouseA', 'two_mice_mouseB'`.
  768. - sleap
  769. .slp and .h5/.hdf5 files generated by sleap. For single-animal tracking,
  770. each file yields a single key/value pair in the returned `coordinates`
  771. and `confidences` dictionaries. For multi-animal tracking, a key/vaue
  772. pair will be generated for each track. For example a single file called
  773. `two_mice.h5` will yield the pair of keys `'two_mice_track0',
  774. 'two_mice_track1'`.
  775. - anipose
  776. .csv files generated by anipose. Each file should contain five columns
  777. per keypoint (x,y,z,error,score), plus a last column with the frame
  778. number. The `score` column is used as the keypoint confidence.
  779. - sleap-anipose
  780. .h5/.hdf5 files generated by sleap-anipose. Each file should contain a
  781. dataset called `'tracks'` with shape (n_frames, 1, n_keypoints, 3). If
  782. there is also a `'point_scores'` dataset, it will be used as the
  783. keypoint confidence. Otherwise, the confidence will be set to 1.
  784. - nwb
  785. .nwb files (Neurodata Without Borders). Each file should contain
  786. exactly one `PoseEstimation` object (for multi-animal tracking, each
  787. animal should be stored in its own .nwb file). The `PoseEstimation`
  788. object should contain one `PoseEstimationSeries` object for each
  789. bodypart. Confidence values are optional and will be set to 1 if not
  790. present.
  791. - facemap
  792. .h5 files saved by Facemap. See Facemap documentation for details:
  793. https://facemap.readthedocs.io/en/latest/outputs.html#keypoints-processing
  794. The files should have the format::
  795. [filename].h5
  796. └──Facemap
  797. ├──keypoint1
  798. │ ├──x
  799. │ ├──y
  800. │ └──likelihood
  801. ⋮
  802. - freipose
  803. .json files saved by FreiPose. Each file should contain a list of dicts
  804. that each include a "kp_xyz" key with the 3D coordinates for one frame.
  805. Keypoint scores (saved under "kp_score") are not loaded because they are
  806. not bounded between 0 and 1, which is required for modeling. Since
  807. FreiPose does not save the bodypart names, the `bodyparts` return
  808. value is set to None.
  809. - dannce
  810. .mat files saved by Dannce.
  811. Parameters
  812. ----------
  813. filepath_pattern: str or list of str
  814. Filepath pattern for a set of deeplabcut csv or hdf5 files, or a list
  815. of such patterns. Filepath patterns can be:
  816. - single file (e.g. `/path/to/file.csv`)
  817. - single directory (e.g. `/path/to/dir/`)
  818. - set of files (e.g. `/path/to/fileprefix*`)
  819. - set of directories (e.g. `/path/to/dirprefix*`)
  820. format: str
  821. Format of the files to load. Must be one of `deeplabcut`, `sleap`,
  822. `anipose`, or `sleap-anipose`.
  823. extension: str, default=None
  824. File extension to use when searching for files. If None, then the
  825. extension will be inferred from the `format` argument:
  826. - sleap: 'h5' or 'slp'
  827. - deeplabcut: 'csv' or 'h5'
  828. - anipose: 'csv'
  829. - sleap-anipose: 'h5'
  830. - nwb: 'nwb'
  831. - facemap: 'h5'
  832. - freipose: 'json'
  833. - dannce: 'mat'
  834. recursive: bool, default=True
  835. Whether to search recursively for deeplabcut csv or hdf5 files.
  836. path_in_name: bool, default=False
  837. Whether to name the tracking results from each file by the path to the
  838. file (True) or just the filename (False). If True, the `path_sep`
  839. argument is used to separate the path components.
  840. path_sep: str, default='-'
  841. Separator to use when `path_in_name` is True. For example, if
  842. `path_sep` is `'-'`, then the tracking results from the file
  843. `/path/to/file.csv` will be named `path-to-file`. Using `'/'` as the
  844. separator is discouraged, as it will cause problems saving/loading the
  845. modeling results to/from hdf5 files.
  846. remove_extension: bool, default=True
  847. Whether to remove the file extension when naming the tracking results
  848. from each file.
  849. exclude_individuals: list of str, default=["single"]
  850. List of individuals to exclude from the results. This is only used for
  851. multi-animal tracking with deeplabcut.
  852. Returns
  853. -------
  854. coordinates: dict
  855. Dictionary mapping filenames to keypoint coordinates as ndarrays of
  856. shape (n_frames, n_bodyparts, 2[or 3])
  857. confidences: dict
  858. Dictionary mapping filenames to `likelihood` scores as ndarrays of
  859. shape (n_frames, n_bodyparts)
  860. bodyparts: list of str
  861. List of bodypart names. The order of the names matches the order of the
  862. bodyparts in `coordinates` and `confidences`.
  863. """
  864. formats = {
  865. "deeplabcut": (_deeplabcut_loader, [".csv", ".h5", ".hdf5"]),
  866. "sleap": (_sleap_loader, [".h5", ".hdf5", ".slp"]),
  867. "anipose": (_anipose_loader, [".csv"]),
  868. "sleap-anipose": (_sleap_anipose_loader, [".h5", ".hdf5"]),
  869. "nwb": (_nwb_loader, [".nwb"]),
  870. "facemap": (_facemap_loader, [".h5", ".hdf5"]),
  871. "freipose": (_freipose_loader, [".json"]),
  872. "dannce": (_dannce_loader, [".mat"]),
  873. }
  874. # get format-specific loader and extensions
  875. assert format in formats, fill(
  876. f"Unrecognized format '{format}'. Must be one of {list(formats.keys())}"
  877. )
  878. loader, extensions = formats[format]
  879. # optionally override default extension list
  880. if extension is not None:
  881. extensions = [extension]
  882. # optionally add format-specific arguments
  883. if format == "deeplabcut":
  884. additional_args = {"exclude_individuals": exclude_individuals}
  885. else:
  886. additional_args = {}
  887. # get list of filepaths
  888. filepaths = list_files_with_exts(filepath_pattern, extensions, recursive=recursive)
  889. assert len(filepaths) > 0, fill(
  890. f"No files with extensions {extensions} found for {filepath_pattern}"
  891. )
  892. # load keypoints from each file
  893. coordinates, confidences, bodyparts = {}, {}, None
  894. for filepath in tqdm.tqdm(filepaths, desc=f"Loading keypoints", ncols=72):
  895. name = _name_from_path(filepath, path_in_name, path_sep, remove_extension)
  896. try:
  897. new_coordinates, new_confidences, bodyparts = loader(
  898. filepath, name, **additional_args
  899. )
  900. except Exception as e:
  901. print(fill(f"Error loading {filepath}"))
  902. raise e
  903. if set(new_coordinates.keys()) & set(coordinates.keys()):
  904. raise ValueError(
  905. f"Duplicate names found in {filepath_pattern}:\n\n"
  906. f"{set(new_coordinates.keys()) & set(coordinates.keys())}"
  907. f"\n\nThis may be caused by repeated filenames with "
  908. "different extensions. If so, please set the extension "
  909. "explicitly via the `extension` argument. Another possible"
  910. " cause is commonly-named files in different directories. "
  911. "if that is the case, then set `path_in_name=True`."
  912. )
  913. coordinates.update(new_coordinates)
  914. confidences.update(new_confidences)
  915. # check for valid results
  916. assert len(coordinates) > 0, fill(f"No valid results found for {filepath_pattern}")
  917. check_nan_proportions(coordinates, bodyparts)
  918. return coordinates, confidences, bodyparts
  919. def _deeplabcut_loader(filepath, name, exclude_individuals=["single"]):
  920. """Load tracking results from deeplabcut csv or hdf5 files."""
  921. ext = os.path.splitext(filepath)[1]
  922. if ext == ".h5":
  923. df = pd.read_hdf(filepath)
  924. if ext == ".csv":
  925. with open(filepath) as f:
  926. head = list(islice(f, 0, 5))
  927. if "individuals" in head[1]:
  928. header = [0, 1, 2, 3]
  929. else:
  930. header = [0, 1, 2]
  931. df = pd.read_csv(filepath, header=header, index_col=0)
  932. coordinates, confidences = {}, {}
  933. if "individuals" in df.columns.names:
  934. ind_bodyparts = {}
  935. for ind in df.columns.get_level_values("individuals").unique():
  936. if ind in exclude_individuals:
  937. print(
  938. f'Excluding individual: "{ind}". Set `exclude_individuals=[]` to include.'
  939. )
  940. else:
  941. ind_df = df.xs(ind, axis=1, level="individuals")
  942. bps = ind_df.columns.get_level_values("bodyparts").unique().tolist()
  943. ind_bodyparts[ind] = bps
  944. arr = ind_df.to_numpy().reshape(len(ind_df), -1, 3)
  945. coordinates[f"{name}_{ind}"] = arr[:, :, :-1]
  946. confidences[f"{name}_{ind}"] = arr[:, :, -1]
  947. bodyparts = set(ind_bodyparts[list(ind_bodyparts.keys())[0]])
  948. assert all([set(bps) == bodyparts for bps in ind_bodyparts.values()]), (
  949. f"Bodyparts are not consistent across individuals. The following bodyparts "
  950. f"were found for each individual: {ind_bodyparts}. Use `exclude_individuals`"
  951. "to exclude specific individuals."
  952. )
  953. else:
  954. bodyparts = df.columns.get_level_values("bodyparts").unique().tolist()
  955. arr = df.to_numpy().reshape(len(df), -1, 3)
  956. coordinates[name] = arr[:, :, :-1]
  957. confidences[name] = arr[:, :, -1]
  958. return coordinates, confidences, bodyparts
  959. def _sleap_loader(filepath, name):
  960. """Load keypoints from sleap hdf5 or slp files."""
  961. if os.path.splitext(filepath)[1] == ".slp":
  962. slp_file = sleap_io.load_slp(filepath)
  963. assert len(slp_file.skeletons) == 1, fill(
  964. f"{filepath} contains more than one skeleton. "
  965. "This is not currently supported. Please "
  966. "open a github issue or email [email hidden]"
  967. )
  968. bodyparts = slp_file.skeletons[0].node_names
  969. arr = slp_file.numpy(return_confidence=True)
  970. coords = arr[:, :, :, :-1].transpose((1, 0, 2, 3))
  971. confs = arr[:, :, :, -1].transpose((1, 0, 2))
  972. else:
  973. with h5py.File(filepath, "r") as f:
  974. coords = f["tracks"][()].transpose((0, 3, 2, 1))
  975. confs = f["point_scores"][()].transpose((0, 2, 1))
  976. bodyparts = [name.decode("utf-8") for name in f["node_names"]]
  977. if coords.shape[0] == 1:
  978. coordinates = {name: coords[0]}
  979. confidences = {name: confs[0]}
  980. else:
  981. coordinates = {f"{name}_track{i}": coords[i] for i in range(coords.shape[0])}
  982. confidences = {f"{name}_track{i}": confs[i] for i in range(coords.shape[0])}
  983. return coordinates, confidences, bodyparts
  984. def _anipose_loader(filepath, name):
  985. """Load keypoints from anipose csv files."""
  986. with open(filepath, "r") as f:
  987. header = f.readline()
  988. pattern = r"(?P<string>\w+)_x,(?P=string)_y,(?P=string)_z"
  989. bodyparts = list(re.findall(pattern, header))
  990. df = pd.read_csv(filepath)
  991. coordinates = {
  992. name: np.stack(
  993. [df[[f"{bp}_x", f"{bp}_y", f"{bp}_z"]].to_numpy() for bp in bodyparts],
  994. axis=1,
  995. )
  996. }
  997. confidences = {name: df[[f"{bp}_score" for bp in bodyparts]].to_numpy()}
  998. return coordinates, confidences, bodyparts
  999. def _sleap_anipose_loader(filepath, name):
  1000. """Load keypoints from sleap-anipose hdf5 files."""
  1001. with h5py.File(filepath, "r") as f:
  1002. coords = f["tracks"][()]
  1003. if "point_scores" in f.keys():
  1004. confs = f["point_scores"][()]
  1005. else:
  1006. confs = np.ones_like(coords[..., 0])
  1007. bodyparts = ["bodypart{}".format(i) for i in range(coords.shape[2])]
  1008. if coords.shape[1] == 1:
  1009. coordinates = {name: coords[:, 0]}
  1010. confidences = {name: confs[:, 0]}
  1011. else:
  1012. coordinates = {
  1013. f"{name}_track{i}": coords[:, i] for i in range(coords.shape[1])
  1014. }
  1015. confidences = {
  1016. f"{name}_track{i}": confs[:, i] for i in range(coords.shape[1])
  1017. }
  1018. return coordinates, confidences, bodyparts
  1019. def _load_nwb_pose_obj(io, filepath):
  1020. """Grab PoseEstimation object from an opened .nwb file."""
  1021. all_objs = io.read().all_children()
  1022. pose_objs = [o for o in all_objs if isinstance(o, PoseEstimation)]
  1023. assert len(pose_objs) > 0, fill(f"No PoseEstimation objects found in {filepath}")
  1024. assert len(pose_objs) == 1, fill(
  1025. f"Found multiple PoseEstimation objects in {filepath}. "
  1026. "This is not currently supported. Please open a github "
  1027. "issue to request this feature."
  1028. )
  1029. pose_obj = pose_objs[0]
  1030. return pose_obj
  1031. def _nwb_loader(filepath, name):
  1032. """Load keypoints from nwb files."""
  1033. with NWBHDF5IO(filepath, mode="r", load_namespaces=True) as io:
  1034. pose_obj = _load_nwb_pose_obj(io, filepath)
  1035. bodyparts = list(pose_obj.nodes[:])
  1036. coords = np.stack(
  1037. [pose_obj.pose_estimation_series[bp].data[()] for bp in bodyparts],
  1038. axis=1,
  1039. )
  1040. if "confidence" in pose_obj.pose_estimation_series[bodyparts[0]].fields:
  1041. confs = np.stack(
  1042. [
  1043. pose_obj.pose_estimation_series[bp].confidence[()]
  1044. for bp in bodyparts
  1045. ],
  1046. axis=1,
  1047. )
  1048. else:
  1049. confs = np.ones_like(coords[..., 0])
  1050. coordinates = {name: coords}
  1051. confidences = {name: confs}
  1052. return coordinates, confidences, bodyparts
  1053. def _facemap_loader(filepath, name):
  1054. """Load keypoints from facemap h5 files."""
  1055. with h5py.File(filepath, "r") as h5:
  1056. dset = h5["Facemap"]
  1057. bodyparts = sorted(dset.keys())
  1058. coords, confs = [], []
  1059. for bp in bodyparts:
  1060. coords.append(np.stack([dset[bp]["x"], dset[bp]["y"]], axis=1))
  1061. confs.append(dset[bp]["likelihood"])
  1062. coordinates = {name: np.stack(coords, axis=1)}
  1063. confidences = {name: np.stack(confs, axis=1)}
  1064. return coordinates, confidences, bodyparts
  1065. def _freipose_loader(filepath, name):
  1066. """Load keypoints from freipose json files."""
  1067. with open(filepath, "r") as f:
  1068. data = json.load(f)
  1069. coords = np.concatenate([d["kp_xyz"] for d in data], axis=0)
  1070. coordinates = {name: coords}
  1071. confidences = {name: np.ones_like(coords[..., 0])}
  1072. return coordinates, confidences, None
  1073. def _dannce_loader(filepath, name):
  1074. """Load keypoints from dannce mat files."""
  1075. mat = loadmat(filepath)
  1076. coords = mat["pred"].transpose((0, 2, 1))
  1077. confs = np.ones_like(coords[..., 0])
  1078. coordinates = {name: coords}
  1079. confidences = {name: confs}
  1080. return coordinates, confidences, None
  1081. def save_hdf5(filepath, save_dict, datapath=None, exist_ok=False, overwrite=False):
  1082. """Save a dict of pytrees to an hdf5 file. The leaves of the pytrees must
  1083. be numpy arrays, scalars, or strings.
  1084. Parameters
  1085. ----------
  1086. filepath: str
  1087. Path of the hdf5 file to create.
  1088. save_dict: dict
  1089. Dictionary where the values are pytrees, i.e. recursive collections of
  1090. tuples, lists, dicts, and numpy arrays.
  1091. datapath: str, default=None
  1092. Path within the hdf5 file to save the data. If None, the data is saved
  1093. at the root of the hdf5 file.
  1094. exist_ok: bool, default=False
  1095. If False, will raise an AssertionError when trying to modify an existing file.
  1096. overwrite: bool, default=False
  1097. If False, will raise an AssertionError when trying to overwrite an existing dataset or group.
  1098. """
  1099. assert not (
  1100. os.path.exists(filepath) and not exist_ok
  1101. ), f"{filepath} already exists. Set exist_ok to True to allow for editing an existing file."
  1102. with h5py.File(filepath, "a") as f:
  1103. if datapath is not None:
  1104. _savetree_hdf5(jax.device_get(save_dict), f, datapath, overwrite=overwrite)
  1105. else:
  1106. for k, tree in save_dict.items():
  1107. _savetree_hdf5(jax.device_get(tree), f, k, overwrite=overwrite)
  1108. def load_hdf5(filepath, datapath=None):
  1109. """Load a dict of pytrees from an hdf5 file.
  1110. Parameters
  1111. ----------
  1112. filepath: str
  1113. Path of the hdf5 file to load.
  1114. datapath: str, default=None
  1115. Path within the hdf5 file to load the data from. If None, the data is
  1116. loaded from the root of the hdf5 file.
  1117. Returns
  1118. -------
  1119. save_dict: dict
  1120. Dictionary where the values are pytrees, i.e. recursive collections of
  1121. tuples, lists, dicts, and numpy arrays.
  1122. """
  1123. with h5py.File(filepath, "r") as f:
  1124. if datapath is None:
  1125. return {k: _loadtree_hdf5(f[k]) for k in f}
  1126. else:
  1127. return _loadtree_hdf5(f[datapath])
  1128. def _savetree_hdf5(tree, group, name, overwrite=False):
  1129. """Recursively save a pytree to an h5 file group.
  1130. Parameters
  1131. ----------
  1132. tree: pytree
  1133. The pytree to save.
  1134. group: h5py.Group
  1135. The h5 file group to save the pytree to.
  1136. name: str
  1137. The key in 'group' where the pytree will be saved.
  1138. overwrite: bool, default=False
  1139. If True, will overwrite an existing dataset or group.
  1140. """
  1141. assert not (
  1142. not overwrite and name in group
  1143. ), f"{name} already exists in {group}. Set overwrite to True to overwrite data in an existing file."
  1144. if name in group:
  1145. del group[name]
  1146. if isinstance(tree, np.ndarray):
  1147. if tree.dtype.kind == "U":
  1148. dt = h5py.special_dtype(vlen=str)
  1149. group.create_dataset(name, data=tree.astype(object), dtype=dt)
  1150. else:
  1151. group.create_dataset(name, data=tree)
  1152. elif isinstance(tree, (float, int, str)):
  1153. group.create_dataset(name, data=tree)
  1154. else:
  1155. subgroup = group.create_group(name)
  1156. subgroup.attrs["type"] = type(tree).__name__
  1157. if isinstance(tree, (tuple, list)):
  1158. for k, subtree in enumerate(tree):
  1159. _savetree_hdf5(subtree, subgroup, f"arr{k}", overwrite=overwrite)
  1160. elif isinstance(tree, dict):
  1161. for k, subtree in tree.items():
  1162. _savetree_hdf5(subtree, subgroup, k, overwrite=overwrite)
  1163. else:
  1164. raise ValueError(f"Unrecognized type {type(tree)}")
  1165. def _loadtree_hdf5(leaf):
  1166. """Recursively load a pytree from an h5 file group."""
  1167. if isinstance(leaf, h5py.Dataset):
  1168. data = np.array(leaf[()])
  1169. if h5py.check_dtype(vlen=data.dtype) == str:
  1170. data = np.array([item.decode("utf-8") for item in data])
  1171. elif data.dtype.kind == "S":
  1172. data = data.item().decode("utf-8")
  1173. elif data.shape == ():
  1174. data = data.item()
  1175. return data
  1176. else:
  1177. leaf_type = leaf.attrs["type"]
  1178. values = map(_loadtree_hdf5, leaf.values())
  1179. if leaf_type == "dict":
  1180. return dict(zip(leaf.keys(), values))
  1181. elif leaf_type == "list":
  1182. return list(values)
  1183. elif leaf_type == "tuple":
  1184. return tuple(values)
  1185. else:
  1186. raise ValueError(f"Unrecognized type {leaf_type}")

io.py at commit d676308, under other · at the source

Overview

Authors: M. Natalie Davis1,2,3, Mary Bullock1,2,3, Aanishaa Jhaldiyal1,2,3, Niya Holifield4, Karim Mikhail1,2,3, Yuliya Voskobiynyk1,2,3, Rachael M. Vollmer1,2,3, George C. Prendergast5, Kelli Lauderdale6, Jorge J. Palop6, Karen L. Gamble3,4, Erik D. Roberson1,2,3
  1. Department of Neurology, Heersink School of Medicine, The University of Alabama at Birmingham, Birmingham, Alabama 35294
  2. Killion Center for Neurodegeneration and Experimental Therapeutics, The University of Alabama at Birmingham, Birmingham, Alabama 35294
  3. Alzheimer’s Disease Center, The University of Alabama at Birmingham, Birmingham, Alabama 35294
  4. Department of Psychiatry, Heersink School of Medicine, The University of Alabama at Birmingham, Birmingham, Alabama 35294
  5. Lankenau Institute for Medical Research, Wynnewood, Pennsylvania 19096
  6. Gladstone Institute of Neurological Disease, San Francisco, California 94158
Institutions: University of Alabama at Birmingham (United States); UAB Medicine (United States); Lankenau Institute for Medical Research (United States); Gladstone Institutes (United States)
Journal: eNeuro, volume 13, issue 3, pages ENEURO.0304-25.2026
Dates: received 28 July 2025; accepted 26 February 2026; published online 25 March 2026; in print March 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1523/eneuro.0304-25.2026 · PMID 41895857 · PMCID PMC13064429 · OpenAlex W7141567197
Open access: gold, a free copy (OpenAlex)
Status: code verified (how the code is found and verified)
Categories: EEG (modality), mouse (organism), Alzheimer's / dementia (population), cellular / molecular (subfield)
Methods: Smoothing, state filtering, decompositions, Statistics, Preprocessing, Spectral & time-frequency, Connectivity, fMRI & imaging, Physiology & signal measures
Keywords: Alzheimer's disease, BIN1, network hyperexcitability, parvalbumin
MeSH: Adaptor Proteins, Signal Transducing*, Alzheimer Disease*, Brain*, Cognition*, Neurons*, Nuclear Proteins*, Parvalbumins*, Tumor Suppressor Proteins*, Animals, Disease Models, Animal, Electroencephalography, Exploratory Behavior, Female, Male, Mice, Mice, Knockout, Nerve Tissue Proteins (* major topic)
Journal subjects: Research Article: Negative Results, Disorders of the Nervous System
Topic: Neuroscience and Neuropharmacology Research (Cellular and Molecular Neuroscience, Neuroscience), according to OpenAlex
Funding: HHS | National Institutes of Health (NIH) (RF1AG059405, R01AG061785, P30AG086401, T32HD071866, T32NS061788)
Citations: not cited yet (Europe PMC); 65 references in the paper
Research resources: NeuN RRID:AB_11205760, anti-GAPDH RRID:AB_2107445, BIN1 RRID:AB_2243396, parvalbumin RRID:AB_2927389, immunoblotted with anti-BIN1 99D RRID:AB_309738, RRID:IMSR_JAX:00377, 2007) were crossed with PV-Cre mice RRID:IMSR_JAX:017320, a small group of hAPPJ20 mice RRID:MMRRC_034836-JAX, This model was run in RStudio RRID:SCR_000432, EthoVision XT RRID:SCR_000441, The Spike2 Software RRID:SCR_000903, RRID:SCR_002798, RRID:SCR_007358, RRID:SCR_013715, RRID:SCR_014296, RRID:SCR_014574, RRID:SCR_017107, RRID:SCR_020317, RRID:SCR_021391, RRID:SCR_021496, RRID:SCR_022918, RRID:SCR_023765, RRID:SCR_024783, RRID:SCR_025032, RRID:SCR_025401

Abstract

Bridging integrator 1 (BIN1) is one of the strongest genetic risk factors for Alzheimer’s disease (AD), yet its function in the brain and role in AD remain unclear. Neuronal BIN1 isoform levels are decreased in AD, and recent data show an important role of BIN1 in inhibitory neurons. Inhibitory neurons are key regulators of cognition and network excitability, with parvalbumin-expressing (PV) neurons as the most abundant subtype. We tested the hypothesis that loss of BIN1 from PV neurons contributes to AD-related cognitive dysfunction and network hyperexcitability. We generated a cell type–specific conditional knock-out mouse line, Bin1-pvKO, and examined mice of both sexes. These mice showed few behavioral differences when assessed with traditional or machine learning–based behavioral tests, with only a slight reduction in exploratory behavior in aged cohorts. Bin1-pvKO mice showed no significant differences in network excitability on measures of induced seizure susceptibility and spiking on cortical electroencephalographic recordings. Finally, Bin1-pvKO mice exhibited no major differences in power spectral analysis of cortical electroencephalographic recordings, with only a modest reduction in delta power at high activity levels. These findings suggest that BIN1 loss in PV neurons alone is insufficient to drive the cognitive and network dysfunction observed in AD models. While these results do not exclude a role of BIN1 in PV neurons in AD models, if combined with a “second hit” or alterations in other cell types, they indicate that BIN1 loss in PV neurons alone does not recapitulate key AD-related phenotypes.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 2 matches between paragraphs and lines of code.

robersonlab-uab/Parvalbumin-BIN1-2026

License: none: the authors keep all their rights
State: the link answers, verified on 30 September 2026
Evidence: files inventoried
Commit: 8e5668b4783bde40cf095fb9b1f44160edb37cf4, 20 March 2026
Size: 2 files, 0 scripts
Software Heritage: not archived
Found in: the text, “Single-molecule fluorescent in situ hybridizatio”
Holds: README
Not found: license file, CITATION.cff, environment file, tests, continuous integration, documentation
Availability: 1 check, the latest on 30 September 2026: the link answers
  • 30 September 2026: the link answers
1 file

dattalab/keypoint-moseq

License: other
State: the link answers, verified on 30 September 2026
Evidence: files inventoried
Commit: d67630857635bd9febb8cc5d46310604b01e7dc6, 9 September 2026
Languages: Python (12), Jupyter (3)
Size: 40 files, 15 scripts
Software Heritage: archived
Found in: the text, “Behavioral clustering”
Holds: README, license file, environment (setup.cfg, setup.py), tests, documentation, 3 notebooks
Not found: CITATION.cff, continuous integration
Tools: NumPy (8 files), Matplotlib (6 files), h5py (4 files), JAX (4 files), SciPy (4 files), pandas (3 files), imageio (1 file), NetworkX (1 file), Neurodata Without Borders (PyNWB, MatNWB) (1 file), OpenCV (1 file), Pillow (1 file), Plotly (1 file), scikit-learn (1 file), seaborn (1 file), statsmodels (1 file)
Availability: 1 check, the latest on 30 September 2026: the link answers
  • 30 September 2026: the link answers
17 files

Code accessibility

The code described in the paper is freely available online (https://doi.org/10.17504/protocols.io.bp2l6dzbzvqe/v1).

Reproduced under the paper's license (CC BY), from the paper cited above.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 2 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 15 scripts, each with its path and the digest of its content;
  • 2 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 30 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 12 authors, 4 keywords, 17 MeSH terms, 1 funder, 62 references, 25 RRIDs.

Cite

This paper

Davis, M. N., Bullock, M., Jhaldiyal, A., Holifield, N., Mikhail, K., Voskobiynyk, Y., Vollmer, R. M., Prendergast, G. C., Lauderdale, K., Palop, J. J., Gamble, K. L., & Roberson, E. D. (2026). Parvalbumin Neuron-Targeted Loss of Alzheimer's Disease Risk Gene &lt;i&gt;BIN1&lt;/i&gt; Is Insufficient to Drive Cognitive or Network Excitability Changes. eNeuro, 13(3), ENEURO.0304-25.2026. https://doi.org/10.1523/eneuro.0304-25.2026

BibTeX

@article{davis2026parvalbumin,
author = {Davis, M. Natalie and Bullock, Mary and Jhaldiyal, Aanishaa and Holifield, Niya and Mikhail, Karim and Voskobiynyk, Yuliya and Vollmer, Rachael M. and Prendergast, George C. and Lauderdale, Kelli and Palop, Jorge J. and Gamble, Karen L. and Roberson, Erik D.},
title = {{Parvalbumin Neuron-Targeted Loss of Alzheimer's Disease Risk Gene \&lt;i\&gt;BIN1\&lt;/i\&gt; Is Insufficient to Drive Cognitive or Network Excitability Changes}},
journal = {eNeuro},
year = {2026},
month = mar,
volume = {13},
number = {3},
pages = {ENEURO.0304--25.2026},
publisher = {Society for Neuroscience},
issn = {2373-2822},
doi = {10.1523/eneuro.0304-25.2026},
url = {https://doi.org/10.1523/eneuro.0304-25.2026},
pmid = {41895857},
pmcid = {PMC13064429}
}

RIS

TY - JOUR
AU - Davis, M. Natalie
AU - Bullock, Mary
AU - Jhaldiyal, Aanishaa
AU - Holifield, Niya
AU - Mikhail, Karim
AU - Voskobiynyk, Yuliya
AU - Vollmer, Rachael M.
AU - Prendergast, George C.
AU - Lauderdale, Kelli
AU - Palop, Jorge J.
AU - Gamble, Karen L.
AU - Roberson, Erik D.
TI - Parvalbumin Neuron-Targeted Loss of Alzheimer's Disease Risk Gene &lt;i&gt;BIN1&lt;/i&gt; Is Insufficient to Drive Cognitive or Network Excitability Changes
T2 - eNeuro
J2 - eNeuro
PY - 2026
DA - 2026/03/27
VL - 13
IS - 3
SP - ENEURO.0304
EP - 25.2026
SN - 2373-2822
PB - Society for Neuroscience
DO - 10.1523/eneuro.0304-25.2026
UR - https://doi.org/10.1523/eneuro.0304-25.2026
LA - en
ER -

CSL-JSON

{
"id": "10.1523/eneuro.0304-25.2026",
"type": "article-journal",
"title": "Parvalbumin Neuron-Targeted Loss of Alzheimer's Disease Risk Gene &lt;i&gt;BIN1&lt;/i&gt; Is Insufficient to Drive Cognitive or Network Excitability Changes",
"container-title": "eNeuro",
"author": [
{
"family": "Davis",
"given": "M. Natalie"
},
{
"family": "Bullock",
"given": "Mary"
},
{
"family": "Jhaldiyal",
"given": "Aanishaa"
},
{
"family": "Holifield",
"given": "Niya"
},
{
"family": "Mikhail",
"given": "Karim"
},
{
"family": "Voskobiynyk",
"given": "Yuliya"
},
{
"family": "Vollmer",
"given": "Rachael M."
},
{
"family": "Prendergast",
"given": "George C."
},
{
"family": "Lauderdale",
"given": "Kelli"
},
{
"family": "Palop",
"given": "Jorge J."
},
{
"family": "Gamble",
"given": "Karen L."
},
{
"family": "Roberson",
"given": "Erik D."
}
],
"container-title-short": "eNeuro",
"volume": "13",
"issue": "3",
"page": "ENEURO.0304-25.2026",
"DOI": "10.1523/eneuro.0304-25.2026",
"PMID": "41895857",
"PMCID": "PMC13064429",
"ISSN": "2373-2822",
"publisher": "Society for Neuroscience",
"URL": "https://doi.org/10.1523/eneuro.0304-25.2026",
"language": "en",
"issued": {
"date-parts": [
[
2026,
3,
27
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s41592-024-02318-2 [code]
Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics.
Journal: Nature methods
In common: Neurodata Without Borders (PyNWB, MatNWB), JAX, imageio, 12 other tools, mouse, 3 references
[2] doi:10.1186/s13195-026-02103-7 [code]
BIN1 gain-of-function in the presynaptic compartment leads to isoform-specific synaptotoxicity.
Journal: Alzheimer's research & therapy
In common: h5py, pandas, Matplotlib, 1 other tool, Alzheimer's / dementia, cellular / molecular, 11 references
[3] doi:10.1016/j.neuron.2025.11.021 [code]
Spontaneous behavior is a succession of self-directed tasks.
Journal: Neuron
In common: Neurodata Without Borders (PyNWB, MatNWB), JAX, imageio, 12 other tools, mouse, 1 reference
[4] doi:10.1038/s41593-025-02183-y [code]
Neural population geometry and optimal coding of tasks with shared latent structure.
Journal: Nature neuroscience
In common: Neurodata Without Borders (PyNWB, MatNWB), JAX, imageio, 11 other tools, 2 references
[5] doi:10.1093/braincomms/fcaf443 [code]
Hippocampal and cortical activities reflect early hyperexcitability in an Alzheimer's mouse model.
Journal: Brain communications
In common: pandas, SciPy, Matplotlib, 1 other tool, Alzheimer's / dementia, mouse, 10 references
[6] doi:10.1016/j.celrep.2026.117505 [code]
Impaired spatial coding and neuronal hyperactivity in the medial entorhinal cortex of aged APP knock-in mice.
Journal: Cell reports
In common: pandas, Matplotlib, NumPy, Alzheimer's / dementia, mouse, 10 references
[7] doi:10.1002/alz.13861 [code]
Sleep functional connectivity, hyperexcitability, and cognition in Alzheimer's disease.
Journal: Alzheimer's & dementia : the journal of the Alzheimer's Association
In common: Plotly, h5py, statsmodels, 6 other tools, Alzheimer's / dementia, EEG, 5 references
[8] doi:10.1016/j.celrep.2026.117420 [code]
Neural population dynamics of direct electrical stimulation of neocortex.
Journal: Cell reports
In common: Neurodata Without Borders (PyNWB, MatNWB), JAX, imageio, 10 other tools, mouse
[9] doi:10.7554/elife.109717 [code]
Retrosplenial cortex enables context-dependent goal-directed sensorimotor transformation.
Journal: eLife
In common: Neurodata Without Borders (PyNWB, MatNWB), imageio, OpenCV, 9 other tools, mouse, 2 references
[10] doi:10.1371/journal.pcbi.1013789 [code]
When predict can also explain: Few-shot prediction to select better neural latents.
Journal: PLoS computational biology
In common: Neurodata Without Borders (PyNWB, MatNWB), JAX, imageio, 10 other tools

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Add the badge to the code's README

OSCR: the badge of this paper

If you are an author or a maintainer of this paper's code, you can add this badge. It says what OSCR knows of the paper (its code, its validated tracing map) and links the code to this page. Paste one of these snippets under the README's title. Keep the two marks around it, so that the badge can be replaced in place and is never added twice. OSCR changes nothing in your repository.

Markdown
<!-- oscr-badge:start -->
[![OSCR](https://openscicode.org/badge/doi/10.1523/eneuro.0304-25.2026.svg)](https://openscicode.org/badge/doi/10.1523/eneuro.0304-25.2026)
<!-- oscr-badge:end -->
reStructuredText
.. oscr-badge:start

.. image:: https://openscicode.org/badge/doi/10.1523/eneuro.0304-25.2026.svg
   :target: https://openscicode.org/badge/doi/10.1523/eneuro.0304-25.2026
   :alt: OSCR

.. oscr-badge:end
HTML
<!-- oscr-badge:start -->
<a href="https://openscicode.org/badge/doi/10.1523/eneuro.0304-25.2026"><img src="https://openscicode.org/badge/doi/10.1523/eneuro.0304-25.2026.svg" alt="OSCR"></a>
<!-- oscr-badge:end -->

On GitHub, open the README, choose Edit (the pencil), paste the snippet under the title, then "Commit changes".

If you choose to, you can have the badge kept up to date for you, with oscr badge --write from the command line or with the GitHub Action in your repository's workflows. Every other form and option (a card, AsciiDoc, LaTeX, shields.io) is on the badge page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: code errors and mismatches between the code and the paper about this paper and its code, from readers and its authors, beside the paper and its tracing map.

Report a mismatch between the code and the paper · Report a code error (signed in)

Reproductions: reports from readers who ran the authors' code and did not get the paper's result, what they ran, with which environment, commit and data.

Report a reproduction failure (signed in)

Versions of its code: the releases of its code tied to a version of this paper (the preprint, the accepted manuscript, the version of record…), each with the tracing map versioned with it.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.