OSCR

Measuring surprisal in sound sequences.

Code ↔ Paper

2 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

2 blocks of code here have a permanent identifier (oscr:b:…), listed with the matches below: citable for good, with a page of its own. Select a match and press Cite to copy it. What a block identifier is.

The 2 matches
  1. [1] § General discussion ↔ analysis/SSM.R, lines 1 to 106 · score 0.54 · auditory spectrograms, syllable, tune, STFT, segmentation, SSM · block oscr:b:33EVVKCJZ1WZY-W · Report a mismatch
  2. [2] § General discussion ↔ analysis/surprisal.R, lines 421 to 504 · score 0.50 · amplitude envelope, auditory spectrograms, tune, windows, surprisal · block oscr:b:GRBT6VW1Z4C30-A · Report a mismatch

Paper

Loaded from Europe PMC by your browser; OSCR does not store it: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

R · 692 lines · 24 KB · no license · 1 match

  1. #' Self-similarity matrix
  2. #'
  3. #' Calculates the self-similarity matrix and novelty vector of a sound. The
  4. #' self-similarity matrix is produced by cross-correlating different segments of
  5. #' the input sound. Novelty is calculated by convolving the self-similarity
  6. #' matrix with a tapered checkerboard kernel. The positive lobes of the kernel
  7. #' represent coherence (self-similarity within the regions on either side of the
  8. #' center point) and the negative lobes anti-coherence (cross-similarity
  9. #' between these two regions). Since novelty is the dot product of the
  10. #' checkerboard kernel with the SSM, it is high when the two regions are
  11. #' self-similar (internally consistent) but different from each other.
  12. #'
  13. #' @seealso \code{\link{spectrogram}} \code{\link{modulationSpectrum}}
  14. #' \code{\link{segment}}
  15. #'
  16. #' @references \itemize{
  17. #' \item Foote, J. (1999, October). Visualizing music and
  18. #' audio using self-similarity. In Proceedings of the seventh ACM
  19. #' international conference on Multimedia (Part 1) (pp. 77-80). ACM.
  20. #' \item
  21. #' Foote, J. (2000). Automatic audio segmentation using a measure of audio
  22. #' novelty. In Multimedia and Expo, 2000. ICME 2000. 2000 IEEE International
  23. #' Conference on (Vol. 1, pp. 452-455). IEEE.
  24. #' }
  25. #' @inheritParams spectrogram
  26. #' @inheritParams analyze
  27. #' @param sparse if TRUE, the entire SSM is not calculated, but only the central
  28. #' region needed to extract the novelty contour (speeds up the processing)
  29. #' @param input the spectral representation used to calculate the SSM: "audSpec"
  30. #' = auditory spectrogram returned by \code{\link{audSpectrogram}}, "mfcc" =
  31. #' Mel-Frequency Cepstral coefficients, "melspec" = Mel-transformed STFT
  32. #' spectrogram, "spec" = STFT power spectrogram (all three returned by
  33. #' \code{\link[tuneR]{melfcc}}). Any custom spectrogram-like matrix of
  34. #' features (time in columns labeled in s, features in rows) is also accepted
  35. #' (see examples)
  36. #' @param takeLog if TRUE, the input is log-transformed prior to calculating
  37. #' self-similarity
  38. #' @param MFCC which mel-frequency cepstral coefficients to use; defaults to
  39. #' \code{2:13}
  40. #' @param melfcc_pars a list of parameters passed to \code{\link[tuneR]{melfcc}}
  41. #' @param audSpec_pars a list of parameters passed to
  42. #' \code{\link{audSpectrogram}} (if input = 'audSpec')
  43. #' @param norm if TRUE, the spectrum of each STFT frame is normalized
  44. #' @param simil method for comparing frames: "cosine" = cosine similarity, "cor"
  45. #' = Pearson's correlation
  46. #' @param kernelLen length of checkerboard kernel for calculating novelty, ms
  47. #' (larger values favor global, slow vs. local, fast novelty)
  48. #' @param kernelSD SD of checkerboard kernel for calculating novelty
  49. #' @param padWith how to treat edges when calculating novelty: NA = treat sound
  50. #' before and after the recording as unknown, 0 = treat it as silence
  51. #' @param ssmWin window for averaging SSM, frames (has a smoothing effect and
  52. #' speeds up the processing)
  53. #' @param output what to return (drop "ssm" to save memory when analyzing a lot
  54. #' of files)
  55. #' @param plot if TRUE, plots the SSM
  56. #' @param heights relative sizes of the SSM and spectrogram/novelty plot
  57. #' @param main plot title
  58. #' @param specPars graphical parameters passed to \code{filled.contour.mod} and
  59. #' affecting the \code{\link{spectrogram}}
  60. #' @param ssmPars graphical parameters passed to \code{filled.contour.mod} and
  61. #' affecting the plot of SSM
  62. #' @param noveltyPars graphical parameters passed to
  63. #' \code{\link[graphics]{lines}} and affecting the novelty contour
  64. #' @return Returns a list of two components: $ssm contains the self-similarity
  65. #' matrix, and $novelty contains the novelty vector.
  66. #' @export
  67. #' @examples
  68. #' sound = c(soundgen(),
  69. #' soundgen(nSyl = 4, sylLen = 50, pauseLen = 70,
  70. #' formants = NA, pitch = c(500, 330)))
  71. #' # playme(sound)
  72. #' # detailed, local features (captures each syllable)
  73. #' s1 = ssm(sound, samplingRate = 16000, kernelLen = 100,
  74. #' sparse = TRUE) # much faster with 'sparse'
  75. #' # more global features (captures the transition b/w the two sounds)
  76. #' s2 = ssm(sound, samplingRate = 16000, kernelLen = 400, sparse = TRUE)
  77. #'
  78. #' s2$summary
  79. #' s2$novelty # novelty contour
  80. #' \dontrun{
  81. #' ssm(sound, samplingRate = 16000,
  82. #' input = 'mfcc', simil = 'cor', norm = TRUE,
  83. #' ssmWin = 10, # speed up the processing
  84. #' kernelLen = 300, # global features
  85. #' specPars = list(colorTheme = 'seewave'),
  86. #' ssmPars = list(col = rainbow(100)),
  87. #' noveltyPars = list(type = 'l', lty = 3, lwd = 2))
  88. #'
  89. #' # Custom input: produce a nice spectrogram first, then feed it into ssm()
  90. #' sp = spectrogram(sound, 16000, windowLength = c(5, 40), contrast = .3,
  91. #' output = 'processed') # return the modified spectrogram
  92. #' colnames(sp) = as.numeric(colnames(sp)) / 1000 # convert ms to s
  93. #' ssm(sound, 16000, kernelLen = 400, input = sp)
  94. #'
  95. #' # Custom input: use acoustic features returned by analyze()
  96. #' an = analyze(sound, 16000, windowLength = 20, novelty = NULL)
  97. #' input_an = t(an$detailed[, 4:ncol(an$detailed)]) # or select pitch, HNR, ...
  98. #' input_an = t(apply(input_an, 1, scale)) # z-transform all variables
  99. #' input_an[is.na(input_an)] = 0 # get rid of NAs
  100. #' colnames(input_an) = an$detailed$time / 1000 # time stamps in s
  101. #' rownames(input_an) = 1:nrow(input_an)
  102. #' image(t(input_an)) # not a spectrogram, just a feature matrix
  103. #' ssm(sound, 16000, kernelLen = 500, input = input_an, takeLog = FALSE,
  104. #' specPars = list(ylab = 'Feature'))
  105. #' }
  106. ssm = function(
  107. x,
  108. samplingRate = NULL,
  109. from = NULL,
  110. to = NULL,
  111. sparse = FALSE,
  112. input = c('melspec', 'mfcc', 'spec', 'audSpec')[1],
  113. melfcc_pars = list(windowLength = 125, step = 25, nbands = 50),
  114. MFCC = 2:13,
  115. audSpec_pars = list(nFilters = 16, step = 10),
  116. takeLog = FALSE,
  117. norm = FALSE,
  118. simil = c('cosine', 'cor')[1],
  119. kernelLen = 1000,
  120. kernelSD = .5,
  121. padWith = 0,
  122. ssmWin = NULL,
  123. summaryFun = c('mean', 'sd'),
  124. output = c('ssm', 'novelty', 'summary'),
  125. reportEvery = NULL,
  126. cores = 1,
  127. plot = TRUE,
  128. savePlots = NULL,
  129. main = NULL,
  130. heights = c(2, 1),
  131. width = 900,
  132. height = 500,
  133. units = 'px',
  134. res = NA,
  135. specPars = list(
  136. colorTheme = c('bw', 'seewave', 'heat.colors', '...')[2],
  137. xlab = 'Time, s'
  138. ),
  139. ssmPars = list(
  140. colorTheme = c('bw', 'seewave', 'heat.colors', '...')[2],
  141. xlab = 'Time, s',
  142. ylab = 'Time, s'
  143. ),
  144. noveltyPars = list(
  145. type = 'b',
  146. pch = 16,
  147. col = 'black',
  148. lwd = 3
  149. )) {
  150. ## Prepare a list of arguments to pass to .ssm()
  151. myPars = as.list(environment())
  152. # exclude unnecessary args
  153. myPars = myPars[!names(myPars) %in% c(
  154. 'x', 'samplingRate', 'from', 'to', 'savePlots', 'reportEvery', 'cores',
  155. 'summaryFun', 'specPars', 'ssmPars', 'noveltyPars', 'ssmWin',
  156. 'melfcc_pars', 'audSpec_pars')]
  157. myPars$specPars = specPars
  158. myPars$ssmPars = ssmPars
  159. myPars$noveltyPars = noveltyPars
  160. myPars$melfcc_pars = melfcc_pars
  161. myPars$audSpec_pars = audSpec_pars
  162. myPars$win = ssmWin
  163. # analyze
  164. pa = processAudio(
  165. x,
  166. samplingRate = samplingRate,
  167. from = from,
  168. to = to,
  169. funToCall = '.ssm',
  170. myPars = myPars,
  171. reportEvery = reportEvery,
  172. cores = cores,
  173. savePlots = savePlots
  174. )
  175. # htmlPlots
  176. if (!is.null(pa$input$savePlots) && pa$input$n > 1) {
  177. try(htmlPlots(pa$input, savePlots = savePlots, changesAudio = FALSE,
  178. suffix = "ssm", width = paste0(width, units)))
  179. }
  180. # prepare output
  181. if (!is.null(summaryFun) && any(!is.na(summaryFun))) {
  182. temp = vector('list', pa$input$n)
  183. for (i in 1:pa$input$n) {
  184. if (!pa$input$failed[i]) {
  185. temp[[i]] = summarizeAnalyze(
  186. data.frame(novelty = pa$result[[i]]$novelty),
  187. summaryFun = summaryFun,
  188. var_noSummary = NULL)
  189. }
  190. }
  191. idx_failed = which(pa$input$failed)
  192. if (length(idx_failed) > 0) {
  193. idx_ok = which(!pa$input$failed)
  194. if (length(idx_ok) > 0) {
  195. filler = temp[[idx_ok[1]]] [1, ]
  196. filler[1, ] = NA
  197. } else {
  198. stop('Failed to analyze any input')
  199. }
  200. for (i in idx_failed) temp[[i]] = filler
  201. }
  202. mysum_all = cbind(data.frame(file = pa$input$filenames_base),
  203. do.call('rbind', temp))
  204. } else {
  205. mysum_all = NULL
  206. }
  207. if (pa$input$n == 1) {
  208. # unlist
  209. ssm = pa$result[[1]]$ssm
  210. novelty = pa$result[[1]]$novelty
  211. } else {
  212. ssm = lapply(pa$result, function(x) x[['ssm']])
  213. novelty = lapply(pa$result, function(x) x[['novelty']])
  214. }
  215. out = list(ssm = ssm, novelty = novelty, summary = mysum_all)
  216. invisible(out[which(names(out) %in% output)])
  217. }
  218. #' SSM per sound
  219. #'
  220. #' Internal soundgen function.
  221. #' @inheritParams ssm
  222. #' @param audio a list returned by \code{readAudio}
  223. #' @keywords internal
  224. .ssm = function(
  225. audio,
  226. sparse = FALSE,
  227. input = c('melspec', 'mfcc', 'spec', 'audSpec')[1],
  228. melfcc_pars = list(windowLength = 125, step = 25, nbands = 50),
  229. MFCC = 2:13,
  230. audSpec_pars = list(nFilters = 16, step = 10),
  231. takeLog = FALSE,
  232. norm = FALSE,
  233. simil = c('cosine', 'cor')[1],
  234. kernelLen = 100,
  235. kernelSD = .5,
  236. padWith = 0,
  237. win = 1,
  238. output = c('ssm', 'novelty', 'summary'),
  239. plot = TRUE,
  240. main = NULL,
  241. heights = c(2, 1),
  242. width = 900,
  243. height = 500,
  244. units = 'px',
  245. res = NA,
  246. specPars = list(
  247. colorTheme = c('bw', 'seewave', 'heat.colors', '...')[2],
  248. xlab = 'Time, s',
  249. ylab = 'kHz'
  250. ),
  251. ssmPars = list(
  252. colorTheme = c('bw', 'seewave', 'heat.colors', '...')[2],
  253. xlab = 'Time, s',
  254. ylab = 'Time, s'
  255. ),
  256. noveltyPars = list(
  257. type = 'b',
  258. pch = 16,
  259. col = 'black',
  260. lwd = 3
  261. )) {
  262. nyquist = audio$samplingRate / 2
  263. if (is.matrix(input)) {
  264. # custom input to SSM - use as is
  265. target_spec = as.matrix(input)
  266. step = diff(as.numeric(colnames(target_spec))[1:2])
  267. frame_points = round(audio$samplingRate * step)
  268. input = 'custom'
  269. } else {
  270. ## compute mel-filtered spectrum and MFCCs
  271. if (input == 'audSpec') {
  272. # call audSpectrogram()
  273. if (is.null(audSpec_pars$nFilters)) audSpec_pars$nFilters = 32
  274. if (is.null(audSpec_pars$step)) audSpec_pars$step = 20
  275. frame_points = round(audio$samplingRate * audSpec_pars$step / 1000)
  276. target_spec = do.call(.audSpectrogram, c(audSpec_pars, list(
  277. audio = audio,
  278. plot = FALSE
  279. )))$audSpec # cols = time, rows = freq
  280. colnames(target_spec) = as.numeric(colnames(target_spec)) / 1000 # ms to s
  281. } else if (input %in% c('mfcc', 'spec', 'melspec')) {
  282. # call tuneR::melfcc()
  283. if (is.null(melfcc_pars$windowLength)) melfcc_pars$windowLength = 25
  284. if (is.null(melfcc_pars$step)) melfcc_pars$step = 5
  285. frame_points = round(audio$samplingRate * melfcc_pars$step / 1000)
  286. if (!is.numeric(melfcc_pars$windowLength) | melfcc_pars$windowLength <= 0 |
  287. melfcc_pars$windowLength > (audio$duration / 2 * 1000)) {
  288. melfcc_pars$windowLength = min(50, round(audio$duration / 2 * 1000))
  289. warning(paste0(
  290. '"windowLength" must be between 0 and half the sound duration (in ms);
  291. resetting to ', melfcc_pars$windowLength, ' ms')
  292. )
  293. }
  294. if (is.null(melfcc_pars$step))
  295. melfcc_pars$step = melfcc_pars$windowLength / 4
  296. if (is.null(melfcc_pars$nbands)) {
  297. melfcc_pars$nbands = round(100 * melfcc_pars$windowLength / 20)
  298. }
  299. windowLength_points = floor(melfcc_pars$windowLength / 1000 *
  300. audio$samplingRate / 2) * 2
  301. if (is.null(melfcc_pars$maxfreq)) {
  302. melfcc_pars$maxfreq = floor(audio$samplingRate / 2) # Nyquist
  303. }
  304. sound = tuneR::Wave(left = audio$sound, samp.rate = audio$samplingRate, bit = 16)
  305. mel = do.call(tuneR::melfcc, c(
  306. melfcc_pars[which(!names(melfcc_pars) %in% c('windowLength', 'step'))],
  307. list(
  308. samples = sound,
  309. wintime = melfcc_pars$windowLength / 1000,
  310. hoptime = melfcc_pars$step / 1000,
  311. spec_out = TRUE,
  312. numcep = max(MFCC)
  313. )))
  314. if (input == 'mfcc') {
  315. # the first cepstrum presumably makes no sense with amplitude normalization
  316. # (?), and it overestimates the similarity of different frames
  317. target_spec = t(mel$cepstra)[MFCC, ]
  318. target_spec[is.na(target_spec)] = 0 # MFCC are NaN for silent frames
  319. colnames(target_spec) = seq(audio$timeShift, audio$duration,
  320. length.out = ncol(target_spec))
  321. rownames(target_spec) = 1:nrow(target_spec)
  322. } else if (input == 'melspec') {
  323. target_spec = t(mel$aspectrum) # cols = time, rows = freq
  324. colnames(target_spec) = seq(audio$timeShift, audio$duration,
  325. length.out = ncol(target_spec))
  326. rownames(target_spec) = otherToHz(
  327. seq(0, HzToOther(nyquist, "mel"),
  328. length.out = nrow(target_spec)), "mel") / 1000
  329. } else if (input == 'spec') {
  330. target_spec = t(mel$pspectrum) # cols = time, rows = freq
  331. colnames(target_spec) = seq(audio$timeShift, audio$duration,
  332. length.out = ncol(target_spec))
  333. rownames(target_spec) = seq(0, nyquist,
  334. length.out = nrow(target_spec)) / 1000
  335. }
  336. }
  337. }
  338. if (takeLog) {
  339. target_spec = target_spec - min(target_spec, na.rm = TRUE)
  340. target_spec = log(target_spec + min(target_spec[target_spec > 0], na.rm = TRUE))
  341. }
  342. # image(t(target_spec))
  343. ## compute self-similarity matrix
  344. # kernel size in frames, guaranteed to be even
  345. kernelSize = max(4, round(kernelLen * audio$samplingRate / 1000 /
  346. frame_points / 2) * 2)
  347. s = selfsim(
  348. m = target_spec,
  349. norm = norm,
  350. simil = simil,
  351. win = win,
  352. sparse = sparse,
  353. kernelSize = kernelSize
  354. )
  355. # s = zeroOne(s^2) # hist(s)
  356. # image(s)
  357. ## compute novelty
  358. novelty = getNovelty(ssm = s, kernelSize = kernelSize,
  359. kernelSD = kernelSD, padWith = padWith)
  360. ## PLOTTING
  361. if (is.character(audio$savePlots)) {
  362. plot = TRUE
  363. png(filename = paste0(audio$savePlots, audio$filename_noExt, "_ssm.png"),
  364. width = width, height = height, units = units, res = res)
  365. }
  366. if (plot) {
  367. if (is.null(main)) {
  368. if (audio$filename_base == 'sound') {
  369. main = ''
  370. } else {
  371. main = audio$filename_base
  372. }
  373. }
  374. op = par(c('mar', 'xaxt', 'yaxt', 'mfrow')) # save user's original pars
  375. layout(matrix(c(2, 1), nrow = 2, byrow = TRUE), heights = heights)
  376. par(mar = c(5.1, 4.1, 0, 2.1),
  377. xaxt = 's',
  378. yaxt = 's')
  379. # spectrogram
  380. if (input == 'audSpec') {
  381. # spec = zeroOne(t(log(target_spec + 1e-4)))
  382. spec = t(target_spec)
  383. specPars1 = list(
  384. colorTheme = 'seewave',
  385. xlab = 'Time, s',
  386. ylab = 'kHz'
  387. )
  388. specPars1[names(specPars)] = specPars
  389. specPars1$color.palette = switchColorTheme(specPars1$colorTheme)
  390. specPars1[['colorTheme']] = NULL
  391. do.call(filled.contour.mod, c(list(
  392. x = as.numeric(rownames(spec)),
  393. y = as.numeric(colnames(spec)),
  394. z = spec,
  395. yScale = if (is.null(audSpec_pars$yScale)) 'bark' else audSpec_pars$yScale
  396. ), specPars1))
  397. specPars1$xlim = range(as.numeric(rownames(spec)))
  398. specPars1$ylim = range(as.numeric(colnames(spec)))
  399. } else {
  400. # # log-transform and normalize spectrogram
  401. # if (input == 'melspec') {
  402. # spec = log(zeroOne(mel$aspectrum) + 1e-4) # dynamic range ~ 80 dB or 1e-4
  403. # } else {
  404. # spec = log(zeroOne(mel$pspectrum) + 1e-4)
  405. # }
  406. # spec = zeroOne(spec)
  407. spec = t(target_spec)
  408. specPars1 = list(
  409. colorTheme = 'seewave',
  410. xlab = 'Time, s',
  411. ylab = if (input %in% c('custom', 'mfcc')) input else 'kHz'
  412. )
  413. specPars1[names(specPars)] = specPars
  414. specPars1$color.palette = switchColorTheme(specPars1$colorTheme)
  415. specPars1[['colorTheme']] = NULL
  416. do.call(filled.contour.mod, c(list(
  417. x = as.numeric(rownames(spec)),
  418. y = as.numeric(colnames(spec)),
  419. z = spec,
  420. yScale = if (input == 'melspec') 'mel' else 'linear'
  421. ), specPars1
  422. ))
  423. }
  424. specPars1$xlim = c(audio$timeShift, audio$duration)
  425. specPars1$ylim = range(as.numeric(rownames(target_spec)))
  426. if (input == 'melspec')
  427. specPars1$ylim = HzToOther(specPars1$ylim * 1000, 'mel')
  428. # novelty
  429. noveltyPars1 = list(
  430. type = 'b',
  431. pch = 16,
  432. col = 'black',
  433. lwd = 3
  434. )
  435. noveltyPars1[names(noveltyPars)] = noveltyPars
  436. do.call(lines, c(list(
  437. x = seq(specPars1$xlim[1], specPars1$xlim[2], length.out = length(novelty)),
  438. y = zeroOne(novelty) * specPars1$ylim[2] * .95
  439. ), noveltyPars1
  440. ))
  441. axis(side = 1, labels = TRUE)
  442. par(mar = c(0, 4.1, 2.1, 2.1),
  443. xaxt = 'n',
  444. yaxt = 's')
  445. xlab = ''
  446. # SSM
  447. ssmPars1 = list(
  448. levels = seq(0, 1, length = 30),
  449. colorTheme = 'seewave',
  450. xlab = 'Time, s',
  451. ylab = 'Time, s',
  452. main = main
  453. )
  454. ssmPars1[names(ssmPars)] = ssmPars
  455. ssmPars1$color.palette = switchColorTheme(ssmPars1$colorTheme)
  456. ssmPars1[['colorTheme']] = NULL
  457. timestamps_ssm = seq(0, audio$duration, length.out = nrow(s))
  458. do.call(filled.contour.mod, c(list(
  459. x = timestamps_ssm,
  460. y = timestamps_ssm,
  461. z = s,
  462. y_Hz = FALSE
  463. ), ssmPars1
  464. ))
  465. # restore original pars
  466. par('mar' = op$mar, 'xaxt' = op$xaxt, 'yaxt' = op$yaxt, 'mfrow' = op$mfrow)
  467. if (is.character(audio$savePlots)) dev.off()
  468. }
  469. out = list(ssm = s, novelty = novelty)
  470. invisible(out[which(names(out) %in% output)])
  471. }
  472. #' Compute self-similarity
  473. #'
  474. #' Internal soundgen function.
  475. #'
  476. #' Called by \code{\link{ssm}}.
  477. #' @param m input matrix such as a spectrogram
  478. #' @inheritParams ssm
  479. #' @param win the length of window for averaging self-similarity, frames
  480. #' @return Returns a square self-similarity matrix.
  481. #' @keywords internal
  482. #' @examples
  483. #' m = matrix(rnorm(40), nrow = 5)
  484. #' soundgen:::selfsim(m, sparse = TRUE, kernelSize = 2)
  485. selfsim = function(m,
  486. norm = FALSE,
  487. simil = c('cosine', 'cor')[1],
  488. win = 1,
  489. sparse = FALSE,
  490. kernelSize = NULL) {
  491. nc = ncol(m)
  492. if (win > floor(nc / 2)) {
  493. win = floor(nc / 2)
  494. warning(paste('"win" must be smaller than half the number of frames',
  495. 'resetting to', floor(nc / 2)))
  496. }
  497. if (win %% 2 == 0) {
  498. win = max(ceiling(win / 2) * 2 - 1, 1)
  499. } # win must be odd
  500. # normalize input by column, if needed
  501. if (norm) {
  502. m = apply(m, 2, zeroOne, na.rm = TRUE)
  503. m[is.na(m)] = 0
  504. }
  505. # calculate windows for averaging self-similarity
  506. winIdx = unique(round(seq(1, nc - win + 1, length.out = ceiling(nc / win))))
  507. numWins = length(winIdx)
  508. # calculate the lower triangle of self-similarity matrix
  509. out = matrix(NA, nrow = numWins, ncol = numWins)
  510. rownames(out) = colnames(out) = winIdx
  511. if (!sparse) j_idx = seq_len(numWins)
  512. for (i in seq_along(winIdx)) {
  513. if (sparse) {
  514. j_idx = max(1, i - kernelSize) : max(1, (i - 1))
  515. } else {
  516. j_idx = 1:max(1, (i - 1))
  517. }
  518. for (j in j_idx) {
  519. mi = as.vector(m[, winIdx[i]:(winIdx[i] + win - 1)])
  520. mj = as.vector(m[, winIdx[j]:(winIdx[j] + win - 1)])
  521. if (any(mi != 0) && any(mj != 0)) {
  522. if (simil == 'cosine') {
  523. # http://stackoverflow.com/questions/6597005/cosine-similarity-between-two-vectors-in-language-r
  524. out[i, j] = crossprod(mi, mj) / sqrt(crossprod(mi) * crossprod(mj))
  525. } else if (simil == 'cor') {
  526. out[i, j] = cor(mi, mj)
  527. }
  528. } else {
  529. # if at least one is a vector of zeros, set result to 0 (otherwise NA)
  530. out[i, j] = 0
  531. }
  532. }
  533. }
  534. # fill up the upper triangle as well
  535. diag(out) = 1
  536. out1 = t(out)
  537. out1[lower.tri(out1)] = out[lower.tri(out)]
  538. # isSymmetric(out1)
  539. # image(out1)
  540. zeroOne(t(out1), na.rm = TRUE)
  541. }
  542. #' Checkerboard kernel
  543. #'
  544. #' Internal soundgen function.
  545. #'
  546. #' Prepares a square matrix \code{size x size} specifying a gaussian kernel for
  547. #' measuring novelty of self-similarity matrices. Called by
  548. #' \code{\link{getNovelty}}
  549. #' @param size kernel size (points), preferably an even number
  550. #' @param kernel_mean,kernelSD mean and SD of the gaussian kernel
  551. #' @param plot if TRUE, shows a perspective plot of the kernel
  552. #' @param checker if TRUE, inverts two quadrants
  553. #' @return Returns a square matrix with \code{size} rows and columns.
  554. #' @keywords internal
  555. #' @examples
  556. #' kernel = soundgen:::getCheckerboardKernel(size = 64, kernelSD = 0.1, plot = TRUE)
  557. #' dim(kernel)
  558. #' kernel = soundgen:::getCheckerboardKernel(size = 19, kernelSD = .5,
  559. #' checker = FALSE, plot = TRUE)
  560. #' kernel = soundgen:::getCheckerboardKernel(size = c(9, 45), kernelSD = .5,
  561. #' checker = FALSE, plot = TRUE)
  562. #' kernel = soundgen:::getCheckerboardKernel(size = c(9, 45), kernelSD = .5,
  563. #' checker = TRUE, plot = TRUE)
  564. getCheckerboardKernel = function(size,
  565. kernel_mean = 0,
  566. kernelSD = 0.5,
  567. plot = FALSE,
  568. checker = TRUE) {
  569. if (length(size) == 1) {
  570. x = y = seq(-1, 1, length.out = size)
  571. size = c(size, size)
  572. } else if (length(size) == 2) {
  573. x = seq(-1, 1, length.out = size[1])
  574. y = seq(-1, 1, length.out = size[2])
  575. } else {
  576. stop('size must be of length 1 or 2')
  577. }
  578. kernelSD = kernelSD # just to get rid of the "unused arg" warning in CMD check :-)
  579. if (max(size) < 50) {
  580. # faster than mvtnorm::dmvnorm for small kernels
  581. kernel = matrix(NA, ncol = size[2], nrow = size[1])
  582. for (i in seq_len(nrow(kernel))) {
  583. for (j in seq_len(ncol(kernel))) {
  584. kernel[i, j] = dnorm(x[i], mean = kernel_mean, sd = kernelSD) *
  585. dnorm(y[j], mean = kernel_mean, sd = kernelSD)
  586. }
  587. }
  588. } else {
  589. # this is faster for large kernels
  590. sigma = diag(2) * kernelSD
  591. kernel_long = expand.grid(x1 = x, x2 = y)
  592. kernel_long$dd = mvtnorm::dmvnorm(x = kernel_long,
  593. mean = c(kernel_mean, kernel_mean),
  594. sigma = sigma)
  595. kernel = matrix(kernel_long$dd, nrow = size[1])
  596. # kernel[1:5, 1:5]
  597. }
  598. if (checker) {
  599. fl_row = floor(size[1] / 2)
  600. fl_col = floor(size[2] / 2)
  601. cl_row = ceiling(size[1] / 2)
  602. cl_col = ceiling(size[2] / 2)
  603. # quadrant 0 to 3 o'clock
  604. kernel[seq_len(fl_row), (cl_col + 1):size[2]] = -kernel[seq_len(fl_row), (cl_col + 1):size[2]]
  605. # quadrant 6 to 9 o'clock
  606. kernel[(cl_row + 1):size[1], seq_len(cl_col)] = -kernel[(cl_row + 1):size[1], seq_len(cl_col)]
  607. }
  608. kernel = kernel / max(kernel)
  609. if (plot) {
  610. persp(
  611. kernel,
  612. theta = -20,
  613. phi = 25,
  614. # zlim = c(-1, 4),
  615. ticktype = 'detailed'
  616. )
  617. }
  618. kernel
  619. }
  620. #' SSM novelty
  621. #'
  622. #' Internal soundgen function.
  623. #'
  624. #' Calculates novelty in a self-similarity matrix. Called by \code{\link{ssm}}.
  625. #' @param ssm self-similarity matrix, as produced by \code{\link{selfsim}}
  626. #' @param kernelSize the size of gausisan kernel (points)
  627. #' @param kernelSD the SD of gaussian kernel
  628. #' @param normalize if TRUE, normalizes so that max = 1
  629. #' @return Returns a numeric vector of length \code{nrow(ssm)}
  630. #' @keywords internal
  631. getNovelty = function(ssm,
  632. kernelSize,
  633. kernelSD,
  634. padWith = 0,
  635. normalize = TRUE) {
  636. kernel = getCheckerboardKernel(size = kernelSize, kernelSD = kernelSD)
  637. ## pad matrix with size / 2 zeros, so that we can correlate it with the
  638. # kernel starting from the very edge
  639. ssm_padded = matrix(padWith,
  640. nrow = nrow(ssm) + kernelSize,
  641. ncol = nrow(ssm) + kernelSize)
  642. halfK = kernelSize / 2
  643. # indices in the padded matrix where we'll paste the original ssm
  644. idx = c(halfK + 1, nrow(ssm_padded) - halfK)
  645. # paste original. Now we have a padded ssm
  646. ssm_padded[idx[1]:idx[2], idx[1]:idx[2]] = ssm
  647. ## get novelty
  648. novelty = rep(0, nrow(ssm))
  649. # for each point on the main diagonal, novelty = correlation between the
  650. # checkerboard kernel and the ssm. See Badawy, "Audio novelty-based
  651. # segmentation of music concerts"
  652. for (i in idx[1]:idx[2]) {
  653. n = (i - halfK):(i + halfK - 1)
  654. # suppress warnings, b/c otherwise cor complains of sd = 0 for silent segments
  655. mat_i = ssm_padded[n, n]
  656. diag(mat_i) = NA
  657. novelty[i - halfK] = suppressWarnings(
  658. cor(as.vector(mat_i), as.vector(kernel), use = 'pairwise.complete.obs')
  659. )
  660. }
  661. novelty[is.na(novelty)] = 0
  662. novelty
  663. }

SSM.R, no license · at the source

Overview

Authors: Andrey Anikin1
ORCID iDs: Andrey Anikin
  1. Division of Cognitive Science, Department of Philosophy, Lund University, Box 192, SE-221 00 Lund, Sweden
Institutions: Lund University (Sweden)
Journal: Behavior research methods, volume 58, issue 10, article 278
Dates: received 13 February 2026; accepted 21 July 2026; published online 24 August 2026; in print 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.3758/s13428-026-03153-3 · PMID 42637973 · PMCID PMC13503466 · OpenAlex W7128612809
Open access: hybrid, a free copy (OpenAlex)
Preprint: osf.io/bgzvc
Status: code verified (how the code is found and verified)
Categories: human (organism)
Keywords: Auditory attention, Salience, Shannon surprisal, Bayesian surprise, Self-similarity
MeSH: Acoustic Stimulation*, Auditory Perception*, Sound*, Algorithms, Animals, Attention, Bayes Theorem, Humans, Neural Networks, Computer (* major topic)
Topic: Hearing Loss and Rehabilitation (Speech and Hearing, Health Professions), according to OpenAlex
Funding: Vetenskapsrådet (2023-00850)
Citations: not cited yet (Europe PMC); 68 references in the paper

Abstract

Sensory input that violates prior expectations attracts attention, making unpredictability an important perceptual property to measure. In the auditory modality, knowing what sounds will be perceived as surprising, and therefore salient, is relevant both for studying vocal communication and for applied purposes such as managing noise pollution. Focusing on sequences of animal vocalizations and environmental sounds as ecologically important acoustic stimuli, I describe and benchmark several algorithms for measuring their perceived unpredictability. Information-theoretical approaches include Shannon surprisal and Bayesian surprise, both implemented here to detect deviant stimuli based on distributional acoustic properties. The second group of algorithms is based on detecting spectro-temporal recurrence assessed with autocorrelation functions (ACF surprisal) and self-similarity matrices (SSM novelty). The third approach uses neural networks. Based on the ratings of the predictability of 300 synthetic acoustic sequences by 195 human listeners, Shannon surprisal and SSM novelty capture the perceived unpredictability that is due to spectral variability, whereas ACF surprisal taps into the perceptual impact of irregular rhythm. Most algorithms converge on the time scale of about 1 s as the most perceptually relevant for spectral variability, which is consistent with the hypothesis that the perception of unpredictability stems from a relatively limited amount of auditory input held in short-term memory. Together, the presented open-source algorithms offer powerful and flexible tools for measuring acoustic surprisal and studying auditory attention, while the corpus of predictability ratings offers a resource for future benchmarking. All code and data are freely available from the R package soundgen and supplementary materials at https://osf.io/bgzvc.

Supplementary Information: The online version contains supplementary material available at 10.3758/s13428-026-03153-3.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 2 matches between paragraphs and lines of code.

OSF bgzvc

License: none: the authors keep all their rights
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Languages: R (16)
Size: 54 files, 16 scripts
Software Heritage: not checked
Found in: the text, “Data analysis”
Holds: 6 notebooks
Not found: README, license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: ggplot2 (7 files), brms (5 files), patchwork (1 file), randomForest (1 file), tidyverse (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers (HTTP 200)
  • 27 September 2026: the link answers (HTTP 200)
16 files
At the source: osf.io/bgzvc

OSF kp9mg

License: none: the authors keep all their rights
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Languages: R (16)
Size: 42 files, 16 scripts
Software Heritage: not checked
Found in: the references
Holds: 5 notebooks
Not found: README, license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: brms (3 files), ggplot2 (3 files), tidyverse (3 files), reshape2 (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers (HTTP 200)
  • 27 September 2026: the link answers (HTTP 200)
16 files
At the source: osf.io/kp9mg/

Code availability

All data and materials are available at https://osf.io/bgzvc. This includes R code for running all reported analyses. The functions for measuring surprisal are included in the open-source R package soundgen, freely available from https://cran.r-project.org/package=soundgen.

Reproduced under the paper's license (CC BY), from the paper cited above.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 2 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 32 scripts, each with its path and the digest of its content;
  • 2 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data Availability Statement

All data and materials are available at https://osf.io/bgzvc.

All data and materials are available at https://osf.io/bgzvc. This includes R code for running all reported analyses. The functions for measuring surprisal are included in the open-source R package soundgen, freely available from https://cran.r-project.org/package=soundgen.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 2, 28 September 2026

  • Publisher: n/a → Springer Science+Business Media

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 1 author, 5 keywords, 9 MeSH terms, 1 funder, 44 references.

Cite

This paper

Anikin, A. (2026). Measuring surprisal in sound sequences. Behavior research methods, 58(10), 278. https://doi.org/10.3758/s13428-026-03153-3

BibTeX

@article{anikin2026measuring,
author = {Anikin, Andrey},
title = {{Measuring surprisal in sound sequences}},
journal = {Behavior research methods},
year = {2026},
month = aug,
volume = {58},
number = {10},
pages = {278},
publisher = {Springer Science+Business Media},
issn = {1554-351X},
doi = {10.3758/s13428-026-03153-3},
url = {https://doi.org/10.3758/s13428-026-03153-3},
pmid = {42637973},
pmcid = {PMC13503466}
}

RIS

TY - JOUR
AU - Anikin, Andrey
TI - Measuring surprisal in sound sequences
T2 - Behavior research methods
J2 - Behav Res Methods
PY - 2026
DA - 2026/08/24
VL - 58
IS - 10
SP - 278
SN - 1554-351X
PB - Springer Science+Business Media
DO - 10.3758/s13428-026-03153-3
UR - https://doi.org/10.3758/s13428-026-03153-3
LA - en
ER -

CSL-JSON

{
"id": "10.3758/s13428-026-03153-3",
"type": "article-journal",
"title": "Measuring surprisal in sound sequences",
"container-title": "Behavior research methods",
"author": [
{
"family": "Anikin",
"given": "Andrey"
}
],
"container-title-short": "Behav Res Methods",
"volume": "58",
"issue": "10",
"page": "278",
"DOI": "10.3758/s13428-026-03153-3",
"PMID": "42637973",
"PMCID": "PMC13503466",
"ISSN": "1554-351X",
"publisher": "Springer Science+Business Media",
"URL": "https://doi.org/10.3758/s13428-026-03153-3",
"language": "en",
"issued": {
"date-parts": [
[
2026,
8,
24
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s42003-026-10957-8 [code]
Brain defence by the extracellular matrix protein Cochlin.
Journal: Communications biology
In common: randomForest, brms, patchwork, 3 other tools
[2] doi:10.1186/s13229-025-00678-w [code]
Locus coeruleus tonic upregulation increases selectivity to inconspicuous auditory information in autistic compared to non-autistic individuals: a combined pupillometry and electroencephalography study.
Journal: Molecular autism
In common: brms, reshape2, ggplot2, 1 other tool, 1 reference
[3] doi:10.1371/journal.pone.0360526 [code]
Sex- and age-specific development of ssRNA virus receptor expression in the human brain.
Journal: PloS one
In common: randomForest, patchwork, reshape2, 2 other tools
[4] doi:10.1093/bioinformatics/btag592 [code]
Network-based stratification of allele-specific expression reveals patient subgroups in Huntington's disease.
Journal: Bioinformatics (Oxford, England)
In common: randomForest, patchwork, reshape2, 2 other tools
[5] doi:10.3390/ijms27156925 [code]
XGBoost-SHAP Interpretable Modeling Identifies and Validates an Eight-Gene Biomarker for Hepatic Encephalopathy Risk Prediction in Cirrhosis.
Journal: International journal of molecular sciences
In common: randomForest, patchwork, reshape2, 2 other tools
[6] doi:10.1186/s13073-026-01598-x [code]
Exploring phenotype-related single-cells through attention-enhanced representation learning.
Journal: Genome medicine
In common: randomForest, patchwork, reshape2, 2 other tools
[7] doi:10.1038/s41598-026-34997-6 [code]
Uncovering the role of integrated stress in Alzheimer's disease through single-cell and transcriptomic analysis.
Journal: Scientific reports
In common: randomForest, patchwork, reshape2, 2 other tools
[8] doi:10.1093/braincomms/fcaf493 [code]
Subcortical correlates of developmental language disorder: more than the neostriatum.
Journal: Brain communications
In common: randomForest, patchwork, reshape2, 2 other tools
[9] doi:10.1126/sciadv.adu0726 [code]
CHCHD2 mutant mice link mitochondrial deficits to PD pathophysiology.
Journal: Science advances
In common: randomForest, patchwork, reshape2, 2 other tools
[10] doi:10.1038/s41537-025-00661-7 [code]
Glymphatic system dysfunction correlated with gut dysbiosis and cognitive impairment in schizophrenia.
Journal: Schizophrenia (Heidelberg, Germany)
In common: randomForest, patchwork, reshape2, 2 other tools

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Add the badge to the code's README

OSCR: the badge of this paper

If you are an author or a maintainer of this paper's code, you can add this badge. It says what OSCR knows of the paper (its code, its validated tracing map) and links the code to this page. Paste one of these snippets under the README's title. Keep the two marks around it, so that the badge can be replaced in place and is never added twice. OSCR changes nothing in your repository.

Markdown
<!-- oscr-badge:start -->
[![OSCR](https://openscicode.org/badge/doi/10.3758/s13428-026-03153-3.svg)](https://openscicode.org/badge/doi/10.3758/s13428-026-03153-3)
<!-- oscr-badge:end -->
reStructuredText
.. oscr-badge:start

.. image:: https://openscicode.org/badge/doi/10.3758/s13428-026-03153-3.svg
   :target: https://openscicode.org/badge/doi/10.3758/s13428-026-03153-3
   :alt: OSCR

.. oscr-badge:end
HTML
<!-- oscr-badge:start -->
<a href="https://openscicode.org/badge/doi/10.3758/s13428-026-03153-3"><img src="https://openscicode.org/badge/doi/10.3758/s13428-026-03153-3.svg" alt="OSCR"></a>
<!-- oscr-badge:end -->

If you choose to, you can have the badge kept up to date for you, with oscr badge --write from the command line or with the GitHub Action in your repository's workflows. Every other form and option (a card, AsciiDoc, LaTeX, shields.io) is on the badge page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: code errors and mismatches between the code and the paper about this paper and its code, from readers and its authors, beside the paper and its tracing map.

Report a mismatch between the code and the paper · Report a code error (signed in)

Reproductions: reports from readers who ran the authors' code and did not get the paper's result, what they ran, with which environment, commit and data.

Report a reproduction failure (signed in)

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.