OSCR

Statistically valid explainable black-box machine learning: applications in sex classification across species using brain imaging.

Code ↔ Paper

9 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 9 matches
  1. [1] § Materials and methods › Simulated datasets ↔ treeple/datasets/hyppo.py, lines 428–568 · score 0.74 · identity covariance matrix, Trunk simulation, Gaussian, closer, vector, dimension
  2. [2] § Materials and methods › Oblique random forest ↔ treeple/ensemble/_supervised_forest.py, lines 346–583 · score 0.66 · Sparse Projection Oblique, axis aligned decision, Oblique Random Forest, treeple, traditional, partitions
  3. [3] § Materials and methods › Feature importance testing ↔ examples/sparse_oblique_trees/plot_extra_oblique_random_forest.py, lines 1–78 · score 0.62 · computationally expensive, machine learning, high dimensional, predictions, algorithms, tree
  4. [4] § Results › Hyper-parameter tuning ↔ treeple/ensemble/_supervised_forest.py, lines 645–909 · score 0.61 · max patch dim, OOB score, max_features, split, classification
  5. [5] § Materials and methods › Oblique random forest ↔ yggdrasil_decision_forests/port/python/ydf/learner/specialized_learners_pre_generated.py, lines 53–482 · score 0.60 · cross validation, expensive, overfitting, bag, OOB, bootstrap
  6. [6] § Results › Performance of the sex classifier ↔ yggdrasil_decision_forests/port/python/ydf/metric/display_metric.py, lines 70–185 · score 0.58 · receiver operating characteristic, ROC curves, AUC, volume, thresholds, classifier
  7. [7] § Materials and methods › Feature importance testing ↔ treeple/ensemble/_honest_forest.py, lines 99–449 · score 0.57 · Gini impurity, split nodes, Random Forest, subset, predictions, trees
  8. [8] § Materials and methods › Simulated datasets ↔ treeple/ensemble/_eiforest.py, lines 6–160 · score 0.57 · Gaussian distributions, high dimensional, covariance, sparse, selection, algorithms
  9. [9] § Materials and methods › Feature importance testing ↔ yggdrasil_decision_forests/port/python/ydf/learner/specialized_learners_pre_generated.py, lines 1290–1840 · score 0.51 · model predictions, Additive, machine, global, computationally, permutations

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 1,931 lines · 80 KB · other · 2 matches

  1. from sklearn.utils._param_validation import StrOptions
  2. from .._lib.sklearn.ensemble._forest import ForestClassifier, ForestRegressor
  3. from ..tree import (
  4. ExtraObliqueDecisionTreeClassifier,
  5. ExtraObliqueDecisionTreeRegressor,
  6. ObliqueDecisionTreeClassifier,
  7. ObliqueDecisionTreeRegressor,
  8. PatchObliqueDecisionTreeClassifier,
  9. PatchObliqueDecisionTreeRegressor,
  10. )
  11. from ..tree._neighbors import SimMatrixMixin
  12. from ._extensions import ForestClassifierMixin, ForestMixin
  13. class ObliqueRandomForestClassifier(
  14. SimMatrixMixin, ForestClassifierMixin, ForestMixin, ForestClassifier
  15. ):
  16. """
  17. An oblique random forest classifier.
  18. A oblique random forest is a meta estimator similar to a random
  19. forest that fits a number of oblique decision tree classifiers
  20. on various sub-samples of the dataset and uses averaging to
  21. improve the predictive accuracy and control over-fitting.
  22. The sub-sample size is controlled with the `max_samples` parameter if
  23. `bootstrap=True` (default), otherwise the whole dataset is used to build
  24. each tree.
  25. Read more in the :ref:`User Guide <sklearn:forest>`.
  26. Parameters
  27. ----------
  28. n_estimators : int, default=100
  29. The number of trees in the forest.
  30. criterion : {"gini", "entropy"}, default="gini"
  31. The function to measure the quality of a split. Supported criteria are
  32. "gini" for the Gini impurity and "entropy" for the information gain.
  33. Note: this parameter is tree-specific.
  34. max_depth : int, default=None
  35. The maximum depth of the tree. If None, then nodes are expanded until
  36. all leaves are pure or until all leaves contain less than
  37. min_samples_split samples.
  38. min_samples_split : int or float, default=2
  39. The minimum number of samples required to split an internal node:
  40. - If int, then consider `min_samples_split` as the minimum number.
  41. - If float, then `min_samples_split` is a fraction and
  42. `ceil(min_samples_split * n_samples)` are the minimum
  43. number of samples for each split.
  44. min_samples_leaf : int or float, default=1
  45. The minimum number of samples required to be at a leaf node.
  46. A split point at any depth will only be considered if it leaves at
  47. least ``min_samples_leaf`` training samples in each of the left and
  48. right branches. This may have the effect of smoothing the model,
  49. especially in regression.
  50. - If int, then consider `min_samples_leaf` as the minimum number.
  51. - If float, then `min_samples_leaf` is a fraction and
  52. `ceil(min_samples_leaf * n_samples)` are the minimum
  53. number of samples for each node.
  54. min_weight_fraction_leaf : float, default=0.0
  55. The minimum weighted fraction of the sum total of weights (of all
  56. the input samples) required to be at a leaf node. Samples have
  57. equal weight when sample_weight is not provided.
  58. max_features : {"sqrt", "log2", None}, int or float, default="sqrt"
  59. The number of features to consider when looking for the best split:
  60. - If int, then consider `max_features` features at each split.
  61. - If float, then `max_features` is a fraction and
  62. `round(max_features * n_features)` features are considered at each
  63. split.
  64. - If "auto", then `max_features=sqrt(n_features)`.
  65. - If "sqrt", then `max_features=sqrt(n_features)`.
  66. - If "log2", then `max_features=log2(n_features)`.
  67. - If None, then `max_features=n_features`.
  68. Note: the search for a split does not stop until at least one
  69. valid partition of the node samples is found, even if it requires to
  70. effectively inspect more than ``max_features`` features.
  71. max_leaf_nodes : int, default=None
  72. Grow trees with ``max_leaf_nodes`` in best-first fashion.
  73. Best nodes are defined as relative reduction in impurity.
  74. If None then unlimited number of leaf nodes.
  75. min_impurity_decrease : float, default=0.0
  76. A node will be split if this split induces a decrease of the impurity
  77. greater than or equal to this value.
  78. The weighted impurity decrease equation is the following::
  79. N_t / N * (impurity - N_t_R / N_t * right_impurity
  80. - N_t_L / N_t * left_impurity)
  81. where ``N`` is the total number of samples, ``N_t`` is the number of
  82. samples at the current node, ``N_t_L`` is the number of samples in the
  83. left child, and ``N_t_R`` is the number of samples in the right child.
  84. ``N``, ``N_t``, ``N_t_R`` and ``N_t_L`` all refer to the weighted sum,
  85. if ``sample_weight`` is passed.
  86. bootstrap : bool, default=True
  87. Whether bootstrap samples are used when building trees. If False, the
  88. whole dataset is used to build each tree.
  89. oob_score : bool, default=False
  90. Whether to use out-of-bag samples to estimate the generalization score.
  91. Only available if bootstrap=True.
  92. n_jobs : int, default=None
  93. The number of jobs to run in parallel. :meth:`fit`, :meth:`predict`,
  94. :meth:`decision_path` and :meth:`apply` are all parallelized over the
  95. trees. ``None`` means 1 unless in a `joblib.parallel_backend`
  96. context. ``-1`` means using all processors. See :term:`Glossary
  97. <n_jobs>` for more details.
  98. random_state : int, RandomState instance or None, default=None
  99. Controls both the randomness of the bootstrapping of the samples used
  100. when building trees (if ``bootstrap=True``) and the sampling of the
  101. features to consider when looking for the best split at each node
  102. (if ``max_features < n_features``).
  103. See :term:`Glossary <random_state>` for details.
  104. verbose : int, default=0
  105. Controls the verbosity when fitting and predicting.
  106. warm_start : bool, default=False
  107. When set to ``True``, reuse the solution of the previous call to fit
  108. and add more estimators to the ensemble, otherwise, just fit a whole
  109. new forest. See :term:`the Glossary <warm_start>`.
  110. class_weight : {"balanced", "balanced_subsample"}, dict or list of dicts, \
  111. default=None
  112. Weights associated with classes in the form ``{class_label: weight}``.
  113. If not given, all classes are supposed to have weight one. For
  114. multi-output problems, a list of dicts can be provided in the same
  115. order as the columns of y.
  116. Note that for multioutput (including multilabel) weights should be
  117. defined for each class of every column in its own dict. For example,
  118. for four-class multilabel classification weights should be
  119. [{0: 1, 1: 1}, {0: 1, 1: 5}, {0: 1, 1: 1}, {0: 1, 1: 1}] instead of
  120. [{1:1}, {2:5}, {3:1}, {4:1}].
  121. The "balanced" mode uses the values of y to automatically adjust
  122. weights inversely proportional to class frequencies in the input data
  123. as ``n_samples / (n_classes * np.bincount(y))``
  124. The "balanced_subsample" mode is the same as "balanced" except that
  125. weights are computed based on the bootstrap sample for every tree
  126. grown.
  127. For multi-output, the weights of each column of y will be multiplied.
  128. Note that these weights will be multiplied with sample_weight (passed
  129. through the fit method) if sample_weight is specified.
  130. max_samples : int or float, default=None
  131. If bootstrap is True, the number of samples to draw from X
  132. to train each base estimator.
  133. - If None (default), then draw `X.shape[0]` samples.
  134. - If int, then draw `max_samples` samples.
  135. - If float, then draw `max_samples * X.shape[0]` samples. Thus,
  136. `max_samples` should be in the interval `(0.0, 1.0]`.
  137. feature_combinations : float, default=None
  138. The number of features to combine on average at each split
  139. of the decision trees. If ``None``, then will default to the minimum of
  140. ``(1.5, n_features)``. This controls the number of non-zeros is the
  141. projection matrix. Setting the value to 1.0 is equivalent to a
  142. traditional decision-tree. ``feature_combinations * max_features``
  143. gives the number of expected non-zeros in the projection matrix of shape
  144. ``(max_features, n_features)``. Thus this value must always be less than
  145. ``n_features`` in order to be valid.
  146. Attributes
  147. ----------
  148. estimators_ : list of treeple.tree.ObliqueDecisionTreeClassifier
  149. The collection of fitted sub-estimators.
  150. classes_ : ndarray of shape (n_classes,) or a list of such arrays
  151. The classes labels (single output problem), or a list of arrays of
  152. class labels (multi-output problem).
  153. n_classes_ : int or list
  154. The number of classes (single output problem), or a list containing the
  155. number of classes for each output (multi-output problem).
  156. n_features_ : int
  157. The number of features when ``fit`` is performed.
  158. n_features_in_ : int
  159. Number of features seen during :term:`fit`.
  160. feature_names_in_ : ndarray of shape (`n_features_in_`,)
  161. Names of features seen during :term:`fit`. Defined only when `X`
  162. has feature names that are all strings.
  163. n_outputs_ : int
  164. The number of outputs when ``fit`` is performed.
  165. feature_importances_ : ndarray of shape (n_features,)
  166. The impurity-based feature importances.
  167. The higher, the more important the feature.
  168. The importance of a feature is computed as the (normalized)
  169. total reduction of the criterion brought by that feature. It is also
  170. known as the Gini importance.
  171. Warning: impurity-based feature importances can be misleading for
  172. high cardinality features (many unique values). See
  173. :func:`sklearn.inspection.permutation_importance` as an alternative.
  174. oob_score_ : float
  175. Score of the training dataset obtained using an out-of-bag estimate.
  176. This attribute exists only when ``oob_score`` is True.
  177. oob_decision_function_ : ndarray of shape (n_samples, n_classes) or \
  178. (n_samples, n_classes, n_outputs)
  179. Decision function computed with out-of-bag estimate on the training
  180. set. If n_estimators is small it might be possible that a data point
  181. was never left out during the bootstrap. In this case,
  182. `oob_decision_function_` might contain NaN. This attribute exists
  183. only when ``oob_score`` is True.
  184. See Also
  185. --------
  186. treeple.tree.ObliqueDecisionTreeClassifier : An oblique decision
  187. tree classifier.
  188. sklearn.ensemble.RandomForestClassifier : An axis-aligned decision
  189. forest classifier.
  190. Notes
  191. -----
  192. The default values for the parameters controlling the size of the trees
  193. (e.g. ``max_depth``, ``min_samples_leaf``, etc.) lead to fully grown and
  194. unpruned trees which can potentially be very large on some data sets. To
  195. reduce memory consumption, the complexity and size of the trees should be
  196. controlled by setting those parameter values.
  197. The features are always randomly permuted at each split. Therefore,
  198. the best found split may vary, even with the same training data,
  199. ``max_features=n_features`` and ``bootstrap=False``, if the improvement
  200. of the criterion is identical for several splits enumerated during the
  201. search of the best split. To obtain a deterministic behaviour during
  202. fitting, ``random_state`` has to be fixed.
  203. References
  204. ----------
  205. .. [1] L. Breiman, "Random Forests", Machine Learning, 45(1), 5-32, 2001.
  206. Examples
  207. --------
  208. >>> from treeple.ensemble import ObliqueRandomForestClassifier
  209. >>> from sklearn.datasets import make_classification
  210. >>> X, y = make_classification(n_samples=1000, n_features=4,
  211. ... n_informative=2, n_redundant=0,
  212. ... random_state=0, shuffle=False)
  213. >>> clf = ObliqueRandomForestClassifier(max_depth=2, random_state=0)
  214. >>> clf.fit(X, y)
  215. ObliqueRandomForestClassifier(...)
  216. >>> print(clf.predict([[0, 0, 0, 0]]))
  217. [1]
  218. """
  219. tree_type = "oblique"
  220. _parameter_constraints: dict = {
  221. **ForestClassifier._parameter_constraints,
  222. **ObliqueDecisionTreeClassifier._parameter_constraints,
  223. "class_weight": [
  224. StrOptions({"balanced_subsample", "balanced"}),
  225. dict,
  226. list,
  227. None,
  228. ],
  229. }
  230. _parameter_constraints.pop("splitter")
  231. def __init__(
  232. self,
  233. n_estimators=100,
  234. *,
  235. criterion="gini",
  236. max_depth=None,
  237. min_samples_split=2,
  238. min_samples_leaf=1,
  239. min_weight_fraction_leaf=0.0,
  240. max_features="sqrt",
  241. max_leaf_nodes=None,
  242. min_impurity_decrease=0.0,
  243. bootstrap=True,
  244. oob_score=False,
  245. n_jobs=None,
  246. random_state=None,
  247. verbose=0,
  248. warm_start=False,
  249. class_weight=None,
  250. max_samples=None,
  251. feature_combinations=None,
  252. ):
  253. super().__init__(
  254. estimator=ObliqueDecisionTreeClassifier(),
  255. n_estimators=n_estimators,
  256. estimator_params=(
  257. "criterion",
  258. "max_depth",
  259. "min_samples_split",
  260. "min_samples_leaf",
  261. "min_weight_fraction_leaf",
  262. "max_features",
  263. "max_leaf_nodes",
  264. "min_impurity_decrease",
  265. "random_state",
  266. "feature_combinations",
  267. ),
  268. bootstrap=bootstrap,
  269. oob_score=oob_score,
  270. n_jobs=n_jobs,
  271. random_state=random_state,
  272. verbose=verbose,
  273. warm_start=warm_start,
  274. class_weight=class_weight,
  275. max_samples=max_samples,
  276. )
  277. self.criterion = criterion
  278. self.max_depth = max_depth
  279. self.min_samples_split = min_samples_split
  280. self.min_samples_leaf = min_samples_leaf
  281. self.max_features = max_features
  282. self.feature_combinations = feature_combinations
  283. # unused by oblique forests
  284. self.min_weight_fraction_leaf = min_weight_fraction_leaf
  285. self.max_leaf_nodes = max_leaf_nodes
  286. self.min_impurity_decrease = min_impurity_decrease
  287. class ObliqueRandomForestRegressor(SimMatrixMixin, ForestMixin, ForestRegressor):
  288. """An oblique random forest regressor.
  289. A oblique random forest is a meta estimator similar to a random
  290. forest that fits a number of oblique decision tree regressor
  291. on various sub-samples of the dataset and uses averaging to
  292. improve the predictive accuracy and control over-fitting.
  293. The sub-sample size is controlled with the `max_samples` parameter if
  294. `bootstrap=True` (default), otherwise the whole dataset is used to build
  295. each tree.
  296. Read more in the :ref:`User Guide <sklearn:forest>`.
  297. Parameters
  298. ----------
  299. n_estimators : int, default=100
  300. The number of trees in the forest.
  301. criterion : {"squared_error", "absolute_error", "friedman_mse", "poisson"}, \
  302. default="squared_error"
  303. The function to measure the quality of a split. Supported criteria
  304. are "squared_error" for the mean squared error, which is equal to
  305. variance reduction as feature selection criterion and minimizes the L2
  306. loss using the mean of each terminal node, "friedman_mse", which uses
  307. mean squared error with Friedman's improvement score for potential
  308. splits, "absolute_error" for the mean absolute error, which minimizes
  309. the L1 loss using the median of each terminal node, and "poisson" which
  310. uses reduction in Poisson deviance to find splits.
  311. Training using "absolute_error" is significantly slower
  312. than when using "squared_error".
  313. max_depth : int, default=None
  314. The maximum depth of the tree. If None, then nodes are expanded until
  315. all leaves are pure or until all leaves contain less than
  316. min_samples_split samples.
  317. min_samples_split : int or float, default=2
  318. The minimum number of samples required to split an internal node:
  319. - If int, then consider `min_samples_split` as the minimum number.
  320. - If float, then `min_samples_split` is a fraction and
  321. `ceil(min_samples_split * n_samples)` are the minimum
  322. number of samples for each split.
  323. min_samples_leaf : int or float, default=1
  324. The minimum number of samples required to be at a leaf node.
  325. A split point at any depth will only be considered if it leaves at
  326. least ``min_samples_leaf`` training samples in each of the left and
  327. right branches. This may have the effect of smoothing the model,
  328. especially in regression.
  329. - If int, then consider `min_samples_leaf` as the minimum number.
  330. - If float, then `min_samples_leaf` is a fraction and
  331. `ceil(min_samples_leaf * n_samples)` are the minimum
  332. number of samples for each node.
  333. min_weight_fraction_leaf : float, default=0.0
  334. The minimum weighted fraction of the sum total of weights (of all
  335. the input samples) required to be at a leaf node. Samples have
  336. equal weight when sample_weight is not provided.
  337. max_features : {"sqrt", "log2", None}, int or float, default="sqrt"
  338. The number of features to consider when looking for the best split:
  339. - If int, then consider `max_features` features at each split.
  340. - If float, then `max_features` is a fraction and
  341. `round(max_features * n_features)` features are considered at each
  342. split.
  343. - If "auto", then `max_features=sqrt(n_features)`.
  344. - If "sqrt", then `max_features=sqrt(n_features)`.
  345. - If "log2", then `max_features=log2(n_features)`.
  346. - If None, then `max_features=n_features`.
  347. Note: the search for a split does not stop until at least one
  348. valid partition of the node samples is found, even if it requires to
  349. effectively inspect more than ``max_features`` features.
  350. max_leaf_nodes : int, default=None
  351. Grow trees with ``max_leaf_nodes`` in best-first fashion.
  352. Best nodes are defined as relative reduction in impurity.
  353. If None then unlimited number of leaf nodes.
  354. min_impurity_decrease : float, default=0.0
  355. A node will be split if this split induces a decrease of the impurity
  356. greater than or equal to this value.
  357. The weighted impurity decrease equation is the following::
  358. N_t / N * (impurity - N_t_R / N_t * right_impurity
  359. - N_t_L / N_t * left_impurity)
  360. where ``N`` is the total number of samples, ``N_t`` is the number of
  361. samples at the current node, ``N_t_L`` is the number of samples in the
  362. left child, and ``N_t_R`` is the number of samples in the right child.
  363. ``N``, ``N_t``, ``N_t_R`` and ``N_t_L`` all refer to the weighted sum,
  364. if ``sample_weight`` is passed.
  365. bootstrap : bool, default=True
  366. Whether bootstrap samples are used when building trees. If False, the
  367. whole dataset is used to build each tree.
  368. oob_score : bool, default=False
  369. Whether to use out-of-bag samples to estimate the generalization score.
  370. Only available if bootstrap=True.
  371. n_jobs : int, default=None
  372. The number of jobs to run in parallel. :meth:`fit`, :meth:`predict`,
  373. :meth:`decision_path` and :meth:`apply` are all parallelized over the
  374. trees. ``None`` means 1 unless in a `joblib.parallel_backend`
  375. context. ``-1`` means using all processors. See :term:`Glossary
  376. <n_jobs>` for more details.
  377. random_state : int, RandomState instance or None, default=None
  378. Controls both the randomness of the bootstrapping of the samples used
  379. when building trees (if ``bootstrap=True``) and the sampling of the
  380. features to consider when looking for the best split at each node
  381. (if ``max_features < n_features``).
  382. See :term:`Glossary <random_state>` for details.
  383. verbose : int, default=0
  384. Controls the verbosity when fitting and predicting.
  385. warm_start : bool, default=False
  386. When set to ``True``, reuse the solution of the previous call to fit
  387. and add more estimators to the ensemble, otherwise, just fit a whole
  388. new forest. See :term:`the Glossary <warm_start>`.
  389. max_samples : int or float, default=None
  390. If bootstrap is True, the number of samples to draw from X
  391. to train each base estimator.
  392. - If None (default), then draw `X.shape[0]` samples.
  393. - If int, then draw `max_samples` samples.
  394. - If float, then draw `max_samples * X.shape[0]` samples. Thus,
  395. `max_samples` should be in the interval `(0.0, 1.0]`.
  396. feature_combinations : float, default=None
  397. The number of features to combine on average at each split
  398. of the decision trees. If ``None``, then will default to the minimum of
  399. ``(1.5, n_features)``. This controls the number of non-zeros is the
  400. projection matrix. Setting the value to 1.0 is equivalent to a
  401. traditional decision-tree. ``feature_combinations * max_features``
  402. gives the number of expected non-zeros in the projection matrix of shape
  403. ``(max_features, n_features)``. Thus this value must always be less than
  404. ``n_features`` in order to be valid.
  405. Attributes
  406. ----------
  407. estimators_ : list of ObliqueDecisionTreeRegressor
  408. The collection of fitted sub-estimators.
  409. n_features_ : int
  410. The number of features when ``fit`` is performed.
  411. n_features_in_ : int
  412. Number of features seen during :term:`fit`.
  413. feature_names_in_ : ndarray of shape (`n_features_in_`,)
  414. Names of features seen during :term:`fit`. Defined only when `X`
  415. has feature names that are all strings.
  416. n_outputs_ : int
  417. The number of outputs when ``fit`` is performed.
  418. feature_importances_ : ndarray of shape (n_features,)
  419. The impurity-based feature importances.
  420. The higher, the more important the feature.
  421. The importance of a feature is computed as the (normalized)
  422. total reduction of the criterion brought by that feature. It is also
  423. known as the Gini importance.
  424. Warning: impurity-based feature importances can be misleading for
  425. high cardinality features (many unique values). See
  426. :func:`sklearn.inspection.permutation_importance` as an alternative.
  427. oob_score_ : float
  428. Score of the training dataset obtained using an out-of-bag estimate.
  429. This attribute exists only when ``oob_score`` is True.
  430. oob_decision_function_ : ndarray of shape (n_samples, n_classes) or \
  431. (n_samples, n_classes, n_outputs)
  432. Decision function computed with out-of-bag estimate on the training
  433. set. If n_estimators is small it might be possible that a data point
  434. was never left out during the bootstrap. In this case,
  435. `oob_decision_function_` might contain NaN. This attribute exists
  436. only when ``oob_score`` is True.
  437. See Also
  438. --------
  439. treeple.tree.ObliqueDecisionTreeRegressor : An oblique decision
  440. tree regressor.
  441. sklearn.ensemble.RandomForestRegressor : An axis-aligned decision
  442. forest regressor.
  443. Notes
  444. -----
  445. The default values for the parameters controlling the size of the trees
  446. (e.g. ``max_depth``, ``min_samples_leaf``, etc.) lead to fully grown and
  447. unpruned trees which can potentially be very large on some data sets. To
  448. reduce memory consumption, the complexity and size of the trees should be
  449. controlled by setting those parameter values.
  450. The features are always randomly permuted at each split. Therefore,
  451. the best found split may vary, even with the same training data,
  452. ``max_features=n_features`` and ``bootstrap=False``, if the improvement
  453. of the criterion is identical for several splits enumerated during the
  454. search of the best split. To obtain a deterministic behaviour during
  455. fitting, ``random_state`` has to be fixed.
  456. References
  457. ----------
  458. .. [1] L. Breiman, "Random Forests", Machine Learning, 45(1), 5-32, 2001.
  459. .. [2] T. Tomita, "Sparse Projection Oblique Randomer Forests", \
  460. Journal of Machine Learning Research, 21(104), 1-39, 2020.
  461. Examples
  462. --------
  463. >>> from treeple.ensemble import ObliqueRandomForestRegressor
  464. >>> from sklearn.datasets import make_regression
  465. >>> X, y = make_regression(n_features=4, n_informative=2,
  466. ... random_state=0, shuffle=False)
  467. >>> regr = ObliqueRandomForestRegressor(max_depth=2, random_state=0)
  468. >>> regr.fit(X, y)
  469. ObliqueRandomForestRegressor(...)
  470. >>> print(regr.predict([[0, 0, 0, 0]]))
  471. [-5.86327109]
  472. """
  473. tree_type = "oblique"
  474. _parameter_constraints: dict = {
  475. **ForestRegressor._parameter_constraints,
  476. **ObliqueDecisionTreeRegressor._parameter_constraints,
  477. }
  478. _parameter_constraints.pop("splitter")
  479. def __init__(
  480. self,
  481. n_estimators=100,
  482. *,
  483. criterion="squared_error",
  484. max_depth=None,
  485. min_samples_split=2,
  486. min_samples_leaf=1,
  487. min_weight_fraction_leaf=0.0,
  488. max_features=1.0,
  489. max_leaf_nodes=None,
  490. min_impurity_decrease=0.0,
  491. bootstrap=True,
  492. oob_score=False,
  493. n_jobs=None,
  494. random_state=None,
  495. verbose=0,
  496. warm_start=False,
  497. max_samples=None,
  498. feature_combinations=None,
  499. ):
  500. super().__init__(
  501. estimator=ObliqueDecisionTreeRegressor(),
  502. n_estimators=n_estimators,
  503. estimator_params=(
  504. "criterion",
  505. "max_depth",
  506. "min_samples_split",
  507. "min_samples_leaf",
  508. "min_weight_fraction_leaf",
  509. "max_features",
  510. "max_leaf_nodes",
  511. "min_impurity_decrease",
  512. "random_state",
  513. "feature_combinations",
  514. ),
  515. bootstrap=bootstrap,
  516. oob_score=oob_score,
  517. n_jobs=n_jobs,
  518. random_state=random_state,
  519. verbose=verbose,
  520. warm_start=warm_start,
  521. max_samples=max_samples,
  522. )
  523. self.criterion = criterion
  524. self.max_depth = max_depth
  525. self.min_samples_split = min_samples_split
  526. self.min_samples_leaf = min_samples_leaf
  527. self.min_weight_fraction_leaf = min_weight_fraction_leaf
  528. self.max_features = max_features
  529. self.max_leaf_nodes = max_leaf_nodes
  530. self.min_impurity_decrease = min_impurity_decrease
  531. self.feature_combinations = feature_combinations
  532. # unused by oblique forests
  533. self.min_weight_fraction_leaf = min_weight_fraction_leaf
  534. self.max_leaf_nodes = max_leaf_nodes
  535. self.min_impurity_decrease = min_impurity_decrease
  536. class PatchObliqueRandomForestClassifier(
  537. SimMatrixMixin, ForestClassifierMixin, ForestMixin, ForestClassifier
  538. ):
  539. """A patch-oblique random forest classifier.
  540. A patch-oblique random forest is a meta estimator similar to a random
  541. forest that fits a number of patch oblique decision tree classifiers
  542. on various sub-samples of the dataset and uses averaging to
  543. improve the predictive accuracy and control over-fitting. For more
  544. details, see :footcite:`Li2023manifold`.
  545. Parameters
  546. ----------
  547. n_estimators : int, default=100
  548. The number of trees in the forest.
  549. criterion : {"gini", "entropy"}, default="gini"
  550. The function to measure the quality of a split. Supported criteria are
  551. "gini" for the Gini impurity and "entropy" for the information gain.
  552. Note: this parameter is tree-specific.
  553. max_depth : int, default=None
  554. The maximum depth of the tree. If None, then nodes are expanded until
  555. all leaves are pure or until all leaves contain less than
  556. min_samples_split samples.
  557. min_samples_split : int or float, default=2
  558. The minimum number of samples required to split an internal node:
  559. - If int, then consider `min_samples_split` as the minimum number.
  560. - If float, then `min_samples_split` is a fraction and
  561. `ceil(min_samples_split * n_samples)` are the minimum
  562. number of samples for each split.
  563. min_samples_leaf : int or float, default=1
  564. The minimum number of samples required to be at a leaf node.
  565. A split point at any depth will only be considered if it leaves at
  566. least ``min_samples_leaf`` training samples in each of the left and
  567. right branches. This may have the effect of smoothing the model,
  568. especially in regression.
  569. - If int, then consider `min_samples_leaf` as the minimum number.
  570. - If float, then `min_samples_leaf` is a fraction and
  571. `ceil(min_samples_leaf * n_samples)` are the minimum
  572. number of samples for each node.
  573. min_weight_fraction_leaf : float, default=0.0
  574. The minimum weighted fraction of the sum total of weights (of all
  575. the input samples) required to be at a leaf node. Samples have
  576. equal weight when sample_weight is not provided.
  577. max_features : {"sqrt", "log2", None}, int or float, default="sqrt"
  578. The number of features to consider when looking for the best split:
  579. - If int, then consider `max_features` features at each split.
  580. - If float, then `max_features` is a fraction and
  581. `round(max_features * n_features)` features are considered at each
  582. split.
  583. - If "auto", then `max_features=sqrt(n_features)`.
  584. - If "sqrt", then `max_features=sqrt(n_features)`.
  585. - If "log2", then `max_features=log2(n_features)`.
  586. - If None, then `max_features=n_features`.
  587. Note: the search for a split does not stop until at least one
  588. valid partition of the node samples is found, even if it requires to
  589. effectively inspect more than ``max_features`` features.
  590. max_leaf_nodes : int, default=None
  591. Grow trees with ``max_leaf_nodes`` in best-first fashion.
  592. Best nodes are defined as relative reduction in impurity.
  593. If None then unlimited number of leaf nodes.
  594. min_impurity_decrease : float, default=0.0
  595. A node will be split if this split induces a decrease of the impurity
  596. greater than or equal to this value.
  597. The weighted impurity decrease equation is the following::
  598. N_t / N * (impurity - N_t_R / N_t * right_impurity
  599. - N_t_L / N_t * left_impurity)
  600. where ``N`` is the total number of samples, ``N_t`` is the number of
  601. samples at the current node, ``N_t_L`` is the number of samples in the
  602. left child, and ``N_t_R`` is the number of samples in the right child.
  603. ``N``, ``N_t``, ``N_t_R`` and ``N_t_L`` all refer to the weighted sum,
  604. if ``sample_weight`` is passed.
  605. bootstrap : bool, default=True
  606. Whether bootstrap samples are used when building trees. If False, the
  607. whole dataset is used to build each tree.
  608. oob_score : bool, default=False
  609. Whether to use out-of-bag samples to estimate the generalization score.
  610. Only available if bootstrap=True.
  611. n_jobs : int, default=None
  612. The number of jobs to run in parallel. :meth:`fit`, :meth:`predict`,
  613. :meth:`decision_path` and :meth:`apply` are all parallelized over the
  614. trees. ``None`` means 1 unless in a `joblib.parallel_backend`
  615. context. ``-1`` means using all processors. See :term:`Glossary
  616. <n_jobs>` for more details.
  617. random_state : int, RandomState instance or None, default=None
  618. Controls both the randomness of the bootstrapping of the samples used
  619. when building trees (if ``bootstrap=True``) and the sampling of the
  620. features to consider when looking for the best split at each node
  621. (if ``max_features < n_features``).
  622. See :term:`Glossary <random_state>` for details.
  623. verbose : int, default=0
  624. Controls the verbosity when fitting and predicting.
  625. warm_start : bool, default=False
  626. When set to ``True``, reuse the solution of the previous call to fit
  627. and add more estimators to the ensemble, otherwise, just fit a whole
  628. new forest. See :term:`the Glossary <warm_start>`.
  629. class_weight : {"balanced", "balanced_subsample"}, dict or list of dicts, \
  630. default=None
  631. Weights associated with classes in the form ``{class_label: weight}``.
  632. If not given, all classes are supposed to have weight one. For
  633. multi-output problems, a list of dicts can be provided in the same
  634. order as the columns of y.
  635. Note that for multioutput (including multilabel) weights should be
  636. defined for each class of every column in its own dict. For example,
  637. for four-class multilabel classification weights should be
  638. [{0: 1, 1: 1}, {0: 1, 1: 5}, {0: 1, 1: 1}, {0: 1, 1: 1}] instead of
  639. [{1:1}, {2:5}, {3:1}, {4:1}].
  640. The "balanced" mode uses the values of y to automatically adjust
  641. weights inversely proportional to class frequencies in the input data
  642. as ``n_samples / (n_classes * np.bincount(y))``
  643. The "balanced_subsample" mode is the same as "balanced" except that
  644. weights are computed based on the bootstrap sample for every tree
  645. grown.
  646. For multi-output, the weights of each column of y will be multiplied.
  647. Note that these weights will be multiplied with sample_weight (passed
  648. through the fit method) if sample_weight is specified.
  649. max_samples : int or float, default=None
  650. If bootstrap is True, the number of samples to draw from X
  651. to train each base estimator.
  652. - If None (default), then draw `X.shape[0]` samples.
  653. - If int, then draw `max_samples` samples.
  654. - If float, then draw `max_samples * X.shape[0]` samples. Thus,
  655. `max_samples` should be in the interval `(0.0, 1.0]`.
  656. min_patch_dims : array-like, optional
  657. The minimum dimensions of a patch, by default 1 along all dimensions.
  658. max_patch_dims : array-like, optional
  659. The maximum dimensions of a patch, by default 1 along all dimensions.
  660. dim_contiguous : array-like of bool, optional
  661. Whether or not each patch is sampled contiguously along this dimension.
  662. data_dims : array-like, optional
  663. The presumed dimensions of the un-vectorized feature vector, by default
  664. will be a 1D vector with (1, n_features) shape.
  665. boundary : optional, str {'wrap'}
  666. The boundary condition to use when sampling patches, by default None.
  667. 'wrap' corresponds to the boundary condition as is in numpy and scipy.
  668. feature_weight : array-like of shape (n_features,), default=None
  669. Feature weights. If None, then features are equally weighted as is.
  670. If provided, then the feature weights are used to weight the
  671. patches that are generated. The feature weights are used
  672. as follows: for every patch that is sampled, the feature weights over
  673. the entire patch is summed and normalizes the patch.
  674. Attributes
  675. ----------
  676. estimators_ : list of PatchObliqueDecisionTreeClassifier
  677. The collection of fitted sub-estimators.
  678. classes_ : ndarray of shape (n_classes,) or a list of such arrays
  679. The classes labels (single output problem), or a list of arrays of
  680. class labels (multi-output problem).
  681. n_classes_ : int or list
  682. The number of classes (single output problem), or a list containing the
  683. number of classes for each output (multi-output problem).
  684. n_features_ : int
  685. The number of features when ``fit`` is performed.
  686. n_features_in_ : int
  687. Number of features seen during :term:`fit`.
  688. feature_names_in_ : ndarray of shape (`n_features_in_`,)
  689. Names of features seen during :term:`fit`. Defined only when `X`
  690. has feature names that are all strings.
  691. n_outputs_ : int
  692. The number of outputs when ``fit`` is performed.
  693. feature_importances_ : ndarray of shape (n_features,)
  694. The impurity-based feature importances.
  695. The higher, the more important the feature.
  696. The importance of a feature is computed as the (normalized)
  697. total reduction of the criterion brought by that feature. It is also
  698. known as the Gini importance.
  699. Warning: impurity-based feature importances can be misleading for
  700. high cardinality features (many unique values). See
  701. :func:`sklearn.inspection.permutation_importance` as an alternative.
  702. oob_score_ : float
  703. Score of the training dataset obtained using an out-of-bag estimate.
  704. This attribute exists only when ``oob_score`` is True.
  705. oob_decision_function_ : ndarray of shape (n_samples, n_classes) or \
  706. (n_samples, n_classes, n_outputs)
  707. Decision function computed with out-of-bag estimate on the training
  708. set. If n_estimators is small it might be possible that a data point
  709. was never left out during the bootstrap. In this case,
  710. `oob_decision_function_` might contain NaN. This attribute exists
  711. only when ``oob_score`` is True.
  712. See Also
  713. --------
  714. treeple.tree.ObliqueDecisionTreeClassifier : An oblique decision
  715. tree classifier.
  716. sklearn.ensemble.RandomForestClassifier : An axis-aligned decision
  717. forest classifier.
  718. Notes
  719. -----
  720. The default values for the parameters controlling the size of the trees
  721. (e.g. ``max_depth``, ``min_samples_leaf``, etc.) lead to fully grown and
  722. unpruned trees which can potentially be very large on some data sets. To
  723. reduce memory consumption, the complexity and size of the trees should be
  724. controlled by setting those parameter values.
  725. The features are always randomly permuted at each split. Therefore,
  726. the best found split may vary, even with the same training data,
  727. ``max_features=n_features`` and ``bootstrap=False``, if the improvement
  728. of the criterion is identical for several splits enumerated during the
  729. search of the best split. To obtain a deterministic behaviour during
  730. fitting, ``random_state`` has to be fixed.
  731. References
  732. ----------
  733. .. footbibliography::
  734. """
  735. tree_type = "oblique"
  736. _parameter_constraints: dict = {
  737. **ForestClassifier._parameter_constraints,
  738. **PatchObliqueDecisionTreeClassifier._parameter_constraints,
  739. "class_weight": [
  740. StrOptions({"balanced_subsample", "balanced"}),
  741. dict,
  742. list,
  743. None,
  744. ],
  745. }
  746. _parameter_constraints.pop("splitter")
  747. def __init__(
  748. self,
  749. n_estimators=100,
  750. *,
  751. criterion="gini",
  752. max_depth=None,
  753. min_samples_split=2,
  754. min_samples_leaf=1,
  755. min_weight_fraction_leaf=0.0,
  756. max_features="sqrt",
  757. max_leaf_nodes=None,
  758. min_impurity_decrease=0.0,
  759. bootstrap=True,
  760. oob_score=False,
  761. n_jobs=None,
  762. random_state=None,
  763. verbose=0,
  764. warm_start=False,
  765. class_weight=None,
  766. max_samples=None,
  767. min_patch_dims=None,
  768. max_patch_dims=None,
  769. dim_contiguous=None,
  770. data_dims=None,
  771. boundary=None,
  772. feature_weight=None,
  773. ):
  774. super().__init__(
  775. estimator=PatchObliqueDecisionTreeClassifier(),
  776. n_estimators=n_estimators,
  777. estimator_params=(
  778. "criterion",
  779. "max_depth",
  780. "min_samples_split",
  781. "min_samples_leaf",
  782. "min_weight_fraction_leaf",
  783. "max_features",
  784. "max_leaf_nodes",
  785. "min_impurity_decrease",
  786. "random_state",
  787. "min_patch_dims",
  788. "max_patch_dims",
  789. "dim_contiguous",
  790. "data_dims",
  791. "boundary",
  792. "feature_weight",
  793. ),
  794. bootstrap=bootstrap,
  795. oob_score=oob_score,
  796. n_jobs=n_jobs,
  797. random_state=random_state,
  798. verbose=verbose,
  799. warm_start=warm_start,
  800. class_weight=class_weight,
  801. max_samples=max_samples,
  802. )
  803. self.criterion = criterion
  804. self.max_depth = max_depth
  805. self.min_samples_split = min_samples_split
  806. self.min_samples_leaf = min_samples_leaf
  807. self.max_features = max_features
  808. self.min_patch_dims = min_patch_dims
  809. self.max_patch_dims = max_patch_dims
  810. self.dim_contiguous = dim_contiguous
  811. self.data_dims = data_dims
  812. self.boundary = boundary
  813. self.feature_weight = feature_weight
  814. # unused by oblique forests
  815. self.min_weight_fraction_leaf = min_weight_fraction_leaf
  816. self.max_leaf_nodes = max_leaf_nodes
  817. self.min_impurity_decrease = min_impurity_decrease
  818. class PatchObliqueRandomForestRegressor(SimMatrixMixin, ForestMixin, ForestRegressor):
  819. """A patch-oblique random forest regressor.
  820. A patch-oblique random forest is a meta estimator similar to a random
  821. forest that fits a number of patch oblique decision tree regressors
  822. on various sub-samples of the dataset and uses averaging to
  823. improve the predictive accuracy and control over-fitting. For more
  824. details, see :footcite:`Li2023manifold`.
  825. Parameters
  826. ----------
  827. n_estimators : int, default=100
  828. The number of trees in the forest.
  829. criterion : {"squared_error", "absolute_error", "friedman_mse", "poisson"},\
  830. default="squared_error"
  831. The function to measure the quality of a split. Supported criteria
  832. are "squared_error" for the mean squared error, which is equal to
  833. variance reduction as feature selection criterion and minimizes the L2
  834. loss using the mean of each terminal node, "friedman_mse", which uses
  835. mean squared error with Friedman's improvement score for potential
  836. splits, "absolute_error" for the mean absolute error, which minimizes
  837. the L1 loss using the median of each terminal node, and "poisson" which
  838. uses reduction in Poisson deviance to find splits.
  839. Training using "absolute_error" is significantly slower
  840. than when using "squared_error".
  841. max_depth : int, default=None
  842. The maximum depth of the tree. If None, then nodes are expanded until
  843. all leaves are pure or until all leaves contain less than
  844. min_samples_split samples.
  845. min_samples_split : int or float, default=2
  846. The minimum number of samples required to split an internal node:
  847. - If int, then consider `min_samples_split` as the minimum number.
  848. - If float, then `min_samples_split` is a fraction and
  849. `ceil(min_samples_split * n_samples)` are the minimum
  850. number of samples for each split.
  851. min_samples_leaf : int or float, default=1
  852. The minimum number of samples required to be at a leaf node.
  853. A split point at any depth will only be considered if it leaves at
  854. least ``min_samples_leaf`` training samples in each of the left and
  855. right branches. This may have the effect of smoothing the model,
  856. especially in regression.
  857. - If int, then consider `min_samples_leaf` as the minimum number.
  858. - If float, then `min_samples_leaf` is a fraction and
  859. `ceil(min_samples_leaf * n_samples)` are the minimum
  860. number of samples for each node.
  861. min_weight_fraction_leaf : float, default=0.0
  862. The minimum weighted fraction of the sum total of weights (of all
  863. the input samples) required to be at a leaf node. Samples have
  864. equal weight when sample_weight is not provided.
  865. max_features : {"sqrt", "log2", None}, int or float, default="sqrt"
  866. The number of features to consider when looking for the best split:
  867. - If int, then consider `max_features` features at each split.
  868. - If float, then `max_features` is a fraction and
  869. `round(max_features * n_features)` features are considered at each
  870. split.
  871. - If "auto", then `max_features=sqrt(n_features)`.
  872. - If "sqrt", then `max_features=sqrt(n_features)`.
  873. - If "log2", then `max_features=log2(n_features)`.
  874. - If None, then `max_features=n_features`.
  875. Note: the search for a split does not stop until at least one
  876. valid partition of the node samples is found, even if it requires to
  877. effectively inspect more than ``max_features`` features.
  878. max_leaf_nodes : int, default=None
  879. Grow trees with ``max_leaf_nodes`` in best-first fashion.
  880. Best nodes are defined as relative reduction in impurity.
  881. If None then unlimited number of leaf nodes.
  882. min_impurity_decrease : float, default=0.0
  883. A node will be split if this split induces a decrease of the impurity
  884. greater than or equal to this value.
  885. The weighted impurity decrease equation is the following::
  886. N_t / N * (impurity - N_t_R / N_t * right_impurity
  887. - N_t_L / N_t * left_impurity)
  888. where ``N`` is the total number of samples, ``N_t`` is the number of
  889. samples at the current node, ``N_t_L`` is the number of samples in the
  890. left child, and ``N_t_R`` is the number of samples in the right child.
  891. ``N``, ``N_t``, ``N_t_R`` and ``N_t_L`` all refer to the weighted sum,
  892. if ``sample_weight`` is passed.
  893. bootstrap : bool, default=True
  894. Whether bootstrap samples are used when building trees. If False, the
  895. whole dataset is used to build each tree.
  896. oob_score : bool, default=False
  897. Whether to use out-of-bag samples to estimate the generalization score.
  898. Only available if bootstrap=True.
  899. n_jobs : int, default=None
  900. The number of jobs to run in parallel. :meth:`fit`, :meth:`predict`,
  901. :meth:`decision_path` and :meth:`apply` are all parallelized over the
  902. trees. ``None`` means 1 unless in a `joblib.parallel_backend`
  903. context. ``-1`` means using all processors. See :term:`Glossary
  904. <n_jobs>` for more details.
  905. random_state : int, RandomState instance or None, default=None
  906. Controls both the randomness of the bootstrapping of the samples used
  907. when building trees (if ``bootstrap=True``) and the sampling of the
  908. features to consider when looking for the best split at each node
  909. (if ``max_features < n_features``).
  910. See :term:`Glossary <random_state>` for details.
  911. verbose : int, default=0
  912. Controls the verbosity when fitting and predicting.
  913. warm_start : bool, default=False
  914. When set to ``True``, reuse the solution of the previous call to fit
  915. and add more estimators to the ensemble, otherwise, just fit a whole
  916. new forest. See :term:`the Glossary <warm_start>`.
  917. max_samples : int or float, default=None
  918. If bootstrap is True, the number of samples to draw from X
  919. to train each base estimator.
  920. - If None (default), then draw `X.shape[0]` samples.
  921. - If int, then draw `max_samples` samples.
  922. - If float, then draw `max_samples * X.shape[0]` samples. Thus,
  923. `max_samples` should be in the interval `(0.0, 1.0]`.
  924. min_patch_dims : array-like, optional
  925. The minimum dimensions of a patch, by default 1 along all dimensions.
  926. max_patch_dims : array-like, optional
  927. The maximum dimensions of a patch, by default 1 along all dimensions.
  928. dim_contiguous : array-like of bool, optional
  929. Whether or not each patch is sampled contiguously along this dimension.
  930. data_dims : array-like, optional
  931. The presumed dimensions of the un-vectorized feature vector, by default
  932. will be a 1D vector with (1, n_features) shape.
  933. boundary : optional, str {'wrap'}
  934. The boundary condition to use when sampling patches, by default None.
  935. 'wrap' corresponds to the boundary condition as is in numpy and scipy.
  936. feature_weight : array-like of shape (n_features,), default=None
  937. Feature weights. If None, then features are equally weighted as is.
  938. If provided, then the feature weights are used to weight the
  939. patches that are generated. The feature weights are used
  940. as follows: for every patch that is sampled, the feature weights over
  941. the entire patch is summed and normalizes the patch.
  942. Attributes
  943. ----------
  944. estimators_ : list of PatchObliqueDecisionTreeRegressor
  945. The collection of fitted sub-estimators.
  946. n_features_ : int
  947. The number of features when ``fit`` is performed.
  948. n_features_in_ : int
  949. Number of features seen during :term:`fit`.
  950. feature_names_in_ : ndarray of shape (`n_features_in_`,)
  951. Names of features seen during :term:`fit`. Defined only when `X`
  952. has feature names that are all strings.
  953. n_outputs_ : int
  954. The number of outputs when ``fit`` is performed.
  955. feature_importances_ : ndarray of shape (n_features,)
  956. The impurity-based feature importances.
  957. The higher, the more important the feature.
  958. The importance of a feature is computed as the (normalized)
  959. total reduction of the criterion brought by that feature. It is also
  960. known as the Gini importance.
  961. Warning: impurity-based feature importances can be misleading for
  962. high cardinality features (many unique values). See
  963. :func:`sklearn.inspection.permutation_importance` as an alternative.
  964. oob_score_ : float
  965. Score of the training dataset obtained using an out-of-bag estimate.
  966. This attribute exists only when ``oob_score`` is True.
  967. oob_prediction_ : ndarray of shape (n_samples,) or (n_samples, n_outputs)
  968. Prediction computed with out-of-bag estimate on the training set.
  969. This attribute exists only when ``oob_score`` is True.
  970. See Also
  971. --------
  972. treeple.tree.ObliqueDecisionTreeRegressor : An oblique decision
  973. tree regressor.
  974. sklearn.ensemble.RandomForestRegressor : An axis-aligned decision
  975. forest regressor.
  976. Notes
  977. -----
  978. The default values for the parameters controlling the size of the trees
  979. (e.g. ``max_depth``, ``min_samples_leaf``, etc.) lead to fully grown and
  980. unpruned trees which can potentially be very large on some data sets. To
  981. reduce memory consumption, the complexity and size of the trees should be
  982. controlled by setting those parameter values.
  983. The features are always randomly permuted at each split. Therefore,
  984. the best found split may vary, even with the same training data,
  985. ``max_features=n_features`` and ``bootstrap=False``, if the improvement
  986. of the criterion is identical for several splits enumerated during the
  987. search of the best split. To obtain a deterministic behaviour during
  988. fitting, ``random_state`` has to be fixed.
  989. References
  990. ----------
  991. .. footbibliography::
  992. Examples
  993. --------
  994. >>> from treeple.ensemble import PatchObliqueRandomForestRegressor
  995. >>> from sklearn.datasets import make_regression
  996. >>> X, y = make_regression(n_features=4, n_informative=2,
  997. ... random_state=0, shuffle=False)
  998. >>> regressor = PatchObliqueRandomForestRegressor(max_depth=2, random_state=0)
  999. >>> regressor.fit(X, y)
  1000. PatchObliqueRandomForestRegressor(...)
  1001. >>> print(regressor.predict([[0, 0, 0, 0]]))
  1002. [-5.82818509]
  1003. """
  1004. tree_type = "oblique"
  1005. _parameter_constraints: dict = {
  1006. **ForestRegressor._parameter_constraints,
  1007. **PatchObliqueDecisionTreeRegressor._parameter_constraints,
  1008. }
  1009. _parameter_constraints.pop("splitter")
  1010. def __init__(
  1011. self,
  1012. n_estimators=100,
  1013. *,
  1014. criterion="squared_error",
  1015. max_depth=None,
  1016. min_samples_split=2,
  1017. min_samples_leaf=1,
  1018. min_weight_fraction_leaf=0.0,
  1019. max_features=1.0,
  1020. max_leaf_nodes=None,
  1021. min_impurity_decrease=0.0,
  1022. bootstrap=True,
  1023. oob_score=False,
  1024. n_jobs=None,
  1025. random_state=None,
  1026. verbose=0,
  1027. warm_start=False,
  1028. max_samples=None,
  1029. min_patch_dims=None,
  1030. max_patch_dims=None,
  1031. dim_contiguous=None,
  1032. data_dims=None,
  1033. boundary=None,
  1034. feature_weight=None,
  1035. ):
  1036. super().__init__(
  1037. estimator=PatchObliqueDecisionTreeRegressor(),
  1038. n_estimators=n_estimators,
  1039. estimator_params=(
  1040. "criterion",
  1041. "max_depth",
  1042. "min_samples_split",
  1043. "min_samples_leaf",
  1044. "min_weight_fraction_leaf",
  1045. "max_features",
  1046. "max_leaf_nodes",
  1047. "min_impurity_decrease",
  1048. "random_state",
  1049. "min_patch_dims",
  1050. "max_patch_dims",
  1051. "dim_contiguous",
  1052. "data_dims",
  1053. "boundary",
  1054. "feature_weight",
  1055. ),
  1056. bootstrap=bootstrap,
  1057. oob_score=oob_score,
  1058. n_jobs=n_jobs,
  1059. random_state=random_state,
  1060. verbose=verbose,
  1061. warm_start=warm_start,
  1062. max_samples=max_samples,
  1063. )
  1064. self.criterion = criterion
  1065. self.max_depth = max_depth
  1066. self.min_samples_split = min_samples_split
  1067. self.min_samples_leaf = min_samples_leaf
  1068. self.max_features = max_features
  1069. self.min_patch_dims = min_patch_dims
  1070. self.max_patch_dims = max_patch_dims
  1071. self.dim_contiguous = dim_contiguous
  1072. self.data_dims = data_dims
  1073. self.boundary = boundary
  1074. self.feature_weight = feature_weight
  1075. # unused by oblique forests
  1076. self.min_weight_fraction_leaf = min_weight_fraction_leaf
  1077. self.max_leaf_nodes = max_leaf_nodes
  1078. self.min_impurity_decrease = min_impurity_decrease
  1079. class ExtraObliqueRandomForestClassifier(
  1080. SimMatrixMixin, ForestClassifierMixin, ForestMixin, ForestClassifier
  1081. ):
  1082. """
  1083. An extra oblique random forest classifier.
  1084. An extra oblique random forest is a meta estimator similar to a random
  1085. forest that fits a number of extra oblique decision tree classifiers
  1086. on various sub-samples of the dataset and uses averaging to
  1087. improve the predictive accuracy and control over-fitting.
  1088. The sub-sample size is controlled with the `max_samples` parameter if
  1089. `bootstrap=True` (default), otherwise the whole dataset is used to build
  1090. each tree.
  1091. Read more in the :ref:`User Guide <sklearn:forest>`.
  1092. Parameters
  1093. ----------
  1094. n_estimators : int, default=100
  1095. The number of trees in the forest.
  1096. criterion : {"gini", "entropy"}, default="gini"
  1097. The function to measure the quality of a split. Supported criteria are
  1098. "gini" for the Gini impurity and "entropy" for the information gain.
  1099. Note: this parameter is tree-specific.
  1100. max_depth : int, default=None
  1101. The maximum depth of the tree. If None, then nodes are expanded until
  1102. all leaves are pure or until all leaves contain less than
  1103. min_samples_split samples.
  1104. min_samples_split : int or float, default=2
  1105. The minimum number of samples required to split an internal node:
  1106. - If int, then consider `min_samples_split` as the minimum number.
  1107. - If float, then `min_samples_split` is a fraction and
  1108. `ceil(min_samples_split * n_samples)` are the minimum
  1109. number of samples for each split.
  1110. min_samples_leaf : int or float, default=1
  1111. The minimum number of samples required to be at a leaf node.
  1112. A split point at any depth will only be considered if it leaves at
  1113. least ``min_samples_leaf`` training samples in each of the left and
  1114. right branches. This may have the effect of smoothing the model,
  1115. especially in regression.
  1116. - If int, then consider `min_samples_leaf` as the minimum number.
  1117. - If float, then `min_samples_leaf` is a fraction and
  1118. `ceil(min_samples_leaf * n_samples)` are the minimum
  1119. number of samples for each node.
  1120. min_weight_fraction_leaf : float, default=0.0
  1121. The minimum weighted fraction of the sum total of weights (of all
  1122. the input samples) required to be at a leaf node. Samples have
  1123. equal weight when sample_weight is not provided.
  1124. max_features : {"sqrt", "log2", None}, int or float, default="sqrt"
  1125. The number of features to consider when looking for the best split:
  1126. - If int, then consider `max_features` features at each split.
  1127. - If float, then `max_features` is a fraction and
  1128. `round(max_features * n_features)` features are considered at each
  1129. split.
  1130. - If "auto", then `max_features=sqrt(n_features)`.
  1131. - If "sqrt", then `max_features=sqrt(n_features)`.
  1132. - If "log2", then `max_features=log2(n_features)`.
  1133. - If None, then `max_features=n_features`.
  1134. Note: the search for a split does not stop until at least one
  1135. valid partition of the node samples is found, even if it requires to
  1136. effectively inspect more than ``max_features`` features.
  1137. max_leaf_nodes : int, default=None
  1138. Grow trees with ``max_leaf_nodes`` in best-first fashion.
  1139. Best nodes are defined as relative reduction in impurity.
  1140. If None then unlimited number of leaf nodes.
  1141. min_impurity_decrease : float, default=0.0
  1142. A node will be split if this split induces a decrease of the impurity
  1143. greater than or equal to this value.
  1144. The weighted impurity decrease equation is the following::
  1145. N_t / N * (impurity - N_t_R / N_t * right_impurity
  1146. - N_t_L / N_t * left_impurity)
  1147. where ``N`` is the total number of samples, ``N_t`` is the number of
  1148. samples at the current node, ``N_t_L`` is the number of samples in the
  1149. left child, and ``N_t_R`` is the number of samples in the right child.
  1150. ``N``, ``N_t``, ``N_t_R`` and ``N_t_L`` all refer to the weighted sum,
  1151. if ``sample_weight`` is passed.
  1152. bootstrap : bool, default=True
  1153. Whether bootstrap samples are used when building trees. If False, the
  1154. whole dataset is used to build each tree.
  1155. oob_score : bool, default=False
  1156. Whether to use out-of-bag samples to estimate the generalization score.
  1157. Only available if bootstrap=True.
  1158. n_jobs : int, default=None
  1159. The number of jobs to run in parallel. :meth:`fit`, :meth:`predict`,
  1160. :meth:`decision_path` and :meth:`apply` are all parallelized over the
  1161. trees. ``None`` means 1 unless in a :obj:`joblib.parallel_backend`
  1162. context. ``-1`` means using all processors. See :term:`Glossary
  1163. <n_jobs>` for more details.
  1164. random_state : int, RandomState instance or None, default=None
  1165. Controls both the randomness of the bootstrapping of the samples used
  1166. when building trees (if ``bootstrap=True``) and the sampling of the
  1167. features to consider when looking for the best split at each node
  1168. (if ``max_features < n_features``).
  1169. See :term:`Glossary <random_state>` for details.
  1170. verbose : int, default=0
  1171. Controls the verbosity when fitting and predicting.
  1172. warm_start : bool, default=False
  1173. When set to ``True``, reuse the solution of the previous call to fit
  1174. and add more estimators to the ensemble, otherwise, just fit a whole
  1175. new forest. See :term:`the Glossary <warm_start>`.
  1176. class_weight : {"balanced", "balanced_subsample"}, dict or list of dicts, \
  1177. default=None
  1178. Weights associated with classes in the form ``{class_label: weight}``.
  1179. If not given, all classes are supposed to have weight one. For
  1180. multi-output problems, a list of dicts can be provided in the same
  1181. order as the columns of y.
  1182. Note that for multioutput (including multilabel) weights should be
  1183. defined for each class of every column in its own dict. For example,
  1184. for four-class multilabel classification weights should be
  1185. [{0: 1, 1: 1}, {0: 1, 1: 5}, {0: 1, 1: 1}, {0: 1, 1: 1}] instead of
  1186. [{1:1}, {2:5}, {3:1}, {4:1}].
  1187. The "balanced" mode uses the values of y to automatically adjust
  1188. weights inversely proportional to class frequencies in the input data
  1189. as ``n_samples / (n_classes * np.bincount(y))``
  1190. The "balanced_subsample" mode is the same as "balanced" except that
  1191. weights are computed based on the bootstrap sample for every tree
  1192. grown.
  1193. For multi-output, the weights of each column of y will be multiplied.
  1194. Note that these weights will be multiplied with sample_weight (passed
  1195. through the fit method) if sample_weight is specified.
  1196. max_samples : int or float, default=None
  1197. If bootstrap is True, the number of samples to draw from X
  1198. to train each base estimator.
  1199. - If None (default), then draw `X.shape[0]` samples.
  1200. - If int, then draw `max_samples` samples.
  1201. - If float, then draw `max_samples * X.shape[0]` samples. Thus,
  1202. `max_samples` should be in the interval `(0.0, 1.0]`.
  1203. feature_combinations : float, default=None
  1204. The number of features to combine on average at each split
  1205. of the decision trees. If ``None``, then will default to the minimum of
  1206. ``(1.5, n_features)``. This controls the number of non-zeros is the
  1207. projection matrix. Setting the value to 1.0 is equivalent to a
  1208. traditional decision-tree. ``feature_combinations * max_features``
  1209. gives the number of expected non-zeros in the projection matrix of shape
  1210. ``(max_features, n_features)``. Thus this value must always be less than
  1211. ``n_features`` in order to be valid.
  1212. Attributes
  1213. ----------
  1214. estimators_ : list of treeple.tree.ExtraObliqueDecisionTreeClassifier
  1215. The collection of fitted sub-estimators.
  1216. classes_ : ndarray of shape (n_classes,) or a list of such arrays
  1217. The classes labels (single output problem), or a list of arrays of
  1218. class labels (multi-output problem).
  1219. n_classes_ : int or list
  1220. The number of classes (single output problem), or a list containing the
  1221. number of classes for each output (multi-output problem).
  1222. n_features_ : int
  1223. The number of features when ``fit`` is performed.
  1224. n_features_in_ : int
  1225. Number of features seen during :term:`fit`.
  1226. feature_names_in_ : ndarray of shape (`n_features_in_`,)
  1227. Names of features seen during :term:`fit`. Defined only when `X`
  1228. has feature names that are all strings.
  1229. n_outputs_ : int
  1230. The number of outputs when ``fit`` is performed.
  1231. feature_importances_ : ndarray of shape (n_features,)
  1232. The impurity-based feature importances.
  1233. The higher, the more important the feature.
  1234. The importance of a feature is computed as the (normalized)
  1235. total reduction of the criterion brought by that feature. It is also
  1236. known as the Gini importance.
  1237. Warning: impurity-based feature importances can be misleading for
  1238. high cardinality features (many unique values). See
  1239. :func:`sklearn.inspection.permutation_importance` as an alternative.
  1240. oob_score_ : float
  1241. Score of the training dataset obtained using an out-of-bag estimate.
  1242. This attribute exists only when ``oob_score`` is True.
  1243. oob_decision_function_ : ndarray of shape (n_samples, n_classes) or \
  1244. (n_samples, n_classes, n_outputs)
  1245. Decision function computed with out-of-bag estimate on the training
  1246. set. If n_estimators is small it might be possible that a data point
  1247. was never left out during the bootstrap. In this case,
  1248. `oob_decision_function_` might contain NaN. This attribute exists
  1249. only when ``oob_score`` is True.
  1250. See Also
  1251. --------
  1252. treeple.tree.ExtraObliqueDecisionTreeClassifier : An extremely randomized oblique decision
  1253. tree classifier.
  1254. treeple.tree.ObliqueDecisionTreeClassifier : An oblique decision tree classifier.
  1255. sklearn.ensemble.RandomForestClassifier : An axis-aligned decision
  1256. forest classifier.
  1257. Notes
  1258. -----
  1259. The default values for the parameters controlling the size of the trees
  1260. (e.g. ``max_depth``, ``min_samples_leaf``, etc.) lead to fully grown and
  1261. unpruned trees which can potentially be very large on some data sets. To
  1262. reduce memory consumption, the complexity and size of the trees should be
  1263. controlled by setting those parameter values.
  1264. The features are always randomly permuted at each split. Therefore,
  1265. the best found split may vary, even with the same training data,
  1266. ``max_features=n_features`` and ``bootstrap=False``, if the improvement
  1267. of the criterion is identical for several splits enumerated during the
  1268. search of the best split. To obtain a deterministic behaviour during
  1269. fitting, ``random_state`` has to be fixed.
  1270. References
  1271. ----------
  1272. .. [1] L. Breiman, "Random Forests", Machine Learning, 45(1), 5-32, 2001.
  1273. .. [2] P. Geurts, D. Ernst., and L. Wehenkel, "Extremely randomized trees",
  1274. Machine Learning, 63(1), 3-42, 2006.
  1275. Examples
  1276. --------
  1277. >>> from treeple.ensemble import ExtraObliqueRandomForestClassifier
  1278. >>> from sklearn.datasets import make_classification
  1279. >>> X, y = make_classification(n_samples=1000, n_features=4,
  1280. ... n_informative=2, n_redundant=0,
  1281. ... random_state=0, shuffle=False)
  1282. >>> clf = ExtraObliqueRandomForestClassifier(max_depth=2, random_state=0)
  1283. >>> clf.fit(X, y)
  1284. ExtraObliqueRandomForestClassifier(...)
  1285. >>> print(clf.predict([[0, 0, 0, 0]]))
  1286. [1]
  1287. """
  1288. _parameter_constraints: dict = {
  1289. **ForestClassifier._parameter_constraints,
  1290. **ExtraObliqueDecisionTreeClassifier._parameter_constraints,
  1291. "class_weight": [
  1292. StrOptions({"balanced_subsample", "balanced"}),
  1293. dict,
  1294. list,
  1295. None,
  1296. ],
  1297. }
  1298. _parameter_constraints.pop("splitter")
  1299. def __init__(
  1300. self,
  1301. n_estimators=100,
  1302. *,
  1303. criterion="gini",
  1304. max_depth=None,
  1305. min_samples_split=2,
  1306. min_samples_leaf=1,
  1307. min_weight_fraction_leaf=0.0,
  1308. max_features="sqrt",
  1309. max_leaf_nodes=None,
  1310. min_impurity_decrease=0.0,
  1311. bootstrap=True,
  1312. oob_score=False,
  1313. n_jobs=None,
  1314. random_state=None,
  1315. verbose=0,
  1316. warm_start=False,
  1317. class_weight=None,
  1318. max_samples=None,
  1319. feature_combinations=None,
  1320. ):
  1321. super().__init__(
  1322. estimator=ExtraObliqueDecisionTreeClassifier(),
  1323. n_estimators=n_estimators,
  1324. estimator_params=(
  1325. "criterion",
  1326. "max_depth",
  1327. "min_samples_split",
  1328. "min_samples_leaf",
  1329. "min_weight_fraction_leaf",
  1330. "max_features",
  1331. "max_leaf_nodes",
  1332. "min_impurity_decrease",
  1333. "random_state",
  1334. "feature_combinations",
  1335. ),
  1336. bootstrap=bootstrap,
  1337. oob_score=oob_score,
  1338. n_jobs=n_jobs,
  1339. random_state=random_state,
  1340. verbose=verbose,
  1341. warm_start=warm_start,
  1342. class_weight=class_weight,
  1343. max_samples=max_samples,
  1344. )
  1345. self.criterion = criterion
  1346. self.max_depth = max_depth
  1347. self.min_samples_split = min_samples_split
  1348. self.min_samples_leaf = min_samples_leaf
  1349. self.max_features = max_features
  1350. self.feature_combinations = feature_combinations
  1351. # unused by oblique forests
  1352. self.min_weight_fraction_leaf = min_weight_fraction_leaf
  1353. self.max_leaf_nodes = max_leaf_nodes
  1354. self.min_impurity_decrease = min_impurity_decrease
  1355. class ExtraObliqueRandomForestRegressor(SimMatrixMixin, ForestMixin, ForestRegressor):
  1356. """An extra oblique random forest regressor.
  1357. An extra oblique random forest is a meta estimator similar to a random
  1358. forest that fits a number of extra oblique decision tree regressor
  1359. on various sub-samples of the dataset and uses averaging to
  1360. improve the predictive accuracy and control over-fitting.
  1361. The sub-sample size is controlled with the `max_samples` parameter if
  1362. `bootstrap=True` (default), otherwise the whole dataset is used to build
  1363. each tree.
  1364. Read more in the :ref:`User Guide <sklearn:forest>`.
  1365. Parameters
  1366. ----------
  1367. n_estimators : int, default=100
  1368. The number of trees in the forest.
  1369. criterion : {"squared_error", "absolute_error", "friedman_mse", "poisson"}, \
  1370. default="squared_error"
  1371. The function to measure the quality of a split. Supported criteria
  1372. are "squared_error" for the mean squared error, which is equal to
  1373. variance reduction as feature selection criterion and minimizes the L2
  1374. loss using the mean of each terminal node, "friedman_mse", which uses
  1375. mean squared error with Friedman's improvement score for potential
  1376. splits, "absolute_error" for the mean absolute error, which minimizes
  1377. the L1 loss using the median of each terminal node, and "poisson" which
  1378. uses reduction in Poisson deviance to find splits.
  1379. Training using "absolute_error" is significantly slower
  1380. than when using "squared_error".
  1381. max_depth : int, default=None
  1382. The maximum depth of the tree. If None, then nodes are expanded until
  1383. all leaves are pure or until all leaves contain less than
  1384. min_samples_split samples.
  1385. min_samples_split : int or float, default=2
  1386. The minimum number of samples required to split an internal node:
  1387. - If int, then consider `min_samples_split` as the minimum number.
  1388. - If float, then `min_samples_split` is a fraction and
  1389. `ceil(min_samples_split * n_samples)` are the minimum
  1390. number of samples for each split.
  1391. min_samples_leaf : int or float, default=1
  1392. The minimum number of samples required to be at a leaf node.
  1393. A split point at any depth will only be considered if it leaves at
  1394. least ``min_samples_leaf`` training samples in each of the left and
  1395. right branches. This may have the effect of smoothing the model,
  1396. especially in regression.
  1397. - If int, then consider `min_samples_leaf` as the minimum number.
  1398. - If float, then `min_samples_leaf` is a fraction and
  1399. `ceil(min_samples_leaf * n_samples)` are the minimum
  1400. number of samples for each node.
  1401. min_weight_fraction_leaf : float, default=0.0
  1402. The minimum weighted fraction of the sum total of weights (of all
  1403. the input samples) required to be at a leaf node. Samples have
  1404. equal weight when sample_weight is not provided.
  1405. max_features : {"sqrt", "log2", None}, int or float, default="sqrt"
  1406. The number of features to consider when looking for the best split:
  1407. - If int, then consider `max_features` features at each split.
  1408. - If float, then `max_features` is a fraction and
  1409. `round(max_features * n_features)` features are considered at each
  1410. split.
  1411. - If "auto", then `max_features=sqrt(n_features)`.
  1412. - If "sqrt", then `max_features=sqrt(n_features)`.
  1413. - If "log2", then `max_features=log2(n_features)`.
  1414. - If None, then `max_features=n_features`.
  1415. Note: the search for a split does not stop until at least one
  1416. valid partition of the node samples is found, even if it requires to
  1417. effectively inspect more than ``max_features`` features.
  1418. max_leaf_nodes : int, default=None
  1419. Grow trees with ``max_leaf_nodes`` in best-first fashion.
  1420. Best nodes are defined as relative reduction in impurity.
  1421. If None then unlimited number of leaf nodes.
  1422. min_impurity_decrease : float, default=0.0
  1423. A node will be split if this split induces a decrease of the impurity
  1424. greater than or equal to this value.
  1425. The weighted impurity decrease equation is the following::
  1426. N_t / N * (impurity - N_t_R / N_t * right_impurity
  1427. - N_t_L / N_t * left_impurity)
  1428. where ``N`` is the total number of samples, ``N_t`` is the number of
  1429. samples at the current node, ``N_t_L`` is the number of samples in the
  1430. left child, and ``N_t_R`` is the number of samples in the right child.
  1431. ``N``, ``N_t``, ``N_t_R`` and ``N_t_L`` all refer to the weighted sum,
  1432. if ``sample_weight`` is passed.
  1433. bootstrap : bool, default=True
  1434. Whether bootstrap samples are used when building trees. If False, the
  1435. whole dataset is used to build each tree.
  1436. oob_score : bool, default=False
  1437. Whether to use out-of-bag samples to estimate the generalization score.
  1438. Only available if bootstrap=True.
  1439. n_jobs : int, default=None
  1440. The number of jobs to run in parallel. :meth:`fit`, :meth:`predict`,
  1441. :meth:`decision_path` and :meth:`apply` are all parallelized over the
  1442. trees. ``None`` means 1 unless in a :obj:`joblib.parallel_backend`
  1443. context. ``-1`` means using all processors. See :term:`Glossary
  1444. <n_jobs>` for more details.
  1445. random_state : int, RandomState instance or None, default=None
  1446. Controls both the randomness of the bootstrapping of the samples used
  1447. when building trees (if ``bootstrap=True``) and the sampling of the
  1448. features to consider when looking for the best split at each node
  1449. (if ``max_features < n_features``).
  1450. See :term:`Glossary <random_state>` for details.
  1451. verbose : int, default=0
  1452. Controls the verbosity when fitting and predicting.
  1453. warm_start : bool, default=False
  1454. When set to ``True``, reuse the solution of the previous call to fit
  1455. and add more estimators to the ensemble, otherwise, just fit a whole
  1456. new forest. See :term:`the Glossary <warm_start>`.
  1457. max_samples : int or float, default=None
  1458. If bootstrap is True, the number of samples to draw from X
  1459. to train each base estimator.
  1460. - If None (default), then draw `X.shape[0]` samples.
  1461. - If int, then draw `max_samples` samples.
  1462. - If float, then draw `max_samples * X.shape[0]` samples. Thus,
  1463. `max_samples` should be in the interval `(0.0, 1.0]`.
  1464. feature_combinations : float, default=None
  1465. The number of features to combine on average at each split
  1466. of the decision trees. If ``None``, then will default to the minimum of
  1467. ``(1.5, n_features)``. This controls the number of non-zeros is the
  1468. projection matrix. Setting the value to 1.0 is equivalent to a
  1469. traditional decision-tree. ``feature_combinations * max_features``
  1470. gives the number of expected non-zeros in the projection matrix of shape
  1471. ``(max_features, n_features)``. Thus this value must always be less than
  1472. ``n_features`` in order to be valid.
  1473. Attributes
  1474. ----------
  1475. estimators_ : list of ExtraObliqueDecisionTreeRegressor
  1476. The collection of fitted sub-estimators.
  1477. n_features_ : int
  1478. The number of features when ``fit`` is performed.
  1479. n_features_in_ : int
  1480. Number of features seen during :term:`fit`.
  1481. feature_names_in_ : ndarray of shape (`n_features_in_`,)
  1482. Names of features seen during :term:`fit`. Defined only when `X`
  1483. has feature names that are all strings.
  1484. n_outputs_ : int
  1485. The number of outputs when ``fit`` is performed.
  1486. feature_importances_ : ndarray of shape (n_features,)
  1487. The impurity-based feature importances.
  1488. The higher, the more important the feature.
  1489. The importance of a feature is computed as the (normalized)
  1490. total reduction of the criterion brought by that feature. It is also
  1491. known as the Gini importance.
  1492. Warning: impurity-based feature importances can be misleading for
  1493. high cardinality features (many unique values). See
  1494. :func:`sklearn.inspection.permutation_importance` as an alternative.
  1495. oob_score_ : float
  1496. Score of the training dataset obtained using an out-of-bag estimate.
  1497. This attribute exists only when ``oob_score`` is True.
  1498. oob_decision_function_ : ndarray of shape (n_samples, n_classes) or \
  1499. (n_samples, n_classes, n_outputs)
  1500. Decision function computed with out-of-bag estimate on the training
  1501. set. If n_estimators is small it might be possible that a data point
  1502. was never left out during the bootstrap. In this case,
  1503. `oob_decision_function_` might contain NaN. This attribute exists
  1504. only when ``oob_score`` is True.
  1505. See Also
  1506. --------
  1507. treeple.tree.ExtraObliqueDecisionTreeRegressor : An extra oblique decision
  1508. tree regressor.
  1509. treeple.tree.ObliqueDecisionTreeRegressor : An oblique decision
  1510. tree regressor.
  1511. sklearn.ensemble.RandomForestRegressor : An axis-aligned decision
  1512. forest regressor.
  1513. Notes
  1514. -----
  1515. The default values for the parameters controlling the size of the trees
  1516. (e.g. ``max_depth``, ``min_samples_leaf``, etc.) lead to fully grown and
  1517. unpruned trees which can potentially be very large on some data sets. To
  1518. reduce memory consumption, the complexity and size of the trees should be
  1519. controlled by setting those parameter values.
  1520. The features are always randomly permuted at each split. Therefore,
  1521. the best found split may vary, even with the same training data,
  1522. ``max_features=n_features`` and ``bootstrap=False``, if the improvement
  1523. of the criterion is identical for several splits enumerated during the
  1524. search of the best split. To obtain a deterministic behaviour during
  1525. fitting, ``random_state`` has to be fixed.
  1526. References
  1527. ----------
  1528. .. [1] L. Breiman, "Random Forests", Machine Learning, 45(1), 5-32, 2001.
  1529. .. [2] T. Tomita, "Sparse Projection Oblique Randomer Forests", \
  1530. Journal of Machine Learning Research, 21(104), 1-39, 2020.
  1531. .. [3] P. Geurts, D. Ernst., and L. Wehenkel, "Extremely randomized trees", \
  1532. Machine Learning, 63(1), 3-42, 2006.
  1533. Examples
  1534. --------
  1535. >>> from treeple.ensemble import ExtraObliqueRandomForestRegressor
  1536. >>> from sklearn.datasets import make_regression
  1537. >>> X, y = make_regression(n_features=4, n_informative=2,
  1538. ... random_state=0, shuffle=False)
  1539. >>> regr = ExtraObliqueRandomForestRegressor(max_depth=2, random_state=0)
  1540. >>> regr.fit(X, y)
  1541. ExtraObliqueRandomForestRegressor(...)
  1542. >>> print(regr.predict([[0, 0, 0, 0]]))
  1543. [-3.05063517]
  1544. """
  1545. _parameter_constraints: dict = {
  1546. **ForestRegressor._parameter_constraints,
  1547. **ExtraObliqueDecisionTreeRegressor._parameter_constraints,
  1548. }
  1549. _parameter_constraints.pop("splitter")
  1550. def __init__(
  1551. self,
  1552. n_estimators=100,
  1553. *,
  1554. criterion="squared_error",
  1555. max_depth=None,
  1556. min_samples_split=2,
  1557. min_samples_leaf=1,
  1558. min_weight_fraction_leaf=0.0,
  1559. max_features=1.0,
  1560. max_leaf_nodes=None,
  1561. min_impurity_decrease=0.0,
  1562. bootstrap=True,
  1563. oob_score=False,
  1564. n_jobs=None,
  1565. random_state=None,
  1566. verbose=0,
  1567. warm_start=False,
  1568. max_samples=None,
  1569. feature_combinations=None,
  1570. ):
  1571. super().__init__(
  1572. estimator=ExtraObliqueDecisionTreeRegressor(),
  1573. n_estimators=n_estimators,
  1574. estimator_params=(
  1575. "criterion",
  1576. "max_depth",
  1577. "min_samples_split",
  1578. "min_samples_leaf",
  1579. "min_weight_fraction_leaf",
  1580. "max_features",
  1581. "max_leaf_nodes",
  1582. "min_impurity_decrease",
  1583. "random_state",
  1584. "feature_combinations",
  1585. ),
  1586. bootstrap=bootstrap,
  1587. oob_score=oob_score,
  1588. n_jobs=n_jobs,
  1589. random_state=random_state,
  1590. verbose=verbose,
  1591. warm_start=warm_start,
  1592. max_samples=max_samples,
  1593. )
  1594. self.criterion = criterion
  1595. self.max_depth = max_depth
  1596. self.min_samples_split = min_samples_split
  1597. self.min_samples_leaf = min_samples_leaf
  1598. self.min_weight_fraction_leaf = min_weight_fraction_leaf
  1599. self.max_features = max_features
  1600. self.max_leaf_nodes = max_leaf_nodes
  1601. self.min_impurity_decrease = min_impurity_decrease
  1602. self.feature_combinations = feature_combinations
  1603. # unused by oblique forests
  1604. self.min_weight_fraction_leaf = min_weight_fraction_leaf
  1605. self.max_leaf_nodes = max_leaf_nodes
  1606. self.min_impurity_decrease = min_impurity_decrease

_supervised_forest.py at commit 75c2cf9, under other · at the source

Overview

Authors: Tingshan Liu1, Jayanta Dey1, Beiya Xu1, Eric W. Bridgeford2, Samuel Alldritt3, Karl-Heinz Nenning4, Kyoungseob Byeon3, Ting Xu3, Joshua T. Vogelstein5
  1. Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Maryland, United States of America
  2. Department of Psychology, Stanford University, Stanford, California, United States of America
  3. Center for the Developing Brain, Child Mind Institute, New York, New York, United States of America
  4. Nathan S. Kline Institute for Psychiatric Research, Orangeburg, New York, United States of America
  5. Department of Biomedical Engineering, Institute for Computational Medicine, Kavli Neuroscience Discovery Institute, Johns Hopkins University, Baltimore, Maryland, United States of America
Institutions: Johns Hopkins University (United States); The University of Texas at San Antonio (United States); Child Mind Institute (United States)
Journal: PloS one, volume 21, issue 4, article e0346575
Dates: received 17 July 2025; accepted 21 March 2026; published online 16 April 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1371/journal.pone.0346575 · PMID 41989998 · PMCID PMC13086332 · OpenAlex W4412440324
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: structural MRI / diffusion (modality), human (organism), non-human primate (organism), methods / tools (subfield)
Methods: Connectivity, Statistics, Machine learning, Preprocessing
MeSH: Brain*, Machine Learning*, Neuroimaging*, Algorithms, Animals, Classification Algorithms, Female, Humans, Macaca, Magnetic Resonance Imaging, Male, Random Forest, Sex Characteristics (* major topic)
Journal subjects: Biology and life sciences, Organisms, Eukaryota, Animals, Vertebrates, Amniotes, Mammals, Primates, Monkeys, Old World monkeys, Macaque, Zoology, Medicine and Health Sciences, Diagnostic Medicine, Diagnostic Radiology, Magnetic Resonance Imaging, Research and Analysis Methods, Imaging Techniques, Radiology and Imaging, Neuroimaging, Neuroscience, Plants, Trees, Physical Sciences, Mathematics, Applied Mathematics, Algorithms, Simulation and Modeling, Anatomy, Nervous System, Central Nervous System, Discrete Mathematics, Combinatorics, Permutation, Computer and Information Sciences, Artificial Intelligence, Machine Learning
Topic: Genetic Associations and Epidemiology (Genetics, Biochemistry, Genetics and Molecular Biology), according to OpenAlex
Funding: NIH (2U19NS104653, 1RF1MH128696); National Science Foundation (NSF 20–540)
Citations: not cited yet (Europe PMC); 34 references in the paper

Abstract

Sex classification using neuroimaging data has the potential to revolutionize personalized diagnostics by revealing subtle structural brain differences that underlie sex-specific disease risks. Despite the promise of machine learning, traditional methods often fall short in providing both high classification accuracy and interpretable, statistically validated feature importance scores for high-dimensional imaging data. This gap is particularly evident when conventional techniques such as random forests, LIME, and SHAP are applied, as they struggle with complex feature interactions and managing noise in large datasets. We address this challenge by developing an integrated framework that combines Oblique Random Forests (ORFs) with a novel, permutation-based feature importance testing algorithm. ORFs extend traditional random forests by employing oblique decision boundaries through linear combinations of features, thereby capturing intricate interactions inherent in neuroimaging data. Our feature importance testing method, NEOFIT, rigorously quantifies the significance of each feature by generating null distributions and corrected p-values. We first validate our approach using simulated datasets, establishing its robustness and scalability under controlled conditions. We then apply our method to classify sex from both voxel-wise structural MRI and cortical thickness data in humans and macaques, facilitating direct cross-species comparisons. ORFs achieves AUC > 0.80 on human data, and >0.70 on macaque data, while NEOFIT identifies statistically significant features aligned with sex-dimorphic neuroanatomy. Our results demonstrate that the proposed framework not only enhances classification performance but also provides clear, interpretable insights into the neuroanatomical features that distinguish sexes. These methodological advancements pave the way for improved diagnostic tools and contribute to a deeper understanding of the evolutionary basis of sex differences in brain structure.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 9 matches between paragraphs and lines of code.

google/yggdrasil-decision-forests

License: Apache-2.0
State: the link answers, verified on 29 September 2026
Evidence: files inventoried
Commit: 240fd063a047033d683d739c7b5b2f556cf7a5a9, 29 September 2026
Languages: C++ (350), C/C++ (229), Python (184), Jupyter (42), Go (41), Shell (24), JavaScript (10), TypeScript (7), NEURON (1), Java (1)
Size: 1,608 files, 889 scripts
Software Heritage: archived
Found in: the text, “Future work”
Holds: README, license file, CITATION.cff, environment (documentation/public/requirements.txt, yggdrasil_decision_forests/port/python/Dockerfile, yggdrasil_decision_forests/port/python/requirements.txt, yggdrasil_decision_forests/port/python/requirements_lock.txt, yggdrasil_decision_forests/port/python/config/setup.py, yggdrasil_decision_forests/port/tensorflow/pip_pkg/setup.py), tests, continuous integration, documentation, 42 notebooks
Tools: NumPy (84 files), pandas (63 files), JAX (21 files), TensorFlow (17 files), scikit-learn (11 files), Matplotlib (7 files), xarray (5 files), SciPy (2 files), UMAP (2 files), Plotly (1 file), PyTorch (1 file), seaborn (1 file), Hugging Face Transformers (1 file)
Availability: 1 check, the latest on 29 September 2026: the link answers
  • 29 September 2026: the link answers
891 files

neurodata/sex_classification

License: none: the authors keep all their rights
State: the link answers, verified on 29 September 2026
Evidence: files inventoried
Commit: 27c99d80b11f647b665a2dd5ec4a8430d234453a, 6 June 2024
Languages: Jupyter (22), Python (18)
Size: 46 files, 40 scripts
Software Heritage: archived
Found in: “Code availability”
Holds: README, 11 notebooks
Not found: license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: pandas (40 files), Matplotlib (38 files), NiBabel (38 files), NumPy (38 files), scikit-learn (38 files), SciPy (30 files), Numba (2 files)
Availability: 1 check, the latest on 29 September 2026: the link answers
  • 29 September 2026: the link answers
41 files

neurodata/treeple

License: other
State: the link answers, verified on 29 September 2026
Evidence: files inventoried
Commit: 75c2cf919939574e4240fe261f053162039495cf, 25 February 2025
Languages: Python (110), Jupyter (6)
Size: 233 files, 116 scripts
Software Heritage: archived
Found in: “Code availability”
Holds: README, license file, CITATION.cff, environment (pyproject.toml), tests, continuous integration, documentation, 6 notebooks
Tools: NumPy (81 files), scikit-learn (62 files), Matplotlib (36 files), SciPy (23 files), seaborn (22 files), pandas (17 files)
Availability: 1 check, the latest on 29 September 2026: the link answers
  • 29 September 2026: the link answers
118 files

Code availability

The code used to perform the analysis and generate results for sex classification can be accessed at https://github.com/neurodata/sex_classification. NEOFIT implementation can be tracked at https://github.com/neurodata/treeple.

Reproduced under the paper's license (CC BY), from the paper cited above.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 3 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 1,045 scripts, each with its path and the digest of its content;
  • 9 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data Availability

All relevant links to the data are within the manuscript and its Supporting Information files.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 29 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 9 authors, 13 MeSH terms, 2 funders, 27 references.

Cite

This paper

Liu, T., Dey, J., Xu, B., Bridgeford, E. W., Alldritt, S., Nenning, K.-H., Byeon, K., Xu, T., & Vogelstein, J. T. (2026). Statistically valid explainable black-box machine learning: applications in sex classification across species using brain imaging. PloS one, 21(4), e0346575. https://doi.org/10.1371/journal.pone.0346575

BibTeX

@article{liu2026statistically,
author = {Liu, Tingshan and Dey, Jayanta and Xu, Beiya and Bridgeford, Eric W. and Alldritt, Samuel and Nenning, Karl-Heinz and Byeon, Kyoungseob and Xu, Ting and Vogelstein, Joshua T.},
title = {{Statistically valid explainable black-box machine learning: applications in sex classification across species using brain imaging}},
journal = {PloS one},
year = {2026},
month = apr,
volume = {21},
number = {4},
pages = {e0346575},
publisher = {PLOS},
issn = {1932-6203},
doi = {10.1371/journal.pone.0346575},
url = {https://doi.org/10.1371/journal.pone.0346575},
pmid = {41989998},
pmcid = {PMC13086332}
}

RIS

TY - JOUR
AU - Liu, Tingshan
AU - Dey, Jayanta
AU - Xu, Beiya
AU - Bridgeford, Eric W.
AU - Alldritt, Samuel
AU - Nenning, Karl-Heinz
AU - Byeon, Kyoungseob
AU - Xu, Ting
AU - Vogelstein, Joshua T.
TI - Statistically valid explainable black-box machine learning: applications in sex classification across species using brain imaging
T2 - PloS one
J2 - PLoS One
PY - 2026
DA - 2026/04/16
VL - 21
IS - 4
SP - e0346575
SN - 1932-6203
PB - PLOS
DO - 10.1371/journal.pone.0346575
UR - https://doi.org/10.1371/journal.pone.0346575
LA - en
ER -

CSL-JSON

{
"id": "10.1371/journal.pone.0346575",
"type": "article-journal",
"title": "Statistically valid explainable black-box machine learning: applications in sex classification across species using brain imaging",
"container-title": "PloS one",
"author": [
{
"family": "Liu",
"given": "Tingshan"
},
{
"family": "Dey",
"given": "Jayanta"
},
{
"family": "Xu",
"given": "Beiya"
},
{
"family": "Bridgeford",
"given": "Eric W."
},
{
"family": "Alldritt",
"given": "Samuel"
},
{
"family": "Nenning",
"given": "Karl-Heinz"
},
{
"family": "Byeon",
"given": "Kyoungseob"
},
{
"family": "Xu",
"given": "Ting"
},
{
"family": "Vogelstein",
"given": "Joshua T."
}
],
"container-title-short": "PLoS One",
"volume": "21",
"issue": "4",
"page": "e0346575",
"DOI": "10.1371/journal.pone.0346575",
"PMID": "41989998",
"PMCID": "PMC13086332",
"ISSN": "1932-6203",
"publisher": "PLOS",
"URL": "https://doi.org/10.1371/journal.pone.0346575",
"language": "en",
"issued": {
"date-parts": [
[
2026,
4,
16
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s42003-026-10957-8 [code]
Brain defence by the extracellular matrix protein Cochlin.
Journal: Communications biology
In common: JAX, Hugging Face Transformers, UMAP, 10 other tools
[2] doi:10.1038/s41467-026-75959-w [code]
Charting higher-order models of brain function beyond pairwise interactions.
Journal: Nature communications
In common: JAX, xarray, Numba, 7 other tools, 2 references
[3] doi:10.1038/s41467-026-72253-7 [code]
Spurious alignment between large language models and brains can emerge from non-robust methods and overlooked confounds.
Journal: Nature communications
In common: xarray, Hugging Face Transformers, Plotly, 8 other tools, methods / tools
[4] doi:10.3389/fnsys.2026.1822122 [code]
Convergence-divergence circuits for multimodal integration of innate and learned opponent valences.
Journal: Frontiers in systems neuroscience
In common: UMAP, Numba, TensorFlow, 9 other tools
[5] doi:10.1038/s41467-026-73996-z [code]
Genetic architecture of white matter microstructure captured by unsupervised deep representation learning of fractional anisotropy maps.
Journal: Nature communications
In common: UMAP, Plotly, NiBabel, 7 other tools, structural MRI / diffusion, 2 references
[6] doi:10.1126/sciadv.aed4944 [code]
Focused ultrasound blood-brain barrier opening reveals a paradoxical remote metabolic response in the primate brain.
Journal: Science advances
In common: JAX, Numba, TensorFlow, 7 other tools, non-human primate, structural MRI / diffusion
[7] doi:10.1038/s41467-026-72057-9 [code]
Sex-specific behavioral feedback modulates sensorimotor processing and drives flexible social behavior.
Journal: Nature communications
In common: xarray, UMAP, Numba, 8 other tools
[8] doi:10.1038/s41467-026-76011-7 [code]
Human cortex organizes dynamic co-fluctuations along the sensorimotor-association axis.
Journal: Nature communications
In common: NiBabel, PyTorch, SciPy, 2 other tools, 3 references, author Ting Xu
[9] doi:10.21203/rs.3.rs-9326213/v1 [code]
Multi-task fMRI outperforms resting-state fMRI for revealing task-invariant organization of the human brain
Journal: Research Square (preprint)
In common: Numba, Plotly, NiBabel, 7 other tools, 2 references
[10] doi:10.1523/eneuro.0023-26.2026 [code]
Real-Time Segmentation and Classification of Birdsong Syllables for Learning Experiments.
Journal: eNeuro
In common: UMAP, Numba, TensorFlow, 8 other tools

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.