Modeling_Tool.Explainability

Model explainability layer. ModelExplainer is one entry point for SHAP attribution, Owen value grouped attribution, PDP, ICE, ALE, and LIME. It accepts a GradientBoostingModel, an LRMaster, or a fitted estimator. User guide: Model Explainability.

Optional dependencies

shap and lime load lazily, only when a SHAP, Owen, or LIME method runs, so import Modeling_Tool does not import them. PDP, ICE, and ALE need neither. Install both with:

pip install 'supermodelingfactory[explain]'

Model Explainer: Model_Explainer

Model_Explainer

Model explainability for the credit-modeling toolkit.

Global and local explanations for models trained with SuperModelingFactory:

• SHAP for attribution (global importance, summary / dependence plots, local contributions)
• Owen Value grouped attribution via SHAP PartitionExplainer
• PDP (partial dependence) for average marginal effects
• ICE (individual conditional expectation) for per-sample response curves
• ALE (accumulated local effects) for correlation-aware marginal effects
• LIME for local surrogate explanations and sampled global summaries

shap and lime are optional dependencies. They are imported lazily, only when the corresponding explanation is actually computed, so import Modeling_Tool never pulls them in. Install the full explainability extra with::

pip install supermodelingfactory[explain]

ModelExplainer

Unified explainer for SuperModelingFactory models.

The explainer accepts either a SuperModelingFactory model wrapper (GradientBoostingModel or LRMaster) or a raw fitted estimator. It exposes SHAP attribution, Owen Value grouped attribution, plus model-agnostic effect methods (PDP, ICE, ALE, and LIME).

Parameters:

Name	Type	Description	Default
model	GradientBoostingModel, LRMaster, or estimator	A SuperModelingFactory model wrapper or a fitted estimator (for example a scikit-learn LogisticRegression, or a LightGBM or XGBoost model). The underlying estimator is unwrapped from a wrapper automatically.	required
feature_names	list of str or None	Column order the model expects. When None it is read from the model (varlist, feature_cols, feature_names, feature_names_) or from the estimator (feature_names_in_, feature_name_, feature_name()); it stays None when none of them exists. A DataFrame passed to a later method is aligned to this order by column name (extra columns are ignored) when it contains all of these columns, and a plain array is labelled with these names when its column count matches.	None
model_type	str or None	Model family, case-insensitive. When None it is read from model.model_type or guessed from the estimator class ("lgb", "xgb", "lr", otherwise the lower-cased class name). "lgb", "lightgbm", "xgb" and "xgboost" select the SHAP TreeExplainer; "lr", "linear" and "logisticregression" select the LinearExplainer; any other value selects the model-agnostic shap.Explainer on the positive-class probability.	None
background_data	pandas.DataFrame, array-like, or None	Representative sample of the training features. It is the SHAP background for linear and model-agnostic models (required there) and optional for tree models; it is also the default background for Owen values, the default data for the coalition structure, and the default X_train for LIME.	None

Attributes:

Name	Type	Description
model	object	The model argument as passed.
estimator	object	The fitted estimator unwrapped from model (model itself when it is already a raw estimator).
model_type	str	Lower-cased model family.
feature_names	list of str or None	Resolved feature order (see the feature_names parameter).
background_data	pandas.DataFrame, array-like, or None	The background sample passed to the constructor.
shap_values_	ndarray or None	2-D SHAP values (rows by features) cached by the last SHAP computation (explain, or a SHAP method that was given X); None before that.
expected_value_	ndarray or None	Base value(s) of the cached SHAP explanation.
explanation_	Explanation or None	The cached SHAP explanation object.
coalition_structure_	dict or None	Coalition structure cached by build_coalition_structure or explain_owen.
owen_values_	ndarray or None	2-D Owen values (rows by features) cached by explain_owen.
owen_expected_value_	ndarray or None	Base value(s) of the cached Owen explanation.
owen_explanation_	Explanation or None	The cached Owen explanation object.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

class ModelExplainer:
    """Unified explainer for SuperModelingFactory models.

    The explainer accepts either a SuperModelingFactory model wrapper
    (``GradientBoostingModel`` or ``LRMaster``) or a raw fitted estimator. It
    exposes SHAP attribution, Owen Value grouped attribution, plus model-agnostic
    effect methods (PDP, ICE, ALE, and LIME).

    Parameters
    ----------
    model : GradientBoostingModel, LRMaster, or estimator
        A SuperModelingFactory model wrapper or a fitted estimator (for example a
        scikit-learn ``LogisticRegression``, or a LightGBM or XGBoost model). The
        underlying estimator is unwrapped from a wrapper automatically.
    feature_names : list of str or None, default None
        Column order the model expects. When ``None`` it is read from the model
        (``varlist``, ``feature_cols``, ``feature_names``, ``feature_names_``) or from
        the estimator (``feature_names_in_``, ``feature_name_``, ``feature_name()``);
        it stays ``None`` when none of them exists. A DataFrame passed to a later method
        is aligned to this order by column name (extra columns are ignored) when it
        contains all of these columns, and a plain array is labelled with these names
        when its column count matches.
    model_type : str or None, default None
        Model family, case-insensitive. When ``None`` it is read from
        ``model.model_type`` or guessed from the estimator class (``"lgb"``,
        ``"xgb"``, ``"lr"``, otherwise the lower-cased class name). ``"lgb"``,
        ``"lightgbm"``, ``"xgb"`` and ``"xgboost"`` select the SHAP ``TreeExplainer``;
        ``"lr"``, ``"linear"`` and ``"logisticregression"`` select the
        ``LinearExplainer``; any other value selects the model-agnostic
        ``shap.Explainer`` on the positive-class probability.
    background_data : pandas.DataFrame, array-like, or None, default None
        Representative sample of the training features. It is the SHAP background for
        linear and model-agnostic models (required there) and optional for tree models;
        it is also the default background for Owen values, the default data for the
        coalition structure, and the default ``X_train`` for LIME.

    Attributes
    ----------
    model : object
        The ``model`` argument as passed.
    estimator : object
        The fitted estimator unwrapped from ``model`` (``model`` itself when it is
        already a raw estimator).
    model_type : str
        Lower-cased model family.
    feature_names : list of str or None
        Resolved feature order (see the ``feature_names`` parameter).
    background_data : pandas.DataFrame, array-like, or None
        The background sample passed to the constructor.
    shap_values_ : numpy.ndarray or None
        2-D SHAP values (rows by features) cached by the last SHAP computation
        (`explain`, or a SHAP method that was given ``X``); ``None`` before that.
    expected_value_ : numpy.ndarray or None
        Base value(s) of the cached SHAP explanation.
    explanation_ : shap.Explanation or None
        The cached SHAP explanation object.
    coalition_structure_ : dict or None
        Coalition structure cached by `build_coalition_structure` or `explain_owen`.
    owen_values_ : numpy.ndarray or None
        2-D Owen values (rows by features) cached by `explain_owen`.
    owen_expected_value_ : numpy.ndarray or None
        Base value(s) of the cached Owen explanation.
    owen_explanation_ : shap.Explanation or None
        The cached Owen explanation object.
    """

    def __init__(self, model, feature_names=None, model_type=None, background_data=None):
        self.model = model
        self.background_data = background_data
        self.estimator = self._resolve_estimator(model)
        self.model_type = (model_type or self._infer_model_type(model, self.estimator)).lower()
        self.feature_names = self._resolve_feature_names(feature_names)

        self._explainer = None
        self._explainer_kind = None
        self.shap_values_ = None
        self.expected_value_ = None
        self.explanation_ = None
        self._last_X = None

        self.coalition_structure_ = None
        self._owen_explainer = None
        self._owen_model_output = None
        self._owen_cache = None
        self.owen_values_ = None
        self.owen_expected_value_ = None
        self.owen_explanation_ = None
        self._owen_last_X = None

    # ------------------------------------------------------------------ #
    # Resolution helpers
    # ------------------------------------------------------------------ #
    @staticmethod
    def _resolve_estimator(model):
        """Unwrap the underlying fitted estimator from an SMF wrapper."""
        for wrapper_attr in ("_model", "model_instance"):
            wrapper = getattr(model, wrapper_attr, None)
            if wrapper is not None and getattr(wrapper, "model", None) is not None:
                return wrapper.model
        inner = getattr(model, "model", None)
        if inner is not None and (hasattr(inner, "predict") or hasattr(inner, "coef_")):
            return inner
        return model

    @staticmethod
    def _infer_model_type(model, estimator):
        """Best-effort detection of the model family."""
        declared = getattr(model, "model_type", None)
        if isinstance(declared, str) and declared:
            return declared
        cls = type(estimator).__name__.lower()
        module = type(estimator).__module__.lower()
        if "lightgbm" in module or cls.startswith("lgb"):
            return "lgb"
        if "xgboost" in module or cls.startswith("xgb"):
            return "xgb"
        if "logisticregression" in cls:
            return "lr"
        return cls

    def _is_tree(self):
        return self.model_type in _TREE_MODEL_TYPES

    def _is_linear(self):
        return self.model_type in _LINEAR_MODEL_TYPES

    def _resolve_feature_names(self, feature_names):
        """Infer feature ordering from explicit arg / wrapper / booster."""
        if feature_names is not None:
            return list(feature_names)
        for attr in ("varlist", "feature_cols", "feature_names", "feature_names_"):
            value = getattr(self.model, attr, None)
            if value is not None and not callable(value):
                try:
                    return list(value)
                except TypeError:
                    pass
        estimator = self.estimator
        for attr in ("feature_names_in_", "feature_name_"):
            value = getattr(estimator, attr, None)
            if value is not None and not callable(value):
                try:
                    return list(value)
                except TypeError:
                    pass
        booster_names = getattr(estimator, "feature_name", None)
        if callable(booster_names):
            try:
                return list(booster_names())
            except Exception:  # pragma: no cover - defensive
                pass
        return None

    # ------------------------------------------------------------------ #
    # Data / prediction utilities
    # ------------------------------------------------------------------ #
    def _as_frame(self, X):
        """Coerce input to a DataFrame aligned to ``feature_names`` when known."""
        if X is None:
            return None
        if isinstance(X, pd.DataFrame):
            cols = self.feature_names
            if cols is not None and all(c in X.columns for c in cols):
                return X.loc[:, cols]
            return X
        arr = np.asarray(X)
        if arr.ndim == 1:
            arr = arr.reshape(1, -1)
        cols = self.feature_names
        if cols is not None and len(cols) == arr.shape[1]:
            return pd.DataFrame(arr, columns=cols)
        return pd.DataFrame(arr)

    def _predict_proba_pos(self, X):
        """Positive-class probability callable for model-agnostic explanations."""
        estimator = self.estimator
        frame = self._as_frame(X)
        if hasattr(estimator, "predict_proba"):
            proba = np.asarray(estimator.predict_proba(frame))
            return proba[:, -1] if proba.ndim == 2 else proba
        return np.asarray(estimator.predict(frame)).ravel()

    def _predict_proba_pos_batched(self, X, prediction_batch_size=100000):
        """Predict in bounded row batches while preserving input order."""
        frame = self._as_frame(X)
        if prediction_batch_size is None:
            return np.asarray(self._predict_proba_pos(frame)).ravel()
        batch_size = int(prediction_batch_size)
        if batch_size <= 0:
            raise ValueError("prediction_batch_size must be a positive integer or None")
        outputs = [
            np.asarray(self._predict_proba_pos(frame.iloc[start : start + batch_size])).ravel()
            for start in range(0, len(frame), batch_size)
        ]
        return np.concatenate(outputs) if outputs else np.asarray([], dtype=float)

    def _predict_log_odds(self, X, eps=1e-6):
        """Log-odds prediction callable for reason-code style explanations."""
        p = np.asarray(self._predict_proba_pos(X)).ravel()
        p = np.clip(p, eps, 1.0 - eps)
        return np.log(p / (1.0 - p))

    def _predict_proba_2d(self, X):
        """Two-column probability callable for LIME classification mode."""
        pos = np.asarray(self._predict_proba_pos(X)).ravel()
        return np.column_stack([1.0 - pos, pos])

    def _sample_frame(self, X, sample_size=None, random_state=None):
        frame = self._as_frame(X).copy()
        if sample_size is not None and len(frame) > sample_size:
            frame = frame.sample(n=sample_size, random_state=random_state)
        return frame

    def _feature_name(self, X, feature):
        frame = self._as_frame(X)
        if isinstance(feature, int):
            return frame.columns[feature]
        if feature not in frame.columns:
            raise KeyError(f"Feature {feature!r} not found in X")
        return feature

    @staticmethod
    def _numeric_grid(series: pd.Series, grid_resolution=50, percentiles=(0.05, 0.95)):
        clean = pd.to_numeric(series, errors="coerce").dropna()
        if clean.empty:
            raise ValueError("Effect methods require a numeric feature with at least one non-missing value")
        lo, hi = clean.quantile(list(percentiles)).to_numpy(dtype=float)
        if np.isclose(lo, hi):
            vals = np.array(sorted(clean.unique()), dtype=float)
            return vals[:grid_resolution]
        return np.linspace(lo, hi, int(grid_resolution))

    @staticmethod
    def _normalize_values(values, base_values=None):
        values = np.asarray(values)
        base = np.asarray(base_values) if base_values is not None else None
        if values.ndim == 3:
            values = values[..., -1]
            if base is not None and base.ndim == 2:
                base = base[..., -1]
        return values, base

    def _prediction_fn(self, model_output):
        if model_output == "probability":
            return self._predict_proba_pos
        if model_output in {"log_odds", "logit"}:
            return self._predict_log_odds
        raise ValueError("model_output must be 'probability' or 'log_odds'")

    @staticmethod
    def _coerce_xgb_base_score_scalar(base_score, allow_vector=False):
        """Normalize XGBoost 3.x single-target base_score values for older SHAP loaders."""
        if isinstance(base_score, str):
            text = base_score.strip()
            if text.startswith("[") and text.endswith("]"):
                inner = text[1:-1].strip()
                if not inner:
                    return float(text)
                if "," in inner:
                    if allow_vector:
                        return base_score
                    raise ValueError(
                        "Multi-element XGBoost base_score vectors are not supported "
                        f"for single-target SHAP compatibility ({inner!r})."
                    )
                return float(inner)
            try:
                return float(text)
            except ValueError:
                try:
                    parsed = ast.literal_eval(text)
                except (SyntaxError, ValueError):
                    return base_score
                return ModelExplainer._coerce_xgb_base_score_scalar(
                    parsed,
                    allow_vector=allow_vector,
                )

        if isinstance(base_score, (list, tuple, np.ndarray)):
            flat = np.ravel(base_score)
            if flat.size == 1:
                return float(flat[0])
            if allow_vector:
                return base_score
            raise ValueError(
                "Multi-element XGBoost base_score vectors are not supported "
                f"for single-target SHAP compatibility ({flat.size} elements)."
            )

        return float(base_score)

    @staticmethod
    def _coerce_single_xgb_base_score(base_score):
        """Backward-compatible alias for single-target base_score normalization."""
        try:
            return ModelExplainer._coerce_xgb_base_score_scalar(
                base_score,
                allow_vector=False,
            )
        except ValueError:
            return base_score

    @classmethod
    def _normalize_xgb_base_score_in_dump(cls, model_dump):
        """Normalize XGBoost 3.x single-target base_score for older SHAP loaders."""
        learner = model_dump.get("learner", {})
        params = learner.get("learner_model_param", {})
        if "base_score" not in params:
            return

        try:
            params["base_score"] = cls._coerce_xgb_base_score_scalar(
                params["base_score"],
                allow_vector=False,
            )
        except ValueError:
            return

    @classmethod
    def _patch_decode_ubjson_buffer(cls, module, force=False):
        """Patch one SHAP decode_ubjson_buffer entry point for XGBoost base_score."""
        decoder = getattr(module, "decode_ubjson_buffer", None)
        if decoder is None:
            return

        if getattr(decoder, "_smf_xgb_base_score_patch", False) and not force:
            return

        original = getattr(decoder, "_smf_original_decoder", decoder)

        def patched_decoder(*args, **kwargs):
            model_dump = original(*args, **kwargs)
            try:
                cls._normalize_xgb_base_score_in_dump(model_dump)
            except Exception:
                pass
            return model_dump

        patched_decoder._smf_xgb_base_score_patch = True
        patched_decoder._smf_original_decoder = original
        module.decode_ubjson_buffer = patched_decoder

    @classmethod
    def _patch_shap_xgb_base_score_loader(cls, force=False):
        """Patch SHAP UBJSON decoders that assume XGBoost base_score is scalar."""
        try:
            from shap.explainers import _tree as shap_tree  # noqa: WPS433
            from shap.explainers.other import _ubjson as shap_ubjson  # noqa: WPS433
        except Exception:  # pragma: no cover - depends on optional SHAP internals
            return

        cls._patch_decode_ubjson_buffer(shap_tree, force=force)
        cls._patch_decode_ubjson_buffer(shap_ubjson, force=force)

    @classmethod
    def _patch_shap_xgb_tree_model_loader(cls, force=False):
        """Wrap SHAP's XGBTreeModelLoader so base_score coercion is always applied."""
        try:
            from shap.explainers._tree import XGBTreeModelLoader  # noqa: WPS433
        except Exception:  # pragma: no cover - depends on optional SHAP internals
            return

        init = XGBTreeModelLoader.__init__
        if getattr(init, "_smf_xgb_loader_patch", False) and not force:
            return

        original_init = getattr(init, "_smf_original_init", init)

        def patched_init(self, xgb_model):
            cls._apply_xgb_shap_compat_patches()
            try:
                return original_init(self, xgb_model)
            except ValueError as exc:
                if not cls._is_xgb_shap_base_score_error(exc):
                    raise
                cls._apply_xgb_shap_compat_patches(force=True)
                return original_init(self, xgb_model)

        patched_init._smf_xgb_loader_patch = True
        patched_init._smf_original_init = original_init
        XGBTreeModelLoader.__init__ = patched_init

    @classmethod
    def _apply_xgb_shap_compat_patches(cls, force=False):
        """Apply all XGBoost-only SHAP compatibility patches."""
        cls._patch_shap_xgb_base_score_loader(force=force)
        cls._patch_shap_xgb_tree_model_loader(force=force)

    @staticmethod
    def _is_xgb_shap_base_score_error(exc):
        text = str(exc).lower()
        if "base_score" in text:
            return True
        if "could not convert string to float" in text and "[" in text:
            return True
        if "xgbtreemodelloader" in text:
            return True
        return False

    @staticmethod
    def _format_xgb_shap_version_hint():
        versions = [f"python={sys.version.split()[0]}"]
        for package in ("shap", "xgboost"):
            try:
                versions.append(f"{package}={importlib.metadata.version(package)}")
            except importlib.metadata.PackageNotFoundError:
                continue
        return ", ".join(versions)

    def _raise_xgb_shap_base_score_error(self, exc):
        versions = self._format_xgb_shap_version_hint()
        raise ValueError(
            "SHAP TreeExplainer failed while parsing an XGBoost base_score. "
            "This is a known incompatibility between XGBoost >= 3.1 and older "
            "SHAP versions, where XGBoost serializes base_score as a single-item "
            "array string such as '[5E-1]'. SuperModelingFactory attempted "
            "automatic compatibility patches but SHAP still rejected the model. "
            f"Detected versions: {versions}. "
            "For Python 3.11+, prefer shap>=0.50.0. On Python 3.10, keep the "
            "current SuperModelingFactory release or temporarily pin xgboost<3.1 "
            "only if multi-target models still fail."
        ) from exc

    def _build_xgb_tree_explainer(self, shap, data):
        """Build a TreeExplainer for XGBoost with compatibility retries."""
        self._apply_xgb_shap_compat_patches()
        last_exc = None
        for _ in range(2):
            try:
                return shap.TreeExplainer(self.estimator, data=data)
            except ValueError as exc:
                last_exc = exc
                if not self._is_xgb_shap_base_score_error(exc):
                    raise
                self._apply_xgb_shap_compat_patches(force=True)
        self._raise_xgb_shap_base_score_error(last_exc)

    # ------------------------------------------------------------------ #
    # SHAP computation
    # ------------------------------------------------------------------ #
    def _build_explainer(self):
        shap = _lazy_shap()
        if self._is_tree():
            data = None
            if self.background_data is not None:
                data = self._as_frame(self.background_data)
            if self.model_type in {"xgb", "xgboost"}:
                self._explainer = self._build_xgb_tree_explainer(shap, data)
            else:
                self._explainer = shap.TreeExplainer(self.estimator, data=data)
            self._explainer_kind = "tree"
        elif self._is_linear():
            if self.background_data is None:
                raise ValueError(
                    "Linear models require `background_data` (a representative "
                    "sample of the training features) to build a SHAP LinearExplainer."
                )
            self._explainer = shap.LinearExplainer(self.estimator, self._as_frame(self.background_data))
            self._explainer_kind = "linear"
        else:
            if self.background_data is None:
                raise ValueError(
                    f"Model type {self.model_type!r} needs a model-agnostic explainer, "
                    "which requires `background_data`."
                )
            masker = self._as_frame(self.background_data)
            self._explainer = shap.Explainer(self._predict_proba_pos, masker)
            self._explainer_kind = "generic"
        return self._explainer

    def _shap_values_for(self, X):
        """Compute normalized (2-D) SHAP values for *X* without caching."""
        if self._explainer is None:
            self._build_explainer()
        explanation = self._explainer(X)
        values, base = self._normalize_values(explanation.values, explanation.base_values)
        return values, base, explanation

    def explain(self, X):
        """Compute and cache SHAP values for a dataset.

        Parameters
        ----------
        X : pandas.DataFrame or array-like
            Rows to explain. A DataFrame is aligned to ``feature_names`` by column name
            when it contains all of them (extra columns are ignored); an array is
            labelled with ``feature_names`` when its column count matches, and a 1-D
            array is read as a single row.

        Returns
        -------
        shap.Explanation
            The explanation object returned by the SHAP explainer.

        Raises
        ------
        ValueError
            If the model is linear or needs the model-agnostic explainer and no
            ``background_data`` was given; or, for XGBoost, if SHAP still cannot parse
            the model's ``base_score`` after the built-in compatibility patches.
        ImportError
            If the optional ``shap`` package is not installed.

        Notes
        -----
        The SHAP explainer is built on the first call and reused afterwards:
        ``TreeExplainer`` for LightGBM and XGBoost (``background_data`` optional),
        ``LinearExplainer`` for linear models (``background_data`` required), and the
        model-agnostic ``shap.Explainer`` on the positive-class probability for any
        other model (``background_data`` required). Tree and linear SHAP values are in
        the model's raw output scale (log-odds for a binary classifier), model-agnostic
        values are in probability units.

        Sets ``shap_values_`` (2-D; for a multi-output explanation only the last output
        is kept), ``expected_value_`` and ``explanation_``, and remembers ``X`` for the
        plot and importance methods that are called without data.
        """
        frame = self._as_frame(X)
        values, base, explanation = self._shap_values_for(frame)
        self.shap_values_ = values
        self.expected_value_ = base
        self.explanation_ = explanation
        self._last_X = frame
        return explanation

    def _ensure_values(self, X):
        if X is not None:
            self.explain(X)
        if self.shap_values_ is None or self._last_X is None:
            raise RuntimeError("No SHAP values available. Call explain(X) first or pass X=...")
        return self.shap_values_, self._last_X

    def _resolved_names(self, X, n_features):
        if self.feature_names is not None:
            return list(self.feature_names)
        if isinstance(X, pd.DataFrame):
            return list(X.columns)
        return [f"f{i}" for i in range(n_features)]

    def feature_importance(self, X=None, normalize=False):
        """Global feature importance as mean absolute SHAP value.

        Parameters
        ----------
        X : pandas.DataFrame, array-like, or None, default None
            Rows to explain. When given, the SHAP values are recomputed with `explain`
            and replace the cached ones; when ``None``, the values cached by the last
            SHAP computation are used.
        normalize : bool, default False
            If ``True``, add an ``importance_pct`` column with each feature's share of
            the total mean absolute SHAP value (fractions that sum to 1, not percent).

        Returns
        -------
        pandas.DataFrame
            One row per feature with ``feature`` and ``mean_abs_shap`` (plus
            ``importance_pct`` when ``normalize=True``), sorted by ``mean_abs_shap`` in
            descending order.

        Raises
        ------
        RuntimeError
            If ``X`` is ``None`` and no SHAP values have been computed yet.
        """
        values, X_used = self._ensure_values(X)
        mean_abs = np.abs(values).mean(axis=0)
        names = self._resolved_names(X_used, values.shape[1])
        table = pd.DataFrame({"feature": names, "mean_abs_shap": mean_abs})
        table = table.sort_values("mean_abs_shap", ascending=False).reset_index(drop=True)
        if normalize:
            total = table["mean_abs_shap"].sum()
            table["importance_pct"] = table["mean_abs_shap"] / total if total else 0.0
        return table

    def summary_plot(self, X=None, max_display=20, plot_type="dot", show=True, save_path=None, random_state=0):
        """SHAP summary (beeswarm / bar) plot.

        ``random_state`` seeds the point jitter through shap's ``rng`` argument
        (shap >= 0.47), so the plot no longer depends on NumPy's global RNG;
        older shap versions ignore it.

        Parameters
        ----------
        X : pandas.DataFrame, array-like, or None, default None
            Rows to plot. When given, the SHAP values are recomputed with `explain` and
            replace the cached ones; when ``None``, the cached values are used.
        max_display : int, default 20
            Maximum number of features shown (the most important ones).
        plot_type : str, default "dot"
            Plot kind passed to ``shap.summary_plot``, for example ``"dot"`` (beeswarm)
            or ``"bar"`` (mean absolute SHAP value).
        show : bool, default True
            If ``True``, display the figure with ``plt.show()``. If ``False``, the figure
            is closed after it has been saved, so nothing is displayed.
        save_path : str, path-like, or None, default None
            When given, the figure is saved to this path (``dpi=150``, tight bounding
            box; the format follows the file extension). The folder must already exist.
        random_state : int, default 0
            Seed of the random number generator that jitters the points.

        Returns
        -------
        matplotlib.figure.Figure
            The current matplotlib figure.

        Raises
        ------
        RuntimeError
            If ``X`` is ``None`` and no SHAP values have been computed yet.
        """
        shap = _lazy_shap()
        import matplotlib.pyplot as plt

        values, X_used = self._ensure_values(X)
        kwargs = {"max_display": max_display, "plot_type": plot_type, "show": False}
        if "rng" in inspect.signature(shap.summary_plot).parameters:
            kwargs["rng"] = np.random.default_rng(random_state)
        if not isinstance(X_used, pd.DataFrame) and self.feature_names is not None:
            kwargs["feature_names"] = self.feature_names
        shap.summary_plot(values, X_used, **kwargs)
        return self._finalize_plot(plt, show, save_path)

    def dependence_plot(self, feature, X=None, interaction_index="auto", show=True, save_path=None):
        """SHAP dependence plot for a single feature.

        Parameters
        ----------
        feature : str or int
            Feature to plot, as a column name or a column position (passed to
            ``shap.dependence_plot``).
        X : pandas.DataFrame, array-like, or None, default None
            Rows to plot. When given, the SHAP values are recomputed with `explain` and
            replace the cached ones; when ``None``, the cached values are used.
        interaction_index : str, int, or None, default "auto"
            Feature used to color the points. ``"auto"`` lets SHAP choose the feature
            with the strongest apparent interaction; a name or a position selects one
            feature; ``None`` turns the coloring off.
        show : bool, default True
            If ``True``, display the figure with ``plt.show()``. If ``False``, the figure
            is closed after it has been saved, so nothing is displayed.
        save_path : str, path-like, or None, default None
            When given, the figure is saved to this path (``dpi=150``, tight bounding
            box; the format follows the file extension). The folder must already exist.

        Returns
        -------
        matplotlib.figure.Figure
            The current matplotlib figure.

        Raises
        ------
        RuntimeError
            If ``X`` is ``None`` and no SHAP values have been computed yet.
        """
        shap = _lazy_shap()
        import matplotlib.pyplot as plt

        values, X_used = self._ensure_values(X)
        kwargs = {"interaction_index": interaction_index, "show": False}
        if not isinstance(X_used, pd.DataFrame) and self.feature_names is not None:
            kwargs["feature_names"] = self.feature_names
        shap.dependence_plot(feature, values, X_used, **kwargs)
        return self._finalize_plot(plt, show, save_path)

    def explain_instance(self, x_row):
        """Per-feature SHAP contributions for a single sample.

        Parameters
        ----------
        x_row : pandas.DataFrame, pandas.Series, dict, or array-like
            One observation. A Series or a dict is turned into a one-row frame and a 1-D
            array is read as one row; when several rows are given, only the first is
            explained.

        Returns
        -------
        pandas.DataFrame
            One row per feature with ``feature``, ``value`` (the feature value in
            ``x_row``) and ``shap_value``, sorted by absolute SHAP value (largest
            first). ``attrs["base_value"]`` holds the explainer's base value as a float
            (NaN when it is unavailable).

        Raises
        ------
        ValueError
            If the SHAP explainer has to be built and cannot be (see `explain`).
        ImportError
            If the optional ``shap`` package is not installed.

        Notes
        -----
        The result is not cached: ``shap_values_``, ``expected_value_`` and
        ``explanation_`` keep the values of the last `explain` call.
        """
        if isinstance(x_row, pd.Series):
            x_row = x_row.to_frame().T
        elif isinstance(x_row, dict):
            x_row = pd.DataFrame([x_row])
        frame = self._as_frame(x_row)
        if frame.shape[0] != 1:
            frame = frame.iloc[[0]]

        values, base, _ = self._shap_values_for(frame)
        names = self._resolved_names(frame, values.shape[1])
        table = pd.DataFrame(
            {"feature": names, "value": np.ravel(np.asarray(frame.values)), "shap_value": np.ravel(values[0])}
        )
        order = table["shap_value"].abs().sort_values(ascending=False).index
        table = table.loc[order].reset_index(drop=True)
        table.attrs["base_value"] = float(np.ravel(base)[0]) if base is not None and base.size else float("nan")
        return table

    # ------------------------------------------------------------------ #
    # Owen Value / Coalition SHAP
    # ------------------------------------------------------------------ #
    def build_coalition_structure(
        self,
        X=None,
        prior_groups=None,
        threshold=0.35,
        method="complete",
        corr_method="spearman",
        min_group_size=1,
        intra_dist=0.01,
        inter_dist=0.99,
    ):
        """Build and cache a coalition structure for Owen Value explanations.

        Parameters
        ----------
        X : pandas.DataFrame, array-like, or None, default None
            Data the correlation clustering is computed on. When ``None``, the
            explainer's ``background_data`` is used.
        prior_groups : dict or None, default None
            Business groups as ``{group_name: [feature, ...]}``. They take precedence
            over the automatic clusters; feature names that are not columns of the data
            are ignored, and a feature may not appear in two groups.
        threshold : float, default 0.35
            Cut height of the hierarchical clustering on the distance
            ``1 - abs(association)``; must be between 0 and 1.
        method : str, default "complete"
            Linkage method: ``"complete"``, ``"average"`` or ``"single"``.
        corr_method : str, default "spearman"
            Association measure: ``"spearman"``, ``"pearson"``, ``"kendall"`` or
            ``"MIC"`` (case-insensitive; needs the optional ``minepy`` package).
        min_group_size : int, default 1
            Automatic clusters with fewer features than this are merged into one group
            named ``"auto_singleton"``.
        intra_dist : float, default 0.01
            Distance between two features of the same group in the partition tree passed
            to SHAP.
        inter_dist : float, default 0.99
            Distance between two features of different groups in that partition tree.

        Returns
        -------
        dict
            The coalition structure, as returned by the module-level
            `build_coalition_structure` (keys ``groups``, ``shap_lnk``, ``corr_lnk``,
            ``auto_groups``, ``summary``, ``features``, ``threshold``, ``method`` and
            ``corr_method``). It is also cached as ``coalition_structure_``.

        Raises
        ------
        ValueError
            If both ``X`` and ``background_data`` are ``None``, or if the clustering
            arguments are invalid (see the module-level function).
        """
        data = X if X is not None else self.background_data
        if data is None:
            raise ValueError("build_coalition_structure requires X or background_data")
        frame = self._as_frame(data)
        cs = _build_coalition_structure(
            frame,
            prior_groups=prior_groups,
            threshold=threshold,
            method=method,
            corr_method=corr_method,
            min_group_size=min_group_size,
            intra_dist=intra_dist,
            inter_dist=inter_dist,
        )
        self.coalition_structure_ = cs
        return cs

    def _build_owen_explainer(self, coalition_structure, background_data=None, model_output="probability"):
        shap = _lazy_shap()
        background = background_data if background_data is not None else self.background_data
        if background is None:
            raise ValueError("Owen Value explanations require background_data")
        background = self._as_frame(background)
        features = coalition_structure.get("features")
        if features is not None:
            background = background.loc[:, list(features)]
        masker = shap.maskers.Partition(background, clustering=coalition_structure["shap_lnk"])
        self._owen_explainer = shap.PartitionExplainer(self._prediction_fn(model_output), masker)
        self._owen_model_output = model_output
        self._owen_cache = {
            "model_output": model_output,
            "structure": self._structure_digest(coalition_structure),
            "background": self._frame_digest(background),
        }
        return self._owen_explainer

    @staticmethod
    def _structure_digest(coalition_structure):
        """Hashable summary of what a coalition structure contributes to the partition tree: features and linkage."""
        linkage = np.ascontiguousarray(np.asarray(coalition_structure["shap_lnk"], dtype=float))
        features = coalition_structure.get("features")
        return (
            None if features is None else tuple(str(feature) for feature in features),
            linkage.shape,
            hashlib.md5(linkage.tobytes()).hexdigest(),
        )

    @staticmethod
    def _frame_digest(frame):
        """Hashable summary of a background frame: shape, column names and the values, row by row."""
        row_hashes = pd.util.hash_pandas_object(frame, index=False).to_numpy()
        return (frame.shape, tuple(str(column) for column in frame.columns), hashlib.md5(row_hashes.tobytes()).hexdigest())

    def _owen_cache_is_stale(self, coalition_structure, background_data, model_output):
        """Whether the cached ``PartitionExplainer`` was built from another structure, background or output."""
        cache = self._owen_cache
        if self._owen_explainer is None or cache is None:
            return True
        if cache["model_output"] != model_output or cache["structure"] != self._structure_digest(coalition_structure):
            return True
        background = background_data if background_data is not None else self.background_data
        if background is None:
            return False  # nothing to rebuild from, so keep the explainer that exists
        background = self._as_frame(background)
        features = coalition_structure.get("features")
        if features is not None:
            background = background.loc[:, list(features)]
        return cache["background"] != self._frame_digest(background)

    def explain_owen(
        self,
        X,
        coalition_structure=None,
        prior_groups=None,
        threshold=0.35,
        method="complete",
        corr_method="spearman",
        background_data=None,
        model_output="probability",
        rebuild=False,
        **explain_kwargs,
    ):
        """Compute Owen Value attribution with SHAP PartitionExplainer.

        ``model_output='probability'`` explains positive-class probability.
        ``model_output='log_odds'`` is useful for credit reason-code reporting.

        Parameters
        ----------
        X : pandas.DataFrame or array-like
            Rows to explain. It must contain every feature of the coalition structure;
            its columns are selected in the structure's feature order.
        coalition_structure : dict or None, default None
            Structure returned by `build_coalition_structure`. When ``None``, the cached
            ``coalition_structure_`` is reused; when there is none, one is built from
            ``background_data`` (the argument, else the constructor's, else ``X``).
        prior_groups : dict or None, default None
            Business groups as ``{group_name: [feature, ...]}``. When given, a new
            structure is always built from them (using ``threshold``, ``method`` and
            ``corr_method``) and it replaces ``coalition_structure``, even an explicit
            one.
        threshold : float, default 0.35
            Cut height of the clustering on ``1 - abs(association)``; used only when the
            structure is built here.
        method : str, default "complete"
            Linkage method (``"complete"``, ``"average"`` or ``"single"``); used only
            when the structure is built here.
        corr_method : str, default "spearman"
            Association measure (``"spearman"``, ``"pearson"``, ``"kendall"`` or
            ``"MIC"``); used only when the structure is built here.
        background_data : pandas.DataFrame, array-like, or None, default None
            Background sample for building the structure (when it is built here) and
            for the SHAP ``PartitionExplainer``. Defaults to the constructor's
            ``background_data``. A background whose values differ from those the cached
            ``PartitionExplainer`` was built with makes it rebuild.
        model_output : str, default "probability"
            Quantity to explain: ``"probability"`` (positive-class probability) or
            ``"log_odds"`` (``"logit"`` is accepted as an alias). Any other value raises
            ``ValueError``.
        rebuild : bool, default False
            If ``True``, build a new ``PartitionExplainer`` instead of reusing the cached
            one. It is also rebuilt automatically when the coalition structure (its
            features and linkage), the background values or ``model_output`` differ from
            those the cached explainer was built with.
        **explain_kwargs
            Extra keyword arguments passed to the ``PartitionExplainer`` call, for example
            ``max_evals`` or ``silent``.

        Returns
        -------
        shap.Explanation
            The Owen value explanation of ``X`` (restricted to the features of the
            coalition structure).

        Raises
        ------
        ValueError
            If no ``background_data`` is available (argument or constructor) to build
            the ``PartitionExplainer``, or if ``model_output`` is not supported.
        ImportError
            If the optional ``shap`` package is not installed.

        Notes
        -----
        Sets ``coalition_structure_``, ``owen_values_``, ``owen_expected_value_`` and
        ``owen_explanation_``, and remembers ``X`` for the Owen importance methods.

        The ``PartitionExplainer`` is cached and reused while the coalition structure
        (its features and linkage), the background values and ``model_output`` stay the
        same, for example when only ``X`` changes or when the same ``prior_groups`` are
        passed again. A change of any of them, or ``rebuild=True``, builds a new one. A
        later call without any background keeps the explainer that was built.
        """
        frame = self._as_frame(X)
        if coalition_structure is None:
            coalition_structure = self.coalition_structure_
        if coalition_structure is None or prior_groups is not None:
            base = background_data if background_data is not None else self.background_data
            if base is None:
                base = frame
            coalition_structure = self.build_coalition_structure(
                base,
                prior_groups=prior_groups,
                threshold=threshold,
                method=method,
                corr_method=corr_method,
            )
        features = coalition_structure.get("features")
        if features is not None:
            frame = frame.loc[:, list(features)]

        if rebuild or self._owen_cache_is_stale(coalition_structure, background_data, model_output):
            self._build_owen_explainer(coalition_structure, background_data=background_data, model_output=model_output)

        explanation = self._owen_explainer(frame, **explain_kwargs)
        values, base = self._normalize_values(explanation.values, explanation.base_values)
        self.coalition_structure_ = coalition_structure
        self.owen_values_ = values
        self.owen_expected_value_ = base
        self.owen_explanation_ = explanation
        self._owen_last_X = frame
        return explanation

    def _ensure_owen_values(self, X=None):
        if X is not None:
            self.explain_owen(X)
        if self.owen_values_ is None or self._owen_last_X is None or self.coalition_structure_ is None:
            raise RuntimeError("No Owen values available. Call explain_owen(X) first or pass X=...")
        return self.owen_values_, self._owen_last_X, self.coalition_structure_

    def owen_feature_importance(self, X=None, normalize=False):
        """Global feature importance as mean absolute Owen value.

        Parameters
        ----------
        X : pandas.DataFrame, array-like, or None, default None
            Rows to explain. When given, the Owen values are recomputed with
            ``explain_owen(X)`` using its defaults (the cached coalition structure and
            ``model_output="probability"``) and replace the cached ones; when ``None``,
            the values cached by the last `explain_owen` call are used.
        normalize : bool, default False
            If ``True``, add an ``importance_pct`` column with each feature's share of
            the total mean absolute Owen value (fractions that sum to 1, not percent).

        Returns
        -------
        pandas.DataFrame
            One row per feature with ``feature`` and ``mean_abs_owen`` (plus
            ``importance_pct`` when ``normalize=True``), sorted by ``mean_abs_owen`` in
            descending order.

        Raises
        ------
        RuntimeError
            If no Owen values are cached and ``X`` is ``None``.
        """
        values, X_used, _ = self._ensure_owen_values(X)
        names = self._resolved_names(X_used, values.shape[1])
        table = pd.DataFrame({"feature": names, "mean_abs_owen": np.abs(values).mean(axis=0)})
        table = table.sort_values("mean_abs_owen", ascending=False).reset_index(drop=True)
        if normalize:
            total = table["mean_abs_owen"].sum()
            table["importance_pct"] = table["mean_abs_owen"] / total if total else 0.0
        return table

    def owen_group_importance(self, X=None, normalize=False):
        """Aggregate Owen values to coalition groups.

        Parameters
        ----------
        X : pandas.DataFrame, array-like, or None, default None
            Rows to explain. When given, the Owen values are recomputed with
            ``explain_owen(X)`` using its defaults (the cached coalition structure and
            ``model_output="probability"``) and replace the cached ones; when ``None``,
            the values cached by the last `explain_owen` call are used.
        normalize : bool, default False
            If ``True``, add an ``importance_pct`` column with each group's share of the
            total ``mean_abs_owen`` (fractions that sum to 1, not percent).

        Returns
        -------
        pandas.DataFrame
            One row per coalition group with ``group``, ``n_features``, ``features``
            (the member features found in the data), ``mean_owen`` and ``mean_abs_owen``
            (plus ``importance_pct`` when ``normalize=True``), sorted by
            ``mean_abs_owen`` in descending order. A group's value for a row is the sum
            of its members' Owen values; ``mean_owen`` averages that signed sum and
            ``mean_abs_owen`` averages its absolute value.

        Raises
        ------
        RuntimeError
            If no Owen values are cached and ``X`` is ``None``.
        """
        values, X_used, cs = self._ensure_owen_values(X)
        names = list(X_used.columns)
        rows = []
        for group, feats in cs["groups"].items():
            idxs = [names.index(feat) for feat in feats if feat in names]
            if not idxs:
                continue
            contrib = values[:, idxs].sum(axis=1)
            rows.append(
                {
                    "group": group,
                    "n_features": len(idxs),
                    "features": [names[i] for i in idxs],
                    "mean_owen": float(np.mean(contrib)),
                    "mean_abs_owen": float(np.mean(np.abs(contrib))),
                }
            )
        table = pd.DataFrame(rows).sort_values("mean_abs_owen", ascending=False).reset_index(drop=True)
        if normalize and not table.empty:
            total = table["mean_abs_owen"].sum()
            table["importance_pct"] = table["mean_abs_owen"] / total if total else 0.0
        return table

    def owen_explain_instance(self, x_row=None, aggregate_groups=True):
        """Return local Owen reason codes for one sample.

        Parameters
        ----------
        x_row : pandas.DataFrame, array-like, or None, default None
            One observation. When ``None``, the first row of the cached Owen batch is
            used. When given, only its first row is used and the Owen values are
            recomputed for it with ``explain_owen`` defaults (cached coalition structure,
            ``model_output="probability"``), which replaces the cached results. A Series
            or 1-D array is read by position, so its values must follow
            ``feature_names``; a dict is not supported.
        aggregate_groups : bool, default True
            If ``True``, return one row per coalition group (the sum of its members'
            Owen values). If ``False``, return one row per feature.

        Returns
        -------
        pandas.DataFrame
            With ``aggregate_groups=True``: ``group``, ``n_features``, ``features``,
            ``owen_value`` and ``abs_owen_value``, sorted by ``abs_owen_value`` (largest
            first), with ``attrs["base_value"]`` (float, NaN when unavailable) and
            ``attrs["model_output"]``. With ``aggregate_groups=False``: ``feature``,
            ``value``, ``owen_value`` and ``abs_owen_value`` sorted the same way, without
            those ``attrs``.

        Raises
        ------
        RuntimeError
            If ``x_row`` is ``None`` and no Owen values are cached.
        """
        if x_row is not None:
            self.explain_owen(self._as_frame(x_row).iloc[[0]])
        values, X_used, cs = self._ensure_owen_values(None)
        row_values = values[0]
        row_x = X_used.iloc[0]
        if not aggregate_groups:
            table = pd.DataFrame({"feature": list(X_used.columns), "value": row_x.values, "owen_value": row_values})
            table["abs_owen_value"] = table["owen_value"].abs()
            return table.sort_values("abs_owen_value", ascending=False).reset_index(drop=True)

        rows = []
        names = list(X_used.columns)
        for group, feats in cs["groups"].items():
            idxs = [names.index(feat) for feat in feats if feat in names]
            if not idxs:
                continue
            contrib = float(row_values[idxs].sum())
            rows.append(
                {
                    "group": group,
                    "n_features": len(idxs),
                    "features": [names[i] for i in idxs],
                    "owen_value": contrib,
                    "abs_owen_value": abs(contrib),
                }
            )
        table = pd.DataFrame(rows).sort_values("abs_owen_value", ascending=False).reset_index(drop=True)
        base = self.owen_expected_value_
        table.attrs["base_value"] = float(np.ravel(base)[0]) if base is not None and np.asarray(base).size else float("nan")
        table.attrs["model_output"] = self._owen_model_output
        return table

    # ------------------------------------------------------------------ #
    # PDP / ICE
    # ------------------------------------------------------------------ #
    def partial_dependence(self, X, feature, grid_resolution=50, percentiles=(0.05, 0.95), sample_size=None, random_state=None, prediction_batch_size=100000):
        """Compute one-way partial dependence for a numeric feature.

        Parameters
        ----------
        X : pandas.DataFrame or array-like
            Data to average over. A DataFrame may carry extra columns: the model's
            features are selected by name (see ``feature_names``).
        feature : str or int
            Numeric feature to vary: a column name, or a Python ``int`` position among
            the model's feature columns. A name that is not one of the model's columns
            raises ``KeyError``.
        grid_resolution : int, default 50
            Number of grid points between the lower and upper percentile of the feature.
        percentiles : tuple of float, default (0.05, 0.95)
            Lower and upper quantile of the feature, as fractions between 0 and 1, that
            bound the grid. When they coincide (for example for a constant feature), the
            grid is the sorted distinct values of the feature, at most
            ``grid_resolution`` of them.
        sample_size : int or None, default None
            When ``X`` has more rows than this, a random sample of this size is used;
            ``None`` uses every row.
        random_state : int or None, default None
            Seed of the row sampling (only used when sampling happens); ``None`` draws a
            different sample on each call.
        prediction_batch_size : int or None, default 100000
            Maximum number of rows sent to the model in one prediction call, which bounds
            peak memory; ``None`` predicts all grid points in a single call. A value
            below 1 raises ``ValueError``.

        Returns
        -------
        pandas.DataFrame
            One row per grid point with ``feature`` (the column name), ``grid_value`` and
            ``average_prediction`` (the mean predicted positive-class probability when
            the feature is set to the grid value for every row).

        Raises
        ------
        KeyError
            If ``feature`` is not one of the model's feature columns of ``X``.
        ValueError
            If the feature has no non-missing numeric value, or if
            ``prediction_batch_size`` is not a positive integer or ``None``.
        """
        frame = self._sample_frame(X, sample_size=sample_size, random_state=random_state)
        feature = self._feature_name(frame, feature)
        grid = self._numeric_grid(frame[feature], grid_resolution=grid_resolution, percentiles=percentiles)
        if len(frame) == 0:
            raise ValueError("partial_dependence requires at least one input row")
        batch_size = (
            len(frame) * max(len(grid), 1)
            if prediction_batch_size is None
            else int(prediction_batch_size)
        )
        if batch_size <= 0:
            raise ValueError("prediction_batch_size must be a positive integer or None")
        grids_per_batch = max(1, batch_size // len(frame))
        averages = []
        for start in range(0, len(grid), grids_per_batch):
            grid_block = grid[start : start + grids_per_batch]
            stacked = pd.concat(
                [frame.assign(**{feature: value}) for value in grid_block],
                ignore_index=True,
            )
            predictions = self._predict_proba_pos_batched(
                stacked,
                prediction_batch_size=prediction_batch_size,
            ).reshape(len(grid_block), len(frame))
            averages.extend(predictions.mean(axis=1).astype(float).tolist())
        return pd.DataFrame({"feature": feature, "grid_value": grid, "average_prediction": averages})

    def pdp_plot(self, X, feature, grid_resolution=50, percentiles=(0.05, 0.95), sample_size=None, random_state=None, show=True, save_path=None, prediction_batch_size=100000):
        """Plot one-way partial dependence for a numeric feature.

        Parameters
        ----------
        X : pandas.DataFrame or array-like
            Data to average over; same handling as in `partial_dependence`.
        feature : str or int
            Numeric feature to vary: a column name, or a Python ``int`` position among
            the model's feature columns.
        grid_resolution : int, default 50
            Number of grid points between the lower and upper percentile of the feature.
        percentiles : tuple of float, default (0.05, 0.95)
            Lower and upper quantile of the feature, as fractions between 0 and 1, that
            bound the grid.
        sample_size : int or None, default None
            When ``X`` has more rows than this, a random sample of this size is used;
            ``None`` uses every row.
        random_state : int or None, default None
            Seed of the row sampling (only used when sampling happens).
        show : bool, default True
            If ``True``, display the figure with ``plt.show()``. If ``False``, the figure
            is closed after it has been saved, so nothing is displayed.
        save_path : str, path-like, or None, default None
            When given, the figure is saved to this path (``dpi=150``, tight bounding
            box; the format follows the file extension). The folder must already exist.
        prediction_batch_size : int or None, default 100000
            Maximum number of rows sent to the model in one prediction call; ``None``
            predicts all grid points in a single call.

        Returns
        -------
        matplotlib.figure.Figure
            The figure with the average-prediction curve over the grid.

        Raises
        ------
        KeyError
            If ``feature`` is not one of the model's feature columns of ``X``.
        ValueError
            If the feature has no non-missing numeric value, or if
            ``prediction_batch_size`` is not a positive integer or ``None``.
        """
        import matplotlib.pyplot as plt

        df = self.partial_dependence(
            X,
            feature,
            grid_resolution,
            percentiles,
            sample_size,
            random_state,
            prediction_batch_size,
        )
        fig, ax = plt.subplots(figsize=(7, 4), dpi=120)
        ax.plot(df["grid_value"], df["average_prediction"], color="#336699", linewidth=2)
        ax.set_xlabel(str(df["feature"].iloc[0]))
        ax.set_ylabel("Average prediction")
        ax.set_title(f"PDP: {df['feature'].iloc[0]}")
        ax.grid(alpha=0.25)
        return self._finalize_plot(plt, show, save_path)

    def ice(self, X, feature, grid_resolution=50, percentiles=(0.05, 0.95), sample_size=200, random_state=None, centered=False, prediction_batch_size=100000):
        """Compute individual conditional expectation curves.

        Parameters
        ----------
        X : pandas.DataFrame or array-like
            Data to draw the curves from. A DataFrame may carry extra columns: the
            model's features are selected by name (see ``feature_names``).
        feature : str or int
            Numeric feature to vary: a column name, or a Python ``int`` position among
            the model's feature columns. A name that is not one of the model's columns
            raises ``KeyError``.
        grid_resolution : int, default 50
            Number of grid points between the lower and upper percentile of the feature.
        percentiles : tuple of float, default (0.05, 0.95)
            Lower and upper quantile of the feature, as fractions between 0 and 1, that
            bound the grid. When they coincide (for example for a constant feature), the
            grid is the sorted distinct values of the feature, at most
            ``grid_resolution`` of them.
        sample_size : int or None, default 200
            When ``X`` has more rows than this, a random sample of this many rows (one
            curve each) is used; ``None`` uses every row.
        random_state : int or None, default None
            Seed of the row sampling (only used when sampling happens); ``None`` draws a
            different sample on each call.
        centered : bool, default False
            If ``True``, subtract from each curve its prediction at the first grid point,
            so every curve starts at 0.
        prediction_batch_size : int or None, default 100000
            Maximum number of rows sent to the model in one prediction call; ``None``
            predicts all stacked rows in a single call. A value below 1 raises
            ``ValueError``.

        Returns
        -------
        pandas.DataFrame
            Long format with one row per grid point and sampled row, ordered by grid
            value: ``feature`` (the column name), ``sample_index`` (the row label in
            ``X``), ``grid_value`` and ``prediction`` (the predicted positive-class
            probability, or its change from the first grid point when ``centered=True``).

        Raises
        ------
        KeyError
            If ``feature`` is not one of the model's feature columns of ``X``.
        ValueError
            If the feature has no non-missing numeric value, or if
            ``prediction_batch_size`` is not a positive integer or ``None``.
        """
        frame = self._sample_frame(X, sample_size=sample_size, random_state=random_state)
        feature = self._feature_name(frame, feature)
        grid = self._numeric_grid(frame[feature], grid_resolution=grid_resolution, percentiles=percentiles)
        stacked = pd.concat(
            [frame.assign(**{feature: value}) for value in grid],
            ignore_index=True,
        )
        predictions = self._predict_proba_pos_batched(
            stacked,
            prediction_batch_size=prediction_batch_size,
        )
        out = pd.DataFrame(
            {
                "feature": feature,
                "sample_index": np.tile(frame.index.to_numpy(), len(grid)),
                "grid_value": np.repeat(grid, len(frame)),
                "prediction": predictions,
            }
        )
        if centered:
            prediction_matrix = predictions.reshape(len(grid), len(frame))
            out["prediction"] = (
                prediction_matrix - prediction_matrix[0:1, :]
            ).reshape(-1)
        return out

    def ice_plot(self, X, feature, grid_resolution=50, percentiles=(0.05, 0.95), sample_size=100, random_state=None, centered=False, show=True, save_path=None, prediction_batch_size=100000):
        """Plot ICE curves for a numeric feature.

        Parameters
        ----------
        X : pandas.DataFrame or array-like
            Data to draw the curves from; same handling as in `ice`.
        feature : str or int
            Numeric feature to vary: a column name, or a Python ``int`` position among
            the model's feature columns.
        grid_resolution : int, default 50
            Number of grid points between the lower and upper percentile of the feature.
        percentiles : tuple of float, default (0.05, 0.95)
            Lower and upper quantile of the feature, as fractions between 0 and 1, that
            bound the grid.
        sample_size : int or None, default 100
            When ``X`` has more rows than this, a random sample of this many rows (one
            curve each) is drawn; ``None`` uses every row. Note that this default is
            smaller than the one of `ice` (200).
        random_state : int or None, default None
            Seed of the row sampling (only used when sampling happens).
        centered : bool, default False
            If ``True``, every curve is shifted to start at 0 (the y-axis is then
            labelled "Centered prediction").
        show : bool, default True
            If ``True``, display the figure with ``plt.show()``. If ``False``, the figure
            is closed after it has been saved, so nothing is displayed.
        save_path : str, path-like, or None, default None
            When given, the figure is saved to this path (``dpi=150``, tight bounding
            box; the format follows the file extension). The folder must already exist.
        prediction_batch_size : int or None, default 100000
            Maximum number of rows sent to the model in one prediction call; ``None``
            predicts all stacked rows in a single call.

        Returns
        -------
        matplotlib.figure.Figure
            The figure with one thin curve per sampled row and the average curve in red.

        Raises
        ------
        KeyError
            If ``feature`` is not one of the model's feature columns of ``X``.
        ValueError
            If the feature has no non-missing numeric value, or if
            ``prediction_batch_size`` is not a positive integer or ``None``.
        """
        import matplotlib.pyplot as plt

        df = self.ice(
            X,
            feature,
            grid_resolution,
            percentiles,
            sample_size,
            random_state,
            centered,
            prediction_batch_size,
        )
        fig, ax = plt.subplots(figsize=(7, 4), dpi=120)
        for _, group in df.groupby("sample_index"):
            ax.plot(group["grid_value"], group["prediction"], color="#336699", alpha=0.18, linewidth=0.8)
        avg = df.groupby("grid_value", as_index=False)["prediction"].mean()
        ax.plot(avg["grid_value"], avg["prediction"], color="#CC0033", linewidth=2.2, label="average")
        ax.set_xlabel(str(df["feature"].iloc[0]))
        ax.set_ylabel("Centered prediction" if centered else "Prediction")
        ax.set_title(f"ICE: {df['feature'].iloc[0]}")
        ax.legend()
        ax.grid(alpha=0.25)
        return self._finalize_plot(plt, show, save_path)

    # ------------------------------------------------------------------ #
    # ALE
    # ------------------------------------------------------------------ #
    def ale(self, X, feature, bins=20, sample_size=None, random_state=None, prediction_batch_size=100000):
        """Compute first-order accumulated local effects for a numeric feature.

        Parameters
        ----------
        X : pandas.DataFrame or array-like
            Data the effect is estimated on. A DataFrame may carry extra columns: the
            model's features are selected by name (see ``feature_names``).
        feature : str or int
            Numeric feature to analyse: a column name, or a Python ``int`` position among
            the model's feature columns. A name that is not one of the model's columns
            raises ``KeyError``.
        bins : int, default 20
            Number of equal-count (quantile) intervals the feature is cut into. Repeated
            quantile edges, caused by heavily tied values, are merged, so fewer intervals
            may result.
        sample_size : int or None, default None
            When ``X`` has more rows than this, a random sample of this size is used;
            ``None`` uses every row.
        random_state : int or None, default None
            Seed of the row sampling (only used when sampling happens); ``None`` draws a
            different sample on each call.
        prediction_batch_size : int or None, default 100000
            Maximum number of rows sent to the model in one prediction call; ``None``
            predicts all stacked rows in a single call. A value below 1 raises
            ``ValueError``.

        Returns
        -------
        pandas.DataFrame
            One row per interval with ``feature`` (the column name), ``bin_left``,
            ``bin_right``, ``bin_center``, ``ale_value`` and ``n`` (number of rows in the
            interval). The ``ale_value`` curve is centered so that its count-weighted
            mean is 0.

        Raises
        ------
        KeyError
            If ``feature`` is not one of the model's feature columns of ``X``.
        ValueError
            If the feature has no non-missing numeric value or fewer than two distinct
            interval edges, or if ``prediction_batch_size`` is not a positive integer or
            ``None``.

        Notes
        -----
        Rows whose value of ``feature`` is missing (or not numeric) are dropped before the
        effect is computed, so ``n`` sums to the number of rows with a valid value.
        """
        frame = self._sample_frame(X, sample_size=sample_size, random_state=random_state)
        feature = self._feature_name(frame, feature)
        values = pd.to_numeric(frame[feature], errors="coerce")
        valid = values.notna()
        work = frame.loc[valid].copy()
        values = values.loc[valid]
        if work.empty:
            raise ValueError(f"Feature {feature!r} has no non-missing numeric values")

        quantiles = np.linspace(0, 1, int(bins) + 1)
        edges = np.unique(values.quantile(quantiles).to_numpy(dtype=float))
        if len(edges) < 2:
            raise ValueError(f"Feature {feature!r} does not have enough unique values for ALE")
        edges[0] = values.min()
        edges[-1] = values.max()

        bin_ids = np.searchsorted(edges, values.to_numpy(), side="right") - 1
        bin_ids = np.clip(bin_ids, 0, len(edges) - 2)

        effects = np.zeros(len(edges) - 1, dtype=float)
        counts = []
        centers = []
        prediction_frames = []
        prediction_segments = []
        for idx in range(len(edges) - 1):
            mask = bin_ids == idx
            subset = work.loc[mask].copy()
            counts.append(int(mask.sum()))
            left, right = float(edges[idx]), float(edges[idx + 1])
            centers.append((left + right) / 2.0)
            if subset.empty:
                continue
            low = subset.copy()
            high = subset.copy()
            low[feature] = left
            high[feature] = right
            prediction_segments.append((idx, len(subset)))
            prediction_frames.extend([low, high])

        if prediction_frames:
            stacked = pd.concat(prediction_frames, ignore_index=True)
            predictions = self._predict_proba_pos_batched(
                stacked,
                prediction_batch_size=prediction_batch_size,
            )
            offset = 0
            for idx, count in prediction_segments:
                low_pred = predictions[offset : offset + count]
                high_pred = predictions[offset + count : offset + 2 * count]
                effects[idx] = float(np.mean(high_pred - low_pred))
                offset += 2 * count

        ale_values = np.cumsum(effects)
        counts_arr = np.asarray(counts, dtype=float)
        if counts_arr.sum() > 0:
            ale_values = ale_values - np.average(ale_values, weights=counts_arr)
        else:
            ale_values = ale_values - ale_values.mean()

        return pd.DataFrame({"feature": feature, "bin_left": edges[:-1], "bin_right": edges[1:], "bin_center": centers, "ale_value": ale_values, "n": counts})

    def ale_plot(self, X, feature, bins=20, sample_size=None, random_state=None, show=True, save_path=None, prediction_batch_size=100000):
        """Plot first-order ALE for a numeric feature.

        Parameters
        ----------
        X : pandas.DataFrame or array-like
            Data the effect is estimated on; same handling as in `ale`.
        feature : str or int
            Numeric feature to analyse: a column name, or a Python ``int`` position among
            the model's feature columns.
        bins : int, default 20
            Number of equal-count (quantile) intervals the feature is cut into.
        sample_size : int or None, default None
            When ``X`` has more rows than this, a random sample of this size is used;
            ``None`` uses every row.
        random_state : int or None, default None
            Seed of the row sampling (only used when sampling happens).
        show : bool, default True
            If ``True``, display the figure with ``plt.show()``. If ``False``, the figure
            is closed after it has been saved, so nothing is displayed.
        save_path : str, path-like, or None, default None
            When given, the figure is saved to this path (``dpi=150``, tight bounding
            box; the format follows the file extension). The folder must already exist.
        prediction_batch_size : int or None, default 100000
            Maximum number of rows sent to the model in one prediction call; ``None``
            predicts all stacked rows in a single call.

        Returns
        -------
        matplotlib.figure.Figure
            The figure with the ALE curve plotted at the interval centers.

        Raises
        ------
        KeyError
            If ``feature`` is not one of the model's feature columns of ``X``.
        ValueError
            If the feature has no non-missing numeric value or fewer than two distinct
            interval edges, or if ``prediction_batch_size`` is not a positive integer or
            ``None``.
        """
        import matplotlib.pyplot as plt

        df = self.ale(
            X,
            feature,
            bins=bins,
            sample_size=sample_size,
            random_state=random_state,
            prediction_batch_size=prediction_batch_size,
        )
        fig, ax = plt.subplots(figsize=(7, 4), dpi=120)
        ax.plot(df["bin_center"], df["ale_value"], color="#336699", marker="o", linewidth=2)
        ax.axhline(0, color="#999999", linewidth=1, linestyle="--")
        ax.set_xlabel(str(df["feature"].iloc[0]))
        ax.set_ylabel("Accumulated local effect")
        ax.set_title(f"ALE: {df['feature'].iloc[0]}")
        ax.grid(alpha=0.25)
        return self._finalize_plot(plt, show, save_path)

    # ------------------------------------------------------------------ #
    # LIME
    # ------------------------------------------------------------------ #
    @staticmethod
    def _lime_missing_mask(frame: pd.DataFrame) -> pd.Series:
        numeric = frame.select_dtypes(include=[np.number])
        if numeric.empty:
            return pd.Series(False, index=frame.index)
        return numeric.isna().any(axis=1)

    @classmethod
    def _warn_lime_missing(
        cls,
        frame: pd.DataFrame,
        *,
        role: str,
        missing_strategy: str,
        n_before: int,
        n_after: int,
        fill_values: dict[str, float] | None = None,
    ) -> None:
        numeric = frame.select_dtypes(include=[np.number])
        if numeric.empty:
            return
        null_cols = [col for col in numeric.columns if numeric[col].isna().any()]
        if not null_cols:
            return
        details = []
        for col in null_cols:
            count = int(numeric[col].isna().sum())
            pct = 100.0 * count / len(frame) if len(frame) else 0.0
            detail = f"{col}: {count}/{len(frame)} ({pct:.1f}%)"
            if fill_values and col in fill_values:
                detail += f", median_fill={fill_values[col]:.6g}"
            details.append(detail)
        msg = (
            f"LIME missing_strategy={missing_strategy!r} on {role}: "
            f"null columns={null_cols}; " + "; ".join(details)
        )
        if missing_strategy == "drop":
            msg += f"; rows {n_before} -> {n_after}"
        warnings.warn(msg, UserWarning, stacklevel=3)

    @classmethod
    def _prepare_lime_data(
        cls,
        train: pd.DataFrame,
        rows: pd.DataFrame | None = None,
        *,
        train_role: str = "X_train",
        rows_role: str = "x_row",
        missing_strategy: str = "median",
    ) -> tuple[pd.DataFrame, pd.DataFrame | None]:
        if missing_strategy not in {"median", "drop"}:
            raise ValueError("missing_strategy must be 'median' or 'drop'")

        train_clean = train.copy()
        rows_clean = rows.copy() if rows is not None else None
        numeric_cols = list(train_clean.select_dtypes(include=[np.number]).columns)

        if missing_strategy == "median":
            fill_values: dict[str, float] = {}
            for col in numeric_cols:
                if train_clean[col].isna().any() or (
                    rows_clean is not None and rows_clean[col].isna().any()
                ):
                    median_val = float(train_clean[col].median())
                    fill_values[col] = median_val
                    train_clean[col] = train_clean[col].fillna(median_val)
                    if rows_clean is not None:
                        rows_clean[col] = rows_clean[col].fillna(median_val)
            cls._warn_lime_missing(
                train,
                role=train_role,
                missing_strategy=missing_strategy,
                n_before=len(train),
                n_after=len(train_clean),
                fill_values=fill_values or None,
            )
            if rows_clean is not None:
                cls._warn_lime_missing(
                    rows,
                    role=rows_role,
                    missing_strategy=missing_strategy,
                    n_before=len(rows),
                    n_after=len(rows_clean),
                    fill_values=fill_values or None,
                )
            return train_clean, rows_clean

        n_train_before = len(train_clean)
        train_clean = train_clean.loc[~cls._lime_missing_mask(train_clean)].copy()
        cls._warn_lime_missing(
            train,
            role=train_role,
            missing_strategy=missing_strategy,
            n_before=n_train_before,
            n_after=len(train_clean),
        )
        if train_clean.empty:
            raise ValueError(
                f"LIME missing_strategy='drop' removed all rows from {train_role} "
                f"({n_train_before} -> 0). Provide complete training data or use missing_strategy='median'."
            )

        if rows_clean is not None:
            n_rows_before = len(rows_clean)
            rows_clean = rows_clean.loc[~cls._lime_missing_mask(rows_clean)].copy()
            cls._warn_lime_missing(
                rows,
                role=rows_role,
                missing_strategy=missing_strategy,
                n_before=n_rows_before,
                n_after=len(rows_clean),
            )
            if rows_clean.empty:
                raise ValueError(
                    f"LIME missing_strategy='drop' removed all rows from {rows_role} "
                    f"({n_rows_before} -> 0). Impute missing values or use missing_strategy='median'."
                )
        return train_clean, rows_clean

    def _build_lime_explainer(self, X_train, num_features=None, random_state=None, **lime_kwargs):
        LimeTabularExplainer = _lazy_lime()
        train = self._as_frame(X_train if X_train is not None else self.background_data)
        if train is None:
            raise ValueError("LIME requires X_train or background_data")
        mode = lime_kwargs.pop("mode", "classification")
        class_names = lime_kwargs.pop("class_names", ["class_0", "class_1"])
        return LimeTabularExplainer(
            training_data=np.asarray(train),
            feature_names=list(train.columns),
            class_names=class_names,
            mode=mode,
            random_state=random_state,
            **lime_kwargs,
        )

    def lime_explain_instance(
        self,
        x_row,
        X_train=None,
        num_features=10,
        num_samples=5000,
        random_state=None,
        missing_strategy="median",
        **lime_kwargs,
    ):
        """Explain one sample with LIME.

        Parameters
        ----------
        x_row : pandas.DataFrame, pandas.Series, dict, or array-like
            One observation. A Series or a dict is turned into a one-row frame and a 1-D
            array is read as one row; when several rows are given, only the first is
            explained.
        X_train : pandas.DataFrame, array-like, or None, default None
            Training data used to set up the LIME explainer (feature statistics and
            sampling). When ``None``, the explainer's ``background_data`` is used; one of
            the two must be available.
        num_features : int, default 10
            Maximum number of features in the explanation.
        num_samples : int, default 5000
            Number of perturbed samples used to fit the local surrogate model.
        random_state : int or None, default None
            Seed of the LIME explainer.
        missing_strategy : str, default "median"
            How missing values in the numeric columns of ``X_train`` and ``x_row`` are
            handled: ``"median"`` fills them with the training medians; ``"drop"``
            removes the affected rows. When values are missing, a ``UserWarning`` names
            the affected columns. Any other value raises ``ValueError``.
        **lime_kwargs
            Extra keyword arguments for ``LimeTabularExplainer``, for example
            ``discretize_continuous=False``. ``mode`` defaults to ``"classification"``
            and ``class_names`` to ``["class_0", "class_1"]``.

        Returns
        -------
        pandas.DataFrame
            One row per feature rule with ``feature``, ``feature_rule`` (the LIME rule
            text), ``weight`` and ``abs_weight``, sorted by ``abs_weight`` (largest
            first). ``attrs["intercept"]`` (a dict keyed by class label) and
            ``attrs["score"]`` (the local surrogate's R squared) describe the surrogate
            model. When LIME returns no rule, an empty frame with these four columns is
            returned.

        Raises
        ------
        ValueError
            If neither ``X_train`` nor the explainer's ``background_data`` is available,
            if ``missing_strategy`` is not ``"median"`` or ``"drop"``, or if ``"drop"``
            removes every row of ``X_train`` or ``x_row``.
        ImportError
            If the optional ``lime`` package is not installed.

        Notes
        -----
        The explained quantity is the positive-class probability of the model.
        """
        if isinstance(x_row, pd.Series):
            x_row = x_row.to_frame().T
        elif isinstance(x_row, dict):
            x_row = pd.DataFrame([x_row])
        frame = self._as_frame(x_row)
        if frame.shape[0] != 1:
            frame = frame.iloc[[0]]

        train_raw = self._as_frame(X_train if X_train is not None else self.background_data)
        if train_raw is None:
            raise ValueError("LIME requires X_train or background_data")
        train, frame = self._prepare_lime_data(
            train_raw,
            frame,
            train_role="X_train",
            rows_role="x_row",
            missing_strategy=missing_strategy,
        )
        explainer = self._build_lime_explainer(train, num_features, random_state, **lime_kwargs)
        explanation = explainer.explain_instance(
            data_row=np.asarray(frame.iloc[0]),
            predict_fn=self._predict_proba_2d,
            num_features=num_features,
            num_samples=num_samples,
        )
        rows = []
        names = list(frame.columns)
        for rule, weight in explanation.as_list():
            matched = next((name for name in names if str(rule).startswith(name) or name in str(rule)), rule)
            rows.append({"feature": matched, "feature_rule": rule, "weight": float(weight)})
        out = pd.DataFrame(rows)
        if out.empty:
            return pd.DataFrame(columns=["feature", "feature_rule", "weight", "abs_weight"])
        out["abs_weight"] = out["weight"].abs()
        out = out.sort_values("abs_weight", ascending=False).reset_index(drop=True)
        out.attrs["intercept"] = explanation.intercept
        out.attrs["score"] = explanation.score
        return out

    def lime_global_importance(
        self,
        X,
        X_train=None,
        num_features=10,
        num_samples=2000,
        sample_size=100,
        random_state=None,
        missing_strategy="median",
        **lime_kwargs,
    ):
        """Aggregate LIME local weights across a sample as global importance.

        Parameters
        ----------
        X : pandas.DataFrame or array-like
            Rows to sample from. Each sampled row is explained with
            `lime_explain_instance`.
        X_train : pandas.DataFrame, array-like, or None, default None
            Training data used to set up the LIME explainers. When ``None``, the
            explainer's ``background_data`` is used; one of the two must be available.
        num_features : int, default 10
            Maximum number of features in each local explanation.
        num_samples : int, default 2000
            Number of perturbed samples per explained row (smaller than the default of
            `lime_explain_instance`, 5000).
        sample_size : int or None, default 100
            Maximum number of rows of ``X`` that are explained (a random sample); ``None``
            explains every row.
        random_state : int or None, default None
            Seed of the row sampling and of every LIME explainer.
        missing_strategy : str, default "median"
            How missing values in the numeric columns of ``X_train`` and of the sampled
            rows are handled: ``"median"`` fills them with the training medians;
            ``"drop"`` removes the affected rows (``X_train`` rows, and sampled rows
            before they are explained). When values are missing, a ``UserWarning`` names
            the affected columns. Any other value raises ``ValueError``.
        **lime_kwargs
            Extra keyword arguments passed through to `lime_explain_instance` and on to
            ``LimeTabularExplainer``, for example ``discretize_continuous=False``.

        Returns
        -------
        pandas.DataFrame
            One row per feature with ``feature``, ``mean_abs_lime_weight`` (the mean
            absolute LIME weight over the explained rows in which the feature appears)
            and ``frequency`` (the number of explained rows whose local explanation
            contains the feature), sorted by ``mean_abs_lime_weight`` in descending
            order. An empty frame with these columns is returned when no row is explained.

        Raises
        ------
        ValueError
            If neither ``X_train`` nor the explainer's ``background_data`` is available,
            if ``missing_strategy`` is not ``"median"`` or ``"drop"``, or if ``"drop"``
            removes every row of ``X_train`` or of the sample.
        ImportError
            If the optional ``lime`` package is not installed.

        Notes
        -----
        A new LIME explainer is built and fitted for every explained row, so the run
        time grows with ``sample_size * num_samples``.
        """
        train_raw = self._as_frame(X_train if X_train is not None else self.background_data)
        if train_raw is None:
            raise ValueError("LIME requires X_train or background_data")
        frame = self._sample_frame(X, sample_size=sample_size, random_state=random_state)
        train, frame = self._prepare_lime_data(
            train_raw,
            frame,
            train_role="X_train",
            rows_role="X",
            missing_strategy=missing_strategy,
        )
        rows = []
        for _, row in frame.iterrows():
            local = self.lime_explain_instance(
                row,
                X_train=train,
                num_features=num_features,
                num_samples=num_samples,
                random_state=random_state,
                missing_strategy="median",
                **lime_kwargs,
            )
            rows.append(local)
        if not rows:
            return pd.DataFrame(columns=["feature", "mean_abs_lime_weight", "frequency"])
        all_rows = pd.concat(rows, ignore_index=True)
        summary = all_rows.groupby("feature", as_index=False).agg(
            mean_abs_lime_weight=("abs_weight", "mean"),
            frequency=("feature", "size"),
        )
        return summary.sort_values("mean_abs_lime_weight", ascending=False).reset_index(drop=True)

    # ------------------------------------------------------------------ #
    # Internals
    # ------------------------------------------------------------------ #
    @staticmethod
    def _finalize_plot(plt, show, save_path):
        fig = plt.gcf()
        if save_path:
            fig.savefig(save_path, bbox_inches="tight", dpi=150)
        if show:
            plt.show()
        else:
            plt.close(fig)
        return fig

    def __repr__(self):
        n_feat = len(self.feature_names) if self.feature_names else "?"
        return f"ModelExplainer(model_type={self.model_type!r}, n_features={n_feat}, kind={self._explainer_kind!r})"

explain

explain(X)

Compute and cache SHAP values for a dataset.

Parameters:

Name	Type	Description	Default
X	DataFrame or array - like	Rows to explain. A DataFrame is aligned to feature_names by column name when it contains all of them (extra columns are ignored); an array is labelled with feature_names when its column count matches, and a 1-D array is read as a single row.	required

Returns:

Type	Description
Explanation	The explanation object returned by the SHAP explainer.

Raises:

Type	Description
ValueError	If the model is linear or needs the model-agnostic explainer and no background_data was given; or, for XGBoost, if SHAP still cannot parse the model's base_score after the built-in compatibility patches.
ImportError	If the optional shap package is not installed.

Notes

The SHAP explainer is built on the first call and reused afterwards: TreeExplainer for LightGBM and XGBoost (background_data optional), LinearExplainer for linear models (background_data required), and the model-agnostic shap.Explainer on the positive-class probability for any other model (background_data required). Tree and linear SHAP values are in the model's raw output scale (log-odds for a binary classifier), model-agnostic values are in probability units.

Sets shap_values_ (2-D; for a multi-output explanation only the last output is kept), expected_value_ and explanation_, and remembers X for the plot and importance methods that are called without data.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def explain(self, X):
    """Compute and cache SHAP values for a dataset.

    Parameters
    ----------
    X : pandas.DataFrame or array-like
        Rows to explain. A DataFrame is aligned to ``feature_names`` by column name
        when it contains all of them (extra columns are ignored); an array is
        labelled with ``feature_names`` when its column count matches, and a 1-D
        array is read as a single row.

    Returns
    -------
    shap.Explanation
        The explanation object returned by the SHAP explainer.

    Raises
    ------
    ValueError
        If the model is linear or needs the model-agnostic explainer and no
        ``background_data`` was given; or, for XGBoost, if SHAP still cannot parse
        the model's ``base_score`` after the built-in compatibility patches.
    ImportError
        If the optional ``shap`` package is not installed.

    Notes
    -----
    The SHAP explainer is built on the first call and reused afterwards:
    ``TreeExplainer`` for LightGBM and XGBoost (``background_data`` optional),
    ``LinearExplainer`` for linear models (``background_data`` required), and the
    model-agnostic ``shap.Explainer`` on the positive-class probability for any
    other model (``background_data`` required). Tree and linear SHAP values are in
    the model's raw output scale (log-odds for a binary classifier), model-agnostic
    values are in probability units.

    Sets ``shap_values_`` (2-D; for a multi-output explanation only the last output
    is kept), ``expected_value_`` and ``explanation_``, and remembers ``X`` for the
    plot and importance methods that are called without data.
    """
    frame = self._as_frame(X)
    values, base, explanation = self._shap_values_for(frame)
    self.shap_values_ = values
    self.expected_value_ = base
    self.explanation_ = explanation
    self._last_X = frame
    return explanation

feature_importance

feature_importance(X=None, normalize=False)

Global feature importance as mean absolute SHAP value.

Parameters:

Name	Type	Description	Default
X	pandas.DataFrame, array-like, or None	Rows to explain. When given, the SHAP values are recomputed with explain and replace the cached ones; when None, the values cached by the last SHAP computation are used.	None
normalize	bool	If True, add an importance_pct column with each feature's share of the total mean absolute SHAP value (fractions that sum to 1, not percent).	False

Returns:

Type	Description
DataFrame	One row per feature with feature and mean_abs_shap (plus importance_pct when normalize=True), sorted by mean_abs_shap in descending order.

Raises:

Type	Description
RuntimeError	If X is None and no SHAP values have been computed yet.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def feature_importance(self, X=None, normalize=False):
    """Global feature importance as mean absolute SHAP value.

    Parameters
    ----------
    X : pandas.DataFrame, array-like, or None, default None
        Rows to explain. When given, the SHAP values are recomputed with `explain`
        and replace the cached ones; when ``None``, the values cached by the last
        SHAP computation are used.
    normalize : bool, default False
        If ``True``, add an ``importance_pct`` column with each feature's share of
        the total mean absolute SHAP value (fractions that sum to 1, not percent).

    Returns
    -------
    pandas.DataFrame
        One row per feature with ``feature`` and ``mean_abs_shap`` (plus
        ``importance_pct`` when ``normalize=True``), sorted by ``mean_abs_shap`` in
        descending order.

    Raises
    ------
    RuntimeError
        If ``X`` is ``None`` and no SHAP values have been computed yet.
    """
    values, X_used = self._ensure_values(X)
    mean_abs = np.abs(values).mean(axis=0)
    names = self._resolved_names(X_used, values.shape[1])
    table = pd.DataFrame({"feature": names, "mean_abs_shap": mean_abs})
    table = table.sort_values("mean_abs_shap", ascending=False).reset_index(drop=True)
    if normalize:
        total = table["mean_abs_shap"].sum()
        table["importance_pct"] = table["mean_abs_shap"] / total if total else 0.0
    return table

summary_plot

summary_plot(X=None, max_display=20, plot_type='dot', show=True, save_path=None, random_state=0)

SHAP summary (beeswarm / bar) plot.

random_state seeds the point jitter through shap's rng argument (shap >= 0.47), so the plot no longer depends on NumPy's global RNG; older shap versions ignore it.

Parameters:

Name	Type	Description	Default
X	pandas.DataFrame, array-like, or None	Rows to plot. When given, the SHAP values are recomputed with explain and replace the cached ones; when None, the cached values are used.	None
max_display	int	Maximum number of features shown (the most important ones).	20
plot_type	str	Plot kind passed to shap.summary_plot, for example "dot" (beeswarm) or "bar" (mean absolute SHAP value).	"dot"
show	bool	If True, display the figure with plt.show(). If False, the figure is closed after it has been saved, so nothing is displayed.	True
save_path	str, path-like, or None	When given, the figure is saved to this path (dpi=150, tight bounding box; the format follows the file extension). The folder must already exist.	None
random_state	int	Seed of the random number generator that jitters the points.	0

Returns:

Type	Description
Figure	The current matplotlib figure.

Raises:

Type	Description
RuntimeError	If X is None and no SHAP values have been computed yet.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def summary_plot(self, X=None, max_display=20, plot_type="dot", show=True, save_path=None, random_state=0):
    """SHAP summary (beeswarm / bar) plot.

    ``random_state`` seeds the point jitter through shap's ``rng`` argument
    (shap >= 0.47), so the plot no longer depends on NumPy's global RNG;
    older shap versions ignore it.

    Parameters
    ----------
    X : pandas.DataFrame, array-like, or None, default None
        Rows to plot. When given, the SHAP values are recomputed with `explain` and
        replace the cached ones; when ``None``, the cached values are used.
    max_display : int, default 20
        Maximum number of features shown (the most important ones).
    plot_type : str, default "dot"
        Plot kind passed to ``shap.summary_plot``, for example ``"dot"`` (beeswarm)
        or ``"bar"`` (mean absolute SHAP value).
    show : bool, default True
        If ``True``, display the figure with ``plt.show()``. If ``False``, the figure
        is closed after it has been saved, so nothing is displayed.
    save_path : str, path-like, or None, default None
        When given, the figure is saved to this path (``dpi=150``, tight bounding
        box; the format follows the file extension). The folder must already exist.
    random_state : int, default 0
        Seed of the random number generator that jitters the points.

    Returns
    -------
    matplotlib.figure.Figure
        The current matplotlib figure.

    Raises
    ------
    RuntimeError
        If ``X`` is ``None`` and no SHAP values have been computed yet.
    """
    shap = _lazy_shap()
    import matplotlib.pyplot as plt

    values, X_used = self._ensure_values(X)
    kwargs = {"max_display": max_display, "plot_type": plot_type, "show": False}
    if "rng" in inspect.signature(shap.summary_plot).parameters:
        kwargs["rng"] = np.random.default_rng(random_state)
    if not isinstance(X_used, pd.DataFrame) and self.feature_names is not None:
        kwargs["feature_names"] = self.feature_names
    shap.summary_plot(values, X_used, **kwargs)
    return self._finalize_plot(plt, show, save_path)

dependence_plot

dependence_plot(feature, X=None, interaction_index='auto', show=True, save_path=None)

SHAP dependence plot for a single feature.

Parameters:

Name	Type	Description	Default
feature	str or int	Feature to plot, as a column name or a column position (passed to shap.dependence_plot).	required
X	pandas.DataFrame, array-like, or None	Rows to plot. When given, the SHAP values are recomputed with explain and replace the cached ones; when None, the cached values are used.	None
interaction_index	str, int, or None	Feature used to color the points. "auto" lets SHAP choose the feature with the strongest apparent interaction; a name or a position selects one feature; None turns the coloring off.	"auto"
show	bool	If True, display the figure with plt.show(). If False, the figure is closed after it has been saved, so nothing is displayed.	True
save_path	str, path-like, or None	When given, the figure is saved to this path (dpi=150, tight bounding box; the format follows the file extension). The folder must already exist.	None

Returns:

Type	Description
Figure	The current matplotlib figure.

Raises:

Type	Description
RuntimeError	If X is None and no SHAP values have been computed yet.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def dependence_plot(self, feature, X=None, interaction_index="auto", show=True, save_path=None):
    """SHAP dependence plot for a single feature.

    Parameters
    ----------
    feature : str or int
        Feature to plot, as a column name or a column position (passed to
        ``shap.dependence_plot``).
    X : pandas.DataFrame, array-like, or None, default None
        Rows to plot. When given, the SHAP values are recomputed with `explain` and
        replace the cached ones; when ``None``, the cached values are used.
    interaction_index : str, int, or None, default "auto"
        Feature used to color the points. ``"auto"`` lets SHAP choose the feature
        with the strongest apparent interaction; a name or a position selects one
        feature; ``None`` turns the coloring off.
    show : bool, default True
        If ``True``, display the figure with ``plt.show()``. If ``False``, the figure
        is closed after it has been saved, so nothing is displayed.
    save_path : str, path-like, or None, default None
        When given, the figure is saved to this path (``dpi=150``, tight bounding
        box; the format follows the file extension). The folder must already exist.

    Returns
    -------
    matplotlib.figure.Figure
        The current matplotlib figure.

    Raises
    ------
    RuntimeError
        If ``X`` is ``None`` and no SHAP values have been computed yet.
    """
    shap = _lazy_shap()
    import matplotlib.pyplot as plt

    values, X_used = self._ensure_values(X)
    kwargs = {"interaction_index": interaction_index, "show": False}
    if not isinstance(X_used, pd.DataFrame) and self.feature_names is not None:
        kwargs["feature_names"] = self.feature_names
    shap.dependence_plot(feature, values, X_used, **kwargs)
    return self._finalize_plot(plt, show, save_path)

explain_instance

explain_instance(x_row)

Per-feature SHAP contributions for a single sample.

Parameters:

Name	Type	Description	Default
x_row	pandas.DataFrame, pandas.Series, dict, or array-like	One observation. A Series or a dict is turned into a one-row frame and a 1-D array is read as one row; when several rows are given, only the first is explained.	required

Returns:

Type	Description
DataFrame	One row per feature with feature, value (the feature value in x_row) and shap_value, sorted by absolute SHAP value (largest first). attrs["base_value"] holds the explainer's base value as a float (NaN when it is unavailable).

Raises:

Type	Description
ValueError	If the SHAP explainer has to be built and cannot be (see explain).
ImportError	If the optional shap package is not installed.

Notes

The result is not cached: shap_values_, expected_value_ and explanation_ keep the values of the last explain call.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def explain_instance(self, x_row):
    """Per-feature SHAP contributions for a single sample.

    Parameters
    ----------
    x_row : pandas.DataFrame, pandas.Series, dict, or array-like
        One observation. A Series or a dict is turned into a one-row frame and a 1-D
        array is read as one row; when several rows are given, only the first is
        explained.

    Returns
    -------
    pandas.DataFrame
        One row per feature with ``feature``, ``value`` (the feature value in
        ``x_row``) and ``shap_value``, sorted by absolute SHAP value (largest
        first). ``attrs["base_value"]`` holds the explainer's base value as a float
        (NaN when it is unavailable).

    Raises
    ------
    ValueError
        If the SHAP explainer has to be built and cannot be (see `explain`).
    ImportError
        If the optional ``shap`` package is not installed.

    Notes
    -----
    The result is not cached: ``shap_values_``, ``expected_value_`` and
    ``explanation_`` keep the values of the last `explain` call.
    """
    if isinstance(x_row, pd.Series):
        x_row = x_row.to_frame().T
    elif isinstance(x_row, dict):
        x_row = pd.DataFrame([x_row])
    frame = self._as_frame(x_row)
    if frame.shape[0] != 1:
        frame = frame.iloc[[0]]

    values, base, _ = self._shap_values_for(frame)
    names = self._resolved_names(frame, values.shape[1])
    table = pd.DataFrame(
        {"feature": names, "value": np.ravel(np.asarray(frame.values)), "shap_value": np.ravel(values[0])}
    )
    order = table["shap_value"].abs().sort_values(ascending=False).index
    table = table.loc[order].reset_index(drop=True)
    table.attrs["base_value"] = float(np.ravel(base)[0]) if base is not None and base.size else float("nan")
    return table

build_coalition_structure

build_coalition_structure(X=None, prior_groups=None, threshold=0.35, method='complete', corr_method='spearman', min_group_size=1, intra_dist=0.01, inter_dist=0.99)

Build and cache a coalition structure for Owen Value explanations.

Parameters:

Name	Type	Description	Default
X	pandas.DataFrame, array-like, or None	Data the correlation clustering is computed on. When None, the explainer's background_data is used.	None
prior_groups	dict or None	Business groups as {group_name: [feature, ...]}. They take precedence over the automatic clusters; feature names that are not columns of the data are ignored, and a feature may not appear in two groups.	None
threshold	float	Cut height of the hierarchical clustering on the distance 1 - abs(association); must be between 0 and 1.	0.35
method	str	Linkage method: "complete", "average" or "single".	"complete"
corr_method	str	Association measure: "spearman", "pearson", "kendall" or "MIC" (case-insensitive; needs the optional minepy package).	"spearman"
min_group_size	int	Automatic clusters with fewer features than this are merged into one group named "auto_singleton".	1
intra_dist	float	Distance between two features of the same group in the partition tree passed to SHAP.	0.01
inter_dist	float	Distance between two features of different groups in that partition tree.	0.99

Returns:

Type	Description
dict	The coalition structure, as returned by the module-level build_coalition_structure (keys groups, shap_lnk, corr_lnk, auto_groups, summary, features, threshold, method and corr_method). It is also cached as coalition_structure_.

Raises:

Type	Description
ValueError	If both X and background_data are None, or if the clustering arguments are invalid (see the module-level function).

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def build_coalition_structure(
    self,
    X=None,
    prior_groups=None,
    threshold=0.35,
    method="complete",
    corr_method="spearman",
    min_group_size=1,
    intra_dist=0.01,
    inter_dist=0.99,
):
    """Build and cache a coalition structure for Owen Value explanations.

    Parameters
    ----------
    X : pandas.DataFrame, array-like, or None, default None
        Data the correlation clustering is computed on. When ``None``, the
        explainer's ``background_data`` is used.
    prior_groups : dict or None, default None
        Business groups as ``{group_name: [feature, ...]}``. They take precedence
        over the automatic clusters; feature names that are not columns of the data
        are ignored, and a feature may not appear in two groups.
    threshold : float, default 0.35
        Cut height of the hierarchical clustering on the distance
        ``1 - abs(association)``; must be between 0 and 1.
    method : str, default "complete"
        Linkage method: ``"complete"``, ``"average"`` or ``"single"``.
    corr_method : str, default "spearman"
        Association measure: ``"spearman"``, ``"pearson"``, ``"kendall"`` or
        ``"MIC"`` (case-insensitive; needs the optional ``minepy`` package).
    min_group_size : int, default 1
        Automatic clusters with fewer features than this are merged into one group
        named ``"auto_singleton"``.
    intra_dist : float, default 0.01
        Distance between two features of the same group in the partition tree passed
        to SHAP.
    inter_dist : float, default 0.99
        Distance between two features of different groups in that partition tree.

    Returns
    -------
    dict
        The coalition structure, as returned by the module-level
        `build_coalition_structure` (keys ``groups``, ``shap_lnk``, ``corr_lnk``,
        ``auto_groups``, ``summary``, ``features``, ``threshold``, ``method`` and
        ``corr_method``). It is also cached as ``coalition_structure_``.

    Raises
    ------
    ValueError
        If both ``X`` and ``background_data`` are ``None``, or if the clustering
        arguments are invalid (see the module-level function).
    """
    data = X if X is not None else self.background_data
    if data is None:
        raise ValueError("build_coalition_structure requires X or background_data")
    frame = self._as_frame(data)
    cs = _build_coalition_structure(
        frame,
        prior_groups=prior_groups,
        threshold=threshold,
        method=method,
        corr_method=corr_method,
        min_group_size=min_group_size,
        intra_dist=intra_dist,
        inter_dist=inter_dist,
    )
    self.coalition_structure_ = cs
    return cs

explain_owen

explain_owen(X, coalition_structure=None, prior_groups=None, threshold=0.35, method='complete', corr_method='spearman', background_data=None, model_output='probability', rebuild=False, **explain_kwargs)

Compute Owen Value attribution with SHAP PartitionExplainer.

model_output='probability' explains positive-class probability. model_output='log_odds' is useful for credit reason-code reporting.

Parameters:

Name	Type	Description	Default
X	DataFrame or array - like	Rows to explain. It must contain every feature of the coalition structure; its columns are selected in the structure's feature order.	required
coalition_structure	dict or None	Structure returned by build_coalition_structure. When None, the cached coalition_structure_ is reused; when there is none, one is built from background_data (the argument, else the constructor's, else X).	None
prior_groups	dict or None	Business groups as {group_name: [feature, ...]}. When given, a new structure is always built from them (using threshold, method and corr_method) and it replaces coalition_structure, even an explicit one.	None
threshold	float	Cut height of the clustering on 1 - abs(association); used only when the structure is built here.	0.35
method	str	Linkage method ("complete", "average" or "single"); used only when the structure is built here.	"complete"
corr_method	str	Association measure ("spearman", "pearson", "kendall" or "MIC"); used only when the structure is built here.	"spearman"
background_data	pandas.DataFrame, array-like, or None	Background sample for building the structure (when it is built here) and for the SHAP PartitionExplainer. Defaults to the constructor's background_data. A background whose values differ from those the cached PartitionExplainer was built with makes it rebuild.	None
model_output	str	Quantity to explain: "probability" (positive-class probability) or "log_odds" ("logit" is accepted as an alias). Any other value raises ValueError.	"probability"
rebuild	bool	If True, build a new PartitionExplainer instead of reusing the cached one. It is also rebuilt automatically when the coalition structure (its features and linkage), the background values or model_output differ from those the cached explainer was built with.	False
**explain_kwargs		Extra keyword arguments passed to the PartitionExplainer call, for example max_evals or silent.	{}

Returns:

Type	Description
Explanation	The Owen value explanation of X (restricted to the features of the coalition structure).

Raises:

Type	Description
ValueError	If no background_data is available (argument or constructor) to build the PartitionExplainer, or if model_output is not supported.
ImportError	If the optional shap package is not installed.

Notes

Sets coalition_structure_, owen_values_, owen_expected_value_ and owen_explanation_, and remembers X for the Owen importance methods.

The PartitionExplainer is cached and reused while the coalition structure (its features and linkage), the background values and model_output stay the same, for example when only X changes or when the same prior_groups are passed again. A change of any of them, or rebuild=True, builds a new one. A later call without any background keeps the explainer that was built.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def explain_owen(
    self,
    X,
    coalition_structure=None,
    prior_groups=None,
    threshold=0.35,
    method="complete",
    corr_method="spearman",
    background_data=None,
    model_output="probability",
    rebuild=False,
    **explain_kwargs,
):
    """Compute Owen Value attribution with SHAP PartitionExplainer.

    ``model_output='probability'`` explains positive-class probability.
    ``model_output='log_odds'`` is useful for credit reason-code reporting.

    Parameters
    ----------
    X : pandas.DataFrame or array-like
        Rows to explain. It must contain every feature of the coalition structure;
        its columns are selected in the structure's feature order.
    coalition_structure : dict or None, default None
        Structure returned by `build_coalition_structure`. When ``None``, the cached
        ``coalition_structure_`` is reused; when there is none, one is built from
        ``background_data`` (the argument, else the constructor's, else ``X``).
    prior_groups : dict or None, default None
        Business groups as ``{group_name: [feature, ...]}``. When given, a new
        structure is always built from them (using ``threshold``, ``method`` and
        ``corr_method``) and it replaces ``coalition_structure``, even an explicit
        one.
    threshold : float, default 0.35
        Cut height of the clustering on ``1 - abs(association)``; used only when the
        structure is built here.
    method : str, default "complete"
        Linkage method (``"complete"``, ``"average"`` or ``"single"``); used only
        when the structure is built here.
    corr_method : str, default "spearman"
        Association measure (``"spearman"``, ``"pearson"``, ``"kendall"`` or
        ``"MIC"``); used only when the structure is built here.
    background_data : pandas.DataFrame, array-like, or None, default None
        Background sample for building the structure (when it is built here) and
        for the SHAP ``PartitionExplainer``. Defaults to the constructor's
        ``background_data``. A background whose values differ from those the cached
        ``PartitionExplainer`` was built with makes it rebuild.
    model_output : str, default "probability"
        Quantity to explain: ``"probability"`` (positive-class probability) or
        ``"log_odds"`` (``"logit"`` is accepted as an alias). Any other value raises
        ``ValueError``.
    rebuild : bool, default False
        If ``True``, build a new ``PartitionExplainer`` instead of reusing the cached
        one. It is also rebuilt automatically when the coalition structure (its
        features and linkage), the background values or ``model_output`` differ from
        those the cached explainer was built with.
    **explain_kwargs
        Extra keyword arguments passed to the ``PartitionExplainer`` call, for example
        ``max_evals`` or ``silent``.

    Returns
    -------
    shap.Explanation
        The Owen value explanation of ``X`` (restricted to the features of the
        coalition structure).

    Raises
    ------
    ValueError
        If no ``background_data`` is available (argument or constructor) to build
        the ``PartitionExplainer``, or if ``model_output`` is not supported.
    ImportError
        If the optional ``shap`` package is not installed.

    Notes
    -----
    Sets ``coalition_structure_``, ``owen_values_``, ``owen_expected_value_`` and
    ``owen_explanation_``, and remembers ``X`` for the Owen importance methods.

    The ``PartitionExplainer`` is cached and reused while the coalition structure
    (its features and linkage), the background values and ``model_output`` stay the
    same, for example when only ``X`` changes or when the same ``prior_groups`` are
    passed again. A change of any of them, or ``rebuild=True``, builds a new one. A
    later call without any background keeps the explainer that was built.
    """
    frame = self._as_frame(X)
    if coalition_structure is None:
        coalition_structure = self.coalition_structure_
    if coalition_structure is None or prior_groups is not None:
        base = background_data if background_data is not None else self.background_data
        if base is None:
            base = frame
        coalition_structure = self.build_coalition_structure(
            base,
            prior_groups=prior_groups,
            threshold=threshold,
            method=method,
            corr_method=corr_method,
        )
    features = coalition_structure.get("features")
    if features is not None:
        frame = frame.loc[:, list(features)]

    if rebuild or self._owen_cache_is_stale(coalition_structure, background_data, model_output):
        self._build_owen_explainer(coalition_structure, background_data=background_data, model_output=model_output)

    explanation = self._owen_explainer(frame, **explain_kwargs)
    values, base = self._normalize_values(explanation.values, explanation.base_values)
    self.coalition_structure_ = coalition_structure
    self.owen_values_ = values
    self.owen_expected_value_ = base
    self.owen_explanation_ = explanation
    self._owen_last_X = frame
    return explanation

owen_feature_importance

owen_feature_importance(X=None, normalize=False)

Global feature importance as mean absolute Owen value.

Parameters:

Name	Type	Description	Default
X	pandas.DataFrame, array-like, or None	Rows to explain. When given, the Owen values are recomputed with explain_owen(X) using its defaults (the cached coalition structure and model_output="probability") and replace the cached ones; when None, the values cached by the last explain_owen call are used.	None
normalize	bool	If True, add an importance_pct column with each feature's share of the total mean absolute Owen value (fractions that sum to 1, not percent).	False

Returns:

Type	Description
DataFrame	One row per feature with feature and mean_abs_owen (plus importance_pct when normalize=True), sorted by mean_abs_owen in descending order.

Raises:

Type	Description
RuntimeError	If no Owen values are cached and X is None.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def owen_feature_importance(self, X=None, normalize=False):
    """Global feature importance as mean absolute Owen value.

    Parameters
    ----------
    X : pandas.DataFrame, array-like, or None, default None
        Rows to explain. When given, the Owen values are recomputed with
        ``explain_owen(X)`` using its defaults (the cached coalition structure and
        ``model_output="probability"``) and replace the cached ones; when ``None``,
        the values cached by the last `explain_owen` call are used.
    normalize : bool, default False
        If ``True``, add an ``importance_pct`` column with each feature's share of
        the total mean absolute Owen value (fractions that sum to 1, not percent).

    Returns
    -------
    pandas.DataFrame
        One row per feature with ``feature`` and ``mean_abs_owen`` (plus
        ``importance_pct`` when ``normalize=True``), sorted by ``mean_abs_owen`` in
        descending order.

    Raises
    ------
    RuntimeError
        If no Owen values are cached and ``X`` is ``None``.
    """
    values, X_used, _ = self._ensure_owen_values(X)
    names = self._resolved_names(X_used, values.shape[1])
    table = pd.DataFrame({"feature": names, "mean_abs_owen": np.abs(values).mean(axis=0)})
    table = table.sort_values("mean_abs_owen", ascending=False).reset_index(drop=True)
    if normalize:
        total = table["mean_abs_owen"].sum()
        table["importance_pct"] = table["mean_abs_owen"] / total if total else 0.0
    return table

owen_group_importance

owen_group_importance(X=None, normalize=False)

Aggregate Owen values to coalition groups.

Parameters:

Name	Type	Description	Default
X	pandas.DataFrame, array-like, or None	Rows to explain. When given, the Owen values are recomputed with explain_owen(X) using its defaults (the cached coalition structure and model_output="probability") and replace the cached ones; when None, the values cached by the last explain_owen call are used.	None
normalize	bool	If True, add an importance_pct column with each group's share of the total mean_abs_owen (fractions that sum to 1, not percent).	False

Returns:

Type	Description
DataFrame	One row per coalition group with group, n_features, features (the member features found in the data), mean_owen and mean_abs_owen (plus importance_pct when normalize=True), sorted by mean_abs_owen in descending order. A group's value for a row is the sum of its members' Owen values; mean_owen averages that signed sum and mean_abs_owen averages its absolute value.

Raises:

Type	Description
RuntimeError	If no Owen values are cached and X is None.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def owen_group_importance(self, X=None, normalize=False):
    """Aggregate Owen values to coalition groups.

    Parameters
    ----------
    X : pandas.DataFrame, array-like, or None, default None
        Rows to explain. When given, the Owen values are recomputed with
        ``explain_owen(X)`` using its defaults (the cached coalition structure and
        ``model_output="probability"``) and replace the cached ones; when ``None``,
        the values cached by the last `explain_owen` call are used.
    normalize : bool, default False
        If ``True``, add an ``importance_pct`` column with each group's share of the
        total ``mean_abs_owen`` (fractions that sum to 1, not percent).

    Returns
    -------
    pandas.DataFrame
        One row per coalition group with ``group``, ``n_features``, ``features``
        (the member features found in the data), ``mean_owen`` and ``mean_abs_owen``
        (plus ``importance_pct`` when ``normalize=True``), sorted by
        ``mean_abs_owen`` in descending order. A group's value for a row is the sum
        of its members' Owen values; ``mean_owen`` averages that signed sum and
        ``mean_abs_owen`` averages its absolute value.

    Raises
    ------
    RuntimeError
        If no Owen values are cached and ``X`` is ``None``.
    """
    values, X_used, cs = self._ensure_owen_values(X)
    names = list(X_used.columns)
    rows = []
    for group, feats in cs["groups"].items():
        idxs = [names.index(feat) for feat in feats if feat in names]
        if not idxs:
            continue
        contrib = values[:, idxs].sum(axis=1)
        rows.append(
            {
                "group": group,
                "n_features": len(idxs),
                "features": [names[i] for i in idxs],
                "mean_owen": float(np.mean(contrib)),
                "mean_abs_owen": float(np.mean(np.abs(contrib))),
            }
        )
    table = pd.DataFrame(rows).sort_values("mean_abs_owen", ascending=False).reset_index(drop=True)
    if normalize and not table.empty:
        total = table["mean_abs_owen"].sum()
        table["importance_pct"] = table["mean_abs_owen"] / total if total else 0.0
    return table

owen_explain_instance

owen_explain_instance(x_row=None, aggregate_groups=True)

Return local Owen reason codes for one sample.

Parameters:

Name	Type	Description	Default
x_row	pandas.DataFrame, array-like, or None	One observation. When None, the first row of the cached Owen batch is used. When given, only its first row is used and the Owen values are recomputed for it with explain_owen defaults (cached coalition structure, model_output="probability"), which replaces the cached results. A Series or 1-D array is read by position, so its values must follow feature_names; a dict is not supported.	None
aggregate_groups	bool	If True, return one row per coalition group (the sum of its members' Owen values). If False, return one row per feature.	True

Returns:

Type	Description
DataFrame	With aggregate_groups=True: group, n_features, features, owen_value and abs_owen_value, sorted by abs_owen_value (largest first), with attrs["base_value"] (float, NaN when unavailable) and attrs["model_output"]. With aggregate_groups=False: feature, value, owen_value and abs_owen_value sorted the same way, without those attrs.

Raises:

Type	Description
RuntimeError	If x_row is None and no Owen values are cached.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def owen_explain_instance(self, x_row=None, aggregate_groups=True):
    """Return local Owen reason codes for one sample.

    Parameters
    ----------
    x_row : pandas.DataFrame, array-like, or None, default None
        One observation. When ``None``, the first row of the cached Owen batch is
        used. When given, only its first row is used and the Owen values are
        recomputed for it with ``explain_owen`` defaults (cached coalition structure,
        ``model_output="probability"``), which replaces the cached results. A Series
        or 1-D array is read by position, so its values must follow
        ``feature_names``; a dict is not supported.
    aggregate_groups : bool, default True
        If ``True``, return one row per coalition group (the sum of its members'
        Owen values). If ``False``, return one row per feature.

    Returns
    -------
    pandas.DataFrame
        With ``aggregate_groups=True``: ``group``, ``n_features``, ``features``,
        ``owen_value`` and ``abs_owen_value``, sorted by ``abs_owen_value`` (largest
        first), with ``attrs["base_value"]`` (float, NaN when unavailable) and
        ``attrs["model_output"]``. With ``aggregate_groups=False``: ``feature``,
        ``value``, ``owen_value`` and ``abs_owen_value`` sorted the same way, without
        those ``attrs``.

    Raises
    ------
    RuntimeError
        If ``x_row`` is ``None`` and no Owen values are cached.
    """
    if x_row is not None:
        self.explain_owen(self._as_frame(x_row).iloc[[0]])
    values, X_used, cs = self._ensure_owen_values(None)
    row_values = values[0]
    row_x = X_used.iloc[0]
    if not aggregate_groups:
        table = pd.DataFrame({"feature": list(X_used.columns), "value": row_x.values, "owen_value": row_values})
        table["abs_owen_value"] = table["owen_value"].abs()
        return table.sort_values("abs_owen_value", ascending=False).reset_index(drop=True)

    rows = []
    names = list(X_used.columns)
    for group, feats in cs["groups"].items():
        idxs = [names.index(feat) for feat in feats if feat in names]
        if not idxs:
            continue
        contrib = float(row_values[idxs].sum())
        rows.append(
            {
                "group": group,
                "n_features": len(idxs),
                "features": [names[i] for i in idxs],
                "owen_value": contrib,
                "abs_owen_value": abs(contrib),
            }
        )
    table = pd.DataFrame(rows).sort_values("abs_owen_value", ascending=False).reset_index(drop=True)
    base = self.owen_expected_value_
    table.attrs["base_value"] = float(np.ravel(base)[0]) if base is not None and np.asarray(base).size else float("nan")
    table.attrs["model_output"] = self._owen_model_output
    return table

partial_dependence

partial_dependence(X, feature, grid_resolution=50, percentiles=(0.05, 0.95), sample_size=None, random_state=None, prediction_batch_size=100000)

Compute one-way partial dependence for a numeric feature.

Parameters:

Name	Type	Description	Default
X	DataFrame or array - like	Data to average over. A DataFrame may carry extra columns: the model's features are selected by name (see feature_names).	required
feature	str or int	Numeric feature to vary: a column name, or a Python int position among the model's feature columns. A name that is not one of the model's columns raises KeyError.	required
grid_resolution	int	Number of grid points between the lower and upper percentile of the feature.	50
percentiles	tuple of float	Lower and upper quantile of the feature, as fractions between 0 and 1, that bound the grid. When they coincide (for example for a constant feature), the grid is the sorted distinct values of the feature, at most grid_resolution of them.	(0.05, 0.95)
sample_size	int or None	When X has more rows than this, a random sample of this size is used; None uses every row.	None
random_state	int or None	Seed of the row sampling (only used when sampling happens); None draws a different sample on each call.	None
prediction_batch_size	int or None	Maximum number of rows sent to the model in one prediction call, which bounds peak memory; None predicts all grid points in a single call. A value below 1 raises ValueError.	100000

Returns:

Type	Description
DataFrame	One row per grid point with feature (the column name), grid_value and average_prediction (the mean predicted positive-class probability when the feature is set to the grid value for every row).

Raises:

Type	Description
KeyError	If feature is not one of the model's feature columns of X.
ValueError	If the feature has no non-missing numeric value, or if prediction_batch_size is not a positive integer or None.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def partial_dependence(self, X, feature, grid_resolution=50, percentiles=(0.05, 0.95), sample_size=None, random_state=None, prediction_batch_size=100000):
    """Compute one-way partial dependence for a numeric feature.

    Parameters
    ----------
    X : pandas.DataFrame or array-like
        Data to average over. A DataFrame may carry extra columns: the model's
        features are selected by name (see ``feature_names``).
    feature : str or int
        Numeric feature to vary: a column name, or a Python ``int`` position among
        the model's feature columns. A name that is not one of the model's columns
        raises ``KeyError``.
    grid_resolution : int, default 50
        Number of grid points between the lower and upper percentile of the feature.
    percentiles : tuple of float, default (0.05, 0.95)
        Lower and upper quantile of the feature, as fractions between 0 and 1, that
        bound the grid. When they coincide (for example for a constant feature), the
        grid is the sorted distinct values of the feature, at most
        ``grid_resolution`` of them.
    sample_size : int or None, default None
        When ``X`` has more rows than this, a random sample of this size is used;
        ``None`` uses every row.
    random_state : int or None, default None
        Seed of the row sampling (only used when sampling happens); ``None`` draws a
        different sample on each call.
    prediction_batch_size : int or None, default 100000
        Maximum number of rows sent to the model in one prediction call, which bounds
        peak memory; ``None`` predicts all grid points in a single call. A value
        below 1 raises ``ValueError``.

    Returns
    -------
    pandas.DataFrame
        One row per grid point with ``feature`` (the column name), ``grid_value`` and
        ``average_prediction`` (the mean predicted positive-class probability when
        the feature is set to the grid value for every row).

    Raises
    ------
    KeyError
        If ``feature`` is not one of the model's feature columns of ``X``.
    ValueError
        If the feature has no non-missing numeric value, or if
        ``prediction_batch_size`` is not a positive integer or ``None``.
    """
    frame = self._sample_frame(X, sample_size=sample_size, random_state=random_state)
    feature = self._feature_name(frame, feature)
    grid = self._numeric_grid(frame[feature], grid_resolution=grid_resolution, percentiles=percentiles)
    if len(frame) == 0:
        raise ValueError("partial_dependence requires at least one input row")
    batch_size = (
        len(frame) * max(len(grid), 1)
        if prediction_batch_size is None
        else int(prediction_batch_size)
    )
    if batch_size <= 0:
        raise ValueError("prediction_batch_size must be a positive integer or None")
    grids_per_batch = max(1, batch_size // len(frame))
    averages = []
    for start in range(0, len(grid), grids_per_batch):
        grid_block = grid[start : start + grids_per_batch]
        stacked = pd.concat(
            [frame.assign(**{feature: value}) for value in grid_block],
            ignore_index=True,
        )
        predictions = self._predict_proba_pos_batched(
            stacked,
            prediction_batch_size=prediction_batch_size,
        ).reshape(len(grid_block), len(frame))
        averages.extend(predictions.mean(axis=1).astype(float).tolist())
    return pd.DataFrame({"feature": feature, "grid_value": grid, "average_prediction": averages})

pdp_plot

pdp_plot(X, feature, grid_resolution=50, percentiles=(0.05, 0.95), sample_size=None, random_state=None, show=True, save_path=None, prediction_batch_size=100000)

Plot one-way partial dependence for a numeric feature.

Parameters:

Name	Type	Description	Default
X	DataFrame or array - like	Data to average over; same handling as in partial_dependence.	required
feature	str or int	Numeric feature to vary: a column name, or a Python int position among the model's feature columns.	required
grid_resolution	int	Number of grid points between the lower and upper percentile of the feature.	50
percentiles	tuple of float	Lower and upper quantile of the feature, as fractions between 0 and 1, that bound the grid.	(0.05, 0.95)
sample_size	int or None	When X has more rows than this, a random sample of this size is used; None uses every row.	None
random_state	int or None	Seed of the row sampling (only used when sampling happens).	None
show	bool	If True, display the figure with plt.show(). If False, the figure is closed after it has been saved, so nothing is displayed.	True
save_path	str, path-like, or None	When given, the figure is saved to this path (dpi=150, tight bounding box; the format follows the file extension). The folder must already exist.	None
prediction_batch_size	int or None	Maximum number of rows sent to the model in one prediction call; None predicts all grid points in a single call.	100000

Returns:

Type	Description
Figure	The figure with the average-prediction curve over the grid.

Raises:

Type	Description
KeyError	If feature is not one of the model's feature columns of X.
ValueError	If the feature has no non-missing numeric value, or if prediction_batch_size is not a positive integer or None.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def pdp_plot(self, X, feature, grid_resolution=50, percentiles=(0.05, 0.95), sample_size=None, random_state=None, show=True, save_path=None, prediction_batch_size=100000):
    """Plot one-way partial dependence for a numeric feature.

    Parameters
    ----------
    X : pandas.DataFrame or array-like
        Data to average over; same handling as in `partial_dependence`.
    feature : str or int
        Numeric feature to vary: a column name, or a Python ``int`` position among
        the model's feature columns.
    grid_resolution : int, default 50
        Number of grid points between the lower and upper percentile of the feature.
    percentiles : tuple of float, default (0.05, 0.95)
        Lower and upper quantile of the feature, as fractions between 0 and 1, that
        bound the grid.
    sample_size : int or None, default None
        When ``X`` has more rows than this, a random sample of this size is used;
        ``None`` uses every row.
    random_state : int or None, default None
        Seed of the row sampling (only used when sampling happens).
    show : bool, default True
        If ``True``, display the figure with ``plt.show()``. If ``False``, the figure
        is closed after it has been saved, so nothing is displayed.
    save_path : str, path-like, or None, default None
        When given, the figure is saved to this path (``dpi=150``, tight bounding
        box; the format follows the file extension). The folder must already exist.
    prediction_batch_size : int or None, default 100000
        Maximum number of rows sent to the model in one prediction call; ``None``
        predicts all grid points in a single call.

    Returns
    -------
    matplotlib.figure.Figure
        The figure with the average-prediction curve over the grid.

    Raises
    ------
    KeyError
        If ``feature`` is not one of the model's feature columns of ``X``.
    ValueError
        If the feature has no non-missing numeric value, or if
        ``prediction_batch_size`` is not a positive integer or ``None``.
    """
    import matplotlib.pyplot as plt

    df = self.partial_dependence(
        X,
        feature,
        grid_resolution,
        percentiles,
        sample_size,
        random_state,
        prediction_batch_size,
    )
    fig, ax = plt.subplots(figsize=(7, 4), dpi=120)
    ax.plot(df["grid_value"], df["average_prediction"], color="#336699", linewidth=2)
    ax.set_xlabel(str(df["feature"].iloc[0]))
    ax.set_ylabel("Average prediction")
    ax.set_title(f"PDP: {df['feature'].iloc[0]}")
    ax.grid(alpha=0.25)
    return self._finalize_plot(plt, show, save_path)

ice

ice(X, feature, grid_resolution=50, percentiles=(0.05, 0.95), sample_size=200, random_state=None, centered=False, prediction_batch_size=100000)

Compute individual conditional expectation curves.

Parameters:

Name	Type	Description	Default
X	DataFrame or array - like	Data to draw the curves from. A DataFrame may carry extra columns: the model's features are selected by name (see feature_names).	required
feature	str or int	Numeric feature to vary: a column name, or a Python int position among the model's feature columns. A name that is not one of the model's columns raises KeyError.	required
grid_resolution	int	Number of grid points between the lower and upper percentile of the feature.	50
percentiles	tuple of float	Lower and upper quantile of the feature, as fractions between 0 and 1, that bound the grid. When they coincide (for example for a constant feature), the grid is the sorted distinct values of the feature, at most grid_resolution of them.	(0.05, 0.95)
sample_size	int or None	When X has more rows than this, a random sample of this many rows (one curve each) is used; None uses every row.	200
random_state	int or None	Seed of the row sampling (only used when sampling happens); None draws a different sample on each call.	None
centered	bool	If True, subtract from each curve its prediction at the first grid point, so every curve starts at 0.	False
prediction_batch_size	int or None	Maximum number of rows sent to the model in one prediction call; None predicts all stacked rows in a single call. A value below 1 raises ValueError.	100000

Returns:

Type	Description
DataFrame	Long format with one row per grid point and sampled row, ordered by grid value: feature (the column name), sample_index (the row label in X), grid_value and prediction (the predicted positive-class probability, or its change from the first grid point when centered=True).

Raises:

Type	Description
KeyError	If feature is not one of the model's feature columns of X.
ValueError	If the feature has no non-missing numeric value, or if prediction_batch_size is not a positive integer or None.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def ice(self, X, feature, grid_resolution=50, percentiles=(0.05, 0.95), sample_size=200, random_state=None, centered=False, prediction_batch_size=100000):
    """Compute individual conditional expectation curves.

    Parameters
    ----------
    X : pandas.DataFrame or array-like
        Data to draw the curves from. A DataFrame may carry extra columns: the
        model's features are selected by name (see ``feature_names``).
    feature : str or int
        Numeric feature to vary: a column name, or a Python ``int`` position among
        the model's feature columns. A name that is not one of the model's columns
        raises ``KeyError``.
    grid_resolution : int, default 50
        Number of grid points between the lower and upper percentile of the feature.
    percentiles : tuple of float, default (0.05, 0.95)
        Lower and upper quantile of the feature, as fractions between 0 and 1, that
        bound the grid. When they coincide (for example for a constant feature), the
        grid is the sorted distinct values of the feature, at most
        ``grid_resolution`` of them.
    sample_size : int or None, default 200
        When ``X`` has more rows than this, a random sample of this many rows (one
        curve each) is used; ``None`` uses every row.
    random_state : int or None, default None
        Seed of the row sampling (only used when sampling happens); ``None`` draws a
        different sample on each call.
    centered : bool, default False
        If ``True``, subtract from each curve its prediction at the first grid point,
        so every curve starts at 0.
    prediction_batch_size : int or None, default 100000
        Maximum number of rows sent to the model in one prediction call; ``None``
        predicts all stacked rows in a single call. A value below 1 raises
        ``ValueError``.

    Returns
    -------
    pandas.DataFrame
        Long format with one row per grid point and sampled row, ordered by grid
        value: ``feature`` (the column name), ``sample_index`` (the row label in
        ``X``), ``grid_value`` and ``prediction`` (the predicted positive-class
        probability, or its change from the first grid point when ``centered=True``).

    Raises
    ------
    KeyError
        If ``feature`` is not one of the model's feature columns of ``X``.
    ValueError
        If the feature has no non-missing numeric value, or if
        ``prediction_batch_size`` is not a positive integer or ``None``.
    """
    frame = self._sample_frame(X, sample_size=sample_size, random_state=random_state)
    feature = self._feature_name(frame, feature)
    grid = self._numeric_grid(frame[feature], grid_resolution=grid_resolution, percentiles=percentiles)
    stacked = pd.concat(
        [frame.assign(**{feature: value}) for value in grid],
        ignore_index=True,
    )
    predictions = self._predict_proba_pos_batched(
        stacked,
        prediction_batch_size=prediction_batch_size,
    )
    out = pd.DataFrame(
        {
            "feature": feature,
            "sample_index": np.tile(frame.index.to_numpy(), len(grid)),
            "grid_value": np.repeat(grid, len(frame)),
            "prediction": predictions,
        }
    )
    if centered:
        prediction_matrix = predictions.reshape(len(grid), len(frame))
        out["prediction"] = (
            prediction_matrix - prediction_matrix[0:1, :]
        ).reshape(-1)
    return out

ice_plot

ice_plot(X, feature, grid_resolution=50, percentiles=(0.05, 0.95), sample_size=100, random_state=None, centered=False, show=True, save_path=None, prediction_batch_size=100000)

Plot ICE curves for a numeric feature.

Parameters:

Name	Type	Description	Default
X	DataFrame or array - like	Data to draw the curves from; same handling as in ice.	required
feature	str or int	Numeric feature to vary: a column name, or a Python int position among the model's feature columns.	required
grid_resolution	int	Number of grid points between the lower and upper percentile of the feature.	50
percentiles	tuple of float	Lower and upper quantile of the feature, as fractions between 0 and 1, that bound the grid.	(0.05, 0.95)
sample_size	int or None	When X has more rows than this, a random sample of this many rows (one curve each) is drawn; None uses every row. Note that this default is smaller than the one of ice (200).	100
random_state	int or None	Seed of the row sampling (only used when sampling happens).	None
centered	bool	If True, every curve is shifted to start at 0 (the y-axis is then labelled "Centered prediction").	False
show	bool	If True, display the figure with plt.show(). If False, the figure is closed after it has been saved, so nothing is displayed.	True
save_path	str, path-like, or None	When given, the figure is saved to this path (dpi=150, tight bounding box; the format follows the file extension). The folder must already exist.	None
prediction_batch_size	int or None	Maximum number of rows sent to the model in one prediction call; None predicts all stacked rows in a single call.	100000

Returns:

Type	Description
Figure	The figure with one thin curve per sampled row and the average curve in red.

Raises:

Type	Description
KeyError	If feature is not one of the model's feature columns of X.
ValueError	If the feature has no non-missing numeric value, or if prediction_batch_size is not a positive integer or None.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def ice_plot(self, X, feature, grid_resolution=50, percentiles=(0.05, 0.95), sample_size=100, random_state=None, centered=False, show=True, save_path=None, prediction_batch_size=100000):
    """Plot ICE curves for a numeric feature.

    Parameters
    ----------
    X : pandas.DataFrame or array-like
        Data to draw the curves from; same handling as in `ice`.
    feature : str or int
        Numeric feature to vary: a column name, or a Python ``int`` position among
        the model's feature columns.
    grid_resolution : int, default 50
        Number of grid points between the lower and upper percentile of the feature.
    percentiles : tuple of float, default (0.05, 0.95)
        Lower and upper quantile of the feature, as fractions between 0 and 1, that
        bound the grid.
    sample_size : int or None, default 100
        When ``X`` has more rows than this, a random sample of this many rows (one
        curve each) is drawn; ``None`` uses every row. Note that this default is
        smaller than the one of `ice` (200).
    random_state : int or None, default None
        Seed of the row sampling (only used when sampling happens).
    centered : bool, default False
        If ``True``, every curve is shifted to start at 0 (the y-axis is then
        labelled "Centered prediction").
    show : bool, default True
        If ``True``, display the figure with ``plt.show()``. If ``False``, the figure
        is closed after it has been saved, so nothing is displayed.
    save_path : str, path-like, or None, default None
        When given, the figure is saved to this path (``dpi=150``, tight bounding
        box; the format follows the file extension). The folder must already exist.
    prediction_batch_size : int or None, default 100000
        Maximum number of rows sent to the model in one prediction call; ``None``
        predicts all stacked rows in a single call.

    Returns
    -------
    matplotlib.figure.Figure
        The figure with one thin curve per sampled row and the average curve in red.

    Raises
    ------
    KeyError
        If ``feature`` is not one of the model's feature columns of ``X``.
    ValueError
        If the feature has no non-missing numeric value, or if
        ``prediction_batch_size`` is not a positive integer or ``None``.
    """
    import matplotlib.pyplot as plt

    df = self.ice(
        X,
        feature,
        grid_resolution,
        percentiles,
        sample_size,
        random_state,
        centered,
        prediction_batch_size,
    )
    fig, ax = plt.subplots(figsize=(7, 4), dpi=120)
    for _, group in df.groupby("sample_index"):
        ax.plot(group["grid_value"], group["prediction"], color="#336699", alpha=0.18, linewidth=0.8)
    avg = df.groupby("grid_value", as_index=False)["prediction"].mean()
    ax.plot(avg["grid_value"], avg["prediction"], color="#CC0033", linewidth=2.2, label="average")
    ax.set_xlabel(str(df["feature"].iloc[0]))
    ax.set_ylabel("Centered prediction" if centered else "Prediction")
    ax.set_title(f"ICE: {df['feature'].iloc[0]}")
    ax.legend()
    ax.grid(alpha=0.25)
    return self._finalize_plot(plt, show, save_path)

ale

ale(X, feature, bins=20, sample_size=None, random_state=None, prediction_batch_size=100000)

Compute first-order accumulated local effects for a numeric feature.

Parameters:

Name	Type	Description	Default
X	DataFrame or array - like	Data the effect is estimated on. A DataFrame may carry extra columns: the model's features are selected by name (see feature_names).	required
feature	str or int	Numeric feature to analyse: a column name, or a Python int position among the model's feature columns. A name that is not one of the model's columns raises KeyError.	required
bins	int	Number of equal-count (quantile) intervals the feature is cut into. Repeated quantile edges, caused by heavily tied values, are merged, so fewer intervals may result.	20
sample_size	int or None	When X has more rows than this, a random sample of this size is used; None uses every row.	None
random_state	int or None	Seed of the row sampling (only used when sampling happens); None draws a different sample on each call.	None
prediction_batch_size	int or None	Maximum number of rows sent to the model in one prediction call; None predicts all stacked rows in a single call. A value below 1 raises ValueError.	100000

Returns:

Type	Description
DataFrame	One row per interval with feature (the column name), bin_left, bin_right, bin_center, ale_value and n (number of rows in the interval). The ale_value curve is centered so that its count-weighted mean is 0.

Raises:

Type	Description
KeyError	If feature is not one of the model's feature columns of X.
ValueError	If the feature has no non-missing numeric value or fewer than two distinct interval edges, or if prediction_batch_size is not a positive integer or None.

Notes

Rows whose value of feature is missing (or not numeric) are dropped before the effect is computed, so n sums to the number of rows with a valid value.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def ale(self, X, feature, bins=20, sample_size=None, random_state=None, prediction_batch_size=100000):
    """Compute first-order accumulated local effects for a numeric feature.

    Parameters
    ----------
    X : pandas.DataFrame or array-like
        Data the effect is estimated on. A DataFrame may carry extra columns: the
        model's features are selected by name (see ``feature_names``).
    feature : str or int
        Numeric feature to analyse: a column name, or a Python ``int`` position among
        the model's feature columns. A name that is not one of the model's columns
        raises ``KeyError``.
    bins : int, default 20
        Number of equal-count (quantile) intervals the feature is cut into. Repeated
        quantile edges, caused by heavily tied values, are merged, so fewer intervals
        may result.
    sample_size : int or None, default None
        When ``X`` has more rows than this, a random sample of this size is used;
        ``None`` uses every row.
    random_state : int or None, default None
        Seed of the row sampling (only used when sampling happens); ``None`` draws a
        different sample on each call.
    prediction_batch_size : int or None, default 100000
        Maximum number of rows sent to the model in one prediction call; ``None``
        predicts all stacked rows in a single call. A value below 1 raises
        ``ValueError``.

    Returns
    -------
    pandas.DataFrame
        One row per interval with ``feature`` (the column name), ``bin_left``,
        ``bin_right``, ``bin_center``, ``ale_value`` and ``n`` (number of rows in the
        interval). The ``ale_value`` curve is centered so that its count-weighted
        mean is 0.

    Raises
    ------
    KeyError
        If ``feature`` is not one of the model's feature columns of ``X``.
    ValueError
        If the feature has no non-missing numeric value or fewer than two distinct
        interval edges, or if ``prediction_batch_size`` is not a positive integer or
        ``None``.

    Notes
    -----
    Rows whose value of ``feature`` is missing (or not numeric) are dropped before the
    effect is computed, so ``n`` sums to the number of rows with a valid value.
    """
    frame = self._sample_frame(X, sample_size=sample_size, random_state=random_state)
    feature = self._feature_name(frame, feature)
    values = pd.to_numeric(frame[feature], errors="coerce")
    valid = values.notna()
    work = frame.loc[valid].copy()
    values = values.loc[valid]
    if work.empty:
        raise ValueError(f"Feature {feature!r} has no non-missing numeric values")

    quantiles = np.linspace(0, 1, int(bins) + 1)
    edges = np.unique(values.quantile(quantiles).to_numpy(dtype=float))
    if len(edges) < 2:
        raise ValueError(f"Feature {feature!r} does not have enough unique values for ALE")
    edges[0] = values.min()
    edges[-1] = values.max()

    bin_ids = np.searchsorted(edges, values.to_numpy(), side="right") - 1
    bin_ids = np.clip(bin_ids, 0, len(edges) - 2)

    effects = np.zeros(len(edges) - 1, dtype=float)
    counts = []
    centers = []
    prediction_frames = []
    prediction_segments = []
    for idx in range(len(edges) - 1):
        mask = bin_ids == idx
        subset = work.loc[mask].copy()
        counts.append(int(mask.sum()))
        left, right = float(edges[idx]), float(edges[idx + 1])
        centers.append((left + right) / 2.0)
        if subset.empty:
            continue
        low = subset.copy()
        high = subset.copy()
        low[feature] = left
        high[feature] = right
        prediction_segments.append((idx, len(subset)))
        prediction_frames.extend([low, high])

    if prediction_frames:
        stacked = pd.concat(prediction_frames, ignore_index=True)
        predictions = self._predict_proba_pos_batched(
            stacked,
            prediction_batch_size=prediction_batch_size,
        )
        offset = 0
        for idx, count in prediction_segments:
            low_pred = predictions[offset : offset + count]
            high_pred = predictions[offset + count : offset + 2 * count]
            effects[idx] = float(np.mean(high_pred - low_pred))
            offset += 2 * count

    ale_values = np.cumsum(effects)
    counts_arr = np.asarray(counts, dtype=float)
    if counts_arr.sum() > 0:
        ale_values = ale_values - np.average(ale_values, weights=counts_arr)
    else:
        ale_values = ale_values - ale_values.mean()

    return pd.DataFrame({"feature": feature, "bin_left": edges[:-1], "bin_right": edges[1:], "bin_center": centers, "ale_value": ale_values, "n": counts})

ale_plot

ale_plot(X, feature, bins=20, sample_size=None, random_state=None, show=True, save_path=None, prediction_batch_size=100000)

Plot first-order ALE for a numeric feature.

Parameters:

Name	Type	Description	Default
X	DataFrame or array - like	Data the effect is estimated on; same handling as in ale.	required
feature	str or int	Numeric feature to analyse: a column name, or a Python int position among the model's feature columns.	required
bins	int	Number of equal-count (quantile) intervals the feature is cut into.	20
sample_size	int or None	When X has more rows than this, a random sample of this size is used; None uses every row.	None
random_state	int or None	Seed of the row sampling (only used when sampling happens).	None
show	bool	If True, display the figure with plt.show(). If False, the figure is closed after it has been saved, so nothing is displayed.	True
save_path	str, path-like, or None	When given, the figure is saved to this path (dpi=150, tight bounding box; the format follows the file extension). The folder must already exist.	None
prediction_batch_size	int or None	Maximum number of rows sent to the model in one prediction call; None predicts all stacked rows in a single call.	100000

Returns:

Type	Description
Figure	The figure with the ALE curve plotted at the interval centers.

Raises:

Type	Description
KeyError	If feature is not one of the model's feature columns of X.
ValueError	If the feature has no non-missing numeric value or fewer than two distinct interval edges, or if prediction_batch_size is not a positive integer or None.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def ale_plot(self, X, feature, bins=20, sample_size=None, random_state=None, show=True, save_path=None, prediction_batch_size=100000):
    """Plot first-order ALE for a numeric feature.

    Parameters
    ----------
    X : pandas.DataFrame or array-like
        Data the effect is estimated on; same handling as in `ale`.
    feature : str or int
        Numeric feature to analyse: a column name, or a Python ``int`` position among
        the model's feature columns.
    bins : int, default 20
        Number of equal-count (quantile) intervals the feature is cut into.
    sample_size : int or None, default None
        When ``X`` has more rows than this, a random sample of this size is used;
        ``None`` uses every row.
    random_state : int or None, default None
        Seed of the row sampling (only used when sampling happens).
    show : bool, default True
        If ``True``, display the figure with ``plt.show()``. If ``False``, the figure
        is closed after it has been saved, so nothing is displayed.
    save_path : str, path-like, or None, default None
        When given, the figure is saved to this path (``dpi=150``, tight bounding
        box; the format follows the file extension). The folder must already exist.
    prediction_batch_size : int or None, default 100000
        Maximum number of rows sent to the model in one prediction call; ``None``
        predicts all stacked rows in a single call.

    Returns
    -------
    matplotlib.figure.Figure
        The figure with the ALE curve plotted at the interval centers.

    Raises
    ------
    KeyError
        If ``feature`` is not one of the model's feature columns of ``X``.
    ValueError
        If the feature has no non-missing numeric value or fewer than two distinct
        interval edges, or if ``prediction_batch_size`` is not a positive integer or
        ``None``.
    """
    import matplotlib.pyplot as plt

    df = self.ale(
        X,
        feature,
        bins=bins,
        sample_size=sample_size,
        random_state=random_state,
        prediction_batch_size=prediction_batch_size,
    )
    fig, ax = plt.subplots(figsize=(7, 4), dpi=120)
    ax.plot(df["bin_center"], df["ale_value"], color="#336699", marker="o", linewidth=2)
    ax.axhline(0, color="#999999", linewidth=1, linestyle="--")
    ax.set_xlabel(str(df["feature"].iloc[0]))
    ax.set_ylabel("Accumulated local effect")
    ax.set_title(f"ALE: {df['feature'].iloc[0]}")
    ax.grid(alpha=0.25)
    return self._finalize_plot(plt, show, save_path)

lime_explain_instance

lime_explain_instance(x_row, X_train=None, num_features=10, num_samples=5000, random_state=None, missing_strategy='median', **lime_kwargs)

Explain one sample with LIME.

Parameters:

Name	Type	Description	Default
x_row	pandas.DataFrame, pandas.Series, dict, or array-like	One observation. A Series or a dict is turned into a one-row frame and a 1-D array is read as one row; when several rows are given, only the first is explained.	required
X_train	pandas.DataFrame, array-like, or None	Training data used to set up the LIME explainer (feature statistics and sampling). When None, the explainer's background_data is used; one of the two must be available.	None
num_features	int	Maximum number of features in the explanation.	10
num_samples	int	Number of perturbed samples used to fit the local surrogate model.	5000
random_state	int or None	Seed of the LIME explainer.	None
missing_strategy	str	How missing values in the numeric columns of X_train and x_row are handled: "median" fills them with the training medians; "drop" removes the affected rows. When values are missing, a UserWarning names the affected columns. Any other value raises ValueError.	"median"
**lime_kwargs		Extra keyword arguments for LimeTabularExplainer, for example discretize_continuous=False. mode defaults to "classification" and class_names to ["class_0", "class_1"].	{}

Returns:

Type	Description
DataFrame	One row per feature rule with feature, feature_rule (the LIME rule text), weight and abs_weight, sorted by abs_weight (largest first). attrs["intercept"] (a dict keyed by class label) and attrs["score"] (the local surrogate's R squared) describe the surrogate model. When LIME returns no rule, an empty frame with these four columns is returned.

Raises:

Type	Description
ValueError	If neither X_train nor the explainer's background_data is available, if missing_strategy is not "median" or "drop", or if "drop" removes every row of X_train or x_row.
ImportError	If the optional lime package is not installed.

Notes

The explained quantity is the positive-class probability of the model.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def lime_explain_instance(
    self,
    x_row,
    X_train=None,
    num_features=10,
    num_samples=5000,
    random_state=None,
    missing_strategy="median",
    **lime_kwargs,
):
    """Explain one sample with LIME.

    Parameters
    ----------
    x_row : pandas.DataFrame, pandas.Series, dict, or array-like
        One observation. A Series or a dict is turned into a one-row frame and a 1-D
        array is read as one row; when several rows are given, only the first is
        explained.
    X_train : pandas.DataFrame, array-like, or None, default None
        Training data used to set up the LIME explainer (feature statistics and
        sampling). When ``None``, the explainer's ``background_data`` is used; one of
        the two must be available.
    num_features : int, default 10
        Maximum number of features in the explanation.
    num_samples : int, default 5000
        Number of perturbed samples used to fit the local surrogate model.
    random_state : int or None, default None
        Seed of the LIME explainer.
    missing_strategy : str, default "median"
        How missing values in the numeric columns of ``X_train`` and ``x_row`` are
        handled: ``"median"`` fills them with the training medians; ``"drop"``
        removes the affected rows. When values are missing, a ``UserWarning`` names
        the affected columns. Any other value raises ``ValueError``.
    **lime_kwargs
        Extra keyword arguments for ``LimeTabularExplainer``, for example
        ``discretize_continuous=False``. ``mode`` defaults to ``"classification"``
        and ``class_names`` to ``["class_0", "class_1"]``.

    Returns
    -------
    pandas.DataFrame
        One row per feature rule with ``feature``, ``feature_rule`` (the LIME rule
        text), ``weight`` and ``abs_weight``, sorted by ``abs_weight`` (largest
        first). ``attrs["intercept"]`` (a dict keyed by class label) and
        ``attrs["score"]`` (the local surrogate's R squared) describe the surrogate
        model. When LIME returns no rule, an empty frame with these four columns is
        returned.

    Raises
    ------
    ValueError
        If neither ``X_train`` nor the explainer's ``background_data`` is available,
        if ``missing_strategy`` is not ``"median"`` or ``"drop"``, or if ``"drop"``
        removes every row of ``X_train`` or ``x_row``.
    ImportError
        If the optional ``lime`` package is not installed.

    Notes
    -----
    The explained quantity is the positive-class probability of the model.
    """
    if isinstance(x_row, pd.Series):
        x_row = x_row.to_frame().T
    elif isinstance(x_row, dict):
        x_row = pd.DataFrame([x_row])
    frame = self._as_frame(x_row)
    if frame.shape[0] != 1:
        frame = frame.iloc[[0]]

    train_raw = self._as_frame(X_train if X_train is not None else self.background_data)
    if train_raw is None:
        raise ValueError("LIME requires X_train or background_data")
    train, frame = self._prepare_lime_data(
        train_raw,
        frame,
        train_role="X_train",
        rows_role="x_row",
        missing_strategy=missing_strategy,
    )
    explainer = self._build_lime_explainer(train, num_features, random_state, **lime_kwargs)
    explanation = explainer.explain_instance(
        data_row=np.asarray(frame.iloc[0]),
        predict_fn=self._predict_proba_2d,
        num_features=num_features,
        num_samples=num_samples,
    )
    rows = []
    names = list(frame.columns)
    for rule, weight in explanation.as_list():
        matched = next((name for name in names if str(rule).startswith(name) or name in str(rule)), rule)
        rows.append({"feature": matched, "feature_rule": rule, "weight": float(weight)})
    out = pd.DataFrame(rows)
    if out.empty:
        return pd.DataFrame(columns=["feature", "feature_rule", "weight", "abs_weight"])
    out["abs_weight"] = out["weight"].abs()
    out = out.sort_values("abs_weight", ascending=False).reset_index(drop=True)
    out.attrs["intercept"] = explanation.intercept
    out.attrs["score"] = explanation.score
    return out

lime_global_importance

lime_global_importance(X, X_train=None, num_features=10, num_samples=2000, sample_size=100, random_state=None, missing_strategy='median', **lime_kwargs)

Aggregate LIME local weights across a sample as global importance.

Parameters:

Name	Type	Description	Default
X	DataFrame or array - like	Rows to sample from. Each sampled row is explained with lime_explain_instance.	required
X_train	pandas.DataFrame, array-like, or None	Training data used to set up the LIME explainers. When None, the explainer's background_data is used; one of the two must be available.	None
num_features	int	Maximum number of features in each local explanation.	10
num_samples	int	Number of perturbed samples per explained row (smaller than the default of lime_explain_instance, 5000).	2000
sample_size	int or None	Maximum number of rows of X that are explained (a random sample); None explains every row.	100
random_state	int or None	Seed of the row sampling and of every LIME explainer.	None
missing_strategy	str	How missing values in the numeric columns of X_train and of the sampled rows are handled: "median" fills them with the training medians; "drop" removes the affected rows (X_train rows, and sampled rows before they are explained). When values are missing, a UserWarning names the affected columns. Any other value raises ValueError.	"median"
**lime_kwargs		Extra keyword arguments passed through to lime_explain_instance and on to LimeTabularExplainer, for example discretize_continuous=False.	{}

Returns:

Type	Description
DataFrame	One row per feature with feature, mean_abs_lime_weight (the mean absolute LIME weight over the explained rows in which the feature appears) and frequency (the number of explained rows whose local explanation contains the feature), sorted by mean_abs_lime_weight in descending order. An empty frame with these columns is returned when no row is explained.

Raises:

Type	Description
ValueError	If neither X_train nor the explainer's background_data is available, if missing_strategy is not "median" or "drop", or if "drop" removes every row of X_train or of the sample.
ImportError	If the optional lime package is not installed.

Notes

A new LIME explainer is built and fitted for every explained row, so the run time grows with sample_size * num_samples.

Source code in Modeling_Tool/Explainability/Model_Explainer.py

def lime_global_importance(
    self,
    X,
    X_train=None,
    num_features=10,
    num_samples=2000,
    sample_size=100,
    random_state=None,
    missing_strategy="median",
    **lime_kwargs,
):
    """Aggregate LIME local weights across a sample as global importance.

    Parameters
    ----------
    X : pandas.DataFrame or array-like
        Rows to sample from. Each sampled row is explained with
        `lime_explain_instance`.
    X_train : pandas.DataFrame, array-like, or None, default None
        Training data used to set up the LIME explainers. When ``None``, the
        explainer's ``background_data`` is used; one of the two must be available.
    num_features : int, default 10
        Maximum number of features in each local explanation.
    num_samples : int, default 2000
        Number of perturbed samples per explained row (smaller than the default of
        `lime_explain_instance`, 5000).
    sample_size : int or None, default 100
        Maximum number of rows of ``X`` that are explained (a random sample); ``None``
        explains every row.
    random_state : int or None, default None
        Seed of the row sampling and of every LIME explainer.
    missing_strategy : str, default "median"
        How missing values in the numeric columns of ``X_train`` and of the sampled
        rows are handled: ``"median"`` fills them with the training medians;
        ``"drop"`` removes the affected rows (``X_train`` rows, and sampled rows
        before they are explained). When values are missing, a ``UserWarning`` names
        the affected columns. Any other value raises ``ValueError``.
    **lime_kwargs
        Extra keyword arguments passed through to `lime_explain_instance` and on to
        ``LimeTabularExplainer``, for example ``discretize_continuous=False``.

    Returns
    -------
    pandas.DataFrame
        One row per feature with ``feature``, ``mean_abs_lime_weight`` (the mean
        absolute LIME weight over the explained rows in which the feature appears)
        and ``frequency`` (the number of explained rows whose local explanation
        contains the feature), sorted by ``mean_abs_lime_weight`` in descending
        order. An empty frame with these columns is returned when no row is explained.

    Raises
    ------
    ValueError
        If neither ``X_train`` nor the explainer's ``background_data`` is available,
        if ``missing_strategy`` is not ``"median"`` or ``"drop"``, or if ``"drop"``
        removes every row of ``X_train`` or of the sample.
    ImportError
        If the optional ``lime`` package is not installed.

    Notes
    -----
    A new LIME explainer is built and fitted for every explained row, so the run
    time grows with ``sample_size * num_samples``.
    """
    train_raw = self._as_frame(X_train if X_train is not None else self.background_data)
    if train_raw is None:
        raise ValueError("LIME requires X_train or background_data")
    frame = self._sample_frame(X, sample_size=sample_size, random_state=random_state)
    train, frame = self._prepare_lime_data(
        train_raw,
        frame,
        train_role="X_train",
        rows_role="X",
        missing_strategy=missing_strategy,
    )
    rows = []
    for _, row in frame.iterrows():
        local = self.lime_explain_instance(
            row,
            X_train=train,
            num_features=num_features,
            num_samples=num_samples,
            random_state=random_state,
            missing_strategy="median",
            **lime_kwargs,
        )
        rows.append(local)
    if not rows:
        return pd.DataFrame(columns=["feature", "mean_abs_lime_weight", "frequency"])
    all_rows = pd.concat(rows, ignore_index=True)
    summary = all_rows.groupby("feature", as_index=False).agg(
        mean_abs_lime_weight=("abs_weight", "mean"),
        frequency=("feature", "size"),
    )
    return summary.sort_values("mean_abs_lime_weight", ascending=False).reset_index(drop=True)

Coalition Structure for Owen Values: Coalition_Structure

Coalition_Structure

Coalition structure utilities for Owen Value explanations.

The functions in this module build feature groups for SHAP PartitionExplainer. They combine data-driven correlation clustering with optional business priors, then convert the final groups to the linkage matrix accepted by shap.maskers.Partition.

compute_correlation_linkage

compute_correlation_linkage(X: DataFrame, method: str = 'complete', corr_method: str = 'spearman') -> ndarray

Return a scipy linkage matrix from absolute correlation distances.

Distance is defined as 1 - abs(corr). Constant or all-missing columns are retained and treated as uncorrelated with other features.

Parameters:

Name	Type	Description	Default
X	DataFrame	Feature frame. Its columns are the leaves of the dendrogram; non-numeric values are coerced to missing.	required
method	str	Linkage method: "complete", "average" or "single".	"complete"
corr_method	str	Association measure: "spearman", "pearson" or "kendall" (pandas correlation methods), or "MIC" (case-insensitive; needs the optional minepy package).	"spearman"

Returns:

Type	Description
ndarray	SciPy linkage matrix of shape (n_features - 1, 4); an empty (0, 4) array when X has fewer than two columns.

Raises:

Type	Description
ValueError	If method is not one of the supported linkage methods.
TypeError	If X has two or more columns and is not a pandas DataFrame.
ImportError	If corr_method="MIC" is used and minepy is not installed.

Source code in Modeling_Tool/Explainability/Coalition_Structure.py

def compute_correlation_linkage(
    X: pd.DataFrame,
    method: str = "complete",
    corr_method: str = "spearman",
) -> np.ndarray:
    """Return a scipy linkage matrix from absolute correlation distances.

    Distance is defined as ``1 - abs(corr)``. Constant or all-missing columns are
    retained and treated as uncorrelated with other features.

    Parameters
    ----------
    X : pandas.DataFrame
        Feature frame. Its columns are the leaves of the dendrogram; non-numeric
        values are coerced to missing.
    method : str, default "complete"
        Linkage method: ``"complete"``, ``"average"`` or ``"single"``.
    corr_method : str, default "spearman"
        Association measure: ``"spearman"``, ``"pearson"`` or ``"kendall"`` (pandas
        correlation methods), or ``"MIC"`` (case-insensitive; needs the optional
        ``minepy`` package).

    Returns
    -------
    numpy.ndarray
        SciPy linkage matrix of shape ``(n_features - 1, 4)``; an empty ``(0, 4)`` array
        when ``X`` has fewer than two columns.

    Raises
    ------
    ValueError
        If ``method`` is not one of the supported linkage methods.
    TypeError
        If ``X`` has two or more columns and is not a pandas DataFrame.
    ImportError
        If ``corr_method="MIC"`` is used and ``minepy`` is not installed.
    """
    method = _check_method(method)
    if X.shape[1] < 2:
        return np.empty((0, 4), dtype=float)
    dist = _correlation_distance(X, corr_method=corr_method)
    condensed = squareform(dist.values, checks=False)
    return linkage(condensed, method=method)

auto_cluster

auto_cluster(X: DataFrame, threshold: float = 0.35, method: str = 'complete', corr_method: str = 'spearman', min_group_size: int = 1) -> Dict[str, List[str]]

Build data-driven feature groups using hierarchical clustering.

Parameters:

Name	Type	Description	Default
X	DataFrame	Feature frame; its columns are the features to group.	required
threshold	float	Cut height of the dendrogram on the distance 1 - abs(association); must lie between 0 and 1 (inclusive). A lower value gives tighter, smaller groups.	0.35
method	str	Linkage method: "complete", "average" or "single".	"complete"
corr_method	str	Association measure: "spearman", "pearson" or "kendall" (pandas correlation methods), or "MIC" (case-insensitive; needs the optional minepy package).	"spearman"
min_group_size	int	Clusters with fewer features than this are merged into a single group named "auto_singleton"; the default 1 keeps every cluster.	1

Returns:

Type	Description
dict	Mapping from group name ("auto_cluster_<k>", plus "auto_singleton" when small clusters were merged) to the list of its feature names. A frame with a single column returns {"auto_cluster_1": [column]}.

Raises:

Type	Description
ValueError	If threshold is outside [0, 1] or method is unsupported.
ImportError	If corr_method="MIC" is used and minepy is not installed.

Source code in Modeling_Tool/Explainability/Coalition_Structure.py

def auto_cluster(
    X: pd.DataFrame,
    threshold: float = 0.35,
    method: str = "complete",
    corr_method: str = "spearman",
    min_group_size: int = 1,
) -> Dict[str, List[str]]:
    """Build data-driven feature groups using hierarchical clustering.

    Parameters
    ----------
    X : pandas.DataFrame
        Feature frame; its columns are the features to group.
    threshold : float, default 0.35
        Cut height of the dendrogram on the distance ``1 - abs(association)``; must lie
        between 0 and 1 (inclusive). A lower value gives tighter, smaller groups.
    method : str, default "complete"
        Linkage method: ``"complete"``, ``"average"`` or ``"single"``.
    corr_method : str, default "spearman"
        Association measure: ``"spearman"``, ``"pearson"`` or ``"kendall"`` (pandas
        correlation methods), or ``"MIC"`` (case-insensitive; needs the optional
        ``minepy`` package).
    min_group_size : int, default 1
        Clusters with fewer features than this are merged into a single group named
        ``"auto_singleton"``; the default 1 keeps every cluster.

    Returns
    -------
    dict
        Mapping from group name (``"auto_cluster_<k>"``, plus ``"auto_singleton"`` when
        small clusters were merged) to the list of its feature names. A frame with a
        single column returns ``{"auto_cluster_1": [column]}``.

    Raises
    ------
    ValueError
        If ``threshold`` is outside ``[0, 1]`` or ``method`` is unsupported.
    ImportError
        If ``corr_method="MIC"`` is used and ``minepy`` is not installed.
    """
    if threshold < 0 or threshold > 1:
        raise ValueError("threshold must be between 0 and 1")
    features = list(X.columns)
    if len(features) == 1:
        return {"auto_cluster_1": features}

    lnk = compute_correlation_linkage(X, method=method, corr_method=corr_method)
    labels = fcluster(lnk, t=threshold, criterion="distance")
    groups: Dict[str, List[str]] = {}
    for feat, gid in zip(features, labels):
        groups.setdefault(f"auto_cluster_{int(gid)}", []).append(feat)

    if min_group_size > 1:
        singletons = [key for key, feats in groups.items() if len(feats) < min_group_size]
        merged: List[str] = []
        for key in singletons:
            merged.extend(groups.pop(key))
        if merged:
            groups["auto_singleton"] = merged
    return groups

apply_prior

apply_prior(auto_groups: Mapping[str, Sequence[str]], prior_groups: Optional[Mapping[str, Sequence[str]]], features: Sequence[str]) -> Dict[str, List[str]]

Merge business priors with data-driven groups.

Business priors win. Remaining features keep their automatic cluster with a residual_ prefix. Priors may contain feature names absent from X; those are ignored. Repeated valid feature names across prior groups are rejected.

Parameters:

Name	Type	Description	Default
auto_groups	mapping of str to sequence of str	Data-driven groups, for example the result of auto_cluster.	required
prior_groups	mapping of str to sequence of str, or None	Business groups as {group_name: [feature, ...]}. When None or empty, auto_groups is returned unchanged (as a plain dict of lists).	required
features	sequence of str	All feature names of the data. Prior feature names that are not in it are ignored, and features that end up in no group are collected in "ungrouped".	required

Returns:

Type	Description
dict	Mapping from group name to a list of feature names: the prior groups first (only their members found in features; a prior group with no such member is dropped), then one "residual_<auto group name>" group per automatic group for its members that no prior group claimed, then "ungrouped" if needed.

Raises:

Type	Description
ValueError	If a feature of features appears in more than one prior group.

Source code in Modeling_Tool/Explainability/Coalition_Structure.py

def apply_prior(
    auto_groups: Mapping[str, Sequence[str]],
    prior_groups: Optional[Mapping[str, Sequence[str]]],
    features: Sequence[str],
) -> Dict[str, List[str]]:
    """Merge business priors with data-driven groups.

    Business priors win. Remaining features keep their automatic cluster with a
    ``residual_`` prefix. Priors may contain feature names absent from X; those
    are ignored. Repeated valid feature names across prior groups are rejected.

    Parameters
    ----------
    auto_groups : mapping of str to sequence of str
        Data-driven groups, for example the result of `auto_cluster`.
    prior_groups : mapping of str to sequence of str, or None
        Business groups as ``{group_name: [feature, ...]}``. When ``None`` or empty,
        ``auto_groups`` is returned unchanged (as a plain dict of lists).
    features : sequence of str
        All feature names of the data. Prior feature names that are not in it are
        ignored, and features that end up in no group are collected in ``"ungrouped"``.

    Returns
    -------
    dict
        Mapping from group name to a list of feature names: the prior groups first (only
        their members found in ``features``; a prior group with no such member is
        dropped), then one ``"residual_<auto group name>"`` group per automatic group
        for its members that no prior group claimed, then ``"ungrouped"`` if needed.

    Raises
    ------
    ValueError
        If a feature of ``features`` appears in more than one prior group.
    """
    if not prior_groups:
        return {str(k): list(v) for k, v in auto_groups.items()}

    duplicates = _prior_duplicates(prior_groups, features)
    if duplicates:
        detail = "; ".join(f"{feat}: {groups}" for feat, groups in duplicates.items())
        raise ValueError(f"Features appear in multiple prior groups: {detail}")

    assigned = set()
    merged: Dict[str, List[str]] = {}
    feature_set = set(features)

    for group_name, feats in prior_groups.items():
        valid = [feat for feat in feats if feat in feature_set]
        if valid:
            merged[str(group_name)] = valid
            assigned.update(valid)

    for group_name, feats in auto_groups.items():
        leftover = [feat for feat in feats if feat not in assigned]
        if leftover:
            key = f"residual_{group_name}"
            merged.setdefault(key, []).extend(leftover)
            assigned.update(leftover)

    uncovered = [feat for feat in features if feat not in assigned]
    if uncovered:
        merged["ungrouped"] = uncovered
    return merged

validate_groups

validate_groups(groups: Mapping[str, Sequence[str]], features: Sequence[str], raise_error: bool = True) -> bool

Validate complete, non-overlapping feature coverage.

Parameters:

Name	Type	Description	Default
groups	mapping of str to sequence of str	Group name to member features.	required
features	sequence of str	The features that must each belong to exactly one group.	required
raise_error	bool	If True, an invalid grouping raises ValueError; if False, the function returns False instead.	True

Returns:

Type	Description
bool	True when every feature is in exactly one group and no group contains an unknown feature; False otherwise (only reached with raise_error=False).

Raises:

Type	Description
ValueError	If the grouping is invalid and raise_error is True; the message lists the missing, duplicated and unknown features.

Source code in Modeling_Tool/Explainability/Coalition_Structure.py

def validate_groups(groups: Mapping[str, Sequence[str]], features: Sequence[str], raise_error: bool = True) -> bool:
    """Validate complete, non-overlapping feature coverage.

    Parameters
    ----------
    groups : mapping of str to sequence of str
        Group name to member features.
    features : sequence of str
        The features that must each belong to exactly one group.
    raise_error : bool, default True
        If ``True``, an invalid grouping raises ``ValueError``; if ``False``, the
        function returns ``False`` instead.

    Returns
    -------
    bool
        ``True`` when every feature is in exactly one group and no group contains an
        unknown feature; ``False`` otherwise (only reached with ``raise_error=False``).

    Raises
    ------
    ValueError
        If the grouping is invalid and ``raise_error`` is ``True``; the message lists the
        missing, duplicated and unknown features.
    """
    all_assigned = [feat for feats in groups.values() for feat in feats]
    feature_set = set(features)
    missing = feature_set - set(all_assigned)
    unknown = set(all_assigned) - feature_set
    duplicated = {feat for feat, count in Counter(all_assigned).items() if count > 1}
    ok = not missing and not duplicated and not unknown
    if not ok and raise_error:
        parts = []
        if missing:
            parts.append(f"missing={sorted(missing)}")
        if duplicated:
            parts.append(f"duplicated={sorted(duplicated)}")
        if unknown:
            parts.append(f"unknown={sorted(unknown)}")
        raise ValueError("Invalid coalition groups: " + ", ".join(parts))
    return ok

group_correlation_summary

group_correlation_summary(X: DataFrame, groups: Mapping[str, Sequence[str]], corr_method: str = 'spearman') -> DataFrame

Summarize within-group absolute association.

For corr_method='spearman' (default) the summary reports mean/max absolute correlation. For corr_method='MIC' the same columns report mean/max MIC.

Parameters:

Name	Type	Description	Default
X	DataFrame	Feature frame; non-numeric values are coerced to missing.	required
groups	mapping of str to sequence of str	Group name to member features. Members that are not columns of X are ignored.	required
corr_method	str	Association measure: "spearman", "pearson" or "kendall" (pandas correlation methods), or "MIC" (case-insensitive; needs the optional minepy package).	"spearman"

Returns:

Type	Description
DataFrame	Indexed by group with columns n_features (members found in X), mean_abs_corr and max_abs_corr (mean and maximum absolute association over the member pairs, rounded to 3 decimals; NaN for a group with fewer than two members) and features (the members found in X).

Raises:

Type	Description
TypeError	If X is not a pandas DataFrame.
ValueError	If X has no columns.
ImportError	If corr_method="MIC" is used and minepy is not installed.

Source code in Modeling_Tool/Explainability/Coalition_Structure.py

def group_correlation_summary(
    X: pd.DataFrame,
    groups: Mapping[str, Sequence[str]],
    corr_method: str = "spearman",
) -> pd.DataFrame:
    """Summarize within-group absolute association.

    For ``corr_method='spearman'`` (default) the summary reports mean/max absolute
    correlation. For ``corr_method='MIC'`` the same columns report mean/max MIC.

    Parameters
    ----------
    X : pandas.DataFrame
        Feature frame; non-numeric values are coerced to missing.
    groups : mapping of str to sequence of str
        Group name to member features. Members that are not columns of ``X`` are ignored.
    corr_method : str, default "spearman"
        Association measure: ``"spearman"``, ``"pearson"`` or ``"kendall"`` (pandas
        correlation methods), or ``"MIC"`` (case-insensitive; needs the optional
        ``minepy`` package).

    Returns
    -------
    pandas.DataFrame
        Indexed by ``group`` with columns ``n_features`` (members found in ``X``),
        ``mean_abs_corr`` and ``max_abs_corr`` (mean and maximum absolute association
        over the member pairs, rounded to 3 decimals; NaN for a group with fewer than two
        members) and ``features`` (the members found in ``X``).

    Raises
    ------
    TypeError
        If ``X`` is not a pandas DataFrame.
    ValueError
        If ``X`` has no columns.
    ImportError
        If ``corr_method="MIC"`` is used and ``minepy`` is not installed.
    """
    frame = _as_numeric_frame(X)
    assoc = _association_matrix(frame, corr_method=corr_method)
    rows = []
    for group_name, feats in groups.items():
        valid = [feat for feat in feats if feat in assoc.columns]
        if len(valid) > 1:
            sub = assoc.loc[valid, valid].values
            vals = sub[np.triu_indices_from(sub, k=1)]
            avg_corr = float(np.nanmean(vals)) if vals.size else float("nan")
            max_corr = float(np.nanmax(vals)) if vals.size else float("nan")
        else:
            avg_corr = float("nan")
            max_corr = float("nan")
        rows.append(
            {
                "group": group_name,
                "n_features": len(valid),
                "mean_abs_corr": round(avg_corr, 3) if np.isfinite(avg_corr) else np.nan,
                "max_abs_corr": round(max_corr, 3) if np.isfinite(max_corr) else np.nan,
                "features": valid,
            }
        )
    return pd.DataFrame(rows).set_index("group")

groups_to_shap_clustering

groups_to_shap_clustering(groups: Mapping[str, Sequence[str]], features: Sequence[str], intra_dist: float = 0.01, inter_dist: float = 0.99) -> ndarray

Convert feature groups to the linkage matrix used by SHAP Partition.

Parameters:

Name	Type	Description	Default
groups	mapping of str to sequence of str	Group name to member features; together the groups must cover every feature in features exactly once.	required
features	sequence of str	Feature names in the column order of the data given to SHAP; they are the leaves of the returned tree.	required
intra_dist	float	Distance between two features of the same group.	0.01
inter_dist	float	Distance between two features of different groups.	0.99

Returns:

Type	Description
ndarray	SciPy linkage matrix (complete linkage over the block distance matrix) of shape (n_features - 1, 4); an empty (0, 4) array when there are fewer than two features.

Raises:

Type	Description
ValueError	If groups does not cover features exactly once (see validate_groups).

Source code in Modeling_Tool/Explainability/Coalition_Structure.py

def groups_to_shap_clustering(
    groups: Mapping[str, Sequence[str]],
    features: Sequence[str],
    intra_dist: float = 0.01,
    inter_dist: float = 0.99,
) -> np.ndarray:
    """Convert feature groups to the linkage matrix used by SHAP Partition.

    Parameters
    ----------
    groups : mapping of str to sequence of str
        Group name to member features; together the groups must cover every feature in
        ``features`` exactly once.
    features : sequence of str
        Feature names in the column order of the data given to SHAP; they are the leaves
        of the returned tree.
    intra_dist : float, default 0.01
        Distance between two features of the same group.
    inter_dist : float, default 0.99
        Distance between two features of different groups.

    Returns
    -------
    numpy.ndarray
        SciPy linkage matrix (complete linkage over the block distance matrix) of shape
        ``(n_features - 1, 4)``; an empty ``(0, 4)`` array when there are fewer than two
        features.

    Raises
    ------
    ValueError
        If ``groups`` does not cover ``features`` exactly once (see `validate_groups`).
    """
    features = list(features)
    validate_groups(groups, features, raise_error=True)
    n = len(features)
    if n < 2:
        return np.empty((0, 4), dtype=float)

    feat_idx = {feat: idx for idx, feat in enumerate(features)}
    dist_mat = np.full((n, n), float(inter_dist), dtype=float)
    np.fill_diagonal(dist_mat, 0.0)

    for feats in groups.values():
        idxs = [feat_idx[feat] for feat in feats if feat in feat_idx]
        if idxs:
            dist_mat[np.ix_(idxs, idxs)] = float(intra_dist)
            dist_mat[idxs, idxs] = 0.0

    condensed = squareform(dist_mat, checks=False)
    return linkage(condensed, method="complete")

build_coalition_structure

build_coalition_structure(X: DataFrame, prior_groups: Optional[Mapping[str, Sequence[str]]] = None, threshold: float = 0.35, method: str = 'complete', corr_method: str = 'spearman', min_group_size: int = 1, intra_dist: float = 0.01, inter_dist: float = 0.99) -> dict

Build a complete coalition structure for Owen Value explanations.

corr_method may be any pandas correlation method (pearson, spearman, kendall) or MIC for maximal information coefficient clustering.

Returns a dict containing final groups, automatic groups, correlation linkage, SHAP-compatible linkage, and a within-group correlation summary.

Parameters:

Name	Type	Description	Default
X	DataFrame	Feature frame whose columns are the features to group. Every column is coerced to numeric (non-numeric values become missing), and constant or all-missing columns count as uncorrelated with the others.	required
prior_groups	mapping of str to sequence of str, or None	Business groups as {group_name: [feature, ...]}. They win over the automatic clusters; feature names that are not columns of X are ignored, and a feature of X may not appear in two groups. Features outside every prior group fall back to their automatic cluster, in groups named "residual_<cluster name>".	None
threshold	float	Cut height of the clustering on the distance 1 - abs(association); must lie between 0 and 1 (inclusive). With method="complete" every pair in an automatic group has an absolute association of at least 1 - threshold.	0.35
method	str	Linkage method: "complete", "average" or "single".	"complete"
corr_method	str	Association measure: "spearman", "pearson" or "kendall" (pandas correlation methods), or "MIC" (case-insensitive; needs the optional minepy package).	"spearman"
min_group_size	int	Automatic clusters with fewer features than this are merged into one group named "auto_singleton" before the priors are applied; the default 1 keeps every cluster.	1
intra_dist	float	Distance between two features of the same final group in the partition tree handed to SHAP (shap_lnk).	0.01
inter_dist	float	Distance between two features of different final groups in that tree.	0.99

Returns:

Type

Description

dict

The coalition structure with these keys: groups Final {group_name: [features]} mapping. shap_lnk Linkage matrix for shap.maskers.Partition, shape (n_features - 1, 4) ((0, 4) for a single feature). corr_lnk Linkage matrix of the correlation dendrogram, same shape rules. auto_groups The automatic clusters before the priors were applied. summary DataFrame indexed by group with n_features, mean_abs_corr, max_abs_corr and features (see group_correlation_summary). features Column names of X, in order. threshold, method, corr_method The arguments that produced the structure.

Raises:

Type	Description
TypeError	If X is not a pandas DataFrame.
ValueError	If X has no columns, threshold is outside [0, 1], method is unsupported, or a feature appears in more than one prior group.
ImportError	If corr_method="MIC" is used and minepy is not installed.

Source code in Modeling_Tool/Explainability/Coalition_Structure.py

def build_coalition_structure(
    X: pd.DataFrame,
    prior_groups: Optional[Mapping[str, Sequence[str]]] = None,
    threshold: float = 0.35,
    method: str = "complete",
    corr_method: str = "spearman",
    min_group_size: int = 1,
    intra_dist: float = 0.01,
    inter_dist: float = 0.99,
) -> dict:
    """Build a complete coalition structure for Owen Value explanations.

    ``corr_method`` may be any pandas correlation method (``pearson``,
    ``spearman``, ``kendall``) or ``MIC`` for maximal information coefficient
    clustering.

    Returns a dict containing final groups, automatic groups, correlation
    linkage, SHAP-compatible linkage, and a within-group correlation summary.

    Parameters
    ----------
    X : pandas.DataFrame
        Feature frame whose columns are the features to group. Every column is coerced
        to numeric (non-numeric values become missing), and constant or all-missing
        columns count as uncorrelated with the others.
    prior_groups : mapping of str to sequence of str, or None, default None
        Business groups as ``{group_name: [feature, ...]}``. They win over the automatic
        clusters; feature names that are not columns of ``X`` are ignored, and a feature
        of ``X`` may not appear in two groups. Features outside every prior group fall
        back to their automatic cluster, in groups named ``"residual_<cluster name>"``.
    threshold : float, default 0.35
        Cut height of the clustering on the distance ``1 - abs(association)``; must lie
        between 0 and 1 (inclusive). With ``method="complete"`` every pair in an
        automatic group has an absolute association of at least ``1 - threshold``.
    method : str, default "complete"
        Linkage method: ``"complete"``, ``"average"`` or ``"single"``.
    corr_method : str, default "spearman"
        Association measure: ``"spearman"``, ``"pearson"`` or ``"kendall"`` (pandas
        correlation methods), or ``"MIC"`` (case-insensitive; needs the optional
        ``minepy`` package).
    min_group_size : int, default 1
        Automatic clusters with fewer features than this are merged into one group named
        ``"auto_singleton"`` before the priors are applied; the default 1 keeps every
        cluster.
    intra_dist : float, default 0.01
        Distance between two features of the same final group in the partition tree
        handed to SHAP (``shap_lnk``).
    inter_dist : float, default 0.99
        Distance between two features of different final groups in that tree.

    Returns
    -------
    dict
        The coalition structure with these keys:

        ``groups``
            Final ``{group_name: [features]}`` mapping.
        ``shap_lnk``
            Linkage matrix for ``shap.maskers.Partition``, shape ``(n_features - 1, 4)``
            (``(0, 4)`` for a single feature).
        ``corr_lnk``
            Linkage matrix of the correlation dendrogram, same shape rules.
        ``auto_groups``
            The automatic clusters before the priors were applied.
        ``summary``
            DataFrame indexed by ``group`` with ``n_features``, ``mean_abs_corr``,
            ``max_abs_corr`` and ``features`` (see `group_correlation_summary`).
        ``features``
            Column names of ``X``, in order.
        ``threshold``, ``method``, ``corr_method``
            The arguments that produced the structure.

    Raises
    ------
    TypeError
        If ``X`` is not a pandas DataFrame.
    ValueError
        If ``X`` has no columns, ``threshold`` is outside ``[0, 1]``, ``method`` is
        unsupported, or a feature appears in more than one prior group.
    ImportError
        If ``corr_method="MIC"`` is used and ``minepy`` is not installed.
    """
    frame = _as_numeric_frame(X)
    features = list(frame.columns)
    corr_lnk = compute_correlation_linkage(frame, method=method, corr_method=corr_method)
    auto = auto_cluster(
        frame,
        threshold=threshold,
        method=method,
        corr_method=corr_method,
        min_group_size=min_group_size,
    )
    final = apply_prior(auto, prior_groups, features)
    validate_groups(final, features, raise_error=True)
    summary = group_correlation_summary(frame, final, corr_method=corr_method)
    shap_lnk = groups_to_shap_clustering(final, features, intra_dist=intra_dist, inter_dist=inter_dist)
    return {
        "groups": final,
        "shap_lnk": shap_lnk,
        "corr_lnk": corr_lnk,
        "auto_groups": auto,
        "summary": summary,
        "features": features,
        "threshold": threshold,
        "method": method,
        "corr_method": corr_method,
    }