Modeling_Tool.Eval

Model evaluation layer: Gains tables, KS/AUC summaries, challenger-versus-champion comparison, slice-by-slice evaluation, and ROC/KDE/percentile/gain figures. Passing weight_col or sample_weight takes the weighted path; omitting them keeps the historical unweighted behavior.

User guide: Model Evaluation, including Sample-Weighted Evaluation.

Gains Tables and Performance Summaries: Model_Eval_Tool

Model_Eval_Tool

GainsTableCalculator

Gains table calculator.

Integrates the various Gains table functions and supports both the basic Gains table and the Gains table with custom metrics. Provides an object-oriented interface for computing grouped Gains tables.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input data table.	required
dep	str	Name of the target variable.	required
nbins	int	Number of bins.	10
precision	int	Precision of the bin boundary values.	5
min_bin_prop	float	Minimum proportion of samples per bin.	0.05
include_missing	bool	Whether missing scores are binned. True fills them with fillna and gives them a bin of their own, (-inf, fillna] (a score already equal to fillna joins it); the other bins are computed from the real scores. False leaves the rows with a missing score out.	True
score	str	Name of the score column.	None
model	sklearn-like model	Machine learning model.	None
varlist	list	List of model features.	None
equal_freq	bool	True for equal-frequency binning.	True
chi2_method	bool	Whether to use chi-square binning.	False
chi2_p	float	Significance level of the chi-square test.	0.95
init_equi_bins	int	Initial number of equal-frequency bins.	100
fillna	any	Fill value for missing values.	-999999
spec_values	list	List of special values.	[]
tree_binning	bool	Whether to use decision-tree binning.	False
random_state	int	Random seed.	42
ascending	bool	Whether the bin order is ascending.	False
weight_col	str	Name of the sample weight column (must be in data). When it is set, calculate returns the weighted Gains table (N is the sum of the weights and N_RAW the number of rows), except when it is called with grp_name.	None
weighted_binning	bool	Accepted for compatibility but without any effect: the weighted Gains table always uses equal-frequency bins by cumulative weight.	None

Examples:

>>> calc = GainsTableCalculator(data, dep='target', score='score', nbins=10)
>>> result = calc.calculate(grp_name='region')

Source code in Modeling_Tool/Eval/Model_Eval_Tool.py

class GainsTableCalculator:
    """
    Gains table calculator.

    Integrates the various Gains table functions and supports both the basic Gains table and the
    Gains table with custom metrics. Provides an object-oriented interface for computing grouped Gains tables.

    Parameters
    ----------
    data : pandas.DataFrame
        Input data table.
    dep : str
        Name of the target variable.
    nbins : int, default 10
        Number of bins.
    precision : int, default 5
        Precision of the bin boundary values.
    min_bin_prop : float, default 0.05
        Minimum proportion of samples per bin.
    include_missing : bool, default True
        Whether missing scores are binned. True fills them with ``fillna`` and gives them a bin of their own, ``(-inf,
        fillna]`` (a score already equal to ``fillna`` joins it); the other bins are computed from the real scores.
        False leaves the rows with a missing score out.
    score : str, optional
        Name of the score column.
    model : sklearn-like model, optional
        Machine learning model.
    varlist : list, optional
        List of model features.
    equal_freq : bool, default True
        True for equal-frequency binning.
    chi2_method : bool, default False
        Whether to use chi-square binning.
    chi2_p : float, default 0.95
        Significance level of the chi-square test.
    init_equi_bins : int, default 100
        Initial number of equal-frequency bins.
    fillna : any, default -999999
        Fill value for missing values.
    spec_values : list, default []
        List of special values.
    tree_binning : bool, default False
        Whether to use decision-tree binning.
    random_state : int, default 42
        Random seed.
    ascending : bool, default False
        Whether the bin order is ascending.
    weight_col : str, optional
        Name of the sample weight column (must be in ``data``). When it is set, ``calculate`` returns the weighted Gains
        table (``N`` is the sum of the weights and ``N_RAW`` the number of rows), except when it is called with
        ``grp_name``.
    weighted_binning : bool, optional
        Accepted for compatibility but without any effect: the weighted Gains table always uses equal-frequency bins by
        cumulative weight.

    Examples
    --------
    >>> calc = GainsTableCalculator(data, dep='target', score='score', nbins=10)
    >>> result = calc.calculate(grp_name='region')
    """

    def __init__(self, data, dep, nbins = 10, precision = 5, min_bin_prop = 0.05,
                 include_missing = True, score = None, model = None, varlist = None,
                 equal_freq = True, chi2_method = False, chi2_p = 0.95, 
                 init_equi_bins = 100, fillna = -999999, spec_values = [],
                 tree_binning = False, random_state = 42, ascending = False,
                 weight_col = None, weighted_binning = None):
        """
        Initialize the Gains table calculator.

        Parameters
        ----------
        data : pandas.DataFrame
            Input data table.
        dep : str
            Name of the target variable.
        nbins : int, default 10
            Number of bins.
        precision : int, default 5
            Precision of the bin boundary values.
        min_bin_prop : float, default 0.05
            Minimum proportion of samples per bin.
        include_missing : bool, default True
            Whether missing scores are binned. True fills them with ``fillna`` and gives them a bin of their own,
            ``(-inf, fillna]`` (a score already equal to ``fillna`` joins it); the other bins are computed from the real
            scores. False leaves the rows with a missing score out.
        score : str, optional
            Name of the score column.
        model : sklearn-like model, optional
            Machine learning model.
        varlist : list, optional
            List of model features.
        equal_freq : bool, default True
            True for equal-frequency binning.
        chi2_method : bool, default False
            Whether to use chi-square binning.
        chi2_p : float, default 0.95
            Significance level of the chi-square test.
        init_equi_bins : int, default 100
            Initial number of equal-frequency bins.
        fillna : any, default -999999
            Fill value for missing values.
        spec_values : list, default []
            List of special values.
        tree_binning : bool, default False
            Whether to use decision-tree binning.
        random_state : int, default 42
            Random seed.
        ascending : bool, default False
            Whether the bin order is ascending.
        weight_col : str, optional
            Name of the sample weight column (must be in ``data``). When it is set, ``calculate`` returns the weighted
            Gains table (``N`` is the sum of the weights and ``N_RAW`` the number of rows), except when it is called with
            ``grp_name``.
        weighted_binning : bool, optional
            Accepted for compatibility but without any effect: the weighted Gains table always uses equal-frequency bins
            by cumulative weight.
        """
        self.data = data
        self.dep = dep
        self.nbins = nbins
        self.precision = precision
        self.min_bin_prop = min_bin_prop
        self.include_missing = include_missing
        self.score = score
        self.model = model
        self.varlist = varlist
        self.equal_freq = equal_freq
        self.chi2_method = chi2_method
        self.chi2_p = chi2_p
        self.init_equi_bins = init_equi_bins
        self.fillna = fillna
        self.spec_values = spec_values
        self.tree_binning = tree_binning
        self.random_state = random_state
        self.ascending = ascending
        self.weight_col = weight_col
        self.weighted_binning = weighted_binning

    def calculate(self, grp_name = None, min_data_size = 100, grp_colname = None,
                  sync_range = True, retSummary = False, withSummary = False,
                  wholeGroup = False, add_func = None, weight_col = None):
        """
        Compute the Gains table.

        Parameters
        ----------
        grp_name : str, optional
            Name of the grouping column.
        min_data_size : int, default 100
            Minimum number of samples per group.
        grp_colname : str, optional
            Name of the group column in the output.
        sync_range : bool, default True
            Whether to synchronize the bin boundaries across groups.
        retSummary : bool, default False
            Whether to return only the summary metrics.
        withSummary : bool, default False
            Whether to include an overall summary row.
        wholeGroup : bool, default False
            Whether to use all the data for binning.
        add_func : callable, optional
            Custom statistics function. It receives the rows of one bin as a DataFrame (all the columns of ``data`` plus
            the bin columns ``_bin_num`` and ``_bin_range``) and returns a Series whose values become extra columns, on
            the weighted path too.
        weight_col : str, optional
            Sample weight column for this call; it overrides the ``weight_col`` of the calculator. Default is None, i.e.
            the calculator's ``weight_col``.

        Returns
        -------
        pandas.DataFrame
            Gains table.

        Notes
        -----
        The call is delegated to ``get_gains_table`` with the settings of the calculator. When a weight column applies and
        ``grp_name`` is None, the weighted Gains table (bins 1 to ``nbins``, each with about ``1 / nbins`` of the total
        weight) is returned, with the ``add_func`` columns and the ``Grand Summary`` row when they are asked for; with
        ``grp_name`` the weights are ignored.
        Like ``get_gains_table``, it returns the integer -1, -2 or -3 (instead of raising) when neither ``score`` nor a
        ``model`` with ``varlist`` was given to the calculator.
        """
        return get_gains_table(
            data = self.data,
            dep = self.dep,
            nbins = self.nbins,
            precision = self.precision,
            min_bin_prop = self.min_bin_prop,
            include_missing = self.include_missing,
            score = self.score,
            model = self.model,
            varlist = self.varlist,
            equal_freq = self.equal_freq,
            chi2_method = self.chi2_method,
            grp_name = grp_name,
            min_data_size = min_data_size,
            grp_colname = grp_colname,
            sync_range = sync_range,
            chi2_p = self.chi2_p,
            init_equi_bins = self.init_equi_bins,
            fillna = self.fillna,
            spec_values = self.spec_values,
            retSummary = retSummary,
            tree_binning = self.tree_binning,
            random_state = self.random_state,
            ascending = self.ascending,
            withSummary = withSummary,
            wholeGroup = wholeGroup,
            add_func = add_func,
            weight_col = self.weight_col if weight_col is None else weight_col,
            weighted_binning = self.weighted_binning,
        )

calculate

calculate(grp_name=None, min_data_size=100, grp_colname=None, sync_range=True, retSummary=False, withSummary=False, wholeGroup=False, add_func=None, weight_col=None)

Compute the Gains table.

Parameters:

Name	Type	Description	Default
grp_name	str	Name of the grouping column.	None
min_data_size	int	Minimum number of samples per group.	100
grp_colname	str	Name of the group column in the output.	None
sync_range	bool	Whether to synchronize the bin boundaries across groups.	True
retSummary	bool	Whether to return only the summary metrics.	False
withSummary	bool	Whether to include an overall summary row.	False
wholeGroup	bool	Whether to use all the data for binning.	False
add_func	callable	Custom statistics function. It receives the rows of one bin as a DataFrame (all the columns of data plus the bin columns _bin_num and _bin_range) and returns a Series whose values become extra columns, on the weighted path too.	None
weight_col	str	Sample weight column for this call; it overrides the weight_col of the calculator. Default is None, i.e. the calculator's weight_col.	None

Returns:

Type	Description
DataFrame	Gains table.

Notes

The call is delegated to get_gains_table with the settings of the calculator. When a weight column applies and grp_name is None, the weighted Gains table (bins 1 to nbins, each with about 1 / nbins of the total weight) is returned, with the add_func columns and the Grand Summary row when they are asked for; with grp_name the weights are ignored. Like get_gains_table, it returns the integer -1, -2 or -3 (instead of raising) when neither score nor a model with varlist was given to the calculator.

Source code in Modeling_Tool/Eval/Model_Eval_Tool.py

def calculate(self, grp_name = None, min_data_size = 100, grp_colname = None,
              sync_range = True, retSummary = False, withSummary = False,
              wholeGroup = False, add_func = None, weight_col = None):
    """
    Compute the Gains table.

    Parameters
    ----------
    grp_name : str, optional
        Name of the grouping column.
    min_data_size : int, default 100
        Minimum number of samples per group.
    grp_colname : str, optional
        Name of the group column in the output.
    sync_range : bool, default True
        Whether to synchronize the bin boundaries across groups.
    retSummary : bool, default False
        Whether to return only the summary metrics.
    withSummary : bool, default False
        Whether to include an overall summary row.
    wholeGroup : bool, default False
        Whether to use all the data for binning.
    add_func : callable, optional
        Custom statistics function. It receives the rows of one bin as a DataFrame (all the columns of ``data`` plus
        the bin columns ``_bin_num`` and ``_bin_range``) and returns a Series whose values become extra columns, on
        the weighted path too.
    weight_col : str, optional
        Sample weight column for this call; it overrides the ``weight_col`` of the calculator. Default is None, i.e.
        the calculator's ``weight_col``.

    Returns
    -------
    pandas.DataFrame
        Gains table.

    Notes
    -----
    The call is delegated to ``get_gains_table`` with the settings of the calculator. When a weight column applies and
    ``grp_name`` is None, the weighted Gains table (bins 1 to ``nbins``, each with about ``1 / nbins`` of the total
    weight) is returned, with the ``add_func`` columns and the ``Grand Summary`` row when they are asked for; with
    ``grp_name`` the weights are ignored.
    Like ``get_gains_table``, it returns the integer -1, -2 or -3 (instead of raising) when neither ``score`` nor a
    ``model`` with ``varlist`` was given to the calculator.
    """
    return get_gains_table(
        data = self.data,
        dep = self.dep,
        nbins = self.nbins,
        precision = self.precision,
        min_bin_prop = self.min_bin_prop,
        include_missing = self.include_missing,
        score = self.score,
        model = self.model,
        varlist = self.varlist,
        equal_freq = self.equal_freq,
        chi2_method = self.chi2_method,
        grp_name = grp_name,
        min_data_size = min_data_size,
        grp_colname = grp_colname,
        sync_range = sync_range,
        chi2_p = self.chi2_p,
        init_equi_bins = self.init_equi_bins,
        fillna = self.fillna,
        spec_values = self.spec_values,
        retSummary = retSummary,
        tree_binning = self.tree_binning,
        random_state = self.random_state,
        ascending = self.ascending,
        withSummary = withSummary,
        wholeGroup = wholeGroup,
        add_func = add_func,
        weight_col = self.weight_col if weight_col is None else weight_col,
        weighted_binning = self.weighted_binning,
    )

PerformanceEvaluator

Performance evaluator.

Integrates the various model performance evaluation functions and supports evaluation across multiple datasets and multiple groups. Provides an object-oriented interface for computing and summarizing performance metrics.

Parameters:

Name	Type	Description	Default
tgt_name	str or list of str	Name of the target variable. A list/tuple of several y labels can be passed: each label is evaluated separately and the results are stacked vertically (the output gains a tgt_name column), and a separate figure is drawn for each label.	required
scr_name	str	Name of the score column.	None
model	sklearn-like model	Machine learning model.	None
feature_cols	list	List of model features.	None
dist_bins	int	Number of bins for the score distribution.	20
pct_bins	int	Number of percentile bins.	10
precision	int	Precision of the bin boundary values.	5
min_bin_prop	float	Minimum share of a bin in the Gains tables of evaluate (summary columns IV, LIFT, KS_IN_GAINS, N_BINS, ...), weighted or not: they use pct_bins bins capped at 1 / min_bin_prop, so each bin holds about that share of the rows (of the weight) or more; ties in the scores can move a few rows. The percentile bands (Top/Btm) keep pct_bins. 0 sets no cap.	0.05
include_missing	bool	Whether to include missing values.	False
equal_freq	bool	True for equal-frequency binning.	True
chi2_method	bool	Whether to use chi-square binning.	False
init_equi_bins	int	Initial number of equal-frequency bins.	1000
chi2_p	float	Significance level of the chi-square test.	0.9
tree_binning	bool	Whether to use decision-tree binning.	False
random_state	int	Random seed.	42
weight_col	str	Default weight column; each add_dataset call can also specify its own.	None
spec_values	list	Sentinel score values (for example -1 for a score override) that carry no ordering information. Rows with such a score are left out of the ranking metrics (AUC, KS, the Top/Btm target rates and their lifts) and of the figures, and are binned separately in the Gains tables; N and avgTrue count every row on both paths, and N_SPECIAL / N_SPECIAL_RAW report the sentinel part. Default is None, which is stored as an empty list.	None
ascending	bool	Score direction applied uniformly to the summary, the Gains tables and the figures, including the weighted paths. Default is None, which keeps the legacy direction of every underlying function.	None

Attributes:

Name	Type	Description
datasets	dict	The datasets added with add_dataset, by name.
dataset_weight_cols	dict	The weight column given to add_dataset for each dataset name (None when none was given).
evaluate_status	str or None	Outcome of the last evaluate call: 'ok', 'no_datasets', 'compute_failed' (neither scr_name nor model with feature_cols is set) or 'empty_input' (the result is empty). None before the first call.

Examples:

>>> evaluator = PerformanceEvaluator(tgt_name='target', model=model, feature_cols=features)
>>> evaluator.add_dataset('train', train_df)
>>> evaluator.add_dataset('validation', val_df)
>>> evaluator.add_dataset('oot', oot_df)
>>> result = evaluator.evaluate()
>>>
>>> # Multiple y labels: pass a list as tgt_name -> a vertically stacked table with a tgt_name column, one figure per label
>>> evaluator = PerformanceEvaluator(tgt_name=['bad_dpd7', 'bad_dpd30'], model=model, feature_cols=features)
>>> evaluator.add_dataset('train', train_df).add_dataset('oot', oot_df)
>>> result = evaluator.evaluate(to_show=True)

Source code in Modeling_Tool/Eval/Model_Eval_Tool.py

class PerformanceEvaluator:
    """
    Performance evaluator.

    Integrates the various model performance evaluation functions and supports evaluation across
    multiple datasets and multiple groups. Provides an object-oriented interface for computing and
    summarizing performance metrics.

    Parameters
    ----------
    tgt_name : str or list of str
        Name of the target variable. A list/tuple of several y labels can be passed: each label is evaluated
        separately and the results are stacked vertically (the output gains a ``tgt_name`` column),
        and a separate figure is drawn for each label.
    scr_name : str, optional
        Name of the score column.
    model : sklearn-like model, optional
        Machine learning model.
    feature_cols : list, optional
        List of model features.
    dist_bins : int, default 20
        Number of bins for the score distribution.
    pct_bins : int, default 10
        Number of percentile bins.
    precision : int, default 5
        Precision of the bin boundary values.
    min_bin_prop : float, default 0.05
        Minimum share of a bin in the Gains tables of ``evaluate`` (summary columns ``IV``, ``LIFT``, ``KS_IN_GAINS``,
        ``N_BINS``, ...), weighted or not: they use ``pct_bins`` bins capped at ``1 / min_bin_prop``, so each bin holds
        about that share of the rows (of the weight) or more; ties in the scores can move a few rows. The percentile
        bands (``Top``/``Btm``) keep ``pct_bins``. 0 sets no cap.
    include_missing : bool, default False
        Whether to include missing values.
    equal_freq : bool, default True
        True for equal-frequency binning.
    chi2_method : bool, default False
        Whether to use chi-square binning.
    init_equi_bins : int, default 1000
        Initial number of equal-frequency bins.
    chi2_p : float, default 0.9
        Significance level of the chi-square test.
    tree_binning : bool, default False
        Whether to use decision-tree binning.
    random_state : int, default 42
        Random seed.
    weight_col : str, optional
        Default weight column; each ``add_dataset`` call can also specify its own.
    spec_values : list, optional
        Sentinel score values (for example -1 for a score override) that carry no ordering information. Rows with such a
        score are left out of the ranking metrics (``AUC``, ``KS``, the Top/Btm target rates and their lifts) and of the
        figures, and are binned separately in the Gains tables; ``N`` and ``avgTrue`` count every row on both paths, and
        ``N_SPECIAL`` / ``N_SPECIAL_RAW`` report the sentinel part. Default is None, which is stored as an empty list.
    ascending : bool, optional
        Score direction applied uniformly to the summary, the Gains tables and the figures, including the weighted paths.
        Default is None, which keeps the legacy direction of every underlying function.

    Attributes
    ----------
    datasets : dict
        The datasets added with ``add_dataset``, by name.
    dataset_weight_cols : dict
        The weight column given to ``add_dataset`` for each dataset name (None when none was given).
    evaluate_status : str or None
        Outcome of the last ``evaluate`` call: ``'ok'``, ``'no_datasets'``, ``'compute_failed'`` (neither ``scr_name`` nor
        ``model`` with ``feature_cols`` is set) or ``'empty_input'`` (the result is empty). None before the first call.

    Examples
    --------
    >>> evaluator = PerformanceEvaluator(tgt_name='target', model=model, feature_cols=features)
    >>> evaluator.add_dataset('train', train_df)
    >>> evaluator.add_dataset('validation', val_df)
    >>> evaluator.add_dataset('oot', oot_df)
    >>> result = evaluator.evaluate()
    >>>
    >>> # Multiple y labels: pass a list as tgt_name -> a vertically stacked table with a tgt_name column, one figure per label
    >>> evaluator = PerformanceEvaluator(tgt_name=['bad_dpd7', 'bad_dpd30'], model=model, feature_cols=features)
    >>> evaluator.add_dataset('train', train_df).add_dataset('oot', oot_df)
    >>> result = evaluator.evaluate(to_show=True)
    """

    def __init__(self, tgt_name, scr_name = None, model = None, feature_cols = None,
                 dist_bins = 20, pct_bins = 10, precision = 5, min_bin_prop = 0.05,
                 include_missing = False, equal_freq = True, chi2_method = False,
                 init_equi_bins = 1000, chi2_p = 0.9, tree_binning = False, random_state = 42,
                 weight_col = None, spec_values = None, ascending = None):
        """
        Initialize the performance evaluator.

        Parameters
        ----------
        tgt_name : str or list of str
            Name of the target variable. A list/tuple of several y labels can be passed: each label is evaluated
            separately and the results are stacked vertically (the output gains a ``tgt_name`` column),
            and a separate figure is drawn for each label.
        scr_name : str, optional
            Name of the score column.
        model : sklearn-like model, optional
            Machine learning model.
        feature_cols : list, optional
            List of model features.
        dist_bins : int, default 20
            Number of bins for the score distribution.
        pct_bins : int, default 10
            Number of percentile bins.
        precision : int, default 5
            Precision of the bin boundary values.
        min_bin_prop : float, default 0.05
            Minimum share of a bin in the Gains tables of ``evaluate`` (summary columns ``IV``, ``LIFT``, ``KS_IN_GAINS``,
            ``N_BINS``, ...), weighted or not: they use ``pct_bins`` bins capped at ``1 / min_bin_prop``, so each bin holds
            about that share of the rows (of the weight) or more; ties in the scores can move a few rows. The percentile
            bands (``Top``/``Btm``) keep ``pct_bins``. 0 sets no cap.
        include_missing : bool, default False
            Whether to include missing values.
        equal_freq : bool, default True
            True for equal-frequency binning.
        chi2_method : bool, default False
            Whether to use chi-square binning.
        init_equi_bins : int, default 1000
            Initial number of equal-frequency bins.
        chi2_p : float, default 0.9
            Significance level of the chi-square test.
        tree_binning : bool, default False
            Whether to use decision-tree binning.
        random_state : int, default 42
            Random seed.
        weight_col : str, optional
            Default weight column; each ``add_dataset`` call can also specify its own.
        spec_values : list, optional
            Sentinel score values (for example -1 for a score override) that carry no ordering information. Rows with such
            a score are left out of the ranking metrics (``AUC``, ``KS``, the Top/Btm target rates and their lifts) and of
            the figures, and are binned separately in the Gains tables; ``N`` and ``avgTrue`` count every row on both
            paths, and ``N_SPECIAL`` / ``N_SPECIAL_RAW`` report the sentinel part. Default is None, which is stored as an
            empty list.
        ascending : bool, optional
            Score direction applied uniformly to the summary, the Gains tables and the figures, including the weighted
            paths. Default is None, which keeps the legacy direction of every underlying function.
        """
        self.tgt_name = tgt_name
        self.scr_name = scr_name
        self.model = model
        self.feature_cols = feature_cols
        self.dist_bins = dist_bins
        self.pct_bins = pct_bins
        self.precision = precision
        self.min_bin_prop = min_bin_prop
        self.include_missing = include_missing
        self.equal_freq = equal_freq
        self.chi2_method = chi2_method
        self.init_equi_bins = init_equi_bins
        self.chi2_p = chi2_p
        self.tree_binning = tree_binning
        self.random_state = random_state
        self.weight_col = weight_col
        # spec_values: sentinel scores (e.g. -1 for all-missing overrides)
        # that get their own evaluation bin and are excluded from quantile
        # edges and ranking metrics. ascending: None keeps every underlying
        # function's legacy direction; an explicit bool is threaded through
        # summary, gains, and figures (including weighted paths) uniformly.
        self.spec_values = list(spec_values) if spec_values else []
        self.ascending = ascending
        self.datasets = {}
        self.dataset_weight_cols = {}
        self.evaluate_status = None

    def _gains_nbins(self):
        """Bins of the Gains tables: ``pct_bins`` capped at ``1 / min_bin_prop`` (without the floor of 5 bins that the
        generic binning applies), the same on the weighted and the unweighted path."""
        nbins = int(self.pct_bins)
        prop = float(self.min_bin_prop or 0.0)
        if prop > 0.0:
            nbins = min(nbins, int(np.floor(1.0 / prop + 1e-9)))
        return max(1, nbins)

    def _governed_eval_kwargs(self):
        """Kwargs to thread spec_values/ascending into underlying eval calls.

        Nothing is passed when unset, so every legacy default stays intact
        (unweighted summary False, comparison paths True, weighted False).
        """
        kwargs = {}
        if self.spec_values:
            kwargs["spec_values"] = list(self.spec_values)
        if self.ascending is not None:
            kwargs["ascending"] = bool(self.ascending)
        return kwargs

    def add_dataset(self, name, data, weight_col = None, overwrite = False):
        """
        Add a dataset.

        Parameters
        ----------
        name : str
            Name of the dataset (e.g. 'train', 'validation', 'oot').
        data : pandas.DataFrame
            Dataset.
        weight_col : str, optional
            Sample weight column of this dataset (must be in ``data``); it takes precedence over the weight column given to
            ``evaluate`` and to the constructor. Default is None.
        overwrite : bool, default False
            Whether a dataset that already has this ``name`` may be replaced (a ``RuntimeWarning`` is then issued; the
            weight column is replaced too).

        Returns
        -------
        self
            The evaluator itself, to allow method chaining.

        Raises
        ------
        KeyError
            If a dataset named ``name`` already exists and ``overwrite`` is False.
        """
        if name in self.datasets and not overwrite:
            raise KeyError(
                f"Dataset name {name!r} already exists. Use overwrite=True to replace it, "
                "or pass a different dataset name."
            )
        if name in self.datasets and overwrite:
            warnings.warn(
                f"PerformanceEvaluator.add_dataset overwriting existing dataset {name!r}.",
                RuntimeWarning,
                stacklevel=2,
            )
        self.datasets[name] = data
        self.dataset_weight_cols[name] = weight_col
        return self

    def evaluate(self, oot_grp_name = None, min_data_size = 100, grp_colname = None,
                 fig_save_path = None, rpt_save_path = None, to_show = False, 
                 display = True, gains_table = False, benchmark_dataset = None,
                 weight_col = None):
        """
        Run the performance evaluation.

        Parameters
        ----------
        oot_grp_name : str, optional
            Name of the grouping column. Every added dataset that has this column (not only the OOT data) is evaluated
            separately for each group value, and the datasets without it are skipped. Weights and ``fig_save_path`` are
            ignored in this mode.
        min_data_size : int, default 100
            Minimum number of samples per group.
        grp_colname : str, optional
            Name of the group column in the output. Default is None, i.e. ``oot_grp_name``.
        fig_save_path : str, optional
            Path to save the figure. It is ignored when ``oot_grp_name`` is given, and on the weighted path the figure is
            only drawn when it is set.
        rpt_save_path : str, optional
            Path to save the report (a CSV file of the summary table).
        to_show : bool, default False
            Whether to display the figures.
        display : bool, default True
            Whether to show the result table with ``IPython.display.display`` (it requires IPython).
        gains_table : bool, default False
            Whether the percentile panel and the Top/Btm target rates are built from the Gains-table binning (the
            ``gains_table`` argument of ``evaluate_performance``). The Gains-table summary columns are added either way on
            the unweighted path.
        benchmark_dataset : str or pandas.DataFrame, optional
            Benchmark dataset used to fix the bin boundaries. If a str is passed, the dataset is looked up by name
            among the datasets added with ``add_dataset``; if a DataFrame is passed, it is used directly.
            The default None means that each dataset is binned independently. The weights are ignored when it is given.
        weight_col : str, optional
            Sample weight column for this call; it overrides the ``weight_col`` of the evaluator, while a column given to
            ``add_dataset`` takes precedence over both. Weights are applied only when ``oot_grp_name`` and
            ``benchmark_dataset`` are None and ``tgt_name`` is a single name; otherwise they are silently ignored (when
            ``tgt_name`` is a list or tuple, this argument is ignored, but the weight columns of the evaluator and of the
            datasets are still honored).

        Returns
        -------
        pandas.DataFrame
            Performance evaluation summary table. If ``tgt_name`` of the instance is a list/tuple of several labels,
            each label is evaluated separately and the results are stacked vertically after a new ``tgt_name`` column
            is added; when ``to_show=True``, one figure is drawn for each label, and
            ``fig_save_path`` automatically gets a label suffix (e.g. ``perf.png`` -> ``perf_<label>.png``).
            On the weighted path the table is narrower: ``index``, ``dataset``, ``DATASET``, ``AUC``, ``KS``, ``LIFT``,
            ``IV``, ``N`` (sum of the weights), ``N_RAW``, ``avgTrue`` and ``avgScore`` (plus ``N_SPECIAL`` and
            ``N_SPECIAL_RAW`` when ``spec_values`` match rows). An empty DataFrame is returned when no dataset was added,
            or when neither ``scr_name`` nor ``model`` with ``feature_cols`` is set (a ``DeprecationWarning`` is issued
            then).

        Raises
        ------
        ValueError
            If ``benchmark_dataset`` is a name that was not added with ``add_dataset``, or if no bin edges can be derived
            from it.

        Notes
        -----
        The attribute ``evaluate_status`` records the outcome of the call (``'ok'``, ``'no_datasets'``,
        ``'compute_failed'`` or ``'empty_input'``).
        """
        if len(self.datasets) == 0:
            self.evaluate_status = "no_datasets"
            return pd.DataFrame([])

        if self.scr_name is None and self.model is None and self.feature_cols is None:
            self.evaluate_status = "compute_failed"
            warnings.warn(
                "PerformanceEvaluator.evaluate() now returns an empty DataFrame instead of -1; "
                "inspect .evaluate_status for the failure reason.",
                DeprecationWarning,
                stacklevel=2,
            )
            return pd.DataFrame([])

        if self.scr_name is None and self.model is None:
            self.evaluate_status = "compute_failed"
            warnings.warn(
                "PerformanceEvaluator.evaluate() now returns an empty DataFrame instead of -2; "
                "inspect .evaluate_status for the failure reason.",
                DeprecationWarning,
                stacklevel=2,
            )
            return pd.DataFrame([])

        if self.scr_name is None and self.feature_cols is None:
            self.evaluate_status = "compute_failed"
            warnings.warn(
                "PerformanceEvaluator.evaluate() now returns an empty DataFrame instead of -3; "
                "inspect .evaluate_status for the failure reason.",
                DeprecationWarning,
                stacklevel=2,
            )
            return pd.DataFrame([])

        active_weight_col = weight_col or self.weight_col
        has_dataset_weight = any(v is not None for v in self.dataset_weight_cols.values())
        if (active_weight_col is not None or has_dataset_weight) and oot_grp_name is None and benchmark_dataset is None and not isinstance(self.tgt_name, (list, tuple)):
            rows = []
            for name, data in self.datasets.items():
                if data is None:
                    continue
                wc = self.dataset_weight_cols.get(name) or active_weight_col
                work_data = data.copy()
                scr_name = self.scr_name
                if scr_name is None:
                    scr_name = "_mdl_scr"
                    work_data[scr_name] = self.model.predict_proba(work_data.loc[:, self.feature_cols])[:, 1]
                rows.append(
                    _weighted_eval.dataset_summary(
                        name,
                        work_data,
                        self.tgt_name,
                        scr_name,
                        weight_col=wc,
                        nbins=self._gains_nbins(),
                        ascending=bool(self.ascending) if self.ascending is not None else False,
                        spec_values=self.spec_values or None,
                    )
                )
            fnl_df = pd.DataFrame(rows)
            if display:
                from IPython.display import display as _ipy_display
                _ipy_display(fnl_df)
            if fig_save_path:
                eval_datasets = {}
                for name, data in self.datasets.items():
                    if data is None:
                        continue
                    wc = self.dataset_weight_cols.get(name) or active_weight_col
                    work_data = data.copy()
                    scr_name = self.scr_name
                    if scr_name is None:
                        scr_name = "_mdl_scr"
                        work_data[scr_name] = self.model.predict_proba(work_data.loc[:, self.feature_cols])[:, 1]
                    resolved_weight = _weighted_eval.resolve_weights(
                        work_data,
                        weight_col=wc,
                        expected_len=len(work_data),
                    )
                    y_true = work_data[self.tgt_name].to_numpy()
                    y_score = work_data[scr_name].to_numpy()
                    if self.spec_values:
                        keep = ~pd.Series(y_score).isin(self.spec_values).to_numpy()
                        y_true = y_true[keep]
                        y_score = y_score[keep]
                        if resolved_weight is not None:
                            resolved_weight = np.asarray(resolved_weight)[keep]
                    payload = {
                        "y_true": y_true,
                        "y_score": y_score,
                    }
                    if resolved_weight is not None:
                        payload["sample_weight"] = resolved_weight
                    eval_datasets[name] = payload
                if eval_datasets:
                    evaluate_performance(
                        datasets=eval_datasets,
                        dist_bins=self.dist_bins,
                        pct_bins=self.pct_bins,
                        square_figsize=5,
                        to_show=to_show,
                        save_path=fig_save_path,
                        gains_table=gains_table,
                        equal_freq=self.equal_freq,
                        ascending=self.ascending,
                    )
            if rpt_save_path:
                fnl_df.to_csv(rpt_save_path, index=False)
            self.evaluate_status = "ok" if not fnl_df.empty else "empty_input"
            return fnl_df

        # ── Multiple y labels: when tgt_name is a list/tuple, evaluate label by label, then stack the results vertically ──
        #    Table:   the result of each label gets a new tgt_name column, then they are stacked vertically;
        #    Figures: each label gets its own figure (shown in a loop when to_show=True; fig_save_path automatically gets a label suffix).
        if isinstance(self.tgt_name, (list, tuple)):
            import os as _os

            def _suffix_path(p, lab):
                if not p:
                    return None
                root, ext = _os.path.splitext(str(p))
                return "{0}_{1}{2}".format(root, lab, ext)

            _orig_tgt = self.tgt_name
            multi_results = []
            try:
                for _t in list(self.tgt_name):
                    self.tgt_name = _t
                    sub_df = self.evaluate(
                        oot_grp_name = oot_grp_name,
                        min_data_size = min_data_size,
                        grp_colname = grp_colname,
                        fig_save_path = _suffix_path(fig_save_path, _t),
                        rpt_save_path = None,        # saved once after stacking (see below)
                        to_show = to_show,           # each label gets its own figure
                        display = False,             # displayed once after stacking
                        gains_table = gains_table,
                        benchmark_dataset = benchmark_dataset,
                    )
                    if isinstance(sub_df, pd.DataFrame) and sub_df.shape[0] > 0:
                        sub_df.insert(0, "tgt_name", _t)
                        multi_results.append(sub_df)
            finally:
                self.tgt_name = _orig_tgt

            fnl_df = pd.concat(multi_results, ignore_index = True) if multi_results else pd.DataFrame([])

            if display:
                from IPython.display import display as _ipy_display
                _ipy_display(fnl_df)
            if rpt_save_path:
                fnl_df.to_csv(rpt_save_path, index = False)
            self.evaluate_status = "ok" if not fnl_df.empty else "empty_input"
            return fnl_df

        def _get_score(data):
            if self.scr_name is not None:
                return data[self.scr_name]
            return self.model.predict_proba(data.loc[:, self.feature_cols])[:, 1]

        def _rank_payload(data):
            """(y_true, y_score) with special sentinel scores removed.

            Sentinels like the all-missing override -1 carry no ordering
            information: AUC/KS/distribution figures are computed on the
            non-special rows; their bin-level accounting lives in the gains
            table via spec_values.
            """
            y_true = data[self.tgt_name]
            y_score = _get_score(data)
            if not self.spec_values:
                return y_true, y_score
            keep = ~pd.Series(np.asarray(y_score)).isin(self.spec_values).to_numpy()
            return np.asarray(y_true)[keep], np.asarray(y_score)[keep]

        def _get_benchmark_bin_edges():
            if benchmark_dataset is None:
                return None

            if isinstance(benchmark_dataset, str):
                if benchmark_dataset not in self.datasets:
                    raise ValueError("benchmark_dataset must be one of added datasets: {0}".format(list(self.datasets.keys())))
                benchmark_data = self.datasets[benchmark_dataset]
            else:
                benchmark_data = benchmark_dataset

            benchmark_score = "_benchmark_score"
            benchmark_df = pd.DataFrame({
                self.tgt_name: benchmark_data[self.tgt_name],
                benchmark_score: _get_score(benchmark_data)
            })

            _, benchmark_bin_edges = super_binning(
                data = benchmark_df,
                score = benchmark_score,
                dep = self.tgt_name,
                nbins = self._gains_nbins(),
                precision = self.precision,
                min_bin_prop = self.min_bin_prop,
                include_missing = self.include_missing,
                equal_freq = self.equal_freq,
                chi2_method = self.chi2_method,
                chi2_p = self.chi2_p,
                init_equi_bins = self.init_equi_bins,
                tree_binning = self.tree_binning,
                random_state = self.random_state,
                return_edges = True,
                ascending = True if self.ascending is None else bool(self.ascending)
            )

            if benchmark_bin_edges is None or len(benchmark_bin_edges) < 2:
                raise ValueError("Cannot generate benchmark bin edges from benchmark_dataset.")

            return list(benchmark_bin_edges)

        benchmark_bin_edges = _get_benchmark_bin_edges()

        def _evaluate_dataset_dict(dataset_dict, save_path = None):
            eval_datasets = {}
            for name, data in dataset_dict.items():
                if data is None:
                    continue
                y_true, y_score = _rank_payload(data)
                eval_datasets[name] = {
                    "y_true": y_true,
                    "y_score": y_score
                }

            if len(eval_datasets) == 0:
                self.evaluate_status = "empty_input"
                return pd.DataFrame([])

            model_eval_result_df = evaluate_performance(
                datasets = eval_datasets,
                dist_bins = self.dist_bins,
                pct_bins = self.pct_bins,
                square_figsize = 5,
                to_show = to_show,
                save_path = save_path,
                gains_table = gains_table,
                equal_freq = self.equal_freq,
                pct_bin_edges = benchmark_bin_edges,
                ascending = self.ascending,
            )

            import re
            btm_cols = [x for x in model_eval_result_df.columns if x.startswith("Btm")]
            top_cols = [x for x in model_eval_result_df.columns if x.startswith("Top")]

            if btm_cols and top_cols:
                btm_str = btm_cols[0]
                top_str = top_cols[0]
                numbers = re.findall(r'\d+', btm_str)
                if numbers:
                    quantile = int(numbers[0])
                else:
                    quantile = self.pct_bins

                model_eval_result_df[f"Btm{quantile}%_Lift"] = model_eval_result_df[btm_str] / model_eval_result_df["avgTrue"]
                model_eval_result_df[f"Top{quantile}%_Lift"] = model_eval_result_df[top_str] / model_eval_result_df["avgTrue"]
                model_eval_result_df["AUC_Shift"] = model_eval_result_df["AUC"].shift(1) / model_eval_result_df["AUC"] - 1
                model_eval_result_df["KS_Shift"] = model_eval_result_df["KS"].shift(1) / model_eval_result_df["KS"] - 1

            if self.spec_values:
                # the ranking columns above describe the rows with a real score; N and avgTrue describe every row and the
                # sentinel part is reported apart, as on the weighted path (they used to leave the sentinel rows out)
                all_rows = {}
                for name, data in dataset_dict.items():
                    if data is None:
                        continue
                    y_score = np.asarray(_get_score(data))
                    n_special = int(pd.Series(y_score).isin(self.spec_values).sum())
                    avg_true = _weighted_eval.safe_weighted_average(np.asarray(data[self.tgt_name]), None)
                    all_rows[name] = (len(data), avg_true, n_special)
                if "index" in model_eval_result_df.columns:
                    names = model_eval_result_df["index"]
                    model_eval_result_df["N"] = names.map(lambda key: all_rows.get(key, (np.nan,))[0])
                    model_eval_result_df["avgTrue"] = names.map(lambda key: all_rows.get(key, (np.nan, np.nan))[1])
                    if any(item[2] for item in all_rows.values()):
                        n_special = names.map(lambda key: all_rows.get(key, (0, 0, 0))[2])
                        model_eval_result_df["N_SPECIAL"] = n_special.where(n_special > 0).astype(float)
                        model_eval_result_df["N_SPECIAL_RAW"] = n_special.where(n_special > 0)

            gains_table_cols = ['N_BUMP', 'MIN_RISK_DEP', 'MAX_RISK_DEP', 'KS_IN_GAINS', 'LIFT_IN_GAINS', 'IV', 'N_BINS']
            gains_summ_list = []
            for name, data in dataset_dict.items():
                if data is None:
                    gains_res = pd.DataFrame([], columns = gains_table_cols)
                else:
                    gains_res = get_gains_table(
                        data = data,
                        dep = self.tgt_name,
                        nbins = benchmark_bin_edges if benchmark_bin_edges is not None else self._gains_nbins(),
                        precision = self.precision,
                        min_bin_prop = self.min_bin_prop,
                        include_missing = self.include_missing,
                        score = self.scr_name,
                        equal_freq = self.equal_freq,
                        chi2_method = False if benchmark_bin_edges is not None else self.chi2_method,
                        model = self.model,
                        varlist = self.feature_cols,
                        init_equi_bins = self.init_equi_bins,
                        chi2_p = self.chi2_p,
                        retSummary = True,
                        tree_binning = False if benchmark_bin_edges is not None else self.tree_binning,
                        random_state = self.random_state,
                        **self._governed_eval_kwargs(),
                    )
                gains_res['index'] = name
                gains_summ_list.append(gains_res)

            if gains_summ_list:
                gains_summ = concat_non_empty(gains_summ_list)
                model_eval_result_df = model_eval_result_df.merge(gains_summ, on = ['index'], how = 'left')

            return model_eval_result_df

        if oot_grp_name is None:
            fnl_df = _evaluate_dataset_dict(self.datasets, save_path = fig_save_path)
        else:
            if grp_colname is None:
                grp_colname = oot_grp_name

            grouped_results = []
            for dataset_name, data in self.datasets.items():
                if data is None or oot_grp_name not in data.columns:
                    continue

                grp_list = data[oot_grp_name].sort_values(ascending = True).unique().tolist()
                valid_grp_list = [x for x in grp_list if data[data[oot_grp_name].isin([x])].shape[0] >= min_data_size]

                for grp in valid_grp_list:
                    data_grp = data[data[oot_grp_name].isin([grp])]
                    perf_res = _evaluate_dataset_dict({dataset_name: data_grp}, save_path = None)
                    if perf_res.shape[0] > 0:
                        perf_res[grp_colname] = grp
                        grouped_results.append(perf_res)

            if len(grouped_results) == 0:
                fnl_df = pd.DataFrame([])
            else:
                fnl_df = pd.concat(grouped_results, ignore_index = True)

        if display:
            from IPython.display import display
            display(fnl_df)

        if rpt_save_path:
            fnl_df.to_csv(rpt_save_path, index = False)

        self.evaluate_status = "ok" if isinstance(fnl_df, pd.DataFrame) and not fnl_df.empty else "empty_input"
        return fnl_df

add_dataset

add_dataset(name, data, weight_col=None, overwrite=False)

Add a dataset.

Parameters:

Name	Type	Description	Default
name	str	Name of the dataset (e.g. 'train', 'validation', 'oot').	required
data	DataFrame	Dataset.	required
weight_col	str	Sample weight column of this dataset (must be in data); it takes precedence over the weight column given to evaluate and to the constructor. Default is None.	None
overwrite	bool	Whether a dataset that already has this name may be replaced (a RuntimeWarning is then issued; the weight column is replaced too).	False

Returns:

Type	Description
self	The evaluator itself, to allow method chaining.

Raises:

Type	Description
KeyError	If a dataset named name already exists and overwrite is False.

Source code in Modeling_Tool/Eval/Model_Eval_Tool.py

def add_dataset(self, name, data, weight_col = None, overwrite = False):
    """
    Add a dataset.

    Parameters
    ----------
    name : str
        Name of the dataset (e.g. 'train', 'validation', 'oot').
    data : pandas.DataFrame
        Dataset.
    weight_col : str, optional
        Sample weight column of this dataset (must be in ``data``); it takes precedence over the weight column given to
        ``evaluate`` and to the constructor. Default is None.
    overwrite : bool, default False
        Whether a dataset that already has this ``name`` may be replaced (a ``RuntimeWarning`` is then issued; the
        weight column is replaced too).

    Returns
    -------
    self
        The evaluator itself, to allow method chaining.

    Raises
    ------
    KeyError
        If a dataset named ``name`` already exists and ``overwrite`` is False.
    """
    if name in self.datasets and not overwrite:
        raise KeyError(
            f"Dataset name {name!r} already exists. Use overwrite=True to replace it, "
            "or pass a different dataset name."
        )
    if name in self.datasets and overwrite:
        warnings.warn(
            f"PerformanceEvaluator.add_dataset overwriting existing dataset {name!r}.",
            RuntimeWarning,
            stacklevel=2,
        )
    self.datasets[name] = data
    self.dataset_weight_cols[name] = weight_col
    return self

evaluate

evaluate(oot_grp_name=None, min_data_size=100, grp_colname=None, fig_save_path=None, rpt_save_path=None, to_show=False, display=True, gains_table=False, benchmark_dataset=None, weight_col=None)

Run the performance evaluation.

Parameters:

Name	Type	Description	Default
oot_grp_name	str	Name of the grouping column. Every added dataset that has this column (not only the OOT data) is evaluated separately for each group value, and the datasets without it are skipped. Weights and fig_save_path are ignored in this mode.	None
min_data_size	int	Minimum number of samples per group.	100
grp_colname	str	Name of the group column in the output. Default is None, i.e. oot_grp_name.	None
fig_save_path	str	Path to save the figure. It is ignored when oot_grp_name is given, and on the weighted path the figure is only drawn when it is set.	None
rpt_save_path	str	Path to save the report (a CSV file of the summary table).	None
to_show	bool	Whether to display the figures.	False
display	bool	Whether to show the result table with IPython.display.display (it requires IPython).	True
gains_table	bool	Whether the percentile panel and the Top/Btm target rates are built from the Gains-table binning (the gains_table argument of evaluate_performance). The Gains-table summary columns are added either way on the unweighted path.	False
benchmark_dataset	str or DataFrame	Benchmark dataset used to fix the bin boundaries. If a str is passed, the dataset is looked up by name among the datasets added with add_dataset; if a DataFrame is passed, it is used directly. The default None means that each dataset is binned independently. The weights are ignored when it is given.	None
weight_col	str	Sample weight column for this call; it overrides the weight_col of the evaluator, while a column given to add_dataset takes precedence over both. Weights are applied only when oot_grp_name and benchmark_dataset are None and tgt_name is a single name; otherwise they are silently ignored (when tgt_name is a list or tuple, this argument is ignored, but the weight columns of the evaluator and of the datasets are still honored).	None

Returns:

Type

Description

DataFrame

Performance evaluation summary table. If tgt_name of the instance is a list/tuple of several labels, each label is evaluated separately and the results are stacked vertically after a new tgt_name column is added; when to_show=True, one figure is drawn for each label, and fig_save_path automatically gets a label suffix (e.g. perf.png -> perf_<label>.png). On the weighted path the table is narrower: index, dataset, DATASET, AUC, KS, LIFT, IV, N (sum of the weights), N_RAW, avgTrue and avgScore (plus N_SPECIAL and N_SPECIAL_RAW when spec_values match rows). An empty DataFrame is returned when no dataset was added, or when neither scr_name nor model with feature_cols is set (a DeprecationWarning is issued then).

Raises:

Type	Description
ValueError	If benchmark_dataset is a name that was not added with add_dataset, or if no bin edges can be derived from it.

Notes

The attribute evaluate_status records the outcome of the call ('ok', 'no_datasets', 'compute_failed' or 'empty_input').

Source code in Modeling_Tool/Eval/Model_Eval_Tool.py

def evaluate(self, oot_grp_name = None, min_data_size = 100, grp_colname = None,
             fig_save_path = None, rpt_save_path = None, to_show = False, 
             display = True, gains_table = False, benchmark_dataset = None,
             weight_col = None):
    """
    Run the performance evaluation.

    Parameters
    ----------
    oot_grp_name : str, optional
        Name of the grouping column. Every added dataset that has this column (not only the OOT data) is evaluated
        separately for each group value, and the datasets without it are skipped. Weights and ``fig_save_path`` are
        ignored in this mode.
    min_data_size : int, default 100
        Minimum number of samples per group.
    grp_colname : str, optional
        Name of the group column in the output. Default is None, i.e. ``oot_grp_name``.
    fig_save_path : str, optional
        Path to save the figure. It is ignored when ``oot_grp_name`` is given, and on the weighted path the figure is
        only drawn when it is set.
    rpt_save_path : str, optional
        Path to save the report (a CSV file of the summary table).
    to_show : bool, default False
        Whether to display the figures.
    display : bool, default True
        Whether to show the result table with ``IPython.display.display`` (it requires IPython).
    gains_table : bool, default False
        Whether the percentile panel and the Top/Btm target rates are built from the Gains-table binning (the
        ``gains_table`` argument of ``evaluate_performance``). The Gains-table summary columns are added either way on
        the unweighted path.
    benchmark_dataset : str or pandas.DataFrame, optional
        Benchmark dataset used to fix the bin boundaries. If a str is passed, the dataset is looked up by name
        among the datasets added with ``add_dataset``; if a DataFrame is passed, it is used directly.
        The default None means that each dataset is binned independently. The weights are ignored when it is given.
    weight_col : str, optional
        Sample weight column for this call; it overrides the ``weight_col`` of the evaluator, while a column given to
        ``add_dataset`` takes precedence over both. Weights are applied only when ``oot_grp_name`` and
        ``benchmark_dataset`` are None and ``tgt_name`` is a single name; otherwise they are silently ignored (when
        ``tgt_name`` is a list or tuple, this argument is ignored, but the weight columns of the evaluator and of the
        datasets are still honored).

    Returns
    -------
    pandas.DataFrame
        Performance evaluation summary table. If ``tgt_name`` of the instance is a list/tuple of several labels,
        each label is evaluated separately and the results are stacked vertically after a new ``tgt_name`` column
        is added; when ``to_show=True``, one figure is drawn for each label, and
        ``fig_save_path`` automatically gets a label suffix (e.g. ``perf.png`` -> ``perf_<label>.png``).
        On the weighted path the table is narrower: ``index``, ``dataset``, ``DATASET``, ``AUC``, ``KS``, ``LIFT``,
        ``IV``, ``N`` (sum of the weights), ``N_RAW``, ``avgTrue`` and ``avgScore`` (plus ``N_SPECIAL`` and
        ``N_SPECIAL_RAW`` when ``spec_values`` match rows). An empty DataFrame is returned when no dataset was added,
        or when neither ``scr_name`` nor ``model`` with ``feature_cols`` is set (a ``DeprecationWarning`` is issued
        then).

    Raises
    ------
    ValueError
        If ``benchmark_dataset`` is a name that was not added with ``add_dataset``, or if no bin edges can be derived
        from it.

    Notes
    -----
    The attribute ``evaluate_status`` records the outcome of the call (``'ok'``, ``'no_datasets'``,
    ``'compute_failed'`` or ``'empty_input'``).
    """
    if len(self.datasets) == 0:
        self.evaluate_status = "no_datasets"
        return pd.DataFrame([])

    if self.scr_name is None and self.model is None and self.feature_cols is None:
        self.evaluate_status = "compute_failed"
        warnings.warn(
            "PerformanceEvaluator.evaluate() now returns an empty DataFrame instead of -1; "
            "inspect .evaluate_status for the failure reason.",
            DeprecationWarning,
            stacklevel=2,
        )
        return pd.DataFrame([])

    if self.scr_name is None and self.model is None:
        self.evaluate_status = "compute_failed"
        warnings.warn(
            "PerformanceEvaluator.evaluate() now returns an empty DataFrame instead of -2; "
            "inspect .evaluate_status for the failure reason.",
            DeprecationWarning,
            stacklevel=2,
        )
        return pd.DataFrame([])

    if self.scr_name is None and self.feature_cols is None:
        self.evaluate_status = "compute_failed"
        warnings.warn(
            "PerformanceEvaluator.evaluate() now returns an empty DataFrame instead of -3; "
            "inspect .evaluate_status for the failure reason.",
            DeprecationWarning,
            stacklevel=2,
        )
        return pd.DataFrame([])

    active_weight_col = weight_col or self.weight_col
    has_dataset_weight = any(v is not None for v in self.dataset_weight_cols.values())
    if (active_weight_col is not None or has_dataset_weight) and oot_grp_name is None and benchmark_dataset is None and not isinstance(self.tgt_name, (list, tuple)):
        rows = []
        for name, data in self.datasets.items():
            if data is None:
                continue
            wc = self.dataset_weight_cols.get(name) or active_weight_col
            work_data = data.copy()
            scr_name = self.scr_name
            if scr_name is None:
                scr_name = "_mdl_scr"
                work_data[scr_name] = self.model.predict_proba(work_data.loc[:, self.feature_cols])[:, 1]
            rows.append(
                _weighted_eval.dataset_summary(
                    name,
                    work_data,
                    self.tgt_name,
                    scr_name,
                    weight_col=wc,
                    nbins=self._gains_nbins(),
                    ascending=bool(self.ascending) if self.ascending is not None else False,
                    spec_values=self.spec_values or None,
                )
            )
        fnl_df = pd.DataFrame(rows)
        if display:
            from IPython.display import display as _ipy_display
            _ipy_display(fnl_df)
        if fig_save_path:
            eval_datasets = {}
            for name, data in self.datasets.items():
                if data is None:
                    continue
                wc = self.dataset_weight_cols.get(name) or active_weight_col
                work_data = data.copy()
                scr_name = self.scr_name
                if scr_name is None:
                    scr_name = "_mdl_scr"
                    work_data[scr_name] = self.model.predict_proba(work_data.loc[:, self.feature_cols])[:, 1]
                resolved_weight = _weighted_eval.resolve_weights(
                    work_data,
                    weight_col=wc,
                    expected_len=len(work_data),
                )
                y_true = work_data[self.tgt_name].to_numpy()
                y_score = work_data[scr_name].to_numpy()
                if self.spec_values:
                    keep = ~pd.Series(y_score).isin(self.spec_values).to_numpy()
                    y_true = y_true[keep]
                    y_score = y_score[keep]
                    if resolved_weight is not None:
                        resolved_weight = np.asarray(resolved_weight)[keep]
                payload = {
                    "y_true": y_true,
                    "y_score": y_score,
                }
                if resolved_weight is not None:
                    payload["sample_weight"] = resolved_weight
                eval_datasets[name] = payload
            if eval_datasets:
                evaluate_performance(
                    datasets=eval_datasets,
                    dist_bins=self.dist_bins,
                    pct_bins=self.pct_bins,
                    square_figsize=5,
                    to_show=to_show,
                    save_path=fig_save_path,
                    gains_table=gains_table,
                    equal_freq=self.equal_freq,
                    ascending=self.ascending,
                )
        if rpt_save_path:
            fnl_df.to_csv(rpt_save_path, index=False)
        self.evaluate_status = "ok" if not fnl_df.empty else "empty_input"
        return fnl_df

    # ── Multiple y labels: when tgt_name is a list/tuple, evaluate label by label, then stack the results vertically ──
    #    Table:   the result of each label gets a new tgt_name column, then they are stacked vertically;
    #    Figures: each label gets its own figure (shown in a loop when to_show=True; fig_save_path automatically gets a label suffix).
    if isinstance(self.tgt_name, (list, tuple)):
        import os as _os

        def _suffix_path(p, lab):
            if not p:
                return None
            root, ext = _os.path.splitext(str(p))
            return "{0}_{1}{2}".format(root, lab, ext)

        _orig_tgt = self.tgt_name
        multi_results = []
        try:
            for _t in list(self.tgt_name):
                self.tgt_name = _t
                sub_df = self.evaluate(
                    oot_grp_name = oot_grp_name,
                    min_data_size = min_data_size,
                    grp_colname = grp_colname,
                    fig_save_path = _suffix_path(fig_save_path, _t),
                    rpt_save_path = None,        # saved once after stacking (see below)
                    to_show = to_show,           # each label gets its own figure
                    display = False,             # displayed once after stacking
                    gains_table = gains_table,
                    benchmark_dataset = benchmark_dataset,
                )
                if isinstance(sub_df, pd.DataFrame) and sub_df.shape[0] > 0:
                    sub_df.insert(0, "tgt_name", _t)
                    multi_results.append(sub_df)
        finally:
            self.tgt_name = _orig_tgt

        fnl_df = pd.concat(multi_results, ignore_index = True) if multi_results else pd.DataFrame([])

        if display:
            from IPython.display import display as _ipy_display
            _ipy_display(fnl_df)
        if rpt_save_path:
            fnl_df.to_csv(rpt_save_path, index = False)
        self.evaluate_status = "ok" if not fnl_df.empty else "empty_input"
        return fnl_df

    def _get_score(data):
        if self.scr_name is not None:
            return data[self.scr_name]
        return self.model.predict_proba(data.loc[:, self.feature_cols])[:, 1]

    def _rank_payload(data):
        """(y_true, y_score) with special sentinel scores removed.

        Sentinels like the all-missing override -1 carry no ordering
        information: AUC/KS/distribution figures are computed on the
        non-special rows; their bin-level accounting lives in the gains
        table via spec_values.
        """
        y_true = data[self.tgt_name]
        y_score = _get_score(data)
        if not self.spec_values:
            return y_true, y_score
        keep = ~pd.Series(np.asarray(y_score)).isin(self.spec_values).to_numpy()
        return np.asarray(y_true)[keep], np.asarray(y_score)[keep]

    def _get_benchmark_bin_edges():
        if benchmark_dataset is None:
            return None

        if isinstance(benchmark_dataset, str):
            if benchmark_dataset not in self.datasets:
                raise ValueError("benchmark_dataset must be one of added datasets: {0}".format(list(self.datasets.keys())))
            benchmark_data = self.datasets[benchmark_dataset]
        else:
            benchmark_data = benchmark_dataset

        benchmark_score = "_benchmark_score"
        benchmark_df = pd.DataFrame({
            self.tgt_name: benchmark_data[self.tgt_name],
            benchmark_score: _get_score(benchmark_data)
        })

        _, benchmark_bin_edges = super_binning(
            data = benchmark_df,
            score = benchmark_score,
            dep = self.tgt_name,
            nbins = self._gains_nbins(),
            precision = self.precision,
            min_bin_prop = self.min_bin_prop,
            include_missing = self.include_missing,
            equal_freq = self.equal_freq,
            chi2_method = self.chi2_method,
            chi2_p = self.chi2_p,
            init_equi_bins = self.init_equi_bins,
            tree_binning = self.tree_binning,
            random_state = self.random_state,
            return_edges = True,
            ascending = True if self.ascending is None else bool(self.ascending)
        )

        if benchmark_bin_edges is None or len(benchmark_bin_edges) < 2:
            raise ValueError("Cannot generate benchmark bin edges from benchmark_dataset.")

        return list(benchmark_bin_edges)

    benchmark_bin_edges = _get_benchmark_bin_edges()

    def _evaluate_dataset_dict(dataset_dict, save_path = None):
        eval_datasets = {}
        for name, data in dataset_dict.items():
            if data is None:
                continue
            y_true, y_score = _rank_payload(data)
            eval_datasets[name] = {
                "y_true": y_true,
                "y_score": y_score
            }

        if len(eval_datasets) == 0:
            self.evaluate_status = "empty_input"
            return pd.DataFrame([])

        model_eval_result_df = evaluate_performance(
            datasets = eval_datasets,
            dist_bins = self.dist_bins,
            pct_bins = self.pct_bins,
            square_figsize = 5,
            to_show = to_show,
            save_path = save_path,
            gains_table = gains_table,
            equal_freq = self.equal_freq,
            pct_bin_edges = benchmark_bin_edges,
            ascending = self.ascending,
        )

        import re
        btm_cols = [x for x in model_eval_result_df.columns if x.startswith("Btm")]
        top_cols = [x for x in model_eval_result_df.columns if x.startswith("Top")]

        if btm_cols and top_cols:
            btm_str = btm_cols[0]
            top_str = top_cols[0]
            numbers = re.findall(r'\d+', btm_str)
            if numbers:
                quantile = int(numbers[0])
            else:
                quantile = self.pct_bins

            model_eval_result_df[f"Btm{quantile}%_Lift"] = model_eval_result_df[btm_str] / model_eval_result_df["avgTrue"]
            model_eval_result_df[f"Top{quantile}%_Lift"] = model_eval_result_df[top_str] / model_eval_result_df["avgTrue"]
            model_eval_result_df["AUC_Shift"] = model_eval_result_df["AUC"].shift(1) / model_eval_result_df["AUC"] - 1
            model_eval_result_df["KS_Shift"] = model_eval_result_df["KS"].shift(1) / model_eval_result_df["KS"] - 1

        if self.spec_values:
            # the ranking columns above describe the rows with a real score; N and avgTrue describe every row and the
            # sentinel part is reported apart, as on the weighted path (they used to leave the sentinel rows out)
            all_rows = {}
            for name, data in dataset_dict.items():
                if data is None:
                    continue
                y_score = np.asarray(_get_score(data))
                n_special = int(pd.Series(y_score).isin(self.spec_values).sum())
                avg_true = _weighted_eval.safe_weighted_average(np.asarray(data[self.tgt_name]), None)
                all_rows[name] = (len(data), avg_true, n_special)
            if "index" in model_eval_result_df.columns:
                names = model_eval_result_df["index"]
                model_eval_result_df["N"] = names.map(lambda key: all_rows.get(key, (np.nan,))[0])
                model_eval_result_df["avgTrue"] = names.map(lambda key: all_rows.get(key, (np.nan, np.nan))[1])
                if any(item[2] for item in all_rows.values()):
                    n_special = names.map(lambda key: all_rows.get(key, (0, 0, 0))[2])
                    model_eval_result_df["N_SPECIAL"] = n_special.where(n_special > 0).astype(float)
                    model_eval_result_df["N_SPECIAL_RAW"] = n_special.where(n_special > 0)

        gains_table_cols = ['N_BUMP', 'MIN_RISK_DEP', 'MAX_RISK_DEP', 'KS_IN_GAINS', 'LIFT_IN_GAINS', 'IV', 'N_BINS']
        gains_summ_list = []
        for name, data in dataset_dict.items():
            if data is None:
                gains_res = pd.DataFrame([], columns = gains_table_cols)
            else:
                gains_res = get_gains_table(
                    data = data,
                    dep = self.tgt_name,
                    nbins = benchmark_bin_edges if benchmark_bin_edges is not None else self._gains_nbins(),
                    precision = self.precision,
                    min_bin_prop = self.min_bin_prop,
                    include_missing = self.include_missing,
                    score = self.scr_name,
                    equal_freq = self.equal_freq,
                    chi2_method = False if benchmark_bin_edges is not None else self.chi2_method,
                    model = self.model,
                    varlist = self.feature_cols,
                    init_equi_bins = self.init_equi_bins,
                    chi2_p = self.chi2_p,
                    retSummary = True,
                    tree_binning = False if benchmark_bin_edges is not None else self.tree_binning,
                    random_state = self.random_state,
                    **self._governed_eval_kwargs(),
                )
            gains_res['index'] = name
            gains_summ_list.append(gains_res)

        if gains_summ_list:
            gains_summ = concat_non_empty(gains_summ_list)
            model_eval_result_df = model_eval_result_df.merge(gains_summ, on = ['index'], how = 'left')

        return model_eval_result_df

    if oot_grp_name is None:
        fnl_df = _evaluate_dataset_dict(self.datasets, save_path = fig_save_path)
    else:
        if grp_colname is None:
            grp_colname = oot_grp_name

        grouped_results = []
        for dataset_name, data in self.datasets.items():
            if data is None or oot_grp_name not in data.columns:
                continue

            grp_list = data[oot_grp_name].sort_values(ascending = True).unique().tolist()
            valid_grp_list = [x for x in grp_list if data[data[oot_grp_name].isin([x])].shape[0] >= min_data_size]

            for grp in valid_grp_list:
                data_grp = data[data[oot_grp_name].isin([grp])]
                perf_res = _evaluate_dataset_dict({dataset_name: data_grp}, save_path = None)
                if perf_res.shape[0] > 0:
                    perf_res[grp_colname] = grp
                    grouped_results.append(perf_res)

        if len(grouped_results) == 0:
            fnl_df = pd.DataFrame([])
        else:
            fnl_df = pd.concat(grouped_results, ignore_index = True)

    if display:
        from IPython.display import display
        display(fnl_df)

    if rpt_save_path:
        fnl_df.to_csv(rpt_save_path, index = False)

    self.evaluate_status = "ok" if isinstance(fnl_df, pd.DataFrame) and not fnl_df.empty else "empty_input"
    return fnl_df

get_gains_table

get_gains_table(data, dep, nbins=10, precision=5, min_bin_prop=0.05, include_missing=True, score=None, model=None, varlist=None, equal_freq=True, chi2_method=False, grp_name=None, min_data_size=100, grp_colname=None, sync_range=True, chi2_p=0.95, init_equi_bins=100, fillna=-999999, spec_values=[], retSummary=False, tree_binning=False, random_state=42, ascending=False, withSummary=False, wholeGroup=False, add_func=None, weight_col=None, weighted_binning=None)

Compute the grouped Gains table.

Split the data into groups by the grouping column and compute the Gains table of each group separately. If no grouping column is specified, compute the overall Gains table.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input data table.	required
dep	str	Name of the target variable.	required
nbins	int or list	Number of bins, or an explicit list of bin edges.	10
precision	int	Precision of the bin boundary values.	5
min_bin_prop	float	Minimum proportion of samples per bin.	0.05
include_missing	bool	Whether missing scores are binned. True fills them with fillna and gives them a bin of their own, (-inf, fillna] (a score already equal to fillna joins it); the other bins are computed from the real scores. False leaves the rows with a missing score out.	True
score	str	Name of the score column.	None
model	sklearn-like model	Machine learning model.	None
varlist	list	List of model features.	None
equal_freq	bool	True for equal-frequency binning.	True
chi2_method	bool	Whether to use chi-square binning.	False
grp_name	str	Name of the grouping column.	None
min_data_size	int	Minimum number of samples per group.	100
grp_colname	str	Name of the group column in the output.	None
sync_range	bool	Whether to synchronize the bin boundaries across groups.	True
chi2_p	float	Significance level of the chi-square test.	0.95
init_equi_bins	int	Initial number of equal-frequency bins.	100
fillna	any	Fill value for missing values.	-999999
spec_values	list	List of special values.	[]
retSummary	bool	Whether to return only the summary metrics.	False
tree_binning	bool	Whether to use decision-tree binning.	False
random_state	int	Random seed.	42
ascending	bool	Whether the bin order is ascending.	False
withSummary	bool	Whether to include an overall summary row.	False
wholeGroup	bool	Whether to use all the data for binning.	False
add_func	callable	Custom statistics function. It receives the rows of one bin as a DataFrame (all the columns of data plus the bin columns _bin_num and _bin_range) and returns a Series whose values become extra columns of the table. On the weighted path it is applied to the rows of every row of the table the same way; the weight column is one of the columns it sees.	None
weight_col	str	Name of the sample weight column; when provided and grp_name is not given, the Gains table is aggregated by weight (N in the output is the sum of the weights and N_RAW is the number of rows). It is ignored when grp_name is given.	None
weighted_binning	bool	Accepted for compatibility but without any effect: the weighted Gains table always uses equal-frequency bins by cumulative weight, whatever its value.	None

Returns:

Type

Description

DataFrame

Grouped Gains table. With retSummary=True a one-row summary (N_BUMP, MIN_RISK_DEP, MAX_RISK_DEP, KS_IN_GAINS, LIFT_IN_GAINS, IV, N_BINS) instead, and with grp_name a table that is empty when no group has at least min_data_size rows. When grp_name is None and neither score nor a model together with varlist is given, the integer -1 (nothing given), -2 (varlist without model) or -3 (model without varlist) is returned instead of an error.

Notes

With weight_col and without grp_name the call is delegated to the weighted implementation, which only uses nbins, the score (score, or model with varlist), ascending, include_missing, add_func, withSummary and retSummary: precision, min_bin_prop, equal_freq, chi2_method, chi2_p, init_equi_bins, fillna, spec_values, tree_binning and random_state are ignored. The weighted table has the bins 1 to nbins, each holding about 1 / nbins of the total weight of the rows with a score; rows with a missing score are left out, or reported in a Missing row with include_missing=True, and rows with a missing target count in N but not in PERF_CNT or the bad and good counts. Its Grand Summary row (withSummary=True) covers all the rows of the table (see weighted_eval_utils.get_gains_table). With grp_name the weights are ignored and withSummary is forced to False.

Source code in Modeling_Tool/Eval/Model_Eval_Tool.py

def get_gains_table(data, dep, nbins = 10, precision = 5, min_bin_prop = 0.05, include_missing = True, 
                    score = None, model = None, varlist = None, equal_freq = True, chi2_method = False,
                    grp_name = None, min_data_size = 100, grp_colname = None, sync_range = True,
                    chi2_p = 0.95, init_equi_bins = 100, fillna = -999999, spec_values = [], retSummary = False,
                    tree_binning = False, random_state=42, ascending = False, withSummary = False, wholeGroup = False, 
                    add_func = None, weight_col = None, weighted_binning = None):
    """
    Compute the grouped Gains table.

    Split the data into groups by the grouping column and compute the Gains table of each group separately.
    If no grouping column is specified, compute the overall Gains table.

    Parameters
    ----------
    data : pandas.DataFrame
        Input data table.
    dep : str
        Name of the target variable.
    nbins : int or list, default 10
        Number of bins, or an explicit list of bin edges.
    precision : int, default 5
        Precision of the bin boundary values.
    min_bin_prop : float, default 0.05
        Minimum proportion of samples per bin.
    include_missing : bool, default True
        Whether missing scores are binned. True fills them with ``fillna`` and gives them a bin of their own, ``(-inf,
        fillna]`` (a score already equal to ``fillna`` joins it); the other bins are computed from the real scores.
        False leaves the rows with a missing score out.
    score : str, optional
        Name of the score column.
    model : sklearn-like model, optional
        Machine learning model.
    varlist : list, optional
        List of model features.
    equal_freq : bool, default True
        True for equal-frequency binning.
    chi2_method : bool, default False
        Whether to use chi-square binning.
    grp_name : str, optional
        Name of the grouping column.
    min_data_size : int, default 100
        Minimum number of samples per group.
    grp_colname : str, optional
        Name of the group column in the output.
    sync_range : bool, default True
        Whether to synchronize the bin boundaries across groups.
    chi2_p : float, default 0.95
        Significance level of the chi-square test.
    init_equi_bins : int, default 100
        Initial number of equal-frequency bins.
    fillna : any, default -999999
        Fill value for missing values.
    spec_values : list, default []
        List of special values.
    retSummary : bool, default False
        Whether to return only the summary metrics.
    tree_binning : bool, default False
        Whether to use decision-tree binning.
    random_state : int, default 42
        Random seed.
    ascending : bool, default False
        Whether the bin order is ascending.
    withSummary : bool, default False
        Whether to include an overall summary row.
    wholeGroup : bool, default False
        Whether to use all the data for binning.
    add_func : callable, optional
        Custom statistics function. It receives the rows of one bin as a DataFrame (all the columns of ``data`` plus the bin
        columns ``_bin_num`` and ``_bin_range``) and returns a Series whose values become extra columns of the table.
        On the weighted path it is applied to the rows of every row of the table the same way; the weight column is one
        of the columns it sees.
    weight_col : str, optional
        Name of the sample weight column; when provided and ``grp_name`` is not given, the Gains table is
        aggregated by weight (``N`` in the output is the sum of the weights and ``N_RAW`` is the number of rows).
        It is ignored when ``grp_name`` is given.
    weighted_binning : bool, optional
        Accepted for compatibility but without any effect: the weighted Gains table always uses equal-frequency bins by
        cumulative weight, whatever its value.

    Returns
    -------
    pandas.DataFrame
        Grouped Gains table. With ``retSummary=True`` a one-row summary (``N_BUMP``, ``MIN_RISK_DEP``, ``MAX_RISK_DEP``,
        ``KS_IN_GAINS``, ``LIFT_IN_GAINS``, ``IV``, ``N_BINS``) instead, and with ``grp_name`` a table that is empty when
        no group has at least ``min_data_size`` rows. When ``grp_name`` is None and neither ``score`` nor a ``model``
        together with ``varlist`` is given, the integer -1 (nothing given), -2 (``varlist`` without ``model``) or -3
        (``model`` without ``varlist``) is returned instead of an error.

    Notes
    -----
    With ``weight_col`` and without ``grp_name`` the call is delegated to the weighted implementation, which only uses
    ``nbins``, the score (``score``, or ``model`` with ``varlist``), ``ascending``, ``include_missing``, ``add_func``,
    ``withSummary`` and ``retSummary``: ``precision``, ``min_bin_prop``, ``equal_freq``, ``chi2_method``, ``chi2_p``,
    ``init_equi_bins``, ``fillna``, ``spec_values``, ``tree_binning`` and ``random_state`` are ignored. The weighted table has
    the bins 1 to ``nbins``, each holding about ``1 / nbins`` of the total weight of the rows with a score; rows with a
    missing score are left out, or reported in a ``Missing`` row with ``include_missing=True``, and rows with a missing
    target count in ``N`` but not in ``PERF_CNT`` or the bad and good counts. Its ``Grand Summary`` row
    (``withSummary=True``) covers all the rows of the table (see ``weighted_eval_utils.get_gains_table``). With
    ``grp_name`` the weights are ignored and ``withSummary`` is forced to False.
    """

    if weight_col is not None and grp_name is None:
        work_data = data.copy()
        work_score = score
        if work_score is None:
            if model is None or varlist is None:
                return _get_gains_table_single(
                    data=data,
                    dep=dep,
                    nbins=nbins,
                    precision=precision,
                    min_bin_prop=min_bin_prop,
                    include_missing=include_missing,
                    score=score,
                    model=model,
                    varlist=varlist,
                    equal_freq=equal_freq,
                    chi2_method=chi2_method,
                    chi2_p=chi2_p,
                    init_equi_bins=init_equi_bins,
                    fillna=fillna,
                    spec_values=spec_values,
                    retSummary=retSummary,
                    tree_binning=tree_binning,
                    random_state=random_state,
                    ascending=ascending,
                    withSummary=withSummary,
                    add_func=add_func,
                )
            work_score = "_mdl_scr"
            work_data[work_score] = model.predict_proba(work_data.loc[:, varlist])[:, 1]
        gains = _weighted_eval.get_gains_table(
            work_data,
            dep,
            work_score,
            nbins=nbins,
            weight_col=weight_col,
            weighted_binning=weighted_binning,
            ascending=ascending,
            include_missing=include_missing,
            add_func=None if retSummary else add_func,
            withSummary=withSummary and not retSummary,
        )
        if retSummary:
            return pd.DataFrame({
                "N_BUMP": [gains["RANK_ORDER_BUMP"].sum()],
                "MIN_RISK_DEP": [gains["TRUE_BAD_SHIFT"].round(4).min()],
                "MAX_RISK_DEP": [gains["TRUE_BAD_SHIFT"].round(4).max()],
                "KS_IN_GAINS": [gains["KS_PER_BIN"].round(4).max()],
                "LIFT_IN_GAINS": [gains["LIFT"].round(4).max()],
                "IV": [gains["IV"].replace([np.inf, -np.inf], 0).sum()],
                "N_BINS": [gains.shape[0]],
            })
        return gains

    if grp_colname is None:
        grp_colname = grp_name

    if grp_name is None:
        fnl_df = _get_gains_table_single(data = data, 
                                         dep = dep, 
                                         nbins = nbins, 
                                         precision = precision, 
                                         min_bin_prop = min_bin_prop, 
                                         include_missing = include_missing, 
                                         score = score, 
                                         model = model, 
                                         varlist = varlist, 
                                         equal_freq = equal_freq, 
                                         chi2_method = chi2_method,
                                         chi2_p = chi2_p, 
                                         init_equi_bins = init_equi_bins, 
                                         fillna = fillna, 
                                         spec_values = spec_values,
                                         retSummary = retSummary, 
                                         tree_binning = tree_binning, 
                                         random_state = random_state,
                                         ascending = ascending,
                                         withSummary = withSummary,
                                         add_func = add_func)
        return fnl_df

    withSummary = False
    # Decide the missing bin once from all groups: a group-level decision would number the bins of a group with
    # missing scores differently from the others
    if score is not None and score in data.columns:
        spec_values = _with_missing_bin(data[score], include_missing, fillna, spec_values)
    grp_list = data[grp_name].sort_values(ascending = True).unique().tolist()
    valid_grp_list = [x for x in grp_list if data[data[grp_name].isin([x])].shape[0] >= min_data_size]

    if len(valid_grp_list) == 0:
        fnl_df = _get_gains_table_single(data = data, 
                                         dep = dep, 
                                         nbins = nbins, 
                                         precision = precision, 
                                         min_bin_prop = min_bin_prop, 
                                         include_missing = include_missing, 
                                         score = score, 
                                         model = model, 
                                         varlist = varlist, 
                                         equal_freq = equal_freq, 
                                         chi2_method = chi2_method,
                                         chi2_p = chi2_p, 
                                         init_equi_bins = init_equi_bins, 
                                         fillna = fillna, 
                                         spec_values = spec_values,
                                         retSummary = retSummary, 
                                         tree_binning = tree_binning, 
                                         random_state = random_state,
                                         ascending = ascending,
                                         withSummary = withSummary,
                                         add_func = add_func)
        fnl_df[grp_colname] = np.nan
        col_index = pd.MultiIndex.from_tuples([], names=['_bin_num', '_bin_range'])
        fnl_df = pd.DataFrame([], columns = fnl_df.columns, index = col_index)
        return fnl_df

    first_grp = valid_grp_list[0]
    data_grp = data[data[grp_name].isin([first_grp])]

    grp_for_binning = data_grp if not wholeGroup else data

    if sync_range and retSummary:

        full_gains = _get_gains_table_single(data = grp_for_binning, 
                                            dep = dep, 
                                            nbins = nbins, 
                                            precision = precision, 
                                            min_bin_prop = min_bin_prop, 
                                            include_missing = include_missing, 
                                            score = score, 
                                            model = model, 
                                            varlist = varlist, 
                                            equal_freq = equal_freq, 
                                            chi2_method = chi2_method,
                                            chi2_p = chi2_p, 
                                            init_equi_bins = init_equi_bins, 
                                            fillna = fillna, 
                                            spec_values = spec_values,
                                            retSummary = False, 
                                            tree_binning = tree_binning, 
                                            random_state = random_state,
                                            ascending = ascending,
                                            withSummary = withSummary,
                                            add_func = add_func)

        bin_range = get_bin_range_list(full_gains.reset_index())
        bin_range = bin_range + [-np.inf, np.inf]
        bin_range = sorted(list(set(bin_range)))

        fnl_df = _get_gains_table_single(data = data_grp, 
                                         dep = dep, 
                                         nbins = nbins, 
                                         precision = precision, 
                                         min_bin_prop = min_bin_prop, 
                                         include_missing = include_missing, 
                                         score = score, 
                                         model = model, 
                                         varlist = varlist, 
                                         equal_freq = equal_freq, 
                                         chi2_method = chi2_method,
                                         chi2_p = chi2_p, 
                                         init_equi_bins = init_equi_bins, 
                                         fillna = fillna, 
                                         spec_values = spec_values,
                                         retSummary = retSummary, 
                                         tree_binning = tree_binning, 
                                         random_state = random_state,
                                         ascending = ascending,
                                         withSummary = withSummary,
                                         add_func = add_func)

        fnl_df[grp_colname] = first_grp

        nbins = bin_range

        equal_freq = False
        chi2_method = False

    if sync_range and not retSummary:

        fnl_df = _get_gains_table_single(data = data_grp, 
                                        dep = dep, 
                                        nbins = nbins, 
                                        precision = precision, 
                                        min_bin_prop = min_bin_prop, 
                                        include_missing = include_missing, 
                                        score = score, 
                                        model = model, 
                                        varlist = varlist, 
                                        equal_freq = equal_freq, 
                                        chi2_method = chi2_method,
                                        chi2_p = chi2_p, 
                                        init_equi_bins = init_equi_bins, 
                                        fillna = fillna, 
                                        spec_values = spec_values,
                                        retSummary = retSummary, 
                                        tree_binning = tree_binning, 
                                        random_state = random_state,
                                        ascending = ascending,
                                        withSummary = withSummary,
                                        add_func = add_func)

        fnl_df[grp_colname] = first_grp

        nbins = get_bin_range_list(fnl_df.reset_index())
        nbins = nbins + [-np.inf, np.inf]
        nbins = sorted(list(set(nbins)))

        equal_freq = False
        chi2_method = False

    if (not sync_range and not retSummary) or (not sync_range and retSummary):

        fnl_df = _get_gains_table_single(data = data_grp, 
                                         dep = dep, 
                                         nbins = nbins, 
                                         precision = precision, 
                                         min_bin_prop = min_bin_prop, 
                                         include_missing = include_missing, 
                                         score = score, 
                                         model = model, 
                                         varlist = varlist, 
                                         equal_freq = equal_freq, 
                                         chi2_method = chi2_method,
                                         chi2_p = chi2_p, 
                                         init_equi_bins = init_equi_bins, 
                                         fillna = fillna, 
                                         spec_values = spec_values,
                                         retSummary = retSummary, 
                                         tree_binning = tree_binning, 
                                         random_state = random_state,
                                         ascending = ascending,
                                         withSummary = withSummary,
                                         add_func = add_func)

        fnl_df[grp_colname] = first_grp

    i = 1
    while i < len(valid_grp_list):

        grp = valid_grp_list[i]
        data_grp = data[data[grp_name].isin([grp])]

        perf_res = _get_gains_table_single(data = data_grp, 
                                           dep = dep, 
                                           nbins = nbins, 
                                           precision = precision, 
                                           min_bin_prop = min_bin_prop, 
                                           include_missing = include_missing, 
                                           score = score, 
                                           model = model, 
                                           varlist = varlist, 
                                           equal_freq = equal_freq, 
                                           chi2_method = chi2_method,
                                           chi2_p = chi2_p, 
                                           init_equi_bins = init_equi_bins, 
                                           fillna = fillna, 
                                           spec_values = spec_values,
                                           retSummary = retSummary, 
                                           tree_binning = tree_binning, 
                                           random_state = random_state,
                                           ascending = ascending,
                                           withSummary = withSummary,
                                           add_func = add_func)

        perf_res[grp_colname] = grp

        fnl_df = pd.concat([fnl_df, perf_res])
        i += 1

    return fnl_df

get_perf_summary

get_perf_summary(train, validation, oot, tgt_name, scr_name=None, model=None, feature_cols=None, fig_save_path=None, rpt_save_path=None, to_show=False, display=True, dist_bins=20, pct_bins=10, precision=5, min_bin_prop=0.05, include_missing=False, equal_freq=True, chi2_method=False, init_equi_bins=1000, chi2_p=0.9, oot_grp_name=None, min_data_size=100, grp_colname=None, tree_binning=False, random_state=42, gains_table=False, weight_col=None)

Compute the grouped performance evaluation summary.

Evaluate the performance on the training, validation and OOT samples by group: the data are split by the oot_grp_name column and the performance metrics of each group are computed separately.

Parameters:

Name	Type	Description	Default
train	DataFrame	Training dataset.	required
validation	DataFrame	Validation dataset.	required
oot	DataFrame	Out-of-time (OOT) dataset.	required
tgt_name	str	Name of the target variable.	required
scr_name	str	Name of the score column.	None
model	sklearn-like model	Machine learning model.	None
feature_cols	list	List of model features.	None
fig_save_path	str	Path to save the figure.	None
rpt_save_path	str	Path to save the report.	None
to_show	bool	Whether to display the figures.	False
display	bool	Whether to print the results.	True
dist_bins	int	Number of bins for the score distribution.	20
pct_bins	int	Number of percentile bins.	10
precision	int	Precision of the bin boundary values.	5
min_bin_prop	float	Minimum proportion of samples per bin.	0.05
include_missing	bool	Whether to include missing values.	False
equal_freq	bool	True for equal-frequency binning.	True
chi2_method	bool	Whether to use chi-square binning.	False
init_equi_bins	int	Initial number of equal-frequency bins.	1000
chi2_p	float	Significance level of the chi-square test.	0.9
oot_grp_name	str	Name of the grouping column for the OOT data.	None
min_data_size	int	Minimum number of samples per group.	100
grp_colname	str	Name of the group column in the output.	None
tree_binning	bool	Whether to use decision-tree binning.	False
random_state	int	Random seed.	42
gains_table	bool	Whether the percentile panel and the Top/Btm target rates are built from the Gains-table binning (the gains_table argument of evaluate_performance). The Gains-table summary columns are added either way.	False
weight_col	str	Sample weight column in each dataset DataFrame; used for weighted metrics such as AUC/KS when there is no grouping. It requires scr_name (see Notes) and is ignored when oot_grp_name is given.	None

Returns:

Type

Description

DataFrame

Grouped performance evaluation summary table: one row per given dataset (ins, oos, oot) with the columns of evaluate_performance, the Top/Btm lift, AUC_Shift, KS_Shift and the Gains-table summary columns. With oot_grp_name there is one such block per group of the OOT data, labeled in the grp_colname column (the ins and oos rows are repeated in every block), and an empty DataFrame is returned when no group has at least min_data_size rows. Without oot_grp_name, the integer -1, -2 or -3 is returned instead of an error when scr_name and model with feature_cols are missing.

Notes

With weight_col and without oot_grp_name the call is delegated to the weighted implementation: it needs scr_name (with model and feature_cols instead it fails with KeyError), only uses pct_bins as the number of bins, ignores the figure, report, display and binning arguments, and returns a narrower table (index, dataset, DATASET, AUC, KS, LIFT, IV, N, N_RAW, avgTrue, avgScore).

With oot_grp_name the figure, the report and the display are produced once per group, so fig_save_path and rpt_save_path are overwritten by every group: the files only hold the last group (and the report has no group column).

Source code in Modeling_Tool/Eval/Model_Eval_Tool.py

def get_perf_summary(train, validation, oot, tgt_name, 
                     scr_name = None,
                     model = None, 
                     feature_cols = None, 
                     fig_save_path = None, 
                     rpt_save_path = None,
                     to_show = False, 
                     display = True,
                     dist_bins = 20, 
                     pct_bins = 10, 
                     precision = 5, 
                     min_bin_prop = 0.05,
                     include_missing = False, 
                     equal_freq = True,
                     chi2_method = False, 
                     init_equi_bins = 1000, 
                     chi2_p = 0.9,
                     oot_grp_name = None, 
                     min_data_size = 100, 
                     grp_colname = None,
                     tree_binning = False, 
                     random_state = 42,
                     gains_table = False,
                     weight_col = None):
    """
    Compute the grouped performance evaluation summary.

    Evaluate the performance on the training, validation and OOT samples by group: the data are
    split by the ``oot_grp_name`` column and the performance metrics of each group are computed separately.

    Parameters
    ----------
    train : pandas.DataFrame, optional
        Training dataset.
    validation : pandas.DataFrame, optional
        Validation dataset.
    oot : pandas.DataFrame, optional
        Out-of-time (OOT) dataset.
    tgt_name : str
        Name of the target variable.
    scr_name : str, optional
        Name of the score column.
    model : sklearn-like model, optional
        Machine learning model.
    feature_cols : list, optional
        List of model features.
    fig_save_path : str, optional
        Path to save the figure.
    rpt_save_path : str, optional
        Path to save the report.
    to_show : bool, default False
        Whether to display the figures.
    display : bool, default True
        Whether to print the results.
    dist_bins : int, default 20
        Number of bins for the score distribution.
    pct_bins : int, default 10
        Number of percentile bins.
    precision : int, default 5
        Precision of the bin boundary values.
    min_bin_prop : float, default 0.05
        Minimum proportion of samples per bin.
    include_missing : bool, default False
        Whether to include missing values.
    equal_freq : bool, default True
        True for equal-frequency binning.
    chi2_method : bool, default False
        Whether to use chi-square binning.
    init_equi_bins : int, default 1000
        Initial number of equal-frequency bins.
    chi2_p : float, default 0.9
        Significance level of the chi-square test.
    oot_grp_name : str, optional
        Name of the grouping column for the OOT data.
    min_data_size : int, default 100
        Minimum number of samples per group.
    grp_colname : str, optional
        Name of the group column in the output.
    tree_binning : bool, default False
        Whether to use decision-tree binning.
    random_state : int, default 42
        Random seed.
    gains_table : bool, default False
        Whether the percentile panel and the Top/Btm target rates are built from the Gains-table binning (the
        ``gains_table`` argument of ``evaluate_performance``). The Gains-table summary columns are added either way.
    weight_col : str, optional
        Sample weight column in each dataset DataFrame; used for weighted metrics such as AUC/KS when there is no grouping.
        It requires ``scr_name`` (see Notes) and is ignored when ``oot_grp_name`` is given.

    Returns
    -------
    pandas.DataFrame
        Grouped performance evaluation summary table: one row per given dataset (``ins``, ``oos``, ``oot``) with the
        columns of ``evaluate_performance``, the Top/Btm lift, ``AUC_Shift``, ``KS_Shift`` and the Gains-table summary
        columns. With ``oot_grp_name`` there is one such block per group of the OOT data, labeled in the ``grp_colname``
        column (the ``ins`` and ``oos`` rows are repeated in every block), and an empty DataFrame is returned when no group
        has at least ``min_data_size`` rows. Without ``oot_grp_name``, the integer -1, -2 or -3 is returned instead of an
        error when ``scr_name`` and ``model`` with ``feature_cols`` are missing.

    Notes
    -----
    With ``weight_col`` and without ``oot_grp_name`` the call is delegated to the weighted implementation: it needs
    ``scr_name`` (with ``model`` and ``feature_cols`` instead it fails with ``KeyError``), only uses ``pct_bins`` as the
    number of bins, ignores the figure, report, display and binning arguments, and returns a narrower table (``index``,
    ``dataset``, ``DATASET``, ``AUC``, ``KS``, ``LIFT``, ``IV``, ``N``, ``N_RAW``, ``avgTrue``, ``avgScore``).

    With ``oot_grp_name`` the figure, the report and the display are produced once per group, so ``fig_save_path`` and
    ``rpt_save_path`` are overwritten by every group: the files only hold the last group (and the report has no group
    column).
    """

    if weight_col is not None and oot_grp_name is None:
        return _weighted_eval.get_perf_summary(
            train=train,
            validation=validation,
            oot=oot,
            tgt_name=tgt_name,
            scr_name=scr_name,
            weight_col=weight_col,
            nbins=pct_bins,
        )

    if grp_colname is None:
        grp_colname = oot_grp_name

    if oot_grp_name is None:
        fnl_df = _get_perf_summary_single(train = train, 
                                          validation = validation, 
                                          oot = oot, 
                                          scr_name = scr_name,
                                          model = model, 
                                          tgt_name = tgt_name, 
                                          display=display,
                                          feature_cols=feature_cols, 
                                          to_show=to_show,
                                          fig_save_path=fig_save_path,
                                          rpt_save_path=rpt_save_path,
                                          dist_bins=dist_bins, 
                                          pct_bins=pct_bins,
                                          precision = precision, 
                                          min_bin_prop = min_bin_prop,
                                          include_missing = include_missing, 
                                          equal_freq = equal_freq,
                                          chi2_method = chi2_method, 
                                          init_equi_bins = init_equi_bins, 
                                          chi2_p = chi2_p, 
                                          tree_binning = tree_binning,
                                          random_state = random_state,
                                          gains_table = gains_table)
        return fnl_df

    grp_list = oot[oot_grp_name].sort_values(ascending = True).unique().tolist()
    valid_grp_list = [x for x in grp_list if oot[oot[oot_grp_name].isin([x])].shape[0] >= min_data_size]

    if len(valid_grp_list) == 0:
        return pd.DataFrame([])

    first_grp = valid_grp_list[0]
    oot_grp = oot[oot[oot_grp_name].isin([first_grp])]

    fnl_df = _get_perf_summary_single(train = train, 
                                      validation = validation, 
                                      oot = oot_grp, 
                                      scr_name = scr_name,
                                      model = model, 
                                      tgt_name = tgt_name, 
                                      display=display,
                                      feature_cols=feature_cols, 
                                      to_show=to_show,
                                      fig_save_path=fig_save_path,
                                      rpt_save_path=rpt_save_path,
                                      dist_bins=dist_bins, 
                                      pct_bins=pct_bins,
                                      precision = precision, 
                                      min_bin_prop = min_bin_prop,
                                      include_missing = include_missing, 
                                      equal_freq = equal_freq,
                                      chi2_method = chi2_method, 
                                      init_equi_bins = init_equi_bins, 
                                      chi2_p = chi2_p,
                                      tree_binning = tree_binning,
                                      random_state = random_state,
                                      gains_table = gains_table)
    fnl_df[grp_colname] = first_grp

    i = 1
    while i < len(valid_grp_list):

        grp = valid_grp_list[i]
        oot_grp = oot[oot[oot_grp_name].isin([grp])]

#         print(grp)

        perf_res = _get_perf_summary_single(train = train, 
                                           validation = validation, 
                                           oot = oot_grp, 
                                           scr_name = scr_name,
                                           model = model, 
                                           tgt_name = tgt_name, 
                                           display=display,
                                           feature_cols=feature_cols, 
                                           to_show=to_show,
                                           fig_save_path=fig_save_path,
                                           rpt_save_path=rpt_save_path,
                                           dist_bins=dist_bins, 
                                           pct_bins=pct_bins,
                                           precision = precision, 
                                           min_bin_prop = min_bin_prop,
                                           include_missing = include_missing, 
                                           equal_freq = equal_freq,
                                           chi2_method = chi2_method, 
                                           init_equi_bins = init_equi_bins, 
                                           chi2_p = chi2_p,
                                           tree_binning = tree_binning,
                                           random_state = random_state,
                                           gains_table = gains_table)
        perf_res[grp_colname] = grp

        fnl_df = pd.concat([fnl_df, perf_res])
        i += 1

    return fnl_df

cross_risk

cross_risk(data, score_list, dep, nbins, agg_col=None, precision=5, min_bin_prop=0.05, include_missing=False, equal_freq=True, binning_numeric=[True, True], agg_func='mean', chi2_method=False, chi2_p=0.95, init_equi_bins=100, fillna=-999999, spec_values=[], tree_binning=False, random_state=42, weight_col=None, sample_weight=None, wgt_col=None)

Create a cross-risk table.

After binning the two score columns, compute the aggregated risk value of each cross group. Numeric variables can be binned automatically.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input data table.	required
score_list	list	List of score columns, of length 2.	required
dep	str	Name of the target variable.	required
nbins	int or list	Number of bins (an integer or a list of length 2).	required
agg_col	str	Name of the column to aggregate; defaults to dep.	None
precision	int or list	Precision of the bin boundary values.	5
min_bin_prop	float or list	Minimum proportion of samples per bin.	0.05
include_missing	bool	Whether missing scores are binned. True fills them with fillna and gives them a bin of their own, (-inf, fillna] (a score already equal to fillna joins it); the other bins are computed from the real scores. False leaves the rows with a missing score out.	False
equal_freq	bool	True for equal-frequency binning.	True
binning_numeric	(list, tuple or bool)	Whether to bin each of the two columns when it is numeric ([first, second]); a single bool applies to both and None means [True, True].	[True, True]
agg_func	(str, callable, tuple or dict)	Aggregation function. Regular usage is the same as the aggfunc of pandas.crosstab, e.g. 'mean', 'sum', 'count' or a custom function. The ratio of two aggregated columns can also be computed directly: Shorthand form: agg_col=(numerator_col, denominator_col), agg_func='ratio' Tuple form: agg_func=('ratio', numerator_col, denominator_col) Dict form: agg_func={ 'func': 'ratio', 'numerator': numerator_col, 'denominator': denominator_col, 'numerator_agg': 'sum', 'denominator_agg': 'sum', 'return_count': True, 'return_count_pct': True, 'count_name': 'N', 'count_pct_name': 'N_Pct', 'ratio_name': 'ratio', 'valid_only': True } When return_count=True, the columns of the result get an additional level of metric names, containing the ratio matrix and the N matrix; when return_count_pct=True, the matrix of the share of N in each cell is also returned, computed as: count of the current cell / Total count in the bottom-right corner.	'mean'
chi2_method	bool	Whether to use chi-square binning.	False
chi2_p	float	Significance level of the chi-square test.	0.95
init_equi_bins	int	Initial number of equal-frequency bins.	100
fillna	any	Fill value for missing values.	-999999
spec_values	list	List of special values.	[]
tree_binning	bool	Whether to use decision-tree binning.	False
random_state	int	Random seed.	42
weight_col	str	Column in data with per-row sample weights (alias: wgt_col).	None
sample_weight	array - like	Explicit per-row weights; must align with len(data).	None
wgt_col	str	Alias for weight_col.	None

Returns:

Type	Description
DataFrame	Cross-risk table.

Raises:

Type	Description
ValueError	If the ratio syntax is used without a numerator and a denominator (for agg_func='ratio', agg_col must be a two-element list or tuple), if a ratio column is not in data, if weights are combined with anything other than the plain agg_func='mean', or if binning_numeric is a list, tuple or array that does not have two elements.
TypeError	If binning_numeric is not None, a bool, a list, a tuple or a numpy array.

Notes

If the first score column is not binned (it is not numeric, or binning_numeric[0] is False), the helper columns _bin_num1 and _bin_range1 (and _bin_num2 and _bin_range2 when the second score column is not binned either) are added to the DataFrame passed as data.

Examples:

>>> cross_risk(data, score_list=['score1', 'score2'], dep='target', nbins=10)

Source code in Modeling_Tool/Eval/Model_Eval_Tool.py

def cross_risk(data, score_list, dep, nbins, agg_col = None, precision = 5, min_bin_prop = 0.05, include_missing = False, 
               equal_freq = True, binning_numeric = [True, True], agg_func = 'mean', chi2_method = False,
               chi2_p = 0.95, init_equi_bins = 100, fillna = -999999, spec_values = [], 
               tree_binning = False, random_state = 42, weight_col = None, sample_weight = None, wgt_col = None):
    """
    Create a cross-risk table.

    After binning the two score columns, compute the aggregated risk value of each cross group.
    Numeric variables can be binned automatically.

    Parameters
    ----------
    data : pandas.DataFrame
        Input data table.
    score_list : list
        List of score columns, of length 2.
    dep : str
        Name of the target variable.
    nbins : int or list
        Number of bins (an integer or a list of length 2).
    agg_col : str, optional
        Name of the column to aggregate; defaults to ``dep``.
    precision : int or list, default 5
        Precision of the bin boundary values.
    min_bin_prop : float or list, default 0.05
        Minimum proportion of samples per bin.
    include_missing : bool, default False
        Whether missing scores are binned. True fills them with ``fillna`` and gives them a bin of their own, ``(-inf,
        fillna]`` (a score already equal to ``fillna`` joins it); the other bins are computed from the real scores.
        False leaves the rows with a missing score out.
    equal_freq : bool, default True
        True for equal-frequency binning.
    binning_numeric : list, tuple or bool, default [True, True]
        Whether to bin each of the two columns when it is numeric (``[first, second]``); a single bool applies to both
        and None means ``[True, True]``.
    agg_func : str, callable, tuple or dict, default 'mean'
        Aggregation function.

        Regular usage is the same as the ``aggfunc`` of ``pandas.crosstab``, e.g. ``'mean'``,
        ``'sum'``, ``'count'`` or a custom function.

        The ratio of two aggregated columns can also be computed directly:

        1. Shorthand form:
           ``agg_col=(numerator_col, denominator_col), agg_func='ratio'``
        2. Tuple form:
           ``agg_func=('ratio', numerator_col, denominator_col)``
        3. Dict form:
           ``agg_func={
               'func': 'ratio',
               'numerator': numerator_col,
               'denominator': denominator_col,
               'numerator_agg': 'sum',
               'denominator_agg': 'sum',
               'return_count': True,
               'return_count_pct': True,
               'count_name': 'N',
               'count_pct_name': 'N_Pct',
               'ratio_name': 'ratio',
               'valid_only': True
           }``

        When ``return_count=True``, the columns of the result get an additional level of metric names,
        containing the ratio matrix and the N matrix; when ``return_count_pct=True``,
        the matrix of the share of N in each cell is also returned, computed as:
        count of the current cell / Total count in the bottom-right corner.
    chi2_method : bool, default False
        Whether to use chi-square binning.
    chi2_p : float, default 0.95
        Significance level of the chi-square test.
    init_equi_bins : int, default 100
        Initial number of equal-frequency bins.
    fillna : any, default -999999
        Fill value for missing values.
    spec_values : list, default []
        List of special values.
    tree_binning : bool, default False
        Whether to use decision-tree binning.
    random_state : int, default 42
        Random seed.
    weight_col : str, optional
        Column in ``data`` with per-row sample weights (alias: ``wgt_col``).
    sample_weight : array-like, optional
        Explicit per-row weights; must align with ``len(data)``.
    wgt_col : str, optional
        Alias for ``weight_col``.

    Returns
    -------
    pandas.DataFrame
        Cross-risk table.

    Raises
    ------
    ValueError
        If the ratio syntax is used without a numerator and a denominator (for ``agg_func='ratio'``, ``agg_col`` must be a
        two-element list or tuple), if a ratio column is not in ``data``, if weights are combined with anything other
        than the plain ``agg_func='mean'``, or if ``binning_numeric`` is a list, tuple or array that does not have two
        elements.
    TypeError
        If ``binning_numeric`` is not None, a bool, a list, a tuple or a numpy array.

    Notes
    -----
    If the first score column is not binned (it is not numeric, or ``binning_numeric[0]`` is False), the helper columns
    ``_bin_num1`` and ``_bin_range1`` (and ``_bin_num2`` and ``_bin_range2`` when the second score column is not binned
    either) are added to the DataFrame passed as ``data``.

    Examples
    --------
    >>> cross_risk(data, score_list=['score1', 'score2'], dep='target', nbins=10)
    """

    from pandas.api.types import is_numeric_dtype

    if agg_col is None:
        agg_col = dep

    # A single bool applies to both columns (it used to be indexed and raised TypeError)
    if binning_numeric is None:
        binning_numeric = [True, True]
    elif isinstance(binning_numeric, (bool, np.bool_)):
        binning_numeric = [bool(binning_numeric), bool(binning_numeric)]
    elif isinstance(binning_numeric, (list, tuple, np.ndarray)):
        if len(binning_numeric) != 2:
            raise ValueError(
                "binning_numeric must be a bool or a two-element list, tuple or array; "
                f"got {len(binning_numeric)} elements."
            )
        binning_numeric = [bool(binning_numeric[0]), bool(binning_numeric[1])]
    else:
        raise TypeError("binning_numeric must be None, a bool, or a two-element list, tuple or array.")

    resolved_weight = None
    if weight_col is not None or sample_weight is not None or wgt_col is not None:
        resolved_weight = _weighted_eval.resolve_weights(
            data=data,
            weight_col=weight_col or wgt_col,
            sample_weight=sample_weight,
            expected_len=len(data),
        )

    def _parse_ratio_agg(agg_col, agg_func):
        """
        Parse ratio aggregation syntax while keeping backward compatibility
        with the original cross_risk agg_func behavior.
        """
        ratio_cfg = None

        if isinstance(agg_func, str) and agg_func.lower() in ["ratio", "rate", "divide"]:
            if not isinstance(agg_col, (list, tuple)) or len(agg_col) != 2:
                raise ValueError(
                    "When agg_func='ratio', `agg_col` must be a two-element "
                    "list/tuple: (numerator_col, denominator_col)."
                )
            ratio_cfg = {
                "numerator": agg_col[0],
                "denominator": agg_col[1],
            }

        elif isinstance(agg_func, (list, tuple)) and len(agg_func) >= 3:
            if str(agg_func[0]).lower() in ["ratio", "rate", "divide"]:
                ratio_cfg = {
                    "numerator": agg_func[1],
                    "denominator": agg_func[2],
                }
                if len(agg_func) >= 4:
                    ratio_cfg["return_count"] = bool(agg_func[3])

        elif isinstance(agg_func, dict):
            func_name = str(
                agg_func.get("func", agg_func.get("type", agg_func.get("agg_func", "")))
            ).lower()
            if func_name in ["ratio", "rate", "divide"]:
                ratio_cfg = dict(agg_func)
                ratio_cfg["numerator"] = ratio_cfg.get(
                    "numerator", ratio_cfg.get("num", ratio_cfg.get("numerator_col"))
                )
                ratio_cfg["denominator"] = ratio_cfg.get(
                    "denominator", ratio_cfg.get("denom", ratio_cfg.get("denominator_col"))
                )

        if ratio_cfg is None:
            return None

        if ratio_cfg.get("numerator") is None or ratio_cfg.get("denominator") is None:
            raise ValueError(
                "Ratio aggregation requires both numerator and denominator columns. "
                "Use agg_func={'func': 'ratio', 'numerator': ..., 'denominator': ...}."
            )

        ratio_cfg.setdefault("numerator_agg", "sum")
        ratio_cfg.setdefault("denominator_agg", "sum")
        ratio_cfg.setdefault("return_count", False)
        ratio_cfg.setdefault("return_count_pct", False)
        ratio_cfg.setdefault("count_name", "N")
        ratio_cfg.setdefault("count_pct_name", "N_Pct")
        ratio_cfg.setdefault("ratio_name", f"{ratio_cfg['numerator']}/{ratio_cfg['denominator']}")
        ratio_cfg.setdefault("valid_only", True)
        ratio_cfg.setdefault("zero_division", np.nan)
        return ratio_cfg

    def _parse_regular_agg(agg_func):
        """
        Parse the extended regular aggregation syntax:
        agg_func={
            'func': ['mean', 'count'],
            'return_count_pct': True,
            'count_func_name': 'count',
            'count_pct_name': 'N_Pct'
        }
        """
        if not isinstance(agg_func, dict):
            return None

        func_name = str(
            agg_func.get("func", agg_func.get("type", agg_func.get("agg_func", "")))
        ).lower()
        if func_name in ["ratio", "rate", "divide"]:
            return None

        if "func" not in agg_func and "agg_func" not in agg_func:
            return None

        regular_cfg = dict(agg_func)
        regular_cfg["func"] = regular_cfg.get("func", regular_cfg.get("agg_func"))
        regular_cfg.setdefault("return_count_pct", False)
        regular_cfg.setdefault("count_func_name", "count")
        regular_cfg.setdefault("count_pct_name", "N_Pct")
        regular_cfg.setdefault("margins_name", "Total_Avg_Risk")
        return regular_cfg

    ratio_cfg = _parse_ratio_agg(agg_col, agg_func)
    regular_cfg = _parse_regular_agg(agg_func) if ratio_cfg is None else None

    if is_numeric_dtype(data[score_list[0]]) and binning_numeric[0]:

        data, edges1 = super_binning(data = data, 
                                     score = score_list[0], 
                                     dep = dep, 
                                     nbins = nbins[0] if isinstance(nbins, list) else nbins, 
                                     precision = precision[0] if isinstance(precision, list) else precision, 
                                     min_bin_prop = min_bin_prop[0] if isinstance(min_bin_prop, list) else min_bin_prop, 
                                     include_missing = include_missing, 
                                     equal_freq = equal_freq, 
                                     chi2_method = chi2_method, 
                                     chi2_p = chi2_p, 
                                     init_equi_bins = init_equi_bins, 
                                     fillna = fillna, 
                                     spec_values = _with_missing_bin(data[score_list[0]], include_missing, fillna, spec_values),
                                     tree_binning = tree_binning, 
                                     random_state = random_state, 
                                     return_edges = True, 
                                     bin_colnames = ("_bin_num1", "_bin_range1"),
                                     ascending = True)

    else:
        data["_bin_num1"] = data[score_list[0]]
        data["_bin_range1"] = data[score_list[0]]

    if is_numeric_dtype(data[score_list[1]]) and binning_numeric[1]:

        data, edges2 = super_binning(data = data, 
                                     score = score_list[1], 
                                     dep = dep, 
                                     nbins = nbins[1] if isinstance(nbins, list) else nbins, 
                                     precision = precision[1] if isinstance(precision, list) else precision, 
                                     min_bin_prop = min_bin_prop[1] if isinstance(min_bin_prop, list) else min_bin_prop, 
                                     include_missing = include_missing, 
                                     equal_freq = equal_freq, 
                                     chi2_method = chi2_method, 
                                     chi2_p = chi2_p, 
                                     init_equi_bins = init_equi_bins, 
                                     fillna = fillna, 
                                     spec_values = _with_missing_bin(data[score_list[1]], include_missing, fillna, spec_values),
                                     tree_binning = tree_binning, 
                                     random_state = random_state, 
                                     return_edges = True, 
                                     bin_colnames = ("_bin_num2", "_bin_range2"),
                                     ascending = True)

    else:

        data["_bin_num2"] = data[score_list[1]]
        data["_bin_range2"] = data[score_list[1]]


    if ratio_cfg is not None:
        numerator = ratio_cfg["numerator"]
        denominator = ratio_cfg["denominator"]
        numerator_agg = ratio_cfg["numerator_agg"]
        denominator_agg = ratio_cfg["denominator_agg"]
        return_count = ratio_cfg["return_count"]
        return_count_pct = ratio_cfg["return_count_pct"]
        count_name = ratio_cfg["count_name"]
        count_pct_name = ratio_cfg["count_pct_name"]
        ratio_name = ratio_cfg["ratio_name"]
        valid_only = ratio_cfg["valid_only"]
        zero_division = ratio_cfg["zero_division"]
        margin_name = ratio_cfg.get("margins_name", "Total_Avg_Risk")

        missing_cols = [c for c in [numerator, denominator] if c not in data.columns]
        if len(missing_cols) > 0:
            raise ValueError(f"Columns not found in data for ratio aggregation: {missing_cols}")

        agg_data = data.copy()
        if valid_only:
            agg_data = agg_data.loc[
                agg_data[numerator].notna()
                & agg_data[denominator].notna()
                & (agg_data[denominator] != 0)
            ].copy()

        numerator_res = pd.crosstab([agg_data['_bin_num1'], agg_data['_bin_range1']],
                                    [agg_data['_bin_num2'], agg_data['_bin_range2']],
                                    rownames=[score_list[0], score_list[0]],
                                    colnames=[score_list[1], score_list[1]],
                                    values=agg_data[numerator],
                                    aggfunc=numerator_agg,
                                    margins=True,
                                    margins_name=margin_name)

        denominator_res = pd.crosstab([agg_data['_bin_num1'], agg_data['_bin_range1']],
                                      [agg_data['_bin_num2'], agg_data['_bin_range2']],
                                      rownames=[score_list[0], score_list[0]],
                                      colnames=[score_list[1], score_list[1]],
                                      values=agg_data[denominator],
                                      aggfunc=denominator_agg,
                                      margins=True,
                                      margins_name=margin_name)

        res = numerator_res / denominator_res.replace(0, np.nan)
        if not pd.isna(zero_division):
            res = res.fillna(zero_division)

        if return_count or return_count_pct:
            count_col = ratio_cfg.get("count_col", numerator)
            count_res = pd.crosstab([agg_data['_bin_num1'], agg_data['_bin_range1']],
                                    [agg_data['_bin_num2'], agg_data['_bin_range2']],
                                    rownames=[score_list[0], score_list[0]],
                                    colnames=[score_list[1], score_list[1]],
                                    values=agg_data[count_col],
                                    aggfunc="count",
                                    margins=True,
                                    margins_name=margin_name)
            output_dict = {ratio_name: res}

            if return_count:
                output_dict[count_name] = count_res

            if return_count_pct:
                total_count = count_res.iloc[-1, -1] if count_res.shape[0] > 0 and count_res.shape[1] > 0 else 0
                if total_count == 0 or pd.isna(total_count):
                    count_pct_res = count_res * np.nan
                else:
                    count_pct_res = count_res / total_count
                output_dict[count_pct_name] = count_pct_res

            res = pd.concat(output_dict, axis=1)

        return res

    actual_agg_func = regular_cfg["func"] if regular_cfg is not None else agg_func
    margin_name = regular_cfg.get("margins_name", "Total_Avg_Risk") if regular_cfg is not None else "Total_Avg_Risk"

    if resolved_weight is not None:
        if ratio_cfg is not None or regular_cfg is not None:
            raise ValueError(
                "weighted cross_risk currently supports only simple mean aggregation "
                "(agg_func='mean' without extended ratio/regular dict syntax)."
            )
        agg_name = getattr(actual_agg_func, "__name__", str(actual_agg_func)).lower()
        if agg_name != "mean":
            raise ValueError(
                "weighted cross_risk currently supports only agg_func='mean'; got {0!r}".format(actual_agg_func)
            )
        return _weighted_eval.cross_risk_weighted_mean(
            data=data,
            agg_col=agg_col,
            sample_weight=resolved_weight,
            score_list=score_list,
            margin_name=margin_name,
        )

    res = pd.crosstab([data['_bin_num1'], data['_bin_range1']], 
                        [data['_bin_num2'], data['_bin_range2']], 
                        rownames=[score_list[0], score_list[0]], 
                        colnames=[score_list[1], score_list[1]], 
                        values = data[agg_col], aggfunc = actual_agg_func, 
                        margins = True, margins_name=margin_name)

    if regular_cfg is not None and regular_cfg.get("return_count_pct", False):
        count_func_name = regular_cfg.get("count_func_name", "count")
        count_pct_name = regular_cfg.get("count_pct_name", "N_Pct")

        count_res = None
        if isinstance(res.columns, pd.MultiIndex) and count_func_name in res.columns.get_level_values(0):
            count_res = res[count_func_name]

        if count_res is None:
            count_res = pd.crosstab([data['_bin_num1'], data['_bin_range1']],
                                    [data['_bin_num2'], data['_bin_range2']],
                                    rownames=[score_list[0], score_list[0]],
                                    colnames=[score_list[1], score_list[1]],
                                    values=data[agg_col],
                                    aggfunc="count",
                                    margins=True,
                                    margins_name=margin_name)

        total_count = count_res.iloc[-1, -1] if count_res.shape[0] > 0 and count_res.shape[1] > 0 else 0
        if total_count == 0 or pd.isna(total_count):
            count_pct_res = count_res * np.nan
        else:
            count_pct_res = count_res / total_count

        if isinstance(res.columns, pd.MultiIndex):
            res = pd.concat([res, pd.concat({count_pct_name: count_pct_res}, axis=1)], axis=1)
        else:
            res = pd.concat({regular_cfg.get("func_name", str(actual_agg_func)): res,
                             count_pct_name: count_pct_res}, axis=1)

    return res

get_gains_table_by_cust_metrics

get_gains_table_by_cust_metrics(data, dep, nbins=10, precision=5, min_bin_prop=0.05, include_missing=True, score=None, model=None, varlist=None, equal_freq=True, chi2_method=False, grp_name=None, min_data_size=100, grp_colname=None, sync_range=True, chi2_p=0.95, init_equi_bins=100, fillna=-999999, spec_values=[], tree_binning=False, random_state=42, ascending=True, eval_metrics=['age', 'monthly_income', 'education'], metric_agg_func='mean', withSummary=False)

Compute the grouped Gains table with custom metrics.

Split the data into groups by the grouping column and compute the custom-metric Gains table of each group separately. The Gains table contains the basic statistics as well as the aggregated values of the custom metrics.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input data table.	required
dep	str	Name of the target variable.	required
nbins	int	Number of bins.	10
precision	int	Precision of the bin boundary values.	5
min_bin_prop	float	Minimum proportion of samples per bin.	0.05
include_missing	bool	Whether missing scores are binned. True fills them with fillna and gives them a bin of their own, (-inf, fillna] (a score already equal to fillna joins it); the other bins are computed from the real scores. False leaves the rows with a missing score out.	True
score	str	Name of the score column.	None
model	sklearn-like model	Machine learning model.	None
varlist	list	List of model features.	None
equal_freq	bool	True for equal-frequency binning.	True
chi2_method	bool	Whether to use chi-square binning.	False
grp_name	str	Name of the grouping column.	None
min_data_size	int	Minimum number of samples per group.	100
grp_colname	str	Name of the group column in the output.	None
sync_range	bool	Whether to synchronize the bin boundaries across groups.	True
chi2_p	float	Significance level of the chi-square test.	0.95
init_equi_bins	int	Initial number of equal-frequency bins.	100
fillna	any	Fill value for missing values.	-999999
spec_values	list	List of special values.	[]
tree_binning	bool	Whether to use decision-tree binning.	False
random_state	int	Random seed.	42
ascending	bool	Whether the bin order is ascending.	True
eval_metrics	list	List of custom metrics to compute.	["age", "monthly_income", "education"]
metric_agg_func	str or callable	Aggregation function for the custom metrics.	"mean"
withSummary	bool	Whether to include an overall summary row.	False

Returns:

Type	Description
DataFrame	Grouped Gains table with custom metrics: the compact Gains columns (MIN, MAX, N, AVG_SCORE, AVG_BAD, N_BAD, N_GOOD) plus one column per entry of eval_metrics.

Notes

eval_metrics names the columns of data that are aggregated in every bin with metric_agg_func; its default refers to the columns of a legacy dataset, so always pass your own list. The function has no weight argument, and its default ascending=True differs from that of get_gains_table (False).

Source code in Modeling_Tool/Eval/Model_Eval_Tool.py

def get_gains_table_by_cust_metrics(data, dep, nbins = 10, precision = 5, min_bin_prop = 0.05, include_missing = True, 
                                    score = None, model = None, varlist = None, equal_freq = True, chi2_method = False,
                                    grp_name = None, min_data_size = 100, grp_colname = None, sync_range = True,
                                    chi2_p = 0.95, init_equi_bins = 100, fillna = -999999, spec_values = [], 
                                    tree_binning = False, random_state=42, ascending = True,
                                    eval_metrics = ["age", "monthly_income", "education"], metric_agg_func = "mean", withSummary = False):
    """
    Compute the grouped Gains table with custom metrics.

    Split the data into groups by the grouping column and compute the custom-metric Gains table of each group separately.
    The Gains table contains the basic statistics as well as the aggregated values of the custom metrics.

    Parameters
    ----------
    data : pandas.DataFrame
        Input data table.
    dep : str
        Name of the target variable.
    nbins : int, default 10
        Number of bins.
    precision : int, default 5
        Precision of the bin boundary values.
    min_bin_prop : float, default 0.05
        Minimum proportion of samples per bin.
    include_missing : bool, default True
        Whether missing scores are binned. True fills them with ``fillna`` and gives them a bin of their own, ``(-inf,
        fillna]`` (a score already equal to ``fillna`` joins it); the other bins are computed from the real scores.
        False leaves the rows with a missing score out.
    score : str, optional
        Name of the score column.
    model : sklearn-like model, optional
        Machine learning model.
    varlist : list, optional
        List of model features.
    equal_freq : bool, default True
        True for equal-frequency binning.
    chi2_method : bool, default False
        Whether to use chi-square binning.
    grp_name : str, optional
        Name of the grouping column.
    min_data_size : int, default 100
        Minimum number of samples per group.
    grp_colname : str, optional
        Name of the group column in the output.
    sync_range : bool, default True
        Whether to synchronize the bin boundaries across groups.
    chi2_p : float, default 0.95
        Significance level of the chi-square test.
    init_equi_bins : int, default 100
        Initial number of equal-frequency bins.
    fillna : any, default -999999
        Fill value for missing values.
    spec_values : list, default []
        List of special values.
    tree_binning : bool, default False
        Whether to use decision-tree binning.
    random_state : int, default 42
        Random seed.
    ascending : bool, default True
        Whether the bin order is ascending.
    eval_metrics : list, default ["age", "monthly_income", "education"]
        List of custom metrics to compute.
    metric_agg_func : str or callable, default "mean"
        Aggregation function for the custom metrics.
    withSummary : bool, default False
        Whether to include an overall summary row.

    Returns
    -------
    pandas.DataFrame
        Grouped Gains table with custom metrics: the compact Gains columns (``MIN``, ``MAX``, ``N``, ``AVG_SCORE``,
        ``AVG_BAD``, ``N_BAD``, ``N_GOOD``) plus one column per entry of ``eval_metrics``.

    Notes
    -----
    ``eval_metrics`` names the columns of ``data`` that are aggregated in every bin with ``metric_agg_func``; its default
    refers to the columns of a legacy dataset, so always pass your own list. The function has no weight argument, and its
    default ``ascending=True`` differs from that of ``get_gains_table`` (False).
    """

    if grp_colname is None:
        grp_colname = grp_name

    if grp_name is None:
        fnl_df = _get_cust_gains_table_single(data = data, 
                                             dep = dep, 
                                             nbins = nbins, 
                                             precision = precision, 
                                             min_bin_prop = min_bin_prop, 
                                             include_missing = include_missing, 
                                             score = score, 
                                             model = model, 
                                             varlist = varlist, 
                                             equal_freq = equal_freq, 
                                             chi2_method = chi2_method,
                                             chi2_p = chi2_p, 
                                             init_equi_bins = init_equi_bins, 
                                             fillna = fillna, 
                                             spec_values = spec_values,
                                             eval_metrics = eval_metrics, 
                                             tree_binning = tree_binning, 
                                             random_state = random_state, 
                                             metric_agg_func = metric_agg_func,
                                             ascending = ascending,
                                             withSummary = withSummary)
        return fnl_df

    # Decide the missing bin once from all groups: a group-level decision would number the bins of a group with
    # missing scores differently from the others
    if score is not None and score in data.columns:
        spec_values = _with_missing_bin(data[score], include_missing, fillna, spec_values)
    grp_list = data[grp_name].sort_values(ascending = True).unique().tolist()
    valid_grp_list = [x for x in grp_list if data[data[grp_name].isin([x])].shape[0] >= min_data_size]

    first_grp = valid_grp_list[0]
    data_grp = data[data[grp_name].isin([first_grp])]

    if sync_range:

        fnl_df = _get_cust_gains_table_single(data = data_grp, 
                                             dep = dep, 
                                             nbins = nbins, 
                                             precision = precision, 
                                             min_bin_prop = min_bin_prop, 
                                             include_missing = include_missing, 
                                             score = score, 
                                             model = model, 
                                             varlist = varlist, 
                                             equal_freq = equal_freq, 
                                             chi2_method = chi2_method,
                                             chi2_p = chi2_p, 
                                             init_equi_bins = init_equi_bins, 
                                             fillna = fillna, 
                                             spec_values = spec_values,
                                             eval_metrics = eval_metrics, 
                                             tree_binning = tree_binning, 
                                             random_state = random_state, 
                                             metric_agg_func = metric_agg_func,
                                             ascending = ascending,
                                             withSummary = withSummary)

        fnl_df[grp_colname] = first_grp

        # The first group's table lists only the bins it fills, so its edges need not reach -inf/inf (the empty
        # missing bin of a group without missing scores is not listed): close them as get_gains_table does
        nbins = sorted(set(get_bin_range_list(fnl_df.reset_index()) + [-np.inf, np.inf]))
        equal_freq = False
        chi2_method = False

    else:

        fnl_df = _get_cust_gains_table_single(data = data_grp, 
                                             dep = dep, 
                                             nbins = nbins, 
                                             precision = precision, 
                                             min_bin_prop = min_bin_prop, 
                                             include_missing = include_missing, 
                                             score = score, 
                                             model = model, 
                                             varlist = varlist, 
                                             equal_freq = equal_freq, 
                                             chi2_method = chi2_method,
                                             chi2_p = chi2_p, 
                                             init_equi_bins = init_equi_bins, 
                                             fillna = fillna, 
                                             spec_values = spec_values,
                                             eval_metrics = eval_metrics, 
                                             tree_binning = tree_binning, 
                                             random_state = random_state, 
                                             metric_agg_func = metric_agg_func,
                                             ascending = ascending,
                                             withSummary = withSummary)

        fnl_df[grp_colname] = first_grp

    i = 1
    while i < len(valid_grp_list):

        grp = valid_grp_list[i]
        data_grp = data[data[grp_name].isin([grp])]

        perf_res = _get_cust_gains_table_single(data = data_grp, 
                                               dep = dep, 
                                               nbins = nbins, 
                                               precision = precision, 
                                               min_bin_prop = min_bin_prop, 
                                               include_missing = include_missing, 
                                               score = score, 
                                               model = model, 
                                               varlist = varlist, 
                                               equal_freq = equal_freq, 
                                               chi2_method = chi2_method,
                                               chi2_p = chi2_p, 
                                               init_equi_bins = init_equi_bins, 
                                               fillna = fillna, 
                                               spec_values = spec_values, 
                                               eval_metrics = eval_metrics, 
                                               tree_binning = tree_binning, 
                                               random_state = random_state, 
                                               metric_agg_func = metric_agg_func,
                                               ascending = ascending,
                                               withSummary = withSummary)
        perf_res[grp_colname] = grp

        fnl_df = pd.concat([fnl_df, perf_res])
        i += 1

    return fnl_df

tie_score_rate

tie_score_rate(data, score)

Compute the score tie rate.

Compute the proportion of non-unique score values: one minus the number of distinct score values divided by the number of rows (in a group of rows that share a score, all the rows but one are counted).

Parameters:

Name	Type	Description	Default
data	DataFrame	Input data table.	required
score	str	Name of the score column.	required

Returns:

Type	Description
float	Score tie rate (between 0 and 1).

Examples:

>>> tie_score_rate(data, 'score')
0.15  # means that the number of distinct scores is 85% of the number of rows

Source code in Modeling_Tool/Eval/Model_Eval_Tool.py

def tie_score_rate(data, score):
    """
    Compute the score tie rate.

    Compute the proportion of non-unique score values: one minus the number of distinct score values divided by the number
    of rows (in a group of rows that share a score, all the rows but one are counted).

    Parameters
    ----------
    data : pandas.DataFrame
        Input data table.
    score : str
        Name of the score column.

    Returns
    -------
    float
        Score tie rate (between 0 and 1).

    Examples
    --------
    >>> tie_score_rate(data, 'score')
    0.15  # means that the number of distinct scores is 85% of the number of rows
    """

    n_unique_scr = len(data[score].unique())
    unique_scr_prop = n_unique_scr / data.shape[0]
    return (1 - unique_scr_prop)

score_unique_rate

score_unique_rate(data, score)

Compute the score unique rate.

Compute the proportion of unique score values in the total number of samples.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input data table (this parameter is retained but not used).	required
score	array - like	Score column or array.	required

Returns:

Type	Description
float	Score unique rate (between 0 and 1).

Examples:

>>> score_unique_rate(data, data['score'])
0.85  # means that the scores of 85% of the samples are unique

Source code in Modeling_Tool/Eval/Model_Eval_Tool.py

def score_unique_rate(data, score):
    """
    Compute the score unique rate.

    Compute the proportion of unique score values in the total number of samples.

    Parameters
    ----------
    data : pandas.DataFrame
        Input data table (this parameter is retained but not used).
    score : array-like
        Score column or array.

    Returns
    -------
    float
        Score unique rate (between 0 and 1).

    Examples
    --------
    >>> score_unique_rate(data, data['score'])
    0.85  # means that the scores of 85% of the samples are unique
    """

    return len(np.unique(score)) / len(score)

Score Comparison and Evaluation Pipeline: Evaluation_Tool

Evaluation_Tool

Model Evaluation Tool - Optimized Version

This module provides classes and functions for model evaluation, performance comparison, and gains analysis. It includes utilities for calculating various metrics and generating cross-risk summaries.

Author: Matrix Agent

EvaluationPipeline

Chainable pipeline object for slice-by-slice evaluation.

Conditions are added by chaining .group_by() and .subset_by(), and the pipeline is executed with .apply(func). func may return a pandas.DataFrame or a dict whose values are DataFrames.

Parameters:

Name	Type	Description	Default
m_eval	Model_Evaluation_Tool	Evaluation object whose methods func can call. While apply runs, m_eval.data is temporarily replaced by the current slice.	required
data	DataFrame	Data to slice. Default is None, i.e. m_eval.data at the time the pipeline is created.	None

Attributes:

Name	Type	Description
m_eval	Model_Evaluation_Tool	The evaluation object.
original_data	DataFrame	The data that is sliced.
steps	list of dict	The slicing steps added so far by group_by and subset_by, in the order of the calls.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

class EvaluationPipeline:
    """
    Chainable pipeline object for slice-by-slice evaluation.

    Conditions are added by chaining ``.group_by()`` and ``.subset_by()``, and the
    pipeline is executed with ``.apply(func)``.
    ``func`` may return a ``pandas.DataFrame`` or a ``dict`` whose values are DataFrames.

    Parameters
    ----------
    m_eval : Model_Evaluation_Tool
        Evaluation object whose methods ``func`` can call. While ``apply`` runs, ``m_eval.data`` is temporarily replaced
        by the current slice.
    data : pandas.DataFrame, optional
        Data to slice. Default is None, i.e. ``m_eval.data`` at the time the pipeline is created.

    Attributes
    ----------
    m_eval : Model_Evaluation_Tool
        The evaluation object.
    original_data : pandas.DataFrame
        The data that is sliced.
    steps : list of dict
        The slicing steps added so far by ``group_by`` and ``subset_by``, in the order of the calls.
    """
    def __init__(self, m_eval, data=None):
        self.m_eval = m_eval
        self.original_data = m_eval.data if data is None else data
        self.steps = []
        self._expected_columns = None   # column-name cache for DataFrame results
        self._expected_dict_keys = None # key cache for dict results

    def group_by(self, group_name, min_size=100, group_var_name=None):
        """
        Add a step that splits the data by the values of a column.

        Parameters
        ----------
        group_name : str
            Name of the column to group by.
        min_size : int, default 100
            Groups with fewer than ``min_size`` rows are skipped.
        group_var_name : str or None, default None
            Name of the result column that holds the group value. Default is None, i.e. ``group_name``.

        Returns
        -------
        EvaluationPipeline
            The pipeline itself, to allow method chaining.
        """
        if group_var_name is None:
            group_var_name = group_name
        self.steps.append({
            'type': 'group',
            'group_name': group_name,
            'group_var_name': group_var_name,
            'min_size': min_size
        })
        return self

    def subset_by(self, condition_dict, name='eval_subset', min_size=100):
        """
        Add a step that filters the data with a set of named conditions.

        Parameters
        ----------
        condition_dict : dict
            Mapping from a subset label to a ``DataFrame.query`` string; an empty string ``""`` keeps every row.
        name : str, default 'eval_subset'
            Name of the result column that holds the subset label.
        min_size : int, default 100
            Subsets with fewer than ``min_size`` rows are skipped.

        Returns
        -------
        EvaluationPipeline
            The pipeline itself, to allow method chaining.
        """
        self.steps.append({
            'type': 'subset',
            'condition_dict': condition_dict,
            'subset_var_name': name,
            'min_size': min_size
        })
        return self

    def apply(self, func, **kwargs):
        """
        Run ``func`` on every slice and combine the results.

        The steps are applied in the order in which they were added, and each step splits the output of the previous one
        (``group_by`` followed by ``subset_by`` evaluates every group once per subset). With no step, ``func`` runs once
        on the whole data.

        Parameters
        ----------
        func : callable
            Function to run on each slice. If it has a parameter named ``current_data``, it receives the slice there;
            otherwise it is called without data and is expected to be a method of ``m_eval``, whose ``data`` is replaced by
            the slice while ``func`` runs and restored afterwards. It returns a DataFrame or a dict of DataFrames.
        **kwargs
            Extra keyword arguments passed to ``func`` on every call.

        Returns
        -------
        pandas.DataFrame or dict
            The results of all the slices concatenated, with one extra column per step that holds the group value or the
            subset label (see ``group_var_name`` and ``name``). If ``func`` returns a dict, a dict with the same keys of
            concatenated DataFrames is returned. The result is empty when no slice reaches its ``min_size``.

        Notes
        -----
        An exception raised by ``func`` is logged at INFO level and the slice yields an empty result instead of
        stopping the run, so an unexpectedly empty result may hide an error: call ``func`` on one slice to see it.
        """

        import inspect
        sig = inspect.signature(func)

        def _execute(step_idx, current_data, current_meta):
            if step_idx >= len(self.steps):
                old_data = self.m_eval.data
                self.m_eval.data = current_data
                try:
                    # Inject the current data automatically (if the function accepts a 'current_data' parameter)
                    if 'current_data' in sig.parameters:
                        kwargs['current_data'] = current_data
                    res = func(**kwargs)
                except Exception as e:
                    logger.info(f"Error in group {current_meta}: {e}")
                    if self._expected_dict_keys is not None:
                        res = {k: pd.DataFrame() for k in self._expected_dict_keys}
                    else:
                        res = pd.DataFrame()
                finally:
                    self.m_eval.data = old_data

                # (handle the dict and DataFrame cases)
                if isinstance(res, dict):
                    if self._expected_dict_keys is None and res:
                        self._expected_dict_keys = list(res.keys())
                    for key, df in res.items():
                        if df is not None and not df.empty:
                            for col_name, col_val in current_meta.items():
                                df[col_name] = col_val
                        else:
                            empty_df = pd.DataFrame()
                            for col_name, col_val in current_meta.items():
                                empty_df[col_name] = [col_val]
                            res[key] = empty_df
                    return res
                else:
                    if self._expected_columns is None and res is not None and not res.empty:
                        self._expected_columns = res.columns.tolist()
                    if res is not None and not res.empty:
                        res = res.copy()
                        for key, val in current_meta.items():
                            res[key] = val
                    else:
                        if self._expected_columns is not None:
                            res = pd.DataFrame(columns=self._expected_columns)
                        else:
                            res = pd.DataFrame()
                    return res

            step = self.steps[step_idx]
            if step['type'] == 'group':
                group_name = step['group_name']
                group_var_name = step['group_var_name']
                min_size = step['min_size']
                results = []
                for gval in current_data[group_name].unique():
                    sub_data = current_data[current_data[group_name] == gval]
                    if len(sub_data) >= min_size:
                        new_meta = current_meta.copy()
                        new_meta[group_var_name] = gval
                        sub_res = _execute(step_idx + 1, sub_data, new_meta)
                        if sub_res is not None:
                            results.append(sub_res)
                if not results:
                    if self._expected_dict_keys is not None:
                        return {k: pd.DataFrame() for k in self._expected_dict_keys}
                    else:
                        return pd.DataFrame()
                # Merge the results
                if isinstance(results[0], dict):
                    merged = {k: pd.concat([r[k] for r in results], ignore_index=True) for k in results[0].keys()}
                    return merged
                else:
                    return pd.concat(results, ignore_index=True)

            elif step['type'] == 'subset':
                condition_dict = step['condition_dict']
                subset_var_name = step['subset_var_name']
                min_size = step['min_size']
                results = []
                for label, query in condition_dict.items():
                    if query == "":
                        sub_data = current_data.copy()
                    else:
                        sub_data = current_data.query(query)
                    if len(sub_data) >= min_size:
                        new_meta = current_meta.copy()
                        new_meta[subset_var_name] = label
                        sub_res = _execute(step_idx + 1, sub_data, new_meta)
                        if sub_res is not None:
                            results.append(sub_res)
                if not results:
                    if self._expected_dict_keys is not None:
                        return {k: pd.DataFrame() for k in self._expected_dict_keys}
                    else:
                        return pd.DataFrame()
                if isinstance(results[0], dict):
                    merged = {k: pd.concat([r[k] for r in results], ignore_index=True) for k in results[0].keys()}
                    return merged
                else:
                    return pd.concat(results, ignore_index=True)

        return _execute(0, self.original_data, {})

group_by

group_by(group_name, min_size=100, group_var_name=None)

Add a step that splits the data by the values of a column.

Parameters:

Name	Type	Description	Default
group_name	str	Name of the column to group by.	required
min_size	int	Groups with fewer than min_size rows are skipped.	100
group_var_name	str or None	Name of the result column that holds the group value. Default is None, i.e. group_name.	None

Returns:

Type	Description
EvaluationPipeline	The pipeline itself, to allow method chaining.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def group_by(self, group_name, min_size=100, group_var_name=None):
    """
    Add a step that splits the data by the values of a column.

    Parameters
    ----------
    group_name : str
        Name of the column to group by.
    min_size : int, default 100
        Groups with fewer than ``min_size`` rows are skipped.
    group_var_name : str or None, default None
        Name of the result column that holds the group value. Default is None, i.e. ``group_name``.

    Returns
    -------
    EvaluationPipeline
        The pipeline itself, to allow method chaining.
    """
    if group_var_name is None:
        group_var_name = group_name
    self.steps.append({
        'type': 'group',
        'group_name': group_name,
        'group_var_name': group_var_name,
        'min_size': min_size
    })
    return self

subset_by

subset_by(condition_dict, name='eval_subset', min_size=100)

Add a step that filters the data with a set of named conditions.

Parameters:

Name	Type	Description	Default
condition_dict	dict	Mapping from a subset label to a DataFrame.query string; an empty string "" keeps every row.	required
name	str	Name of the result column that holds the subset label.	'eval_subset'
min_size	int	Subsets with fewer than min_size rows are skipped.	100

Returns:

Type	Description
EvaluationPipeline	The pipeline itself, to allow method chaining.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def subset_by(self, condition_dict, name='eval_subset', min_size=100):
    """
    Add a step that filters the data with a set of named conditions.

    Parameters
    ----------
    condition_dict : dict
        Mapping from a subset label to a ``DataFrame.query`` string; an empty string ``""`` keeps every row.
    name : str, default 'eval_subset'
        Name of the result column that holds the subset label.
    min_size : int, default 100
        Subsets with fewer than ``min_size`` rows are skipped.

    Returns
    -------
    EvaluationPipeline
        The pipeline itself, to allow method chaining.
    """
    self.steps.append({
        'type': 'subset',
        'condition_dict': condition_dict,
        'subset_var_name': name,
        'min_size': min_size
    })
    return self

apply

apply(func, **kwargs)

Run func on every slice and combine the results.

The steps are applied in the order in which they were added, and each step splits the output of the previous one (group_by followed by subset_by evaluates every group once per subset). With no step, func runs once on the whole data.

Parameters:

Name	Type	Description	Default
func	callable	Function to run on each slice. If it has a parameter named current_data, it receives the slice there; otherwise it is called without data and is expected to be a method of m_eval, whose data is replaced by the slice while func runs and restored afterwards. It returns a DataFrame or a dict of DataFrames.	required
**kwargs		Extra keyword arguments passed to func on every call.	{}

Returns:

Type	Description
DataFrame or dict	The results of all the slices concatenated, with one extra column per step that holds the group value or the subset label (see group_var_name and name). If func returns a dict, a dict with the same keys of concatenated DataFrames is returned. The result is empty when no slice reaches its min_size.

Notes

An exception raised by func is logged at INFO level and the slice yields an empty result instead of stopping the run, so an unexpectedly empty result may hide an error: call func on one slice to see it.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def apply(self, func, **kwargs):
    """
    Run ``func`` on every slice and combine the results.

    The steps are applied in the order in which they were added, and each step splits the output of the previous one
    (``group_by`` followed by ``subset_by`` evaluates every group once per subset). With no step, ``func`` runs once
    on the whole data.

    Parameters
    ----------
    func : callable
        Function to run on each slice. If it has a parameter named ``current_data``, it receives the slice there;
        otherwise it is called without data and is expected to be a method of ``m_eval``, whose ``data`` is replaced by
        the slice while ``func`` runs and restored afterwards. It returns a DataFrame or a dict of DataFrames.
    **kwargs
        Extra keyword arguments passed to ``func`` on every call.

    Returns
    -------
    pandas.DataFrame or dict
        The results of all the slices concatenated, with one extra column per step that holds the group value or the
        subset label (see ``group_var_name`` and ``name``). If ``func`` returns a dict, a dict with the same keys of
        concatenated DataFrames is returned. The result is empty when no slice reaches its ``min_size``.

    Notes
    -----
    An exception raised by ``func`` is logged at INFO level and the slice yields an empty result instead of
    stopping the run, so an unexpectedly empty result may hide an error: call ``func`` on one slice to see it.
    """

    import inspect
    sig = inspect.signature(func)

    def _execute(step_idx, current_data, current_meta):
        if step_idx >= len(self.steps):
            old_data = self.m_eval.data
            self.m_eval.data = current_data
            try:
                # Inject the current data automatically (if the function accepts a 'current_data' parameter)
                if 'current_data' in sig.parameters:
                    kwargs['current_data'] = current_data
                res = func(**kwargs)
            except Exception as e:
                logger.info(f"Error in group {current_meta}: {e}")
                if self._expected_dict_keys is not None:
                    res = {k: pd.DataFrame() for k in self._expected_dict_keys}
                else:
                    res = pd.DataFrame()
            finally:
                self.m_eval.data = old_data

            # (handle the dict and DataFrame cases)
            if isinstance(res, dict):
                if self._expected_dict_keys is None and res:
                    self._expected_dict_keys = list(res.keys())
                for key, df in res.items():
                    if df is not None and not df.empty:
                        for col_name, col_val in current_meta.items():
                            df[col_name] = col_val
                    else:
                        empty_df = pd.DataFrame()
                        for col_name, col_val in current_meta.items():
                            empty_df[col_name] = [col_val]
                        res[key] = empty_df
                return res
            else:
                if self._expected_columns is None and res is not None and not res.empty:
                    self._expected_columns = res.columns.tolist()
                if res is not None and not res.empty:
                    res = res.copy()
                    for key, val in current_meta.items():
                        res[key] = val
                else:
                    if self._expected_columns is not None:
                        res = pd.DataFrame(columns=self._expected_columns)
                    else:
                        res = pd.DataFrame()
                return res

        step = self.steps[step_idx]
        if step['type'] == 'group':
            group_name = step['group_name']
            group_var_name = step['group_var_name']
            min_size = step['min_size']
            results = []
            for gval in current_data[group_name].unique():
                sub_data = current_data[current_data[group_name] == gval]
                if len(sub_data) >= min_size:
                    new_meta = current_meta.copy()
                    new_meta[group_var_name] = gval
                    sub_res = _execute(step_idx + 1, sub_data, new_meta)
                    if sub_res is not None:
                        results.append(sub_res)
            if not results:
                if self._expected_dict_keys is not None:
                    return {k: pd.DataFrame() for k in self._expected_dict_keys}
                else:
                    return pd.DataFrame()
            # Merge the results
            if isinstance(results[0], dict):
                merged = {k: pd.concat([r[k] for r in results], ignore_index=True) for k in results[0].keys()}
                return merged
            else:
                return pd.concat(results, ignore_index=True)

        elif step['type'] == 'subset':
            condition_dict = step['condition_dict']
            subset_var_name = step['subset_var_name']
            min_size = step['min_size']
            results = []
            for label, query in condition_dict.items():
                if query == "":
                    sub_data = current_data.copy()
                else:
                    sub_data = current_data.query(query)
                if len(sub_data) >= min_size:
                    new_meta = current_meta.copy()
                    new_meta[subset_var_name] = label
                    sub_res = _execute(step_idx + 1, sub_data, new_meta)
                    if sub_res is not None:
                        results.append(sub_res)
            if not results:
                if self._expected_dict_keys is not None:
                    return {k: pd.DataFrame() for k in self._expected_dict_keys}
                else:
                    return pd.DataFrame()
            if isinstance(results[0], dict):
                merged = {k: pd.concat([r[k] for r in results], ignore_index=True) for k in results[0].keys()}
                return merged
            else:
                return pd.concat(results, ignore_index=True)

    return _execute(0, self.original_data, {})

Utility_Functions

Utility functions for data processing and calculations.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

class Utility_Functions:
    """Utility functions for data processing and calculations."""

    @staticmethod
    def valid_average(x):
        """
        Calculate the average of positive values in a Series.

        This function filters out non-positive values (including zeros and negatives)
        before calculating the mean, useful for financial or count data where
        zero or negative values should not contribute to the average.

        Parameters
        ----------
        x : pandas.Series
            Input series of numeric values.

        Returns
        -------
        float
            Rounded average of positive values, or 0 if no valid values exist.

        Examples
        --------
        >>> import pandas as pd
        >>> Utility_Functions.valid_average(pd.Series([1, 2, -1, 0, 3]))
        2.0
        """
        valid_mask = x > 0
        valid_count = valid_mask.sum()
        valid_values = x[valid_mask]
        value_sum = valid_values.sum()

        if valid_count > 0:
            return round(value_sum / valid_count, 2)
        return 0

valid_average staticmethod

valid_average(x)

Calculate the average of positive values in a Series.

This function filters out non-positive values (including zeros and negatives) before calculating the mean, useful for financial or count data where zero or negative values should not contribute to the average.

Parameters:

Name	Type	Description	Default
x	Series	Input series of numeric values.	required

Returns:

Type	Description
float	Rounded average of positive values, or 0 if no valid values exist.

Examples:

>>> import pandas as pd
>>> Utility_Functions.valid_average(pd.Series([1, 2, -1, 0, 3]))
2.0

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

@staticmethod
def valid_average(x):
    """
    Calculate the average of positive values in a Series.

    This function filters out non-positive values (including zeros and negatives)
    before calculating the mean, useful for financial or count data where
    zero or negative values should not contribute to the average.

    Parameters
    ----------
    x : pandas.Series
        Input series of numeric values.

    Returns
    -------
    float
        Rounded average of positive values, or 0 if no valid values exist.

    Examples
    --------
    >>> import pandas as pd
    >>> Utility_Functions.valid_average(pd.Series([1, 2, -1, 0, 3]))
    2.0
    """
    valid_mask = x > 0
    valid_count = valid_mask.sum()
    valid_values = x[valid_mask]
    value_sum = valid_values.sum()

    if valid_count > 0:
        return round(value_sum / valid_count, 2)
    return 0

Model_Evaluation_Tool

A comprehensive tool for model evaluation, performance comparison, and gains analysis.

This class provides methods for: - Base score calculation and correlation analysis - Model performance comparison across multiple metrics - Gains table generation and analysis - Cross-risk analysis - Multi-dimensional evaluation with subsets and groupings

Parameters:

Name	Type	Description	Default
data	DataFrame	Input data containing features and target variables.	required
dep	str	Name of the dependent/target variable.	required
comp_scrlist	list	List of comparison score column names.	required
model	object	Trained model with predict_proba method. Default is None.	None
base_score	str	Name of the base score column. Default is None.	None
min_data_size	int	Minimum data size threshold: the minimum number of rows of a group in get_gains_summary (used with grp_name). Default is 500.	500
equal_freq	bool	Whether to use equal frequency binning. Default is True.	True
precision	int	Decimal precision for rounding. Default is 5.	5
chi2_method	bool	Whether to use chi-square method. Default is False.	False
chi2_p	float	Chi-square p-value threshold. Default is 0.999.	0.999
init_equi_bins	int	Initial number of equal frequency bins. Default is 500.	500
tree_binning	bool	Whether to use tree-based binning. Default is False.	False
random_seed	int	Random seed for reproducibility. Default is 3407.	3407
nbins	int	Number of bins for analysis. Default is 10.	10
min_bin_prop	float	Minimum proportion for each bin. Default is 0.05.	0.05
include_missing	bool	Whether to include missing values. Default is True.	True
missing_rate_ref	int or float	Reference value for missing rate: the fill value for missing values, passed as fillna to the score binning of get_cross_risk_summary and used to fill the analyzed variables in run_variable_analysis_summary. Default is -99999999.	-99999999
excel_path	str	Path for Excel output. Default is None. It is only stored as an attribute: no method of this class writes an Excel file.	None
fillna	any	Fill value for missing scores in the Gains-table binning of get_gains_summary. Default is -999999.	-999999
spec_values	list	Special score values that get their own bins in get_gains_summary and get_cross_risk_summary. Default is None, which is stored as an empty list.	None
cross_agg_dict	dict	Columns aggregated in the cells of get_cross_risk_summary, as {column: aggregation or list of aggregations}. Default is None, which uses a legacy dictionary for the columns is_dpd7, credit_limit, monthlyincome and education; pass your own dictionary whenever you use that method.	None
subset_condition_dict	dict	Subsets read by multi_subset_wrapper and multi_dim_eval, as {label: DataFrame.query string} (an empty string keeps every row). Default is None, which uses a legacy dictionary of queries on the column aprvvrsn_2 (Overall, AE, NON_AE, FT, NON_FT and NON_FT_AE).	None
eval_ylabels	list of str	Target columns evaluated by multi_ylabel_wrapper and multi_dim_eval. Default is None, which uses the legacy ['is_dpd7'].	None
grp_namelist	list of str	Grouping columns evaluated by multi_dim_eval. Default is None, which uses the legacy ['sample_ind_fnl', 'aprvvrsn_2', 'week_start_date'].	None
gains_display_metric_list	list of str	Gains-table columns kept by get_gains_summary when it gets no add_func. Default is None, which keeps MIN, MAX, N, PROP, AVG_SCORE, AVG_BAD, CUM_BAD_PCT, KS_PER_BIN, LIFT and RANK_ORDER_BUMP.	None
weight_col	str	Name of the sample weight column in data. It makes model_perf_compare and get_gains_summary weighted, except when grp_name is given (the grouped paths are unweighted); get_cross_risk_summary and cross_perf_eval never use it, and neither does multi_dim_eval with its default eval_func, which always groups. Default is None, i.e. unweighted.	None
positive_score_only	bool	Whether model_perf_compare keeps only the rows whose score is greater than 0 (0 and negative values conventionally mean "no score"). Default is True.	True

Attributes:

Name	Type	Description
data	DataFrame	The input data (may be modified during processing).
dep	str	Target variable name.
base_score	str or None	Name of the base score column.
comp_scrlist	list	List of comparison score names.
eval_ylabels	list	Labels for y-axis evaluation.
gains_display_metric_list	list	Metrics to display in gains summary.

Notes

Every constructor parameter is also stored as an attribute of the same name, except random_seed, which is stored as seed; udf holds a Utility_Functions instance. cross_agg_dict, subset_condition_dict, eval_ylabels and grp_namelist default to legacy values tied to one production dataset, so override them whenever you use the methods that read them. get_cross_risk_summary also needs a flow_id column in data.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

class Model_Evaluation_Tool:
    """
    A comprehensive tool for model evaluation, performance comparison, and gains analysis.

    This class provides methods for:
    - Base score calculation and correlation analysis
    - Model performance comparison across multiple metrics
    - Gains table generation and analysis
    - Cross-risk analysis
    - Multi-dimensional evaluation with subsets and groupings

    Parameters
    ----------
    data : pandas.DataFrame
        Input data containing features and target variables.
    dep : str
        Name of the dependent/target variable.
    comp_scrlist : list
        List of comparison score column names.
    model : object, optional
        Trained model with predict_proba method. Default is None.
    base_score : str, optional
        Name of the base score column. Default is None.
    min_data_size : int, optional
        Minimum data size threshold: the minimum number of rows of a group in ``get_gains_summary`` (used with
        ``grp_name``). Default is 500.
    equal_freq : bool, optional
        Whether to use equal frequency binning. Default is True.
    precision : int, optional
        Decimal precision for rounding. Default is 5.
    chi2_method : bool, optional
        Whether to use chi-square method. Default is False.
    chi2_p : float, optional
        Chi-square p-value threshold. Default is 0.999.
    init_equi_bins : int, optional
        Initial number of equal frequency bins. Default is 500.
    tree_binning : bool, optional
        Whether to use tree-based binning. Default is False.
    random_seed : int, optional
        Random seed for reproducibility. Default is 3407.
    nbins : int, optional
        Number of bins for analysis. Default is 10.
    min_bin_prop : float, optional
        Minimum proportion for each bin. Default is 0.05.
    include_missing : bool, optional
        Whether to include missing values. Default is True.
    missing_rate_ref : int or float, optional
        Reference value for missing rate: the fill value for missing values, passed as ``fillna`` to the score binning of
        ``get_cross_risk_summary`` and used to fill the analyzed variables in ``run_variable_analysis_summary``.
        Default is -99999999.
    excel_path : str, optional
        Path for Excel output. Default is None. It is only stored as an attribute: no method of this class writes an
        Excel file.
    fillna : any, optional
        Fill value for missing scores in the Gains-table binning of ``get_gains_summary``. Default is -999999.
    spec_values : list, optional
        Special score values that get their own bins in ``get_gains_summary`` and ``get_cross_risk_summary``.
        Default is None, which is stored as an empty list.
    cross_agg_dict : dict, optional
        Columns aggregated in the cells of ``get_cross_risk_summary``, as ``{column: aggregation or list of aggregations}``.
        Default is None, which uses a legacy dictionary for the columns ``is_dpd7``, ``credit_limit``, ``monthlyincome``
        and ``education``; pass your own dictionary whenever you use that method.
    subset_condition_dict : dict, optional
        Subsets read by ``multi_subset_wrapper`` and ``multi_dim_eval``, as ``{label: DataFrame.query string}`` (an empty
        string keeps every row). Default is None, which uses a legacy dictionary of queries on the column ``aprvvrsn_2``
        (``Overall``, ``AE``, ``NON_AE``, ``FT``, ``NON_FT`` and ``NON_FT_AE``).
    eval_ylabels : list of str, optional
        Target columns evaluated by ``multi_ylabel_wrapper`` and ``multi_dim_eval``. Default is None, which uses the legacy
        ``['is_dpd7']``.
    grp_namelist : list of str, optional
        Grouping columns evaluated by ``multi_dim_eval``. Default is None, which uses the legacy
        ``['sample_ind_fnl', 'aprvvrsn_2', 'week_start_date']``.
    gains_display_metric_list : list of str, optional
        Gains-table columns kept by ``get_gains_summary`` when it gets no ``add_func``. Default is None, which keeps
        ``MIN``, ``MAX``, ``N``, ``PROP``, ``AVG_SCORE``, ``AVG_BAD``, ``CUM_BAD_PCT``, ``KS_PER_BIN``, ``LIFT`` and
        ``RANK_ORDER_BUMP``.
    weight_col : str, optional
        Name of the sample weight column in ``data``. It makes ``model_perf_compare`` and ``get_gains_summary`` weighted,
        except when ``grp_name`` is given (the grouped paths are unweighted); ``get_cross_risk_summary`` and
        ``cross_perf_eval`` never use it, and neither does ``multi_dim_eval`` with its default ``eval_func``, which always
        groups. Default is None, i.e. unweighted.
    positive_score_only : bool, optional
        Whether ``model_perf_compare`` keeps only the rows whose score is greater than 0 (0 and negative values
        conventionally mean "no score"). Default is True.

    Attributes
    ----------
    data : pandas.DataFrame
        The input data (may be modified during processing).
    dep : str
        Target variable name.
    base_score : str or None
        Name of the base score column.
    comp_scrlist : list
        List of comparison score names.
    eval_ylabels : list
        Labels for y-axis evaluation.
    gains_display_metric_list : list
        Metrics to display in gains summary.

    Notes
    -----
    Every constructor parameter is also stored as an attribute of the same name, except ``random_seed``, which is stored as
    ``seed``; ``udf`` holds a ``Utility_Functions`` instance. ``cross_agg_dict``, ``subset_condition_dict``,
    ``eval_ylabels`` and ``grp_namelist`` default to legacy values tied to one production dataset, so override them
    whenever you use the methods that read them. ``get_cross_risk_summary`` also needs a ``flow_id`` column in ``data``.
    """

    def __init__(
        self,
        data: pd.DataFrame,
        dep: str,
        comp_scrlist: List[str],
        model: Any = None,
        base_score: Optional[str] = None,
        min_data_size: int = 500,
        equal_freq: bool = True,
        precision: int = 5,
        chi2_method: bool = False,
        chi2_p: float = 0.999,
        init_equi_bins: int = 500,
        tree_binning: bool = False,
        random_seed: int = 3407,
        nbins: int = 10,
        min_bin_prop: float = 0.05,
        include_missing: bool = True,
        missing_rate_ref: Union[int, float] = -99999999,
        excel_path: Optional[str] = None,
        fillna=-999999,
        spec_values=None,
        cross_agg_dict: Optional[Dict] = None,
        subset_condition_dict: Optional[Dict[str, str]] = None,
        eval_ylabels: Optional[List[str]] = None,
        grp_namelist: Optional[List[str]] = None,
        gains_display_metric_list: Optional[List[str]] = None,
        weight_col: Optional[str] = None,
        positive_score_only: bool = True,
    ):
        """
        Initialize the Model_Evaluation_Tool with configuration parameters.

        Parameters
        ----------
        data : pandas.DataFrame
            Input dataset for evaluation.
        dep : str
            Name of the target/dependent variable.
        comp_scrlist : list
            List of comparison score column names.
        model : object, optional
            Trained model with predict_proba method.
        base_score : str, optional
            Base score column name.
        min_data_size : int, optional
            Minimum data size threshold: the minimum number of rows of a group in ``get_gains_summary`` (used with
            ``grp_name``). Default is 500.
        equal_freq : bool, optional
            Use equal frequency binning if True.
        precision : int, optional
            Decimal precision for calculations.
        chi2_method : bool, optional
            Use chi-square method if True.
        chi2_p : float, optional
            Chi-square p-value threshold.
        init_equi_bins : int, optional
            Initial equal bins count.
        tree_binning : bool, optional
            Use tree-based binning if True.
        random_seed : int, optional
            Random seed value.
        nbins : int, optional
            Number of bins.
        min_bin_prop : float, optional
            Minimum bin proportion.
        include_missing : bool, optional
            Include missing values if True.
        missing_rate_ref : int or float, optional
            Missing rate reference value: the fill value for missing values, passed as ``fillna`` to the score binning of
            ``get_cross_risk_summary`` and used to fill the analyzed variables in ``run_variable_analysis_summary``.
        excel_path : str, optional
            Path for Excel output file. It is only stored as an attribute: no method of this class writes an Excel file.
        fillna : any, optional
            Fill value for missing scores in the Gains-table binning of ``get_gains_summary``. Default is -999999.
        spec_values : list, optional
            Special score values that get their own bins in ``get_gains_summary`` and ``get_cross_risk_summary``.
            Default is None, which is stored as an empty list.
        cross_agg_dict : dict, optional
            Columns aggregated in the cells of ``get_cross_risk_summary``, as ``{column: aggregation or list of
            aggregations}``. Default is None, which uses a legacy dictionary for the columns ``is_dpd7``,
            ``credit_limit``, ``monthlyincome`` and ``education``.
        subset_condition_dict : dict, optional
            Subsets read by ``multi_subset_wrapper`` and ``multi_dim_eval``, as ``{label: DataFrame.query string}`` (an
            empty string keeps every row). Default is None, which uses a legacy dictionary of queries on the column
            ``aprvvrsn_2``.
        eval_ylabels : list of str, optional
            Target columns evaluated by ``multi_ylabel_wrapper`` and ``multi_dim_eval``. Default is None, which uses the
            legacy ``['is_dpd7']``.
        grp_namelist : list of str, optional
            Grouping columns evaluated by ``multi_dim_eval``. Default is None, which uses the legacy
            ``['sample_ind_fnl', 'aprvvrsn_2', 'week_start_date']``.
        gains_display_metric_list : list of str, optional
            Gains-table columns kept by ``get_gains_summary`` when it gets no ``add_func``. Default is None, which keeps
            ``MIN``, ``MAX``, ``N``, ``PROP``, ``AVG_SCORE``, ``AVG_BAD``, ``CUM_BAD_PCT``, ``KS_PER_BIN``, ``LIFT`` and
            ``RANK_ORDER_BUMP``.
        weight_col : str, optional
            Name of the sample weight column in ``data``. It makes ``model_perf_compare`` and ``get_gains_summary``
            weighted, except when ``grp_name`` is given; ``get_cross_risk_summary`` and ``cross_perf_eval`` never use it,
            and neither does ``multi_dim_eval`` with its default ``eval_func``. Default is None, i.e. unweighted.
        positive_score_only : bool, optional
            Whether ``model_perf_compare`` keeps only the rows whose score is greater than 0. Default is True.
        """
        self.data = data
        self.dep = dep
        self.model = model
        self.comp_scrlist = comp_scrlist
        self.base_score = base_score
        self.min_data_size = min_data_size
        self.equal_freq = equal_freq
        self.precision = precision
        self.nbins = nbins
        self.min_bin_prop = min_bin_prop
        self.include_missing = include_missing
        self.tree_binning = tree_binning
        self.seed = random_seed
        self.excel_path = excel_path
        self.missing_rate_ref = missing_rate_ref
        self.fillna = fillna
        self.spec_values = [] if spec_values is None else spec_values
        self.weight_col = weight_col
        self.positive_score_only = positive_score_only

        self.chi2_method = chi2_method
        self.chi2_p = chi2_p
        self.init_equi_bins = init_equi_bins

        self.udf = Utility_Functions()
        self.cross_agg_dict = (
            cross_agg_dict if cross_agg_dict is not None
            else _legacy_cross_agg_dict(self.udf)
        )
        self.subset_condition_dict = (
            subset_condition_dict if subset_condition_dict is not None
            else _legacy_subset_condition_dict()
        )
        self.eval_ylabels = (
            eval_ylabels if eval_ylabels is not None else list(_LEGACY_EVAL_YLABELS)
        )
        self.grp_namelist = (
            grp_namelist if grp_namelist is not None else list(_LEGACY_GRP_NAMELIST)
        )
        self.gains_display_metric_list = (
            gains_display_metric_list if gains_display_metric_list is not None
            else list(_DEFAULT_GAINS_DISPLAY_METRICS)
        )

    def _score_order(self) -> List[str]:
        scores = []
        if self.base_score is not None:
            scores.append(self.base_score)
        scores.extend(self.comp_scrlist or [])
        return scores

    def _valid_score_mask(self, values: pd.Series) -> pd.Series:
        """Rows where the score is a finite number (and above 0 with ``positive_score_only``)."""
        numeric = pd.to_numeric(values, errors="coerce").astype(float)
        mask = pd.Series(np.isfinite(numeric.to_numpy()), index=values.index)
        if self.positive_score_only:
            mask &= numeric.gt(0).to_numpy()
        return mask

    def _build_performance_evaluator(
        self,
        score: str,
        dist_bins: int,
        pct_bins: int,
        min_bin_prop: Optional[float] = None,
        include_missing: Optional[bool] = None,
        equal_freq: Optional[bool] = None,
    ) -> PerformanceEvaluator:
        return PerformanceEvaluator(
            tgt_name=self.dep,
            scr_name=score,
            dist_bins=dist_bins,
            pct_bins=pct_bins,
            precision=self.precision,
            min_bin_prop=self.min_bin_prop if min_bin_prop is None else min_bin_prop,
            include_missing=False if include_missing is None else include_missing,
            equal_freq=self.equal_freq if equal_freq is None else equal_freq,
            chi2_method=self.chi2_method,
            init_equi_bins=self.init_equi_bins,
            chi2_p=self.chi2_p,
            tree_binning=self.tree_binning,
            random_state=self.seed,
            weight_col=self.weight_col,
        )

    def _build_gains_calculator(
        self,
        data: pd.DataFrame,
        score: str,
        nbins: int,
        fillna,
        spec_values,
        include_missing: bool,
        ascending: bool = True,
    ) -> GainsTableCalculator:
        return GainsTableCalculator(
            data=data,
            dep=self.dep,
            score=score,
            nbins=nbins,
            precision=self.precision,
            min_bin_prop=self.min_bin_prop,
            include_missing=include_missing,
            equal_freq=self.equal_freq,
            chi2_method=self.chi2_method,
            chi2_p=self.chi2_p,
            init_equi_bins=self.init_equi_bins,
            fillna=fillna,
            spec_values=spec_values,
            tree_binning=self.tree_binning,
            random_state=self.seed,
            ascending=ascending,
            weight_col=self.weight_col,
        )

    def __init_excel_master(self):
        """
        Initialize Excel Master for report generation.

        Returns
        -------
        ExcelMaster
            Initialized ExcelMaster instance.

        Notes
        -----
        Requires ExcelMaster package to be installed.
        """
        from ExcelMaster.ExcelMaster import ExcelMaster

        em = ExcelMaster(filepath=self.excel_path, verbose=False)
        return em

    def get_base_score(self, scorename: str = '_base_model_score_', disp: bool = False):
        """
        Calculate base model scores using the provided model.

        This method uses the model's predict_proba method to generate scores
        for the input data and optionally displays correlation analysis.

        Parameters
        ----------
        scorename : str, optional
            Name for the score column. Default is '_base_model_score_'.
        disp : bool, optional
            Whether to display intermediate results. Default is False.

        Returns
        -------
        Model_Evaluation_Tool
            Returns self for method chaining.

        Raises
        ------
        AttributeError
            If model is None or lacks required attributes.

        Notes
        -----
        The model must expose ``feature_names_in_``: those columns are passed to ``predict_proba``. As a side effect,
        ``self.data`` is replaced by a copy that has the new score column and ``self.base_score`` is set to ``scorename``.
        ``disp=True`` needs IPython.
        """
        if self.model is None:
            raise AttributeError("Model must be provided to calculate base score.")

        data = self.data.copy()
        model = self.model

        inputs = model.feature_names_in_.tolist()
        data[scorename] = model.predict_proba(data[inputs])[:, 1]
        self.data = data
        self.base_score = scorename

        scrlist = [scorename] + self.comp_scrlist
        if disp:
            from IPython.display import display
            from Modeling_Tool.Feature.Distribution_Tool import proc_means_by_grp
            display(proc_means_by_grp(data, scrlist))

        tmp_data = data[data[scrlist] > 0]

        if disp:
            from IPython.display import display
            display(tmp_data[scrlist].corr())

        return self

    def get_score_correlation(
        self,
        score_list: Optional[List[str]] = None,
        method: str = 'pearson'
    ) -> pd.DataFrame:
        """
        Calculate correlation matrix between score columns.

        Parameters
        ----------
        score_list : list, optional
            List of score column names. If None, uses base_score and comp_scrlist.
        method : str, optional
            Correlation method ('pearson', 'spearman', 'kendall'). Default is 'pearson'.

        Returns
        -------
        pandas.DataFrame
            Correlation summary in long format with columns: 'base', 'compare', 'corr'.

        Raises
        ------
        ValueError
            If ``base_score`` is not set.

        Notes
        -----
        Only the rows in which every score of ``score_list`` is greater than 0 are used (rows with a missing score are
        dropped as well), whatever ``positive_score_only`` is.
        """
        if score_list is None:
            score_list = [self.base_score] + self.comp_scrlist

        if self.base_score is None:
            raise ValueError("Base score must be set before calculating correlation.")

        data = self.data.copy()

        score_query = " and ".join([s + " > 0" for s in score_list])
        data = data.query(score_query)

        corr_summary = data[score_list]\
            .corr(method=method)\
            .reset_index(drop=False)\
            .melt(id_vars=['index'], var_name='compare', value_name='corr')\
            .rename(columns={"index": "base"})

        return corr_summary

    def model_perf_compare(
        self,
        data: Optional[pd.DataFrame] = None,
        grp_name: Optional[str] = None,
        dist_bins: int = 100,
        pct_bins: int = 10,
        min_data_size: int = 50,
        sync_data_size: bool = True,
        min_bin_prop: Optional[float] = None,
        include_missing: Optional[bool] = None,
        equal_freq: Optional[bool] = None,
        sample_name: Optional[str] = None,
    ) -> pd.DataFrame:
        """
        Compare model performance across different scores.

        This method calculates performance metrics (AUC, KS, etc.) for the base score
        and all comparison scores via ``PerformanceEvaluator``, optionally filtering
        by group.

        Parameters
        ----------
        data : pandas.DataFrame, optional
            Data to use. If None, uses self.data.
        grp_name : str, optional
            Group name column for stratification. Every score is then evaluated separately for each group value (the
            result gains a column named ``grp_name``), and ``weight_col`` is ignored.
        dist_bins : int, optional
            Number of distribution bins. Default is 100.
        pct_bins : int, optional
            Number of percentile bins. Default is 10.
        min_data_size : int, optional
            Minimum number of rows of a group for it to be evaluated; it only matters when ``grp_name`` is given. Default
            is 50 (the ``min_data_size`` of the instance is not used here).
        sync_data_size : bool, optional
            Whether every score, the base score included, is evaluated on the same rows: the rows with an observed target
            where every score is valid (see Notes). False evaluates each score on its own valid rows. Default is True.
        min_bin_prop : float, optional
            Minimum bin proportion. Defaults to the instance setting.
        include_missing : bool, optional
            Whether to include missing values in performance binning.
            Defaults to ``False`` for performance comparison.
        equal_freq : bool, optional
            Use equal-frequency binning. Defaults to the instance setting.
        sample_name : str, optional
            Label applied to the evaluated dataset and emitted in the result's
            ``index`` column. Defaults to ``'all'`` (the whole ``data`` treated as
            a single sample). Callers evaluating a specific slice can pass e.g.
            ``'global'`` or a split name.

        Returns
        -------
        pandas.DataFrame
            Performance comparison results sorted by score order: the ``PerformanceEvaluator`` summary of every score
            (``AUC_Shift`` and ``KS_Shift`` are dropped) with the extra columns ``score_name`` and ``N_OWN``. An empty
            DataFrame is returned when ``base_score`` is not set or no valid row is left.

        Notes
        -----
        A score is valid in a row when it is a finite number and, with ``positive_score_only=True``, greater than 0; rows
        whose target is missing are never used. The column ``N_OWN`` gives the number of rows (not weights) in which the
        score alone is valid, so with ``sync_data_size=True`` it shows how much each score's coverage was narrowed to the
        common rows. A score without any valid row is left out of the result with a ``UserWarning`` instead of emptying
        the common rows of the others; when the scores have valid rows but none in common, a ``UserWarning`` is issued
        and an empty DataFrame returned. With ``weight_col`` set on the instance (and no ``grp_name``) the weighted,
        narrower summary is returned.
        """
        data = self.data.copy() if data is None else data.copy()
        score_list = self.comp_scrlist
        dep = self.dep
        base_score = self.base_score
        # Label for the single (unsplit) dataset being evaluated; surfaces as the
        # 'index' column of the result. Defaults to a neutral 'all' rather than the
        # historical (and misleading) 'oot', which implied an out-of-time sample.
        sample_name = sample_name if sample_name is not None else 'all'

        if base_score is None:
            return pd.DataFrame()

        score_order = [score for score in self._score_order() if score in data.columns]
        target_ok = data[dep].notna()
        valid = {score: target_ok & self._valid_score_mask(data[score]) for score in score_order}
        own_n = {score: int(mask.sum()) for score, mask in valid.items()}
        without_rows = [score for score in score_order if own_n[score] == 0]
        if without_rows:
            rule = "a finite value above 0" if self.positive_score_only else "a finite value"
            warnings.warn(
                f"model_perf_compare: {without_rows} have no row with an observed target and {rule}; "
                f"they are left out of the comparison.",
                UserWarning,
                stacklevel=2,
            )
        active = [score for score in score_order if own_n[score] > 0]
        if not active:
            return pd.DataFrame()
        if sync_data_size:
            common = target_ok.copy()
            for score in active:
                common &= valid[score]
            if not bool(common.any()):
                warnings.warn(
                    f"model_perf_compare: the scores {active} have valid rows but none in common, so they cannot be "
                    f"compared on the same rows; pass sync_data_size=False to evaluate each score on its own rows.",
                    UserWarning,
                    stacklevel=2,
                )
                return pd.DataFrame()
            rows_for = {score: common for score in active}
        else:
            rows_for = {score: valid[score] for score in active}

        perf_comp_dict = {}
        for score in active:
            score_data = data.loc[rows_for[score]]

            evaluator = self._build_performance_evaluator(
                score=score,
                dist_bins=dist_bins,
                pct_bins=pct_bins,
                min_bin_prop=min_bin_prop,
                include_missing=include_missing,
                equal_freq=equal_freq,
            )
            perf = evaluator.add_dataset(sample_name, score_data).evaluate(
                oot_grp_name=grp_name,
                min_data_size=min_data_size,
                to_show=False,
                display=False,
            )
            if isinstance(perf, pd.DataFrame) and not perf.empty and 'index' in perf.columns:
                perf = perf.query(f"index == '{sample_name}'")
            if isinstance(perf, pd.DataFrame):
                perf = perf.copy()
                perf['N_OWN'] = own_n[score]
            perf_comp_dict[score] = perf

        if not perf_comp_dict:
            return pd.DataFrame()

        perf_comp_res = []
        for score_name, perf in perf_comp_dict.items():
            perf = perf.copy()
            perf['score_name'] = score_name
            perf_comp_res.append(perf)

        perf_comp_res = pd.concat(perf_comp_res)

        drop_cols = ['AUC_Shift', 'KS_Shift']
        for col in drop_cols:
            if col in perf_comp_res.columns:
                perf_comp_res = perf_comp_res.drop(columns=[col])

        order_map = {val: i for i, val in enumerate(active) if val in perf_comp_dict}
        perf_comp_res['sort_key'] = perf_comp_res['score_name'].map(order_map)
        perf_comp_res_sorted = perf_comp_res.sort_values('sort_key').drop('sort_key', axis=1)

        return perf_comp_res_sorted

    def get_gains_summary(
        self,
        data: Optional[pd.DataFrame] = None,
        grp_name: Optional[str] = None,
        disp: bool = True,
        grp_disp_metric: List[str] = None,
        grp_nbins: int = 5,
        withSummary: bool = True,
        add_func: Optional[Callable] = None,
        sync_range: bool = True,
        spec_values=None,
        include_missing: Optional[bool] = None,
        fillna=None,
    ) -> pd.DataFrame:
        """
        Generate gains summary table for score analysis.

        This method creates gains tables via ``GainsTableCalculator``, showing the
        distribution of targets across score bins, with optional group stratification.
        Custom metrics can be injected through ``add_func``.

        Parameters
        ----------
        data : pandas.DataFrame, optional
            Data to use. If None, uses self.data.
        grp_name : str, optional
            Group name for stratified analysis: one Gains table per group value, stacked, with the group value in a column
            named ``grp_name``. The number of bins is then ``grp_nbins`` instead of the instance's ``nbins``, groups with
            fewer rows than the instance's ``min_data_size`` are skipped, a score with 10 or fewer positive values is
            skipped, and the weights are ignored.
        disp : bool, optional
            Whether to display, with ``IPython.display.display``, one pivot table (one column per group) for each metric
            of ``grp_disp_metric``. It only applies when ``grp_name`` is given. Default is True.
        grp_disp_metric : list, optional
            Metrics shown by ``disp`` for grouped analysis; it does not change the returned table. Default is None, i.e.
            ``['N', 'PROP', 'AVG_BAD', 'LIFT']``.
        grp_nbins : int, optional
            Number of bins for grouped analysis. Default is 5.
        withSummary : bool, optional
            Include summary row. Default is True. It is not applied with ``grp_name``.
        add_func : callable, optional
            Custom metric function merged into each gains table. When it is given, all the columns of the Gains table are
            returned instead of only ``gains_display_metric_list``. It is applied on the weighted path too.
        sync_range : bool, optional
            Synchronize bin ranges across the groups (used with ``grp_name`` only). Default is True.
        spec_values : list, optional
            Special values to treat separately during binning. Default is None, i.e. the instance setting.
        include_missing : bool, optional
            Whether to include missing values. Defaults to the instance setting.
        fillna : any, optional
            Missing-value fill for score binning. Defaults to the instance setting.

        Returns
        -------
        pandas.DataFrame
            Gains summary results with score names: the Gains tables of the base score and of every comparison score,
            stacked, with the bin columns ``_bin_num`` and ``_bin_range`` and a ``score_name`` column (plus the group
            column when ``grp_name`` is given). Only the ``gains_display_metric_list`` columns are kept unless ``add_func``
            is given. An empty DataFrame is returned when none of the scores is a column of the data.

        Notes
        -----
        When the instance has a ``weight_col`` and ``grp_name`` is None, the weighted Gains table is returned (``N`` is the
        sum of the weights and ``N_RAW`` the number of rows), with the ``add_func`` columns and the summary row when they
        are asked for.
        """
        if grp_disp_metric is None:
            grp_disp_metric = ['N', 'PROP', 'AVG_BAD', 'LIFT']

        if fillna is None:
            fillna = self.fillna
        if spec_values is None:
            spec_values = self.spec_values
        if include_missing is None:
            include_missing = self.include_missing

        new_oot_df = self.data.copy() if data is None else data.copy()

        dep = self.dep
        base_score = self.base_score
        min_data_size = self.min_data_size
        nbins = self.nbins if grp_name is None else grp_nbins

        score_list = self._score_order()
        display_metric_list = self.gains_display_metric_list if add_func is None else None

        if self.data is None or len(self.data) == 0:
            empty_df = pd.DataFrame()
            empty_df.index = pd.MultiIndex.from_tuples([], names=['_bin_num', '_bin_range'])
            return empty_df

        gains_table_dict = {}
        for score in score_list:
            if score not in new_oot_df.columns:
                continue

            calculator = self._build_gains_calculator(
                data=new_oot_df,
                score=score,
                nbins=nbins,
                fillna=fillna,
                spec_values=spec_values,
                include_missing=include_missing,
                ascending=True,
            )

            if grp_name is None:
                gains_table_dict[score] = calculator.calculate(
                    withSummary=withSummary,
                    add_func=add_func,
                )
                if display_metric_list is not None:
                    gains_table_dict[score] = gains_table_dict[score][display_metric_list]
                gains_table_dict[score] = gains_table_dict[score].reset_index(drop=False)
            else:
                if new_oot_df.query(f"{score} > 0").shape[0] > 10:
                    oot_by_group = calculator.calculate(
                        grp_name=grp_name,
                        min_data_size=min_data_size,
                        sync_range=sync_range,
                        retSummary=False,
                        withSummary=withSummary,
                        add_func=add_func,
                    )
                    if display_metric_list is not None:
                        oot_by_group = oot_by_group[[*display_metric_list, grp_name]]
                    oot_by_group = oot_by_group.reset_index(drop=False)

                    grp_metric_dict = {}
                    for value in [x for x in grp_disp_metric if x not in ['PROP']]:
                        grp_metric_dict[value] = oot_by_group\
                            .reset_index(drop=False)\
                            .pivot_table(
                                index=['_bin_num', '_bin_range'],
                                columns=[grp_name],
                                values=value,
                                margins=True,
                                margins_name="Grand_Total",
                            )

                    if 'PROP' in grp_disp_metric:
                        tot_cnt = oot_by_group\
                            .reset_index(drop=False)\
                            .pivot_table(
                                index=['_bin_num', '_bin_range'],
                                columns=[grp_name],
                                values=['N'],
                                margins=True,
                                margins_name="Grand_Total",
                                aggfunc='sum',
                            )

                        if tot_cnt.shape[0] > 0:
                            grp_metric_dict['PROP'] = tot_cnt.iloc[0:tot_cnt.shape[0]:] / np.array(
                                tot_cnt.iloc[-1:].T['Grand_Total'].tolist()
                            )
                            grp_metric_dict['PROP'].columns = grp_metric_dict['PROP'].rename(
                                columns={'N': ''}
                            ).columns.map("".join)

                    fnl_grp_gains_res = []
                    for colname, gains in grp_metric_dict.items():
                        gains = gains.copy()
                        gains['variable'] = colname
                        if disp:
                            from IPython.display import display
                            display(gains)
                        fnl_grp_gains_res.append(gains)
                    fnl_grp_gains_res = pd.concat(fnl_grp_gains_res)

                    gains_table_dict[score] = oot_by_group

        fnl_gains_res = []
        for score_name, gains_res in gains_table_dict.items():
            gains_res = gains_res.copy()
            gains_res['score_name'] = score_name
            fnl_gains_res.append(gains_res)

        if not fnl_gains_res:
            return pd.DataFrame()

        return pd.concat(fnl_gains_res)

    def get_cross_risk_summary(
        self,
        cross_agg_dict: Optional[Dict] = None,
        nbins: int = 5,
        equal_freq: Optional[bool] = None,
        disp: bool = True,
        spec_values=None,
        binning_numeric=None,
    ) -> pd.DataFrame:
        """
        Generate cross-risk analysis summary between base and comparison scores.

        This method creates a comprehensive cross-tabulation showing risk metrics
        across different score ranges for both base and comparison scores.

        Parameters
        ----------
        cross_agg_dict : dict, optional
            Dictionary of column names and aggregation functions, as ``{column: aggregation or list of aggregations}``
            (the ``agg_func`` of ``cross_risk``). Default is None, i.e. the ``cross_agg_dict`` of the instance, which is a
            legacy dictionary (``is_dpd7``, ``credit_limit``, ``monthlyincome``, ``education``) unless you set it. An entry
            for the column ``flow_id`` (count and share of rows) is always added, so the data needs a ``flow_id`` column.
        nbins : int, optional
            Number of bins. Default is 5.
        equal_freq : bool, optional
            Use equal frequency binning. If None, uses self.equal_freq.
        disp : bool, optional
            Whether to display, with ``IPython.display.display``, the cross table of every metric for every comparison
            score. Default is True.
        spec_values : list, optional
            Special values to keep separate during binning. Default is None, i.e. the instance setting.
        binning_numeric : bool or list/tuple of bool, optional
            Whether numeric score columns should be binned before cross-risk
            aggregation. If None, defaults to [True, True] for backward
            compatibility. Passing a bool applies the same setting to both
            base and comparison score.

        Returns
        -------
        pandas.DataFrame
            Cross-risk summary with base score range, comparison score range, and metrics, in long format with the columns
            ``base_scr_range``, ``eval_metric``, ``score_name``, ``comp_scr_range`` and ``value``. An empty DataFrame is
            returned when, for some comparison score, no row has a positive base score and a positive comparison score.

        Raises
        ------
        ValueError
            If ``binning_numeric`` is a list or tuple that does not have exactly two elements.
        TypeError
            If ``binning_numeric`` is not None, a bool or a list/tuple of bools.

        Notes
        -----
        For every comparison score only the rows in which both the base score and that score are greater than 0 are used,
        whatever ``positive_score_only`` is (rows with a missing score are therefore dropped too). The summary is
        unweighted: ``weight_col`` is not used.
        """
        data = self.data.copy()
        dep = self.dep
        min_bin_prop = self.min_bin_prop
        precision = self.precision
        tree_binning = self.tree_binning
        equal_freq = equal_freq if equal_freq is not None else self.equal_freq
        fillna = self.missing_rate_ref
        include_missing = self.include_missing

        base_score = self.base_score
        compare_scrlist = self.comp_scrlist

        if cross_agg_dict is None:
            cross_agg_dict = self.cross_agg_dict.copy()
        if spec_values is None:
            spec_values = self.spec_values
        if binning_numeric is None:
            binning_numeric = [True, True]
        elif isinstance(binning_numeric, (bool, np.bool_)):
            binning_numeric = [bool(binning_numeric), bool(binning_numeric)]
        elif isinstance(binning_numeric, (list, tuple)):
            if len(binning_numeric) != 2:
                raise ValueError(
                    "binning_numeric must be a bool or a two-element list/tuple."
                )
            if not all(isinstance(value, (bool, np.bool_)) for value in binning_numeric):
                raise TypeError("binning_numeric values must be booleans.")
            binning_numeric = [bool(binning_numeric[0]), bool(binning_numeric[1])]
        else:
            raise TypeError("binning_numeric must be None, a bool, or a two-element list/tuple.")

        def _ensure_three_level_columns(frame: pd.DataFrame) -> pd.DataFrame:
            frame = frame.copy()
            if isinstance(frame.columns, pd.MultiIndex):
                normalized_columns = []
                for col in frame.columns:
                    col_tuple = tuple(col)
                    if len(col_tuple) < 3:
                        col_tuple = col_tuple + ("",) * (3 - len(col_tuple))
                    elif len(col_tuple) > 3:
                        col_tuple = col_tuple[:2] + ("_".join(map(str, col_tuple[2:])),)
                    normalized_columns.append(col_tuple)
                frame.columns = pd.MultiIndex.from_tuples(normalized_columns)
            else:
                frame.columns = pd.MultiIndex.from_tuples(
                    [(col, "", "") for col in frame.columns]
                )
            return frame

        multi_scr_res = {}
        for compare_scr in compare_scrlist:
            logger.info(compare_scr)
            score_list = [base_score, compare_scr]
            score_query = " and ".join([s + " > 0" for s in score_list])

            prepared_data = data.query(score_query)

            if prepared_data.empty:
                return pd.DataFrame()  # or skip this group

            tot_cnt = prepared_data[score_list].shape[0]
            cross_agg_dict_copy = cross_agg_dict.copy()
            cross_agg_dict_copy.update({'flow_id': ['count', lambda x: x.count() / tot_cnt]})

            cross_res = {}
            for colname, aggfunc in cross_agg_dict_copy.items():
                if data.shape[0] > nbins:
                    cross_res[colname] = cross_risk(
                        data=prepared_data,
                        score_list=[base_score, compare_scr],
                        dep=dep,
                        nbins=nbins,
                        agg_col=colname,
                        precision=precision,
                        min_bin_prop=min_bin_prop,
                        include_missing=include_missing,
                        equal_freq=equal_freq,
                        binning_numeric=binning_numeric,
                        tree_binning=tree_binning,
                        agg_func=aggfunc,
                        fillna=fillna,
                        spec_values=spec_values
                    )

            fnl_res = []
            for colname, res in cross_res.items():
                res = _ensure_three_level_columns(res)
                res[('', 'eval_metric', '')] = colname
                res = res.rename(columns={"<lambda>": "", "count": "n"})
                if disp:
                    from IPython.display import display
                    display(res)
                fnl_res.append(res)

            fnl_res = pd.concat(fnl_res)

            fnl_res[('', 'score_name', '')] = compare_scr
            multi_scr_res[compare_scr] = fnl_res

        combined_cross_res = []
        for score_name, cross_data in multi_scr_res.items():
            cross_data = cross_data.copy()
            cross_data.columns = cross_data.columns.map(
                lambda x: f"{x[0]}_{x[1] if len(str(x[1])) >= 2 else ('0' + str(x[1]))}_{x[2]}".strip("_")
            )
            cross_data.index = cross_data.index.map(
                lambda x: f"{x[0] if len(str(x[0])) >= 2 else ('0' + str(x[0]))}_{x[1]}".strip("_")
            )
            cross_data = cross_data.reset_index(drop=False).melt(
                id_vars=['index', 'eval_metric', 'score_name'],
                var_name='comp_scr_range'
            ).rename(columns={"index": "base_scr_range"})
            combined_cross_res.append(cross_data)
        combined_cross_res = pd.concat(combined_cross_res)

        return combined_cross_res

    def multi_subset_wrapper(
        self,
        condition_dict: Optional[Dict[str, str]] = None,
        subset_var_name: str = 'eval_subset',
        func: Optional[Callable] = None,
        min_subset_size = 10,
        **kwargs
    ) -> pd.DataFrame:
        """
        Apply evaluation function across multiple data subsets.

        This method iterates over predefined subset conditions, filters the data
        accordingly, and applies the evaluation function to each subset.

        Parameters
        ----------
        condition_dict : dict, optional
            Dictionary mapping subset names to query conditions (``DataFrame.query`` strings; an empty string keeps every
            row). If None, uses self.subset_condition_dict.
        subset_var_name : str, optional
            Column name for subset identifier in results. Default is 'eval_subset'.
        func : callable, optional
            Function to apply to each subset. If None, uses model_perf_compare.
        min_subset_size : int, optional
            Only subsets with more than this many rows are evaluated. Default is 10.
        **kwargs
            Additional keyword arguments passed to the evaluation function.

        Returns
        -------
        pandas.DataFrame
            Combined results from all subsets, with the subset label in the column ``subset_var_name``. With no subset
            condition at all (an empty dict) the result is an empty DataFrame that has only the column ``subset_var_name``.
            If there are conditions but no subset is evaluated or none returns a non-empty result, a one-row DataFrame that
            only holds the label of the last subset in ``subset_var_name`` is returned.

        Notes
        -----
        ``self.data`` is replaced by the subset while ``func`` runs and restored afterwards (it is not restored if ``func``
        raises an exception).
        """
        if condition_dict is None:
            condition_dict = self.subset_condition_dict

        original_data = self.data.copy()
        if func is None:
            func = self.model_perf_compare

        fnl_subset_results = []

        if condition_dict:
            for group_value, query in condition_dict.items():
                logger.info(f"INFO:: Multi_Subset_Eval: Running Subset {group_value}")

                subset_data = self.data.query(query).copy() if query != "" else self.data.copy()
                subset_data_size = subset_data.shape[0]

                if subset_data_size > min_subset_size:
                    self.data = subset_data

                    subset_result = func(**kwargs)
                    if subset_result is not None and not subset_result.empty:
                        subset_result = subset_result.copy()
                        subset_result[subset_var_name] = group_value
                        fnl_subset_results.append(subset_result)
                    else:
#                         fnl_subset_results.append(pd.DataFrame({subset_var_name: [group_value]}))
                        self.data = original_data
                        continue

                    self.data = original_data


        if fnl_subset_results:
            return pd.concat(fnl_subset_results)

        if not condition_dict:
            return pd.DataFrame({subset_var_name: []})
        return pd.DataFrame({subset_var_name: [group_value]})

    def multi_ylabel_wrapper(
        self,
        ylabels: Optional[List[str]] = None,
        ylabel_var_name: str = 'eval_ylabel',
        eval_func: Optional[Callable] = None,
        **kwargs
    ) -> pd.DataFrame:
        """
        Apply evaluation function across multiple target labels.

        This method iterates over different target/label columns, temporarily
        updating the dependent variable for each iteration.

        Parameters
        ----------
        ylabels : list, optional
            List of target column names. If None, uses self.eval_ylabels.
        ylabel_var_name : str, optional
            Column name for label identifier in results. Default is 'eval_ylabel'.
        eval_func : callable, optional
            Function to apply for each label. If None, uses model_perf_compare.
        **kwargs
            Additional keyword arguments passed to the evaluation function.

        Returns
        -------
        pandas.DataFrame
            Combined results from all labels, with the label in the column ``ylabel_var_name``. Empty if no label has a
            row with a non-missing value.

        Notes
        -----
        For each label the rows in which the label is missing are dropped, and ``self.dep`` and ``self.data`` are replaced
        while ``eval_func`` runs and restored afterwards (they are not restored if ``eval_func`` raises an exception).
        """
        original_dep = self.dep
        original_data = self.data.copy()
        if eval_func is None:
            eval_func = self.model_perf_compare

        fnl_ylabel_results = []

        if ylabels is None:
            ylabels = self.eval_ylabels

        for current_dep in ylabels:
            logger.info(f"INFO:: Multi_yLabel_Eval: Running Label {current_dep}")

            input_data = self.data.query(f"{current_dep} == {current_dep}").copy()
            subset_data_size = input_data.shape[0]

            if subset_data_size > 0:
                self.dep = current_dep
                self.data = input_data

                subset_result = eval_func(**kwargs)
                subset_result = subset_result.copy()
                subset_result[ylabel_var_name] = current_dep
                fnl_ylabel_results.append(subset_result)

                self.dep = original_dep
                self.data = original_data

        if fnl_ylabel_results:
            return pd.concat(fnl_ylabel_results)
        return pd.DataFrame()

    def multi_group_wrapper(
        self,
        group_name: Optional[str] = None,
        group_var_name: str = 'group_name',
        group_eval_func: Optional[Callable] = None,
        min_subset_size: int = 10,
        **kwargs
    ) -> pd.DataFrame:
        """
        Apply evaluation function across multiple groups.

        This method splits data by a grouping column and applies the evaluation
        function to each group separately.

        Parameters
        ----------
        group_name : str, optional
            Column name to group by. If None, ``group_eval_func`` is called once on all the data (an empty DataFrame is
            returned when ``group_eval_func`` is None as well).
        group_var_name : str, optional
            Column name for group identifier in results. Default is 'group_name'.
        group_eval_func : callable, optional
            Function to apply to each group. If None, uses model_perf_compare.
        min_subset_size : int, optional
            Only groups with more than this many rows are evaluated. Default is 10.
        **kwargs
            Additional keyword arguments passed to the evaluation function.

        Returns
        -------
        pandas.DataFrame
            Combined results from all groups, with the group value in the column ``group_var_name``. Empty if no group is
            evaluated.

        Notes
        -----
        The groups are the distinct non-missing values of ``group_name`` in order of appearance, and the rows are selected
        by value, so numeric group columns and values containing quotes work. ``self.data`` is replaced by the group while
        ``group_eval_func`` runs and restored afterwards (it is not restored if the function raises an exception).
        """
        original_data = self.data.copy()

        if group_name is None:
            return group_eval_func(**kwargs) if group_eval_func else pd.DataFrame()

        group_column = original_data[group_name]
        # Missing values form no group; the values are matched as values, not through a query string,
        # so numeric groups and values with quotes work
        group_value_list = pd.unique(group_column.dropna())

        if group_eval_func is None:
            group_eval_func = self.model_perf_compare

        fnl_group_results = []
        for group_value in group_value_list:
            logger.info(f"INFO:: Multi_Group_Eval: Running Group {group_value}")

            in_group = group_column.eq(group_value).fillna(False).astype(bool)
            input_data = original_data.loc[in_group].copy()
            subset_data_size = input_data.shape[0]

            if subset_data_size > min_subset_size:
                self.data = input_data

                subset_result = group_eval_func(**kwargs)
                subset_result = subset_result.copy()
                subset_result[group_var_name] = group_value
                fnl_group_results.append(subset_result)

                self.data = original_data

        if fnl_group_results:
            return pd.concat(fnl_group_results)
        return pd.DataFrame()

    def multi_dim_eval(
        self,
        condition_dict: Optional[Dict[str, str]] = None,
        eval_ylabels: Optional[List[str]] = None,
        grp_namelist: Optional[List[str]] = None,
        eval_func: Optional[Callable] = None,
        subset_var_name: str = 'eval_subset',
        ylabel_var_name: str = 'eval_ylabel',
        var_name: str = 'group_name',
        value_name: str = 'group_value',
        **kwargs
    ) -> pd.DataFrame:
        """
        Perform multi-dimensional evaluation across subsets, labels, and groups.

        This method combines multi_subset_wrapper and multi_ylabel_wrapper to
        evaluate model performance across different dimensions.

        Parameters
        ----------
        condition_dict : dict, optional
            Subset conditions. If None, uses self.subset_condition_dict.
        eval_ylabels : list, optional
            Target labels. If None, uses self.eval_ylabels.
        grp_namelist : list, optional
            Group columns. If None, uses self.grp_namelist.
        eval_func : callable, optional
            Evaluation function. If None, uses model_perf_compare. It is called for every subset, label and group column;
            the group column is passed to it as ``grp_name`` only when ``eval_func`` has a ``grp_name`` parameter.
        subset_var_name : str, optional
            Column name for subset identifier. Default is 'eval_subset'.
        ylabel_var_name : str, optional
            Column name for label identifier. Default is 'eval_ylabel'.
        var_name : str, optional
            Column name for group name in results. Default is 'group_name'.
        value_name : str, optional
            Column name for group value in results. Default is 'group_value'.
        **kwargs
            Additional keyword arguments, passed on to ``eval_func``.

        Returns
        -------
        pandas.DataFrame
            Multi-dimensional evaluation results: the results of ``eval_func`` for every subset, label and group column,
            stacked, with the columns ``subset_var_name``, ``ylabel_var_name`` and ``var_name`` (the group column that was
            evaluated). When ``eval_func`` has a ``grp_name`` parameter, the group columns are merged into the single
            column ``value_name``.

        Notes
        -----
        If ``eval_func`` has no ``grp_name`` parameter, the same evaluation is simply repeated for each entry of
        ``grp_namelist`` (no grouping is done). With the default ``eval_func`` every call is grouped, so the evaluation is
        unweighted.
        """
        if condition_dict is None:
            condition_dict = self.subset_condition_dict

        if eval_ylabels is None:
            eval_ylabels = self.eval_ylabels

        if grp_namelist is None:
            grp_namelist = self.grp_namelist

        if eval_func is None:
            eval_func = self.model_perf_compare

        fnl_perf_res = []
        for group_name in grp_namelist:
            logger.info(f"INFO:: Multi_Dim_Eval: Running Group {group_name}")

            sig = inspect.signature(eval_func)
            parameters = sig.parameters

            if 'grp_name' in parameters:
                sub_perf_res = self.multi_subset_wrapper(
                    condition_dict=condition_dict,
                    subset_var_name=subset_var_name,
                    func=self.multi_ylabel_wrapper, 
                    ylabels=eval_ylabels, 
                    ylabel_var_name=ylabel_var_name,
                    eval_func=eval_func, 
                    grp_name=group_name,
                    **kwargs
                )
            else:
                sub_perf_res = self.multi_subset_wrapper(
                    condition_dict=condition_dict,
                    subset_var_name=subset_var_name,
                    func=self.multi_ylabel_wrapper, 
                    ylabels=eval_ylabels, 
                    ylabel_var_name=ylabel_var_name,
                    eval_func=eval_func, 
                    **kwargs
                )

            sub_perf_res = sub_perf_res.copy()
            sub_perf_res[var_name] = group_name
            fnl_perf_res.append(sub_perf_res)

        fnl_perf_res = pd.concat(fnl_perf_res)

        if 'grp_name' in parameters:
            if all(x in fnl_perf_res.columns for x in grp_namelist):
                fnl_perf_res[value_name] = fnl_perf_res[grp_namelist].bfill(axis=1).iloc[:, 0]
                fnl_perf_res = fnl_perf_res.drop(columns=grp_namelist)

        return fnl_perf_res

    def cross_perf_eval(
        self,
        bad_list: List[str],
        scr_list: List[str],
        data: Optional[pd.DataFrame] = None,
        eval_metric: List[str] = None,
        melt: bool = True
    ) -> pd.DataFrame:
        """
        Cross-evaluate model performance across multiple bad flags and scores.

        This method calculates performance metrics for all combinations of
        bad flags and scores, useful for comparing model behavior across
        different definitions of "bad" outcomes.

        Parameters
        ----------
        bad_list : list
            List of bad/negative outcome column names.
        scr_list : list
            List of score column names.
        data : pandas.DataFrame, optional
            Data to use. If None, uses self.data.
        eval_metric : list, optional
            Metrics to extract. If None, uses ['AUC', 'KS', 'N'].
        melt : bool, optional
            Whether to melt the pivot table. Default is True.

        Returns
        -------
        pandas.DataFrame
            Cross performance evaluation results. With ``melt=True`` a long table with the columns ``score_name``,
            ``variable`` (``<metric>_<label>``, e.g. ``AUC_bad_flag``) and ``value``; with ``melt=False`` the pivot table
            itself, indexed by ``score_name`` with one ``<metric>_<label>`` column per metric and label. If the data has
            no more rows than ``nbins`` of the instance, an empty DataFrame with the columns ``eval_ylabel`` and
            ``score_name`` is returned.

        Notes
        -----
        The rows in which a label is missing or a score is not positive are dropped. The bins come from the instance
        (``nbins`` for both the distribution and the percentile bins, ``precision`` and ``min_bin_prop``) and the
        evaluation is unweighted.
        """
        if eval_metric is None:
            eval_metric = ['AUC', 'KS', 'N']

        if data is None:
            data = self.data.copy()

        score_query = " and ".join([s + " > 0" for s in scr_list])
        bad_query = " and ".join([s + " == " + s for s in bad_list])

        data = data.query(bad_query)
        data = data.query(score_query)

        if data.shape[0] > self.nbins:
            fnl_res = []
            for bad in bad_list:
                for scr in scr_list:
                    perf = get_perf_summary(
                        None,
                        None,
                        data,
                        tgt_name=bad,
                        scr_name=scr,
                        to_show=False,
                        display=False,
                        dist_bins=self.nbins,
                        pct_bins=self.nbins,
                        precision=self.precision,
                        min_bin_prop=self.min_bin_prop,
                        equal_freq=True
                    )
                    perf = perf.copy()
                    perf['eval_ylabel'] = bad
                    perf['score_name'] = scr
                    fnl_res.append(perf)

            fnl_res = pd.concat(fnl_res)

            summary = fnl_res.pivot_table(index="score_name", columns='eval_ylabel', values=eval_metric)
            summary.columns = summary.columns.map(lambda x: f"{x[0]}_{x[1]}")

            if melt:
                summary = summary.reset_index(drop=False).melt(id_vars=['score_name'])

            return summary

        return pd.DataFrame(columns=['eval_ylabel', 'score_name'])

    def run_variable_analysis_summary(
        self,
        varlist: List[str],
        data: Optional[pd.DataFrame] = None,
        dep: Optional[str] = None,
        nbins: Optional[int] = None,
        equal_freq: Optional[bool] = None,
        min_bin_prop: Optional[float] = None,
        precision: Optional[int] = None,
        chi2_method: Optional[bool] = None,
        chi2_p: Optional[float] = None,
        init_equi_bins: Optional[int] = None,
        tree_binning: Optional[bool] = None,
        include_missing: Optional[bool] = None,
        missing_rate_ref: Optional[Union[int, float]] = None,
        seed: Optional[int] = None,
        spec_values = []
    ) -> pd.DataFrame:
        """
        Run comprehensive variable analysis and generate summary report.

        This method performs binning analysis on specified variables,
        calculating metrics like IV (Information Value), KS and Lift of each variable.

        Parameters
        ----------
        varlist : list
            List of variable names to analyze.
        data : pandas.DataFrame, optional
            Data to use. If None, uses self.data.
        dep : str, optional
            Target variable name. If None, uses self.dep.
        nbins : int, optional
            Number of bins. If None, uses self.nbins.
        equal_freq : bool, optional
            Use equal frequency binning. If None, uses self.equal_freq.
        min_bin_prop : float, optional
            Minimum bin proportion. If None, uses self.min_bin_prop.
        precision : int, optional
            Decimal precision. If None, uses self.precision.
        chi2_method : bool, optional
            Use chi-square method. If None, uses self.chi2_method.
        chi2_p : float, optional
            Chi-square p-value. If None, uses self.chi2_p.
        init_equi_bins : int, optional
            Initial equal bins. If None, uses self.init_equi_bins.
        tree_binning : bool, optional
            Use tree binning. If None, uses self.tree_binning.
        include_missing : bool, optional
            Include missing values. If None, uses self.include_missing.
        missing_rate_ref : int or float, optional
            Missing reference. If None, uses self.missing_rate_ref.
        seed : int, optional
            Random seed. If None, uses self.seed.
        spec_values : list, default []
            Special values that get their own bins in the analysis. Unlike the other arguments, the default is not taken
            from the instance.

        Returns
        -------
        pandas.DataFrame
            Variable analysis summary with IV and other metrics: one row per variable, sorted by decreasing IV, with the
            columns ``var``, ``n_all`` (rows), ``n`` (non-missing rows), ``ks_in_gains``, ``lift_in_gains``, ``iv``,
            ``n_bump``, ``missing_rate``, ``min``, ``mean``, ``max`` and ``n_bins``. Constant variables and variables whose
            computation fails are left out (a warning lists the failed ones).

        Notes
        -----
        Before the analysis the missing values of the variables in ``varlist`` are filled with ``missing_rate_ref`` (in a
        copy of the data). The summary is unweighted.
        """
        if data is None:
            data = self.data.copy()

        if dep is None:
            dep = self.dep

        nbins = nbins if nbins is not None else self.nbins
        equal_freq = equal_freq if equal_freq is not None else self.equal_freq
        min_bin_prop = min_bin_prop if min_bin_prop is not None else self.min_bin_prop
        precision = precision if precision is not None else self.precision
        chi2_method = chi2_method if chi2_method is not None else self.chi2_method
        chi2_p = chi2_p if chi2_p is not None else self.chi2_p
        init_equi_bins = init_equi_bins if init_equi_bins is not None else self.init_equi_bins
        tree_binning = tree_binning if tree_binning is not None else self.tree_binning
        include_missing = include_missing if include_missing is not None else self.include_missing
        missing_rate_ref = missing_rate_ref if missing_rate_ref is not None else self.missing_rate_ref
        seed = seed if seed is not None else self.seed

        from Modeling_Tool.Feature.Feature_Insights import VarExtractionInsights

        varInsights = VarExtractionInsights(
            data=data,
            dep=dep,
            plot_path=None,
            nbins=nbins,
            equal_freq=equal_freq,
            min_bin_prop=min_bin_prop,
            precision=precision,
            chi2_method=chi2_method,
            chi2_p=chi2_p,
            init_equi_bins=init_equi_bins,
            tree_binning=tree_binning,
            include_missing=include_missing,
            seed=seed,
            missing_rate_ref=missing_rate_ref,
            spec_values=spec_values
        )

        data = data.copy()
        for var in varlist:
            data[var] = data[var].fillna(missing_rate_ref)

        var_summary = varInsights.get_var_analysis_report(data=data, varlist=varlist, dep=dep, iv_cut=0)

        return var_summary


    def pipe(self, data=None):
            """
            Return an ``EvaluationPipeline`` for chained group and subset evaluation.

            Parameters
            ----------
            data : pd.DataFrame, optional
                Data to evaluate. Defaults to ``self.data``.

            Returns
            -------
            EvaluationPipeline
            """
            return EvaluationPipeline(self, data)

get_base_score

get_base_score(scorename: str = '_base_model_score_', disp: bool = False)

Calculate base model scores using the provided model.

This method uses the model's predict_proba method to generate scores for the input data and optionally displays correlation analysis.

Parameters:

Name	Type	Description	Default
scorename	str	Name for the score column. Default is 'base_model_score'.	'_base_model_score_'
disp	bool	Whether to display intermediate results. Default is False.	False

Returns:

Type	Description
Model_Evaluation_Tool	Returns self for method chaining.

Raises:

Type	Description
AttributeError	If model is None or lacks required attributes.

Notes

The model must expose feature_names_in_: those columns are passed to predict_proba. As a side effect, self.data is replaced by a copy that has the new score column and self.base_score is set to scorename. disp=True needs IPython.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def get_base_score(self, scorename: str = '_base_model_score_', disp: bool = False):
    """
    Calculate base model scores using the provided model.

    This method uses the model's predict_proba method to generate scores
    for the input data and optionally displays correlation analysis.

    Parameters
    ----------
    scorename : str, optional
        Name for the score column. Default is '_base_model_score_'.
    disp : bool, optional
        Whether to display intermediate results. Default is False.

    Returns
    -------
    Model_Evaluation_Tool
        Returns self for method chaining.

    Raises
    ------
    AttributeError
        If model is None or lacks required attributes.

    Notes
    -----
    The model must expose ``feature_names_in_``: those columns are passed to ``predict_proba``. As a side effect,
    ``self.data`` is replaced by a copy that has the new score column and ``self.base_score`` is set to ``scorename``.
    ``disp=True`` needs IPython.
    """
    if self.model is None:
        raise AttributeError("Model must be provided to calculate base score.")

    data = self.data.copy()
    model = self.model

    inputs = model.feature_names_in_.tolist()
    data[scorename] = model.predict_proba(data[inputs])[:, 1]
    self.data = data
    self.base_score = scorename

    scrlist = [scorename] + self.comp_scrlist
    if disp:
        from IPython.display import display
        from Modeling_Tool.Feature.Distribution_Tool import proc_means_by_grp
        display(proc_means_by_grp(data, scrlist))

    tmp_data = data[data[scrlist] > 0]

    if disp:
        from IPython.display import display
        display(tmp_data[scrlist].corr())

    return self

get_score_correlation

get_score_correlation(score_list: Optional[List[str]] = None, method: str = 'pearson') -> DataFrame

Calculate correlation matrix between score columns.

Parameters:

Name	Type	Description	Default
score_list	list	List of score column names. If None, uses base_score and comp_scrlist.	None
method	str	Correlation method ('pearson', 'spearman', 'kendall'). Default is 'pearson'.	'pearson'

Returns:

Type	Description
DataFrame	Correlation summary in long format with columns: 'base', 'compare', 'corr'.

Raises:

Type	Description
ValueError	If base_score is not set.

Notes

Only the rows in which every score of score_list is greater than 0 are used (rows with a missing score are dropped as well), whatever positive_score_only is.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def get_score_correlation(
    self,
    score_list: Optional[List[str]] = None,
    method: str = 'pearson'
) -> pd.DataFrame:
    """
    Calculate correlation matrix between score columns.

    Parameters
    ----------
    score_list : list, optional
        List of score column names. If None, uses base_score and comp_scrlist.
    method : str, optional
        Correlation method ('pearson', 'spearman', 'kendall'). Default is 'pearson'.

    Returns
    -------
    pandas.DataFrame
        Correlation summary in long format with columns: 'base', 'compare', 'corr'.

    Raises
    ------
    ValueError
        If ``base_score`` is not set.

    Notes
    -----
    Only the rows in which every score of ``score_list`` is greater than 0 are used (rows with a missing score are
    dropped as well), whatever ``positive_score_only`` is.
    """
    if score_list is None:
        score_list = [self.base_score] + self.comp_scrlist

    if self.base_score is None:
        raise ValueError("Base score must be set before calculating correlation.")

    data = self.data.copy()

    score_query = " and ".join([s + " > 0" for s in score_list])
    data = data.query(score_query)

    corr_summary = data[score_list]\
        .corr(method=method)\
        .reset_index(drop=False)\
        .melt(id_vars=['index'], var_name='compare', value_name='corr')\
        .rename(columns={"index": "base"})

    return corr_summary

model_perf_compare

model_perf_compare(data: Optional[DataFrame] = None, grp_name: Optional[str] = None, dist_bins: int = 100, pct_bins: int = 10, min_data_size: int = 50, sync_data_size: bool = True, min_bin_prop: Optional[float] = None, include_missing: Optional[bool] = None, equal_freq: Optional[bool] = None, sample_name: Optional[str] = None) -> DataFrame

Compare model performance across different scores.

This method calculates performance metrics (AUC, KS, etc.) for the base score and all comparison scores via PerformanceEvaluator, optionally filtering by group.

Parameters:

Name	Type	Description	Default
data	DataFrame	Data to use. If None, uses self.data.	None
grp_name	str	Group name column for stratification. Every score is then evaluated separately for each group value (the result gains a column named grp_name), and weight_col is ignored.	None
dist_bins	int	Number of distribution bins. Default is 100.	100
pct_bins	int	Number of percentile bins. Default is 10.	10
min_data_size	int	Minimum number of rows of a group for it to be evaluated; it only matters when grp_name is given. Default is 50 (the min_data_size of the instance is not used here).	50
sync_data_size	bool	Whether every score, the base score included, is evaluated on the same rows: the rows with an observed target where every score is valid (see Notes). False evaluates each score on its own valid rows. Default is True.	True
min_bin_prop	float	Minimum bin proportion. Defaults to the instance setting.	None
include_missing	bool	Whether to include missing values in performance binning. Defaults to False for performance comparison.	None
equal_freq	bool	Use equal-frequency binning. Defaults to the instance setting.	None
sample_name	str	Label applied to the evaluated dataset and emitted in the result's index column. Defaults to 'all' (the whole data treated as a single sample). Callers evaluating a specific slice can pass e.g. 'global' or a split name.	None

Returns:

Type	Description
DataFrame	Performance comparison results sorted by score order: the PerformanceEvaluator summary of every score (AUC_Shift and KS_Shift are dropped) with the extra columns score_name and N_OWN. An empty DataFrame is returned when base_score is not set or no valid row is left.

Notes

A score is valid in a row when it is a finite number and, with positive_score_only=True, greater than 0; rows whose target is missing are never used. The column N_OWN gives the number of rows (not weights) in which the score alone is valid, so with sync_data_size=True it shows how much each score's coverage was narrowed to the common rows. A score without any valid row is left out of the result with a UserWarning instead of emptying the common rows of the others; when the scores have valid rows but none in common, a UserWarning is issued and an empty DataFrame returned. With weight_col set on the instance (and no grp_name) the weighted, narrower summary is returned.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def model_perf_compare(
    self,
    data: Optional[pd.DataFrame] = None,
    grp_name: Optional[str] = None,
    dist_bins: int = 100,
    pct_bins: int = 10,
    min_data_size: int = 50,
    sync_data_size: bool = True,
    min_bin_prop: Optional[float] = None,
    include_missing: Optional[bool] = None,
    equal_freq: Optional[bool] = None,
    sample_name: Optional[str] = None,
) -> pd.DataFrame:
    """
    Compare model performance across different scores.

    This method calculates performance metrics (AUC, KS, etc.) for the base score
    and all comparison scores via ``PerformanceEvaluator``, optionally filtering
    by group.

    Parameters
    ----------
    data : pandas.DataFrame, optional
        Data to use. If None, uses self.data.
    grp_name : str, optional
        Group name column for stratification. Every score is then evaluated separately for each group value (the
        result gains a column named ``grp_name``), and ``weight_col`` is ignored.
    dist_bins : int, optional
        Number of distribution bins. Default is 100.
    pct_bins : int, optional
        Number of percentile bins. Default is 10.
    min_data_size : int, optional
        Minimum number of rows of a group for it to be evaluated; it only matters when ``grp_name`` is given. Default
        is 50 (the ``min_data_size`` of the instance is not used here).
    sync_data_size : bool, optional
        Whether every score, the base score included, is evaluated on the same rows: the rows with an observed target
        where every score is valid (see Notes). False evaluates each score on its own valid rows. Default is True.
    min_bin_prop : float, optional
        Minimum bin proportion. Defaults to the instance setting.
    include_missing : bool, optional
        Whether to include missing values in performance binning.
        Defaults to ``False`` for performance comparison.
    equal_freq : bool, optional
        Use equal-frequency binning. Defaults to the instance setting.
    sample_name : str, optional
        Label applied to the evaluated dataset and emitted in the result's
        ``index`` column. Defaults to ``'all'`` (the whole ``data`` treated as
        a single sample). Callers evaluating a specific slice can pass e.g.
        ``'global'`` or a split name.

    Returns
    -------
    pandas.DataFrame
        Performance comparison results sorted by score order: the ``PerformanceEvaluator`` summary of every score
        (``AUC_Shift`` and ``KS_Shift`` are dropped) with the extra columns ``score_name`` and ``N_OWN``. An empty
        DataFrame is returned when ``base_score`` is not set or no valid row is left.

    Notes
    -----
    A score is valid in a row when it is a finite number and, with ``positive_score_only=True``, greater than 0; rows
    whose target is missing are never used. The column ``N_OWN`` gives the number of rows (not weights) in which the
    score alone is valid, so with ``sync_data_size=True`` it shows how much each score's coverage was narrowed to the
    common rows. A score without any valid row is left out of the result with a ``UserWarning`` instead of emptying
    the common rows of the others; when the scores have valid rows but none in common, a ``UserWarning`` is issued
    and an empty DataFrame returned. With ``weight_col`` set on the instance (and no ``grp_name``) the weighted,
    narrower summary is returned.
    """
    data = self.data.copy() if data is None else data.copy()
    score_list = self.comp_scrlist
    dep = self.dep
    base_score = self.base_score
    # Label for the single (unsplit) dataset being evaluated; surfaces as the
    # 'index' column of the result. Defaults to a neutral 'all' rather than the
    # historical (and misleading) 'oot', which implied an out-of-time sample.
    sample_name = sample_name if sample_name is not None else 'all'

    if base_score is None:
        return pd.DataFrame()

    score_order = [score for score in self._score_order() if score in data.columns]
    target_ok = data[dep].notna()
    valid = {score: target_ok & self._valid_score_mask(data[score]) for score in score_order}
    own_n = {score: int(mask.sum()) for score, mask in valid.items()}
    without_rows = [score for score in score_order if own_n[score] == 0]
    if without_rows:
        rule = "a finite value above 0" if self.positive_score_only else "a finite value"
        warnings.warn(
            f"model_perf_compare: {without_rows} have no row with an observed target and {rule}; "
            f"they are left out of the comparison.",
            UserWarning,
            stacklevel=2,
        )
    active = [score for score in score_order if own_n[score] > 0]
    if not active:
        return pd.DataFrame()
    if sync_data_size:
        common = target_ok.copy()
        for score in active:
            common &= valid[score]
        if not bool(common.any()):
            warnings.warn(
                f"model_perf_compare: the scores {active} have valid rows but none in common, so they cannot be "
                f"compared on the same rows; pass sync_data_size=False to evaluate each score on its own rows.",
                UserWarning,
                stacklevel=2,
            )
            return pd.DataFrame()
        rows_for = {score: common for score in active}
    else:
        rows_for = {score: valid[score] for score in active}

    perf_comp_dict = {}
    for score in active:
        score_data = data.loc[rows_for[score]]

        evaluator = self._build_performance_evaluator(
            score=score,
            dist_bins=dist_bins,
            pct_bins=pct_bins,
            min_bin_prop=min_bin_prop,
            include_missing=include_missing,
            equal_freq=equal_freq,
        )
        perf = evaluator.add_dataset(sample_name, score_data).evaluate(
            oot_grp_name=grp_name,
            min_data_size=min_data_size,
            to_show=False,
            display=False,
        )
        if isinstance(perf, pd.DataFrame) and not perf.empty and 'index' in perf.columns:
            perf = perf.query(f"index == '{sample_name}'")
        if isinstance(perf, pd.DataFrame):
            perf = perf.copy()
            perf['N_OWN'] = own_n[score]
        perf_comp_dict[score] = perf

    if not perf_comp_dict:
        return pd.DataFrame()

    perf_comp_res = []
    for score_name, perf in perf_comp_dict.items():
        perf = perf.copy()
        perf['score_name'] = score_name
        perf_comp_res.append(perf)

    perf_comp_res = pd.concat(perf_comp_res)

    drop_cols = ['AUC_Shift', 'KS_Shift']
    for col in drop_cols:
        if col in perf_comp_res.columns:
            perf_comp_res = perf_comp_res.drop(columns=[col])

    order_map = {val: i for i, val in enumerate(active) if val in perf_comp_dict}
    perf_comp_res['sort_key'] = perf_comp_res['score_name'].map(order_map)
    perf_comp_res_sorted = perf_comp_res.sort_values('sort_key').drop('sort_key', axis=1)

    return perf_comp_res_sorted

get_gains_summary

get_gains_summary(data: Optional[DataFrame] = None, grp_name: Optional[str] = None, disp: bool = True, grp_disp_metric: List[str] = None, grp_nbins: int = 5, withSummary: bool = True, add_func: Optional[Callable] = None, sync_range: bool = True, spec_values=None, include_missing: Optional[bool] = None, fillna=None) -> DataFrame

Generate gains summary table for score analysis.

This method creates gains tables via GainsTableCalculator, showing the distribution of targets across score bins, with optional group stratification. Custom metrics can be injected through add_func.

Parameters:

Name	Type	Description	Default
data	DataFrame	Data to use. If None, uses self.data.	None
grp_name	str	Group name for stratified analysis: one Gains table per group value, stacked, with the group value in a column named grp_name. The number of bins is then grp_nbins instead of the instance's nbins, groups with fewer rows than the instance's min_data_size are skipped, a score with 10 or fewer positive values is skipped, and the weights are ignored.	None
disp	bool	Whether to display, with IPython.display.display, one pivot table (one column per group) for each metric of grp_disp_metric. It only applies when grp_name is given. Default is True.	True
grp_disp_metric	list	Metrics shown by disp for grouped analysis; it does not change the returned table. Default is None, i.e. ['N', 'PROP', 'AVG_BAD', 'LIFT'].	None
grp_nbins	int	Number of bins for grouped analysis. Default is 5.	5
withSummary	bool	Include summary row. Default is True. It is not applied with grp_name.	True
add_func	callable	Custom metric function merged into each gains table. When it is given, all the columns of the Gains table are returned instead of only gains_display_metric_list. It is applied on the weighted path too.	None
sync_range	bool	Synchronize bin ranges across the groups (used with grp_name only). Default is True.	True
spec_values	list	Special values to treat separately during binning. Default is None, i.e. the instance setting.	None
include_missing	bool	Whether to include missing values. Defaults to the instance setting.	None
fillna	any	Missing-value fill for score binning. Defaults to the instance setting.	None

Returns:

Type	Description
DataFrame	Gains summary results with score names: the Gains tables of the base score and of every comparison score, stacked, with the bin columns _bin_num and _bin_range and a score_name column (plus the group column when grp_name is given). Only the gains_display_metric_list columns are kept unless add_func is given. An empty DataFrame is returned when none of the scores is a column of the data.

Notes

When the instance has a weight_col and grp_name is None, the weighted Gains table is returned (N is the sum of the weights and N_RAW the number of rows), with the add_func columns and the summary row when they are asked for.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def get_gains_summary(
    self,
    data: Optional[pd.DataFrame] = None,
    grp_name: Optional[str] = None,
    disp: bool = True,
    grp_disp_metric: List[str] = None,
    grp_nbins: int = 5,
    withSummary: bool = True,
    add_func: Optional[Callable] = None,
    sync_range: bool = True,
    spec_values=None,
    include_missing: Optional[bool] = None,
    fillna=None,
) -> pd.DataFrame:
    """
    Generate gains summary table for score analysis.

    This method creates gains tables via ``GainsTableCalculator``, showing the
    distribution of targets across score bins, with optional group stratification.
    Custom metrics can be injected through ``add_func``.

    Parameters
    ----------
    data : pandas.DataFrame, optional
        Data to use. If None, uses self.data.
    grp_name : str, optional
        Group name for stratified analysis: one Gains table per group value, stacked, with the group value in a column
        named ``grp_name``. The number of bins is then ``grp_nbins`` instead of the instance's ``nbins``, groups with
        fewer rows than the instance's ``min_data_size`` are skipped, a score with 10 or fewer positive values is
        skipped, and the weights are ignored.
    disp : bool, optional
        Whether to display, with ``IPython.display.display``, one pivot table (one column per group) for each metric
        of ``grp_disp_metric``. It only applies when ``grp_name`` is given. Default is True.
    grp_disp_metric : list, optional
        Metrics shown by ``disp`` for grouped analysis; it does not change the returned table. Default is None, i.e.
        ``['N', 'PROP', 'AVG_BAD', 'LIFT']``.
    grp_nbins : int, optional
        Number of bins for grouped analysis. Default is 5.
    withSummary : bool, optional
        Include summary row. Default is True. It is not applied with ``grp_name``.
    add_func : callable, optional
        Custom metric function merged into each gains table. When it is given, all the columns of the Gains table are
        returned instead of only ``gains_display_metric_list``. It is applied on the weighted path too.
    sync_range : bool, optional
        Synchronize bin ranges across the groups (used with ``grp_name`` only). Default is True.
    spec_values : list, optional
        Special values to treat separately during binning. Default is None, i.e. the instance setting.
    include_missing : bool, optional
        Whether to include missing values. Defaults to the instance setting.
    fillna : any, optional
        Missing-value fill for score binning. Defaults to the instance setting.

    Returns
    -------
    pandas.DataFrame
        Gains summary results with score names: the Gains tables of the base score and of every comparison score,
        stacked, with the bin columns ``_bin_num`` and ``_bin_range`` and a ``score_name`` column (plus the group
        column when ``grp_name`` is given). Only the ``gains_display_metric_list`` columns are kept unless ``add_func``
        is given. An empty DataFrame is returned when none of the scores is a column of the data.

    Notes
    -----
    When the instance has a ``weight_col`` and ``grp_name`` is None, the weighted Gains table is returned (``N`` is the
    sum of the weights and ``N_RAW`` the number of rows), with the ``add_func`` columns and the summary row when they
    are asked for.
    """
    if grp_disp_metric is None:
        grp_disp_metric = ['N', 'PROP', 'AVG_BAD', 'LIFT']

    if fillna is None:
        fillna = self.fillna
    if spec_values is None:
        spec_values = self.spec_values
    if include_missing is None:
        include_missing = self.include_missing

    new_oot_df = self.data.copy() if data is None else data.copy()

    dep = self.dep
    base_score = self.base_score
    min_data_size = self.min_data_size
    nbins = self.nbins if grp_name is None else grp_nbins

    score_list = self._score_order()
    display_metric_list = self.gains_display_metric_list if add_func is None else None

    if self.data is None or len(self.data) == 0:
        empty_df = pd.DataFrame()
        empty_df.index = pd.MultiIndex.from_tuples([], names=['_bin_num', '_bin_range'])
        return empty_df

    gains_table_dict = {}
    for score in score_list:
        if score not in new_oot_df.columns:
            continue

        calculator = self._build_gains_calculator(
            data=new_oot_df,
            score=score,
            nbins=nbins,
            fillna=fillna,
            spec_values=spec_values,
            include_missing=include_missing,
            ascending=True,
        )

        if grp_name is None:
            gains_table_dict[score] = calculator.calculate(
                withSummary=withSummary,
                add_func=add_func,
            )
            if display_metric_list is not None:
                gains_table_dict[score] = gains_table_dict[score][display_metric_list]
            gains_table_dict[score] = gains_table_dict[score].reset_index(drop=False)
        else:
            if new_oot_df.query(f"{score} > 0").shape[0] > 10:
                oot_by_group = calculator.calculate(
                    grp_name=grp_name,
                    min_data_size=min_data_size,
                    sync_range=sync_range,
                    retSummary=False,
                    withSummary=withSummary,
                    add_func=add_func,
                )
                if display_metric_list is not None:
                    oot_by_group = oot_by_group[[*display_metric_list, grp_name]]
                oot_by_group = oot_by_group.reset_index(drop=False)

                grp_metric_dict = {}
                for value in [x for x in grp_disp_metric if x not in ['PROP']]:
                    grp_metric_dict[value] = oot_by_group\
                        .reset_index(drop=False)\
                        .pivot_table(
                            index=['_bin_num', '_bin_range'],
                            columns=[grp_name],
                            values=value,
                            margins=True,
                            margins_name="Grand_Total",
                        )

                if 'PROP' in grp_disp_metric:
                    tot_cnt = oot_by_group\
                        .reset_index(drop=False)\
                        .pivot_table(
                            index=['_bin_num', '_bin_range'],
                            columns=[grp_name],
                            values=['N'],
                            margins=True,
                            margins_name="Grand_Total",
                            aggfunc='sum',
                        )

                    if tot_cnt.shape[0] > 0:
                        grp_metric_dict['PROP'] = tot_cnt.iloc[0:tot_cnt.shape[0]:] / np.array(
                            tot_cnt.iloc[-1:].T['Grand_Total'].tolist()
                        )
                        grp_metric_dict['PROP'].columns = grp_metric_dict['PROP'].rename(
                            columns={'N': ''}
                        ).columns.map("".join)

                fnl_grp_gains_res = []
                for colname, gains in grp_metric_dict.items():
                    gains = gains.copy()
                    gains['variable'] = colname
                    if disp:
                        from IPython.display import display
                        display(gains)
                    fnl_grp_gains_res.append(gains)
                fnl_grp_gains_res = pd.concat(fnl_grp_gains_res)

                gains_table_dict[score] = oot_by_group

    fnl_gains_res = []
    for score_name, gains_res in gains_table_dict.items():
        gains_res = gains_res.copy()
        gains_res['score_name'] = score_name
        fnl_gains_res.append(gains_res)

    if not fnl_gains_res:
        return pd.DataFrame()

    return pd.concat(fnl_gains_res)

get_cross_risk_summary

get_cross_risk_summary(cross_agg_dict: Optional[Dict] = None, nbins: int = 5, equal_freq: Optional[bool] = None, disp: bool = True, spec_values=None, binning_numeric=None) -> DataFrame

Generate cross-risk analysis summary between base and comparison scores.

This method creates a comprehensive cross-tabulation showing risk metrics across different score ranges for both base and comparison scores.

Parameters:

Name	Type	Description	Default
cross_agg_dict	dict	Dictionary of column names and aggregation functions, as {column: aggregation or list of aggregations} (the agg_func of cross_risk). Default is None, i.e. the cross_agg_dict of the instance, which is a legacy dictionary (is_dpd7, credit_limit, monthlyincome, education) unless you set it. An entry for the column flow_id (count and share of rows) is always added, so the data needs a flow_id column.	None
nbins	int	Number of bins. Default is 5.	5
equal_freq	bool	Use equal frequency binning. If None, uses self.equal_freq.	None
disp	bool	Whether to display, with IPython.display.display, the cross table of every metric for every comparison score. Default is True.	True
spec_values	list	Special values to keep separate during binning. Default is None, i.e. the instance setting.	None
binning_numeric	bool or list/tuple of bool	Whether numeric score columns should be binned before cross-risk aggregation. If None, defaults to [True, True] for backward compatibility. Passing a bool applies the same setting to both base and comparison score.	None

Returns:

Type	Description
DataFrame	Cross-risk summary with base score range, comparison score range, and metrics, in long format with the columns base_scr_range, eval_metric, score_name, comp_scr_range and value. An empty DataFrame is returned when, for some comparison score, no row has a positive base score and a positive comparison score.

Raises:

Type	Description
ValueError	If binning_numeric is a list or tuple that does not have exactly two elements.
TypeError	If binning_numeric is not None, a bool or a list/tuple of bools.

Notes

For every comparison score only the rows in which both the base score and that score are greater than 0 are used, whatever positive_score_only is (rows with a missing score are therefore dropped too). The summary is unweighted: weight_col is not used.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def get_cross_risk_summary(
    self,
    cross_agg_dict: Optional[Dict] = None,
    nbins: int = 5,
    equal_freq: Optional[bool] = None,
    disp: bool = True,
    spec_values=None,
    binning_numeric=None,
) -> pd.DataFrame:
    """
    Generate cross-risk analysis summary between base and comparison scores.

    This method creates a comprehensive cross-tabulation showing risk metrics
    across different score ranges for both base and comparison scores.

    Parameters
    ----------
    cross_agg_dict : dict, optional
        Dictionary of column names and aggregation functions, as ``{column: aggregation or list of aggregations}``
        (the ``agg_func`` of ``cross_risk``). Default is None, i.e. the ``cross_agg_dict`` of the instance, which is a
        legacy dictionary (``is_dpd7``, ``credit_limit``, ``monthlyincome``, ``education``) unless you set it. An entry
        for the column ``flow_id`` (count and share of rows) is always added, so the data needs a ``flow_id`` column.
    nbins : int, optional
        Number of bins. Default is 5.
    equal_freq : bool, optional
        Use equal frequency binning. If None, uses self.equal_freq.
    disp : bool, optional
        Whether to display, with ``IPython.display.display``, the cross table of every metric for every comparison
        score. Default is True.
    spec_values : list, optional
        Special values to keep separate during binning. Default is None, i.e. the instance setting.
    binning_numeric : bool or list/tuple of bool, optional
        Whether numeric score columns should be binned before cross-risk
        aggregation. If None, defaults to [True, True] for backward
        compatibility. Passing a bool applies the same setting to both
        base and comparison score.

    Returns
    -------
    pandas.DataFrame
        Cross-risk summary with base score range, comparison score range, and metrics, in long format with the columns
        ``base_scr_range``, ``eval_metric``, ``score_name``, ``comp_scr_range`` and ``value``. An empty DataFrame is
        returned when, for some comparison score, no row has a positive base score and a positive comparison score.

    Raises
    ------
    ValueError
        If ``binning_numeric`` is a list or tuple that does not have exactly two elements.
    TypeError
        If ``binning_numeric`` is not None, a bool or a list/tuple of bools.

    Notes
    -----
    For every comparison score only the rows in which both the base score and that score are greater than 0 are used,
    whatever ``positive_score_only`` is (rows with a missing score are therefore dropped too). The summary is
    unweighted: ``weight_col`` is not used.
    """
    data = self.data.copy()
    dep = self.dep
    min_bin_prop = self.min_bin_prop
    precision = self.precision
    tree_binning = self.tree_binning
    equal_freq = equal_freq if equal_freq is not None else self.equal_freq
    fillna = self.missing_rate_ref
    include_missing = self.include_missing

    base_score = self.base_score
    compare_scrlist = self.comp_scrlist

    if cross_agg_dict is None:
        cross_agg_dict = self.cross_agg_dict.copy()
    if spec_values is None:
        spec_values = self.spec_values
    if binning_numeric is None:
        binning_numeric = [True, True]
    elif isinstance(binning_numeric, (bool, np.bool_)):
        binning_numeric = [bool(binning_numeric), bool(binning_numeric)]
    elif isinstance(binning_numeric, (list, tuple)):
        if len(binning_numeric) != 2:
            raise ValueError(
                "binning_numeric must be a bool or a two-element list/tuple."
            )
        if not all(isinstance(value, (bool, np.bool_)) for value in binning_numeric):
            raise TypeError("binning_numeric values must be booleans.")
        binning_numeric = [bool(binning_numeric[0]), bool(binning_numeric[1])]
    else:
        raise TypeError("binning_numeric must be None, a bool, or a two-element list/tuple.")

    def _ensure_three_level_columns(frame: pd.DataFrame) -> pd.DataFrame:
        frame = frame.copy()
        if isinstance(frame.columns, pd.MultiIndex):
            normalized_columns = []
            for col in frame.columns:
                col_tuple = tuple(col)
                if len(col_tuple) < 3:
                    col_tuple = col_tuple + ("",) * (3 - len(col_tuple))
                elif len(col_tuple) > 3:
                    col_tuple = col_tuple[:2] + ("_".join(map(str, col_tuple[2:])),)
                normalized_columns.append(col_tuple)
            frame.columns = pd.MultiIndex.from_tuples(normalized_columns)
        else:
            frame.columns = pd.MultiIndex.from_tuples(
                [(col, "", "") for col in frame.columns]
            )
        return frame

    multi_scr_res = {}
    for compare_scr in compare_scrlist:
        logger.info(compare_scr)
        score_list = [base_score, compare_scr]
        score_query = " and ".join([s + " > 0" for s in score_list])

        prepared_data = data.query(score_query)

        if prepared_data.empty:
            return pd.DataFrame()  # or skip this group

        tot_cnt = prepared_data[score_list].shape[0]
        cross_agg_dict_copy = cross_agg_dict.copy()
        cross_agg_dict_copy.update({'flow_id': ['count', lambda x: x.count() / tot_cnt]})

        cross_res = {}
        for colname, aggfunc in cross_agg_dict_copy.items():
            if data.shape[0] > nbins:
                cross_res[colname] = cross_risk(
                    data=prepared_data,
                    score_list=[base_score, compare_scr],
                    dep=dep,
                    nbins=nbins,
                    agg_col=colname,
                    precision=precision,
                    min_bin_prop=min_bin_prop,
                    include_missing=include_missing,
                    equal_freq=equal_freq,
                    binning_numeric=binning_numeric,
                    tree_binning=tree_binning,
                    agg_func=aggfunc,
                    fillna=fillna,
                    spec_values=spec_values
                )

        fnl_res = []
        for colname, res in cross_res.items():
            res = _ensure_three_level_columns(res)
            res[('', 'eval_metric', '')] = colname
            res = res.rename(columns={"<lambda>": "", "count": "n"})
            if disp:
                from IPython.display import display
                display(res)
            fnl_res.append(res)

        fnl_res = pd.concat(fnl_res)

        fnl_res[('', 'score_name', '')] = compare_scr
        multi_scr_res[compare_scr] = fnl_res

    combined_cross_res = []
    for score_name, cross_data in multi_scr_res.items():
        cross_data = cross_data.copy()
        cross_data.columns = cross_data.columns.map(
            lambda x: f"{x[0]}_{x[1] if len(str(x[1])) >= 2 else ('0' + str(x[1]))}_{x[2]}".strip("_")
        )
        cross_data.index = cross_data.index.map(
            lambda x: f"{x[0] if len(str(x[0])) >= 2 else ('0' + str(x[0]))}_{x[1]}".strip("_")
        )
        cross_data = cross_data.reset_index(drop=False).melt(
            id_vars=['index', 'eval_metric', 'score_name'],
            var_name='comp_scr_range'
        ).rename(columns={"index": "base_scr_range"})
        combined_cross_res.append(cross_data)
    combined_cross_res = pd.concat(combined_cross_res)

    return combined_cross_res

multi_subset_wrapper

multi_subset_wrapper(condition_dict: Optional[Dict[str, str]] = None, subset_var_name: str = 'eval_subset', func: Optional[Callable] = None, min_subset_size=10, **kwargs) -> DataFrame

Apply evaluation function across multiple data subsets.

This method iterates over predefined subset conditions, filters the data accordingly, and applies the evaluation function to each subset.

Parameters:

Name	Type	Description	Default
condition_dict	dict	Dictionary mapping subset names to query conditions (DataFrame.query strings; an empty string keeps every row). If None, uses self.subset_condition_dict.	None
subset_var_name	str	Column name for subset identifier in results. Default is 'eval_subset'.	'eval_subset'
func	callable	Function to apply to each subset. If None, uses model_perf_compare.	None
min_subset_size	int	Only subsets with more than this many rows are evaluated. Default is 10.	10
**kwargs		Additional keyword arguments passed to the evaluation function.	{}

Returns:

Type	Description
DataFrame	Combined results from all subsets, with the subset label in the column subset_var_name. With no subset condition at all (an empty dict) the result is an empty DataFrame that has only the column subset_var_name. If there are conditions but no subset is evaluated or none returns a non-empty result, a one-row DataFrame that only holds the label of the last subset in subset_var_name is returned.

Notes

self.data is replaced by the subset while func runs and restored afterwards (it is not restored if func raises an exception).

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

    def multi_subset_wrapper(
        self,
        condition_dict: Optional[Dict[str, str]] = None,
        subset_var_name: str = 'eval_subset',
        func: Optional[Callable] = None,
        min_subset_size = 10,
        **kwargs
    ) -> pd.DataFrame:
        """
        Apply evaluation function across multiple data subsets.

        This method iterates over predefined subset conditions, filters the data
        accordingly, and applies the evaluation function to each subset.

        Parameters
        ----------
        condition_dict : dict, optional
            Dictionary mapping subset names to query conditions (``DataFrame.query`` strings; an empty string keeps every
            row). If None, uses self.subset_condition_dict.
        subset_var_name : str, optional
            Column name for subset identifier in results. Default is 'eval_subset'.
        func : callable, optional
            Function to apply to each subset. If None, uses model_perf_compare.
        min_subset_size : int, optional
            Only subsets with more than this many rows are evaluated. Default is 10.
        **kwargs
            Additional keyword arguments passed to the evaluation function.

        Returns
        -------
        pandas.DataFrame
            Combined results from all subsets, with the subset label in the column ``subset_var_name``. With no subset
            condition at all (an empty dict) the result is an empty DataFrame that has only the column ``subset_var_name``.
            If there are conditions but no subset is evaluated or none returns a non-empty result, a one-row DataFrame that
            only holds the label of the last subset in ``subset_var_name`` is returned.

        Notes
        -----
        ``self.data`` is replaced by the subset while ``func`` runs and restored afterwards (it is not restored if ``func``
        raises an exception).
        """
        if condition_dict is None:
            condition_dict = self.subset_condition_dict

        original_data = self.data.copy()
        if func is None:
            func = self.model_perf_compare

        fnl_subset_results = []

        if condition_dict:
            for group_value, query in condition_dict.items():
                logger.info(f"INFO:: Multi_Subset_Eval: Running Subset {group_value}")

                subset_data = self.data.query(query).copy() if query != "" else self.data.copy()
                subset_data_size = subset_data.shape[0]

                if subset_data_size > min_subset_size:
                    self.data = subset_data

                    subset_result = func(**kwargs)
                    if subset_result is not None and not subset_result.empty:
                        subset_result = subset_result.copy()
                        subset_result[subset_var_name] = group_value
                        fnl_subset_results.append(subset_result)
                    else:
#                         fnl_subset_results.append(pd.DataFrame({subset_var_name: [group_value]}))
                        self.data = original_data
                        continue

                    self.data = original_data


        if fnl_subset_results:
            return pd.concat(fnl_subset_results)

        if not condition_dict:
            return pd.DataFrame({subset_var_name: []})
        return pd.DataFrame({subset_var_name: [group_value]})

multi_ylabel_wrapper

multi_ylabel_wrapper(ylabels: Optional[List[str]] = None, ylabel_var_name: str = 'eval_ylabel', eval_func: Optional[Callable] = None, **kwargs) -> DataFrame

Apply evaluation function across multiple target labels.

This method iterates over different target/label columns, temporarily updating the dependent variable for each iteration.

Parameters:

Name	Type	Description	Default
ylabels	list	List of target column names. If None, uses self.eval_ylabels.	None
ylabel_var_name	str	Column name for label identifier in results. Default is 'eval_ylabel'.	'eval_ylabel'
eval_func	callable	Function to apply for each label. If None, uses model_perf_compare.	None
**kwargs		Additional keyword arguments passed to the evaluation function.	{}

Returns:

Type	Description
DataFrame	Combined results from all labels, with the label in the column ylabel_var_name. Empty if no label has a row with a non-missing value.

Notes

For each label the rows in which the label is missing are dropped, and self.dep and self.data are replaced while eval_func runs and restored afterwards (they are not restored if eval_func raises an exception).

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def multi_ylabel_wrapper(
    self,
    ylabels: Optional[List[str]] = None,
    ylabel_var_name: str = 'eval_ylabel',
    eval_func: Optional[Callable] = None,
    **kwargs
) -> pd.DataFrame:
    """
    Apply evaluation function across multiple target labels.

    This method iterates over different target/label columns, temporarily
    updating the dependent variable for each iteration.

    Parameters
    ----------
    ylabels : list, optional
        List of target column names. If None, uses self.eval_ylabels.
    ylabel_var_name : str, optional
        Column name for label identifier in results. Default is 'eval_ylabel'.
    eval_func : callable, optional
        Function to apply for each label. If None, uses model_perf_compare.
    **kwargs
        Additional keyword arguments passed to the evaluation function.

    Returns
    -------
    pandas.DataFrame
        Combined results from all labels, with the label in the column ``ylabel_var_name``. Empty if no label has a
        row with a non-missing value.

    Notes
    -----
    For each label the rows in which the label is missing are dropped, and ``self.dep`` and ``self.data`` are replaced
    while ``eval_func`` runs and restored afterwards (they are not restored if ``eval_func`` raises an exception).
    """
    original_dep = self.dep
    original_data = self.data.copy()
    if eval_func is None:
        eval_func = self.model_perf_compare

    fnl_ylabel_results = []

    if ylabels is None:
        ylabels = self.eval_ylabels

    for current_dep in ylabels:
        logger.info(f"INFO:: Multi_yLabel_Eval: Running Label {current_dep}")

        input_data = self.data.query(f"{current_dep} == {current_dep}").copy()
        subset_data_size = input_data.shape[0]

        if subset_data_size > 0:
            self.dep = current_dep
            self.data = input_data

            subset_result = eval_func(**kwargs)
            subset_result = subset_result.copy()
            subset_result[ylabel_var_name] = current_dep
            fnl_ylabel_results.append(subset_result)

            self.dep = original_dep
            self.data = original_data

    if fnl_ylabel_results:
        return pd.concat(fnl_ylabel_results)
    return pd.DataFrame()

multi_group_wrapper

multi_group_wrapper(group_name: Optional[str] = None, group_var_name: str = 'group_name', group_eval_func: Optional[Callable] = None, min_subset_size: int = 10, **kwargs) -> DataFrame

Apply evaluation function across multiple groups.

This method splits data by a grouping column and applies the evaluation function to each group separately.

Parameters:

Name	Type	Description	Default
group_name	str	Column name to group by. If None, group_eval_func is called once on all the data (an empty DataFrame is returned when group_eval_func is None as well).	None
group_var_name	str	Column name for group identifier in results. Default is 'group_name'.	'group_name'
group_eval_func	callable	Function to apply to each group. If None, uses model_perf_compare.	None
min_subset_size	int	Only groups with more than this many rows are evaluated. Default is 10.	10
**kwargs		Additional keyword arguments passed to the evaluation function.	{}

Returns:

Type	Description
DataFrame	Combined results from all groups, with the group value in the column group_var_name. Empty if no group is evaluated.

Notes

The groups are the distinct non-missing values of group_name in order of appearance, and the rows are selected by value, so numeric group columns and values containing quotes work. self.data is replaced by the group while group_eval_func runs and restored afterwards (it is not restored if the function raises an exception).

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def multi_group_wrapper(
    self,
    group_name: Optional[str] = None,
    group_var_name: str = 'group_name',
    group_eval_func: Optional[Callable] = None,
    min_subset_size: int = 10,
    **kwargs
) -> pd.DataFrame:
    """
    Apply evaluation function across multiple groups.

    This method splits data by a grouping column and applies the evaluation
    function to each group separately.

    Parameters
    ----------
    group_name : str, optional
        Column name to group by. If None, ``group_eval_func`` is called once on all the data (an empty DataFrame is
        returned when ``group_eval_func`` is None as well).
    group_var_name : str, optional
        Column name for group identifier in results. Default is 'group_name'.
    group_eval_func : callable, optional
        Function to apply to each group. If None, uses model_perf_compare.
    min_subset_size : int, optional
        Only groups with more than this many rows are evaluated. Default is 10.
    **kwargs
        Additional keyword arguments passed to the evaluation function.

    Returns
    -------
    pandas.DataFrame
        Combined results from all groups, with the group value in the column ``group_var_name``. Empty if no group is
        evaluated.

    Notes
    -----
    The groups are the distinct non-missing values of ``group_name`` in order of appearance, and the rows are selected
    by value, so numeric group columns and values containing quotes work. ``self.data`` is replaced by the group while
    ``group_eval_func`` runs and restored afterwards (it is not restored if the function raises an exception).
    """
    original_data = self.data.copy()

    if group_name is None:
        return group_eval_func(**kwargs) if group_eval_func else pd.DataFrame()

    group_column = original_data[group_name]
    # Missing values form no group; the values are matched as values, not through a query string,
    # so numeric groups and values with quotes work
    group_value_list = pd.unique(group_column.dropna())

    if group_eval_func is None:
        group_eval_func = self.model_perf_compare

    fnl_group_results = []
    for group_value in group_value_list:
        logger.info(f"INFO:: Multi_Group_Eval: Running Group {group_value}")

        in_group = group_column.eq(group_value).fillna(False).astype(bool)
        input_data = original_data.loc[in_group].copy()
        subset_data_size = input_data.shape[0]

        if subset_data_size > min_subset_size:
            self.data = input_data

            subset_result = group_eval_func(**kwargs)
            subset_result = subset_result.copy()
            subset_result[group_var_name] = group_value
            fnl_group_results.append(subset_result)

            self.data = original_data

    if fnl_group_results:
        return pd.concat(fnl_group_results)
    return pd.DataFrame()

multi_dim_eval

multi_dim_eval(condition_dict: Optional[Dict[str, str]] = None, eval_ylabels: Optional[List[str]] = None, grp_namelist: Optional[List[str]] = None, eval_func: Optional[Callable] = None, subset_var_name: str = 'eval_subset', ylabel_var_name: str = 'eval_ylabel', var_name: str = 'group_name', value_name: str = 'group_value', **kwargs) -> DataFrame

Perform multi-dimensional evaluation across subsets, labels, and groups.

This method combines multi_subset_wrapper and multi_ylabel_wrapper to evaluate model performance across different dimensions.

Parameters:

Name	Type	Description	Default
condition_dict	dict	Subset conditions. If None, uses self.subset_condition_dict.	None
eval_ylabels	list	Target labels. If None, uses self.eval_ylabels.	None
grp_namelist	list	Group columns. If None, uses self.grp_namelist.	None
eval_func	callable	Evaluation function. If None, uses model_perf_compare. It is called for every subset, label and group column; the group column is passed to it as grp_name only when eval_func has a grp_name parameter.	None
subset_var_name	str	Column name for subset identifier. Default is 'eval_subset'.	'eval_subset'
ylabel_var_name	str	Column name for label identifier. Default is 'eval_ylabel'.	'eval_ylabel'
var_name	str	Column name for group name in results. Default is 'group_name'.	'group_name'
value_name	str	Column name for group value in results. Default is 'group_value'.	'group_value'
**kwargs		Additional keyword arguments, passed on to eval_func.	{}

Returns:

Type	Description
DataFrame	Multi-dimensional evaluation results: the results of eval_func for every subset, label and group column, stacked, with the columns subset_var_name, ylabel_var_name and var_name (the group column that was evaluated). When eval_func has a grp_name parameter, the group columns are merged into the single column value_name.

Notes

If eval_func has no grp_name parameter, the same evaluation is simply repeated for each entry of grp_namelist (no grouping is done). With the default eval_func every call is grouped, so the evaluation is unweighted.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def multi_dim_eval(
    self,
    condition_dict: Optional[Dict[str, str]] = None,
    eval_ylabels: Optional[List[str]] = None,
    grp_namelist: Optional[List[str]] = None,
    eval_func: Optional[Callable] = None,
    subset_var_name: str = 'eval_subset',
    ylabel_var_name: str = 'eval_ylabel',
    var_name: str = 'group_name',
    value_name: str = 'group_value',
    **kwargs
) -> pd.DataFrame:
    """
    Perform multi-dimensional evaluation across subsets, labels, and groups.

    This method combines multi_subset_wrapper and multi_ylabel_wrapper to
    evaluate model performance across different dimensions.

    Parameters
    ----------
    condition_dict : dict, optional
        Subset conditions. If None, uses self.subset_condition_dict.
    eval_ylabels : list, optional
        Target labels. If None, uses self.eval_ylabels.
    grp_namelist : list, optional
        Group columns. If None, uses self.grp_namelist.
    eval_func : callable, optional
        Evaluation function. If None, uses model_perf_compare. It is called for every subset, label and group column;
        the group column is passed to it as ``grp_name`` only when ``eval_func`` has a ``grp_name`` parameter.
    subset_var_name : str, optional
        Column name for subset identifier. Default is 'eval_subset'.
    ylabel_var_name : str, optional
        Column name for label identifier. Default is 'eval_ylabel'.
    var_name : str, optional
        Column name for group name in results. Default is 'group_name'.
    value_name : str, optional
        Column name for group value in results. Default is 'group_value'.
    **kwargs
        Additional keyword arguments, passed on to ``eval_func``.

    Returns
    -------
    pandas.DataFrame
        Multi-dimensional evaluation results: the results of ``eval_func`` for every subset, label and group column,
        stacked, with the columns ``subset_var_name``, ``ylabel_var_name`` and ``var_name`` (the group column that was
        evaluated). When ``eval_func`` has a ``grp_name`` parameter, the group columns are merged into the single
        column ``value_name``.

    Notes
    -----
    If ``eval_func`` has no ``grp_name`` parameter, the same evaluation is simply repeated for each entry of
    ``grp_namelist`` (no grouping is done). With the default ``eval_func`` every call is grouped, so the evaluation is
    unweighted.
    """
    if condition_dict is None:
        condition_dict = self.subset_condition_dict

    if eval_ylabels is None:
        eval_ylabels = self.eval_ylabels

    if grp_namelist is None:
        grp_namelist = self.grp_namelist

    if eval_func is None:
        eval_func = self.model_perf_compare

    fnl_perf_res = []
    for group_name in grp_namelist:
        logger.info(f"INFO:: Multi_Dim_Eval: Running Group {group_name}")

        sig = inspect.signature(eval_func)
        parameters = sig.parameters

        if 'grp_name' in parameters:
            sub_perf_res = self.multi_subset_wrapper(
                condition_dict=condition_dict,
                subset_var_name=subset_var_name,
                func=self.multi_ylabel_wrapper, 
                ylabels=eval_ylabels, 
                ylabel_var_name=ylabel_var_name,
                eval_func=eval_func, 
                grp_name=group_name,
                **kwargs
            )
        else:
            sub_perf_res = self.multi_subset_wrapper(
                condition_dict=condition_dict,
                subset_var_name=subset_var_name,
                func=self.multi_ylabel_wrapper, 
                ylabels=eval_ylabels, 
                ylabel_var_name=ylabel_var_name,
                eval_func=eval_func, 
                **kwargs
            )

        sub_perf_res = sub_perf_res.copy()
        sub_perf_res[var_name] = group_name
        fnl_perf_res.append(sub_perf_res)

    fnl_perf_res = pd.concat(fnl_perf_res)

    if 'grp_name' in parameters:
        if all(x in fnl_perf_res.columns for x in grp_namelist):
            fnl_perf_res[value_name] = fnl_perf_res[grp_namelist].bfill(axis=1).iloc[:, 0]
            fnl_perf_res = fnl_perf_res.drop(columns=grp_namelist)

    return fnl_perf_res

cross_perf_eval

cross_perf_eval(bad_list: List[str], scr_list: List[str], data: Optional[DataFrame] = None, eval_metric: List[str] = None, melt: bool = True) -> DataFrame

Cross-evaluate model performance across multiple bad flags and scores.

This method calculates performance metrics for all combinations of bad flags and scores, useful for comparing model behavior across different definitions of "bad" outcomes.

Parameters:

Name	Type	Description	Default
bad_list	list	List of bad/negative outcome column names.	required
scr_list	list	List of score column names.	required
data	DataFrame	Data to use. If None, uses self.data.	None
eval_metric	list	Metrics to extract. If None, uses ['AUC', 'KS', 'N'].	None
melt	bool	Whether to melt the pivot table. Default is True.	True

Returns:

Type

Description

DataFrame

Cross performance evaluation results. With melt=True a long table with the columns score_name, variable (<metric>_<label>, e.g. AUC_bad_flag) and value; with melt=False the pivot table itself, indexed by score_name with one <metric>_<label> column per metric and label. If the data has no more rows than nbins of the instance, an empty DataFrame with the columns eval_ylabel and score_name is returned.

Notes

The rows in which a label is missing or a score is not positive are dropped. The bins come from the instance (nbins for both the distribution and the percentile bins, precision and min_bin_prop) and the evaluation is unweighted.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def cross_perf_eval(
    self,
    bad_list: List[str],
    scr_list: List[str],
    data: Optional[pd.DataFrame] = None,
    eval_metric: List[str] = None,
    melt: bool = True
) -> pd.DataFrame:
    """
    Cross-evaluate model performance across multiple bad flags and scores.

    This method calculates performance metrics for all combinations of
    bad flags and scores, useful for comparing model behavior across
    different definitions of "bad" outcomes.

    Parameters
    ----------
    bad_list : list
        List of bad/negative outcome column names.
    scr_list : list
        List of score column names.
    data : pandas.DataFrame, optional
        Data to use. If None, uses self.data.
    eval_metric : list, optional
        Metrics to extract. If None, uses ['AUC', 'KS', 'N'].
    melt : bool, optional
        Whether to melt the pivot table. Default is True.

    Returns
    -------
    pandas.DataFrame
        Cross performance evaluation results. With ``melt=True`` a long table with the columns ``score_name``,
        ``variable`` (``<metric>_<label>``, e.g. ``AUC_bad_flag``) and ``value``; with ``melt=False`` the pivot table
        itself, indexed by ``score_name`` with one ``<metric>_<label>`` column per metric and label. If the data has
        no more rows than ``nbins`` of the instance, an empty DataFrame with the columns ``eval_ylabel`` and
        ``score_name`` is returned.

    Notes
    -----
    The rows in which a label is missing or a score is not positive are dropped. The bins come from the instance
    (``nbins`` for both the distribution and the percentile bins, ``precision`` and ``min_bin_prop``) and the
    evaluation is unweighted.
    """
    if eval_metric is None:
        eval_metric = ['AUC', 'KS', 'N']

    if data is None:
        data = self.data.copy()

    score_query = " and ".join([s + " > 0" for s in scr_list])
    bad_query = " and ".join([s + " == " + s for s in bad_list])

    data = data.query(bad_query)
    data = data.query(score_query)

    if data.shape[0] > self.nbins:
        fnl_res = []
        for bad in bad_list:
            for scr in scr_list:
                perf = get_perf_summary(
                    None,
                    None,
                    data,
                    tgt_name=bad,
                    scr_name=scr,
                    to_show=False,
                    display=False,
                    dist_bins=self.nbins,
                    pct_bins=self.nbins,
                    precision=self.precision,
                    min_bin_prop=self.min_bin_prop,
                    equal_freq=True
                )
                perf = perf.copy()
                perf['eval_ylabel'] = bad
                perf['score_name'] = scr
                fnl_res.append(perf)

        fnl_res = pd.concat(fnl_res)

        summary = fnl_res.pivot_table(index="score_name", columns='eval_ylabel', values=eval_metric)
        summary.columns = summary.columns.map(lambda x: f"{x[0]}_{x[1]}")

        if melt:
            summary = summary.reset_index(drop=False).melt(id_vars=['score_name'])

        return summary

    return pd.DataFrame(columns=['eval_ylabel', 'score_name'])

run_variable_analysis_summary

run_variable_analysis_summary(varlist: List[str], data: Optional[DataFrame] = None, dep: Optional[str] = None, nbins: Optional[int] = None, equal_freq: Optional[bool] = None, min_bin_prop: Optional[float] = None, precision: Optional[int] = None, chi2_method: Optional[bool] = None, chi2_p: Optional[float] = None, init_equi_bins: Optional[int] = None, tree_binning: Optional[bool] = None, include_missing: Optional[bool] = None, missing_rate_ref: Optional[Union[int, float]] = None, seed: Optional[int] = None, spec_values=[]) -> DataFrame

Run comprehensive variable analysis and generate summary report.

This method performs binning analysis on specified variables, calculating metrics like IV (Information Value), KS and Lift of each variable.

Parameters:

Name	Type	Description	Default
varlist	list	List of variable names to analyze.	required
data	DataFrame	Data to use. If None, uses self.data.	None
dep	str	Target variable name. If None, uses self.dep.	None
nbins	int	Number of bins. If None, uses self.nbins.	None
equal_freq	bool	Use equal frequency binning. If None, uses self.equal_freq.	None
min_bin_prop	float	Minimum bin proportion. If None, uses self.min_bin_prop.	None
precision	int	Decimal precision. If None, uses self.precision.	None
chi2_method	bool	Use chi-square method. If None, uses self.chi2_method.	None
chi2_p	float	Chi-square p-value. If None, uses self.chi2_p.	None
init_equi_bins	int	Initial equal bins. If None, uses self.init_equi_bins.	None
tree_binning	bool	Use tree binning. If None, uses self.tree_binning.	None
include_missing	bool	Include missing values. If None, uses self.include_missing.	None
missing_rate_ref	int or float	Missing reference. If None, uses self.missing_rate_ref.	None
seed	int	Random seed. If None, uses self.seed.	None
spec_values	list	Special values that get their own bins in the analysis. Unlike the other arguments, the default is not taken from the instance.	[]

Returns:

Type	Description
DataFrame	Variable analysis summary with IV and other metrics: one row per variable, sorted by decreasing IV, with the columns var, n_all (rows), n (non-missing rows), ks_in_gains, lift_in_gains, iv, n_bump, missing_rate, min, mean, max and n_bins. Constant variables and variables whose computation fails are left out (a warning lists the failed ones).

Notes

Before the analysis the missing values of the variables in varlist are filled with missing_rate_ref (in a copy of the data). The summary is unweighted.

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def run_variable_analysis_summary(
    self,
    varlist: List[str],
    data: Optional[pd.DataFrame] = None,
    dep: Optional[str] = None,
    nbins: Optional[int] = None,
    equal_freq: Optional[bool] = None,
    min_bin_prop: Optional[float] = None,
    precision: Optional[int] = None,
    chi2_method: Optional[bool] = None,
    chi2_p: Optional[float] = None,
    init_equi_bins: Optional[int] = None,
    tree_binning: Optional[bool] = None,
    include_missing: Optional[bool] = None,
    missing_rate_ref: Optional[Union[int, float]] = None,
    seed: Optional[int] = None,
    spec_values = []
) -> pd.DataFrame:
    """
    Run comprehensive variable analysis and generate summary report.

    This method performs binning analysis on specified variables,
    calculating metrics like IV (Information Value), KS and Lift of each variable.

    Parameters
    ----------
    varlist : list
        List of variable names to analyze.
    data : pandas.DataFrame, optional
        Data to use. If None, uses self.data.
    dep : str, optional
        Target variable name. If None, uses self.dep.
    nbins : int, optional
        Number of bins. If None, uses self.nbins.
    equal_freq : bool, optional
        Use equal frequency binning. If None, uses self.equal_freq.
    min_bin_prop : float, optional
        Minimum bin proportion. If None, uses self.min_bin_prop.
    precision : int, optional
        Decimal precision. If None, uses self.precision.
    chi2_method : bool, optional
        Use chi-square method. If None, uses self.chi2_method.
    chi2_p : float, optional
        Chi-square p-value. If None, uses self.chi2_p.
    init_equi_bins : int, optional
        Initial equal bins. If None, uses self.init_equi_bins.
    tree_binning : bool, optional
        Use tree binning. If None, uses self.tree_binning.
    include_missing : bool, optional
        Include missing values. If None, uses self.include_missing.
    missing_rate_ref : int or float, optional
        Missing reference. If None, uses self.missing_rate_ref.
    seed : int, optional
        Random seed. If None, uses self.seed.
    spec_values : list, default []
        Special values that get their own bins in the analysis. Unlike the other arguments, the default is not taken
        from the instance.

    Returns
    -------
    pandas.DataFrame
        Variable analysis summary with IV and other metrics: one row per variable, sorted by decreasing IV, with the
        columns ``var``, ``n_all`` (rows), ``n`` (non-missing rows), ``ks_in_gains``, ``lift_in_gains``, ``iv``,
        ``n_bump``, ``missing_rate``, ``min``, ``mean``, ``max`` and ``n_bins``. Constant variables and variables whose
        computation fails are left out (a warning lists the failed ones).

    Notes
    -----
    Before the analysis the missing values of the variables in ``varlist`` are filled with ``missing_rate_ref`` (in a
    copy of the data). The summary is unweighted.
    """
    if data is None:
        data = self.data.copy()

    if dep is None:
        dep = self.dep

    nbins = nbins if nbins is not None else self.nbins
    equal_freq = equal_freq if equal_freq is not None else self.equal_freq
    min_bin_prop = min_bin_prop if min_bin_prop is not None else self.min_bin_prop
    precision = precision if precision is not None else self.precision
    chi2_method = chi2_method if chi2_method is not None else self.chi2_method
    chi2_p = chi2_p if chi2_p is not None else self.chi2_p
    init_equi_bins = init_equi_bins if init_equi_bins is not None else self.init_equi_bins
    tree_binning = tree_binning if tree_binning is not None else self.tree_binning
    include_missing = include_missing if include_missing is not None else self.include_missing
    missing_rate_ref = missing_rate_ref if missing_rate_ref is not None else self.missing_rate_ref
    seed = seed if seed is not None else self.seed

    from Modeling_Tool.Feature.Feature_Insights import VarExtractionInsights

    varInsights = VarExtractionInsights(
        data=data,
        dep=dep,
        plot_path=None,
        nbins=nbins,
        equal_freq=equal_freq,
        min_bin_prop=min_bin_prop,
        precision=precision,
        chi2_method=chi2_method,
        chi2_p=chi2_p,
        init_equi_bins=init_equi_bins,
        tree_binning=tree_binning,
        include_missing=include_missing,
        seed=seed,
        missing_rate_ref=missing_rate_ref,
        spec_values=spec_values
    )

    data = data.copy()
    for var in varlist:
        data[var] = data[var].fillna(missing_rate_ref)

    var_summary = varInsights.get_var_analysis_report(data=data, varlist=varlist, dep=dep, iv_cut=0)

    return var_summary

pipe

pipe(data=None)

Return an EvaluationPipeline for chained group and subset evaluation.

Parameters:

Name	Type	Description	Default
data	DataFrame	Data to evaluate. Defaults to self.data.	None

Returns:

Type	Description
EvaluationPipeline

Source code in Modeling_Tool/Eval/Evaluation_Tool.py

def pipe(self, data=None):
        """
        Return an ``EvaluationPipeline`` for chained group and subset evaluation.

        Parameters
        ----------
        data : pd.DataFrame, optional
            Data to evaluate. Defaults to ``self.data``.

        Returns
        -------
        EvaluationPipeline
        """
        return EvaluationPipeline(self, data)

Curves, Summaries, and Figures: evaluate_model

evaluate_model

timeit_decorator

timeit_decorator(func)

Decorator that times every call of func.

The elapsed time is measured but not reported (the line that prints it is commented out), so the wrapped function behaves exactly like func: same arguments, same return value, and the metadata is preserved by functools.wraps.

Parameters:

Name	Type	Description	Default
func		Function to wrap.	required

Returns:

Name	Type	Description
wrapper	callable	The wrapped function.

Source code in Modeling_Tool/Eval/evaluate_model.py

def timeit_decorator(func):
    """Decorator that times every call of ``func``.

    The elapsed time is measured but not reported (the line that prints it is commented out), so the wrapped function
    behaves exactly like ``func``: same arguments, same return value, and the metadata is preserved by ``functools.wraps``.

    Parameters
    ----------
    func: callable
        Function to wrap.

    Returns
    -------
    wrapper: callable
        The wrapped function.
    """
    @wraps(func)
    def wrapper(*args, **kwargs):
        start_time = time.time()
        result = func(*args, **kwargs)
        end_time = time.time()
        elapsed_time = end_time - start_time
        # print(f"Function '{func.__name__}' took {elapsed_time:.6f} seconds") # Uncomment this line to print the elapsed time of every function
        return result
    return wrapper

calc_pr

calc_pr(y_true, y_score, sample_weight=None)

Compute the statistics of the P-R curve. Based on sklearn.metrics.precision_recall_curve.

Parameters:

Name	Type	Description	Default
y_true		Sequence of actual sample labels; only 0/1 values are accepted.	required
y_score		Sequence of predicted probabilities.	required
sample_weight		Per-sample weights aligned with y_true. When given, the weighted implementation of weighted_eval_utils is used.	None

Returns:

Name	Type	Description
pr_df	DataFrame	Dataset of PR statistics such as precision, recall, and thresholds.

Notes

The columns are precision, recall, thresholds and, on the unweighted path only, thresholds_percentile (percentage of scores at or below the threshold). The last row has no threshold (NaN).

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def calc_pr(y_true, y_score, sample_weight=None):
    """Compute the statistics of the P-R curve.
    Based on sklearn.metrics.precision_recall_curve.

    Parameters
    ----------
    y_true: array like
        Sequence of actual sample labels; only 0/1 values are accepted.
    y_score: array like
        Sequence of predicted probabilities.
    sample_weight: array like or None, default None
        Per-sample weights aligned with ``y_true``. When given, the weighted implementation of ``weighted_eval_utils`` is used.

    Returns
    -------
    pr_df: pandas.DataFrame
        Dataset of PR statistics such as precision, recall, and thresholds.

    Notes
    -----
    The columns are ``precision``, ``recall``, ``thresholds`` and, on the unweighted path only, ``thresholds_percentile``
    (percentage of scores at or below the threshold). The last row has no threshold (NaN).
    """
    if sample_weight is not None:
        return _weighted_eval.calc_pr(y_true, y_score, sample_weight=sample_weight)

    pr_df = pd.DataFrame(precision_recall_curve(y_true, y_score)).T
    pr_df.columns = ['precision', 'recall', 'thresholds']
    pr_df['thresholds_percentile'] = [100 * np.mean(y_score <= x) for x in pr_df['thresholds']] 

    return pr_df

summarize_pr

summarize_pr(pr_df)

Summarize the P-R curve.

The statistics are:

1. The break-even point (BEP) threshold and the corresponding precision, recall, and other statistics.

Parameters:

Name	Type	Description	Default
pr_df		Dataset of PR statistics such as precision, recall, and thresholds.	required

Returns:

Name	Type	Description
pr_info	dict	Dictionary of P-R curve summary statistics: bep_index (row of the break-even point, where precision and recall are closest), bep_threshold, bep_precision and bep_recall. All four values are NaN when the gap between precision and recall is undefined (all NaN, e.g. a sample with a single class).

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def summarize_pr(pr_df):
    """Summarize the P-R curve.

    The statistics are:

    1. The break-even point (BEP) threshold and the corresponding precision, recall, and other statistics.

    Parameters
    ----------
    pr_df: pandas.DataFrame
        Dataset of PR statistics such as precision, recall, and thresholds.

    Returns
    -------
    pr_info: dict
        Dictionary of P-R curve summary statistics: ``bep_index`` (row of the break-even point, where precision and
        recall are closest), ``bep_threshold``, ``bep_precision`` and ``bep_recall``. All four values are NaN when the gap
        between precision and recall is undefined (all NaN, e.g. a sample with a single class).
    """
    gap = abs(pr_df['precision'] - pr_df['recall'])
    if not gap.notna().any():
        # Precision or recall is all NaN (e.g. the sample has a single class): the break-even point is undefined
        return {'bep_index': np.nan, 'bep_threshold': np.nan, 'bep_precision': np.nan, 'bep_recall': np.nan}
    equalind = np.argmin(gap)
    pr_info = {
        'bep_index': equalind, 
        'bep_threshold': pr_df['thresholds'][equalind], 
        'bep_precision': pr_df['precision'][equalind],
        'bep_recall': pr_df['recall'][equalind],
        }

    return pr_info

plot_pr_curve

plot_pr_curve(pr_dfs, square_figsize=8, to_show=True, save_path=None)

Plot the P-R curve.

Parameters:

Name	Type	Description	Default
pr_dfs		PR datasets for one or more scores, as key-value pairs in the format {name: pr_df}.	required
square_figsize		Side length of the square figure in inches. Defaults to 8.	8
to_show		Whether to display the figure. Defaults to True.	True
save_path		File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.	None

Notes

One entry draws a single curve with its break-even point (BEP); several entries are overlaid, and at most three can be drawn together (the palette has three colors, a fourth raises IndexError).

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def plot_pr_curve(pr_dfs,  square_figsize=8, to_show=True, save_path=None):
    """Plot the P-R curve.

    Parameters
    ----------
    pr_dfs: Dict
        PR datasets for one or more scores, as key-value pairs in the format {name: pr_df}.
    square_figsize: float
        Side length of the square figure in inches. Defaults to 8.
    to_show: bool
        Whether to display the figure. Defaults to True.
    save_path: str
        File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.

    Notes
    -----
    One entry draws a single curve with its break-even point (BEP); several entries are overlaid, and at most three can be
    drawn together (the palette has three colors, a fourth raises ``IndexError``).
    """
    plt.figure(figsize=(square_figsize, square_figsize))
    plt.suptitle('P-R Curve', fontsize=20, fontweight='bold') #, findfont=zhfont)
    ax = plt.subplot(1,1,1)

    models = list(pr_dfs.keys())
    if len(models) == 1:
        pr_df = pr_dfs[models[0]]
        __plot_single_pr_axes(pr_df, ax)
    else:
        __plot_multi_pr_axes(pr_dfs, ax)
    if to_show:
        plt.show()
    if bool(save_path):
        plt.savefig(save_path, bbox_inches='tight')
    plt.close()

calc_roc

calc_roc(y_true, y_score, sample_weight=None)

Compute the statistics of the ROC curve. Based on sklearn.metrics.roc_curve.

Parameters:

Name	Type	Description	Default
y_true		Sequence of actual sample labels; only 0/1 values are accepted.	required
y_score		Sequence of predicted probabilities.	required
sample_weight		Per-sample weights aligned with y_true. When given, the weighted implementation of weighted_eval_utils is used.	None

Returns:

Name	Type	Description
roc_df	DataFrame	Dataset of ROC statistics such as TPR, FPR, and thresholds.

Notes

The columns are fpr, tpr, thresholds and thresholds_percentile (percentage of the samples, or of the total weight when weighted, with a score at or below the threshold). The weighted result also has FPR, TPR and KS (abs(tpr - fpr)). Rows whose label, score or weight is not finite are dropped; if no row is left, an empty DataFrame with these columns is returned.

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def calc_roc(y_true, y_score, sample_weight=None):
    """Compute the statistics of the ROC curve.
    Based on sklearn.metrics.roc_curve.

    Parameters
    ----------
    y_true: array like
        Sequence of actual sample labels; only 0/1 values are accepted.
    y_score: array like
        Sequence of predicted probabilities.
    sample_weight: array like or None, default None
        Per-sample weights aligned with ``y_true``. When given, the weighted implementation of ``weighted_eval_utils`` is used.

    Returns
    -------
    roc_df: pandas.DataFrame
        Dataset of ROC statistics such as TPR, FPR, and thresholds.

    Notes
    -----
    The columns are ``fpr``, ``tpr``, ``thresholds`` and ``thresholds_percentile`` (percentage of the samples, or of the total
    weight when weighted, with a score at or below the threshold). The weighted result also has ``FPR``, ``TPR`` and ``KS``
    (``abs(tpr - fpr)``). Rows whose label, score or weight is not finite are dropped; if no row is left, an empty
    DataFrame with these columns is returned.
    """

    if sample_weight is not None:
        return _weighted_eval.calc_roc(y_true, y_score, sample_weight=sample_weight)

    # Remove invalid values
    mask = np.isfinite(y_score) & np.isfinite(y_true)
    y_true_clean = np.array(y_true)[mask]
    y_score_clean = np.array(y_score)[mask]

    if len(y_true_clean) == 0:
        # Return an empty ROC DataFrame
        return pd.DataFrame(columns=['fpr', 'tpr', 'thresholds', 'thresholds_percentile'])

    roc_df = pd.DataFrame(roc_curve(y_true_clean, y_score_clean)).T
    roc_df.columns = ['fpr', 'tpr', 'thresholds']
    roc_df['thresholds_percentile'] = [100 * np.mean(y_score_clean <= x) for x in roc_df['thresholds']]

    return roc_df

summarize_roc

summarize_roc(roc_df)

Summarize the ROC curve.

The statistics are:

1. AUC
2. KS and its corresponding threshold

Parameters:

Name	Type	Description	Default
roc_df		Dataset of ROC statistics such as TPR, FPR, and thresholds.	required

Returns:

Name	Type	Description
roc_info	dict	Dictionary of ROC curve summary statistics: auc, ks_index (row of the largest tpr - fpr), ks_threshold (score threshold at that row) and ks (largest abs(tpr - fpr)). All four values are NaN when roc_df is empty or TPR/FPR are all NaN (e.g. a sample with a single class).

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def summarize_roc(roc_df):
    """Summarize the ROC curve.

    The statistics are:

    1. AUC
    2. KS and its corresponding threshold

    Parameters
    ----------
    roc_df: pandas.DataFrame
        Dataset of ROC statistics such as TPR, FPR, and thresholds.

    Returns
    -------
    roc_info: dict
        Dictionary of ROC curve summary statistics: ``auc``, ``ks_index`` (row of the largest ``tpr - fpr``),
        ``ks_threshold`` (score threshold at that row) and ``ks`` (largest ``abs(tpr - fpr)``). All four values are NaN when
        ``roc_df`` is empty or TPR/FPR are all NaN (e.g. a sample with a single class).
    """

    if roc_df.empty:
        return {'auc': np.nan, 'ks_index': np.nan, 'ks_threshold': np.nan, 'ks': np.nan}

    f = roc_df['tpr'] - roc_df['fpr']
    if not f.notna().any():
        # With a single class, TPR or FPR is all NaN: AUC, KS, and their thresholds are undefined. Previously argmax
        # returned -1, and looking up thresholds[-1] by label raised a KeyError, so the whole evaluation failed
        return {'auc': np.nan, 'ks_index': np.nan, 'ks_threshold': np.nan, 'ks': np.nan}
    roc_info = {
        'auc': auc(roc_df['fpr'], roc_df['tpr']),
        'ks_index': np.argmax(f),
        'ks_threshold': roc_df['thresholds'][np.argmax(f)],
        'ks': max(abs(f)),
        }

    return roc_info

plot_ks_curve

plot_ks_curve(roc_df, square_figsize=8, to_show=True, save_path=None)

Plot the KS curve. Only the KS curve of a single score can be plotted.

Parameters:

Name	Type	Description	Default
roc_df		Dataset of ROC statistics such as TPR, FPR, and thresholds.	required
square_figsize		Side length of the square figure in inches. Defaults to 8.	8
to_show		Whether to display the figure. Defaults to True.	True
save_path		File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.	None

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def plot_ks_curve(roc_df, square_figsize=8, to_show=True, save_path=None):
    """Plot the KS curve.
    Only the KS curve of a single score can be plotted.

    Parameters
    ----------
    roc_df: pandas.DataFrame
        Dataset of ROC statistics such as TPR, FPR, and thresholds.
    square_figsize: float
        Side length of the square figure in inches. Defaults to 8.
    to_show: bool
        Whether to display the figure. Defaults to True.
    save_path: str
        File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.
    """
    plt.figure(figsize=(square_figsize, square_figsize))
    plt.suptitle('KS Curve', fontsize=20, fontweight='bold') #, findfont=zhfont)
    ax = plt.subplot(1,1,1)
    __plot_ks_axes(roc_df, ax)
    if to_show:
        plt.show()
    if bool(save_path):
        plt.savefig(save_path, bbox_inches='tight')
    plt.close()

plot_roc_curve

plot_roc_curve(roc_dfs, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None)

Plot the ROC curve. Curves for a single score or for multiple scores can be plotted.

Parameters:

Name	Type	Description	Default
roc_dfs		Dictionary of ROC statistics for one or more scores. Key-value pairs in the format {name: roc_df}.	required
square_figsize		Side length of the square figure in inches. Defaults to 8.	8
fontdicts		Dictionary of font settings for the plot, in the format of the module-level presets fontdicts['main'] and fontdicts['sub'] (keys suptitle, subtitle, axislabel and legend).	fontdicts['main']
to_show		Whether to display the figure. Defaults to True.	True
save_path		File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.	None

Notes

One entry draws a single ROC curve with its KS gap; several entries are overlaid, and at most three can be drawn together (the palette has three colors, a fourth raises IndexError).

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def plot_roc_curve(roc_dfs, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None):
    """Plot the ROC curve.
    Curves for a single score or for multiple scores can be plotted.

    Parameters
    ----------
    roc_dfs: dict
        Dictionary of ROC statistics for one or more scores.
        Key-value pairs in the format {name: roc_df}.
    square_figsize: float
        Side length of the square figure in inches. Defaults to 8.
    fontdicts: dict, default fontdicts['main']
        Dictionary of font settings for the plot, in the format of the module-level presets ``fontdicts['main']`` and
        ``fontdicts['sub']`` (keys ``suptitle``, ``subtitle``, ``axislabel`` and ``legend``).
    to_show: bool
        Whether to display the figure. Defaults to True.
    save_path: str
        File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.

    Notes
    -----
    One entry draws a single ROC curve with its KS gap; several entries are overlaid, and at most three can be drawn
    together (the palette has three colors, a fourth raises ``IndexError``).
    """
    plt.figure(figsize=(square_figsize, square_figsize))
    plt.suptitle('ROC Curve', fontsize=fontdicts['suptitle']['size'], fontweight=fontdicts['suptitle']['weight'])
    ax = plt.subplot(1,1,1)
    models = list(roc_dfs.keys())
    if len(models) == 1:
        roc_df = roc_dfs[models[0]]
        __plot_single_roc_axes(roc_df, ax, fontdicts)
    else:
        __plot_multi_roc_axes(roc_dfs, ax, fontdicts)
    if to_show:
        plt.show()
    if bool(save_path):
        plt.savefig(save_path, bbox_inches='tight')
    plt.close()

plot_kde_curve

plot_kde_curve(y_true, y_score_dict, bins=20, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None)

Plot the kernel density estimate (KDE) curves of the scores.

Parameters:

Name	Type	Description	Default
y_true		Sequence of actual sample labels; only 0/1 values are accepted.	required
y_score_dict		Dictionary of one or more score sequences, as key-value pairs in the format {name: Score}.	required
bins		Number of histogram bins; it also sets the bandwidth factor of the KDE (1 / bins / 2).	20
square_figsize		Side length of the square figure in inches. Defaults to 8.	8
fontdicts		Dictionary of font settings for the plot, in the format of the module-level presets fontdicts['main'] and fontdicts['sub'] (keys suptitle, subtitle, axislabel and legend).	fontdicts['main']
to_show		Whether to display the figure. Defaults to True.	True
save_path		File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.	None

Notes

One score draws a histogram plus the KDE curves of the negative and the positive samples; several scores draw one KDE curve per score and at most three can be drawn together (the palette has three colors, a fourth raises IndexError). The plot is unweighted: there is no weight argument.

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def plot_kde_curve(y_true, y_score_dict, bins=20, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None):
    """Plot the kernel density estimate (KDE) curves of the scores.

    Parameters
    ----------
    y_true: array like
        Sequence of actual sample labels; only 0/1 values are accepted.
    y_score_dict: dict
        Dictionary of one or more score sequences, as key-value pairs in the format {name: Score}.
    bins: int, default 20
        Number of histogram bins; it also sets the bandwidth factor of the KDE (``1 / bins / 2``).
    square_figsize: float
        Side length of the square figure in inches. Defaults to 8.
    fontdicts: dict, default fontdicts['main']
        Dictionary of font settings for the plot, in the format of the module-level presets ``fontdicts['main']`` and
        ``fontdicts['sub']`` (keys ``suptitle``, ``subtitle``, ``axislabel`` and ``legend``).
    to_show: bool
        Whether to display the figure. Defaults to True.
    save_path: str
        File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.

    Notes
    -----
    One score draws a histogram plus the KDE curves of the negative and the positive samples; several scores draw one KDE
    curve per score and at most three can be drawn together (the palette has three colors, a fourth raises ``IndexError``).
    The plot is unweighted: there is no weight argument.
    """
    y_true = np.array(y_true)
    plt.figure(figsize=(square_figsize, square_figsize))
    plt.suptitle('Score KDE Curve', fontsize=fontdicts['suptitle']['size'], fontweight=fontdicts['suptitle']['weight']) #, findfont=zhfont)
    ax = plt.subplot(1,1,1)
    models = list(y_score_dict.keys())
    if len(models) == 1:
        y_score = y_score_dict[models[0]]
        y_score = np.array(y_score)
        __plot_single_kde_axes(y_true, y_score, bins, ax, fontdicts)
    else:
        __plot_multi_kde_axes(y_true, y_score_dict, bins, ax, fontdicts)
    if to_show:
        plt.show()
    if bool(save_path):
        plt.savefig(save_path, bbox_inches='tight')
    plt.close()

calc_equid_dist

calc_equid_dist(y_true, y_score, y_group=None, bins=10, sample_weight=None)

Divide the score into equal-width bins and compute the statistics of each bin.

Parameters:

Name	Type	Description	Default
y_true		Sequence of actual sample labels; only 0/1 values are accepted.	required
y_score		Sequence of predicted probabilities.	required
y_group		Sequence of data groups. Defaults to None, i.e. no groups.	None
bins		Number of bins.	10
sample_weight		Per-sample weights aligned with y_true. When given, y_group is ignored and a weighted Gains table is returned instead (see Notes).	None

Returns:

Name	Type	Description
dist_df	DataFrame	Dataset of per-bin statistics after equal-width binning: one row per bin (per group and bin when y_group is given) with the columns thresholds (upper edge of the bin), min_score, max_score, n, proportion, sum_true, sum_score, avg_true, avg_score, capture_rate and the cumulative columns cumsum_n, cumsum_proportion, cumsum_true, cumsum_score, cumavg_true and cumavg_score. y_group is the first column when groups are given.

Notes

The bin edges follow powers of ten: they run from 0 (or, for negative scores, from the negative power of ten at or below the minimum score) to the power of ten at or above the maximum score, so probabilities are cut at 0, 0.1, ..., 1 when bins=10.

With sample_weight the call is delegated to the weighted Gains-table implementation. The result is then a Gains table (columns MIN, MAX, N, N_RAW, AVG_BAD, LIFT, ... indexed by _bin_num and _bin_range) built from bins equal-weight bins, not an equal-width table.

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def calc_equid_dist(y_true, y_score, y_group=None, bins=10, sample_weight=None):
    """Divide the score into equal-width bins and compute the statistics of each bin.

    Parameters
    ----------
    y_true: array like
        Sequence of actual sample labels; only 0/1 values are accepted.
    y_score: array like
        Sequence of predicted probabilities.
    y_group: array like or None, default None
        Sequence of data groups. Defaults to None, i.e. no groups.
    bins: int, default 10
        Number of bins.
    sample_weight: array like or None, default None
        Per-sample weights aligned with ``y_true``. When given, ``y_group`` is ignored and a weighted Gains table is
        returned instead (see Notes).

    Returns
    -------
    dist_df: pandas.DataFrame
        Dataset of per-bin statistics after equal-width binning: one row per bin (per group and bin when ``y_group`` is
        given) with the columns ``thresholds`` (upper edge of the bin), ``min_score``, ``max_score``, ``n``,
        ``proportion``, ``sum_true``, ``sum_score``, ``avg_true``, ``avg_score``, ``capture_rate`` and the cumulative
        columns ``cumsum_n``, ``cumsum_proportion``, ``cumsum_true``, ``cumsum_score``, ``cumavg_true`` and
        ``cumavg_score``. ``y_group`` is the first column when groups are given.

    Notes
    -----
    The bin edges follow powers of ten: they run from 0 (or, for negative scores, from the negative power of ten at or
    below the minimum score) to the power of ten at or above the maximum score, so probabilities are cut at 0, 0.1, ..., 1
    when ``bins=10``.

    With ``sample_weight`` the call is delegated to the weighted Gains-table implementation. The result is then a Gains
    table (columns ``MIN``, ``MAX``, ``N``, ``N_RAW``, ``AVG_BAD``, ``LIFT``, ... indexed by ``_bin_num`` and
    ``_bin_range``) built from ``bins`` equal-weight bins, not an equal-width table.
    """
    if sample_weight is not None:
        return _weighted_eval.calc_equid_dist(y_true, y_score, bins=bins, sample_weight=sample_weight)

    y_true = np.array(y_true)
    y_score = np.array(y_score)

    min_score = __calc_digit_min(np.min(y_score))
    max_score = __calc_digit_max(np.max(y_score))
    step = (max_score - min_score) / bins

    binvalues = list(np.arange(min_score, max_score, step))
    labels = binvalues[1:]
    labels.append(max_score)
    binvalues.append(np.inf)
    thresholds = pd.cut(x=y_score, bins=binvalues, right=False, labels=labels)

    if y_group is not None:
        df = pd.DataFrame({'y_true': y_true, 'y_score': y_score, 'thresholds': thresholds, 'y_group': y_group})
    else:
        df = pd.DataFrame({'y_true': y_true, 'y_score': y_score, 'thresholds': thresholds})
    dist_df = __agg(df)

    return dist_df

calc_equid_pct

calc_equid_pct(y_true, y_score, y_group=None, bins=10, ascending=True, sample_weight=None)

Divide the score into strictly equal-frequency bins and compute the statistics of each bin.

Parameters:

Name	Type	Description	Default
y_true		Sequence of actual sample labels; only 0/1 values are accepted.	required
y_score		Sequence of predicted probabilities.	required
y_group		Sequence of data groups. Defaults to None, i.e. no groups.	None
bins		Number of bins.	10
ascending		Whether to sort y_score in ascending order. Defaults to True, i.e. the first bin holds the lowest scores; with False it holds the highest scores.	True
sample_weight		Per-sample weights aligned with y_true. When given, y_group is ignored and a weighted Gains table is returned instead (see Notes).	None

Returns:

Name	Type	Description
pct_df	DataFrame	Dataset of per-bin statistics after equal-frequency binning. It has the columns of calc_equid_dist plus lift (cumulative bad rate divided by the overall bad rate) and gain (cumulative capture rate); thresholds is the cumulative percentile label of the bin (100 * k / bins for the k-th bin, truncated to an integer).

Notes

The rows are ranked by score and cut into bins bins of int(n / bins) rows each; the remainder goes to the last bin.

With sample_weight the call is delegated to the weighted Gains-table implementation. The result is then a Gains table (columns MIN, MAX, N, N_RAW, AVG_BAD, LIFT, ... indexed by _bin_num and _bin_range) built from bins equal-weight bins (bin 1 holds the lowest scores when ascending=True), not the table described above.

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def calc_equid_pct(y_true, y_score, y_group=None, bins=10, ascending=True, sample_weight=None):
    """Divide the score into strictly equal-frequency bins and compute the statistics of each bin.

    Parameters
    ----------
    y_true: array like
        Sequence of actual sample labels; only 0/1 values are accepted.
    y_score: array like
        Sequence of predicted probabilities.
    y_group: array like or None, default None
        Sequence of data groups. Defaults to None, i.e. no groups.
    bins: int, default 10
        Number of bins.
    ascending: bool, default True
        Whether to sort y_score in ascending order. Defaults to True, i.e. the first bin holds the lowest scores; with
        False it holds the highest scores.
    sample_weight: array like or None, default None
        Per-sample weights aligned with ``y_true``. When given, ``y_group`` is ignored and a weighted Gains table is
        returned instead (see Notes).

    Returns
    -------
    pct_df: pandas.DataFrame
        Dataset of per-bin statistics after equal-frequency binning. It has the columns of ``calc_equid_dist`` plus ``lift``
        (cumulative bad rate divided by the overall bad rate) and ``gain`` (cumulative capture rate); ``thresholds`` is the
        cumulative percentile label of the bin (``100 * k / bins`` for the k-th bin, truncated to an integer).

    Notes
    -----
    The rows are ranked by score and cut into ``bins`` bins of ``int(n / bins)`` rows each; the remainder goes to the last
    bin.

    With ``sample_weight`` the call is delegated to the weighted Gains-table implementation. The result is then a Gains
    table (columns ``MIN``, ``MAX``, ``N``, ``N_RAW``, ``AVG_BAD``, ``LIFT``, ... indexed by ``_bin_num`` and
    ``_bin_range``) built from ``bins`` equal-weight bins (bin 1 holds the lowest scores when ``ascending=True``), not the
    table described above.
    """
    if sample_weight is not None:
        return _weighted_eval.calc_equid_pct(
            y_true,
            y_score,
            bins=bins,
            ascending=ascending,
            sample_weight=sample_weight,
        )

    y_true = np.array(y_true)
    y_score = np.array(y_score)
    size = len(y_true)
    binsize = int(size / bins) # round down
    indices = np.argsort(y_score) if ascending else np.argsort(y_score)[::-1] 

    # Keep the historical integer percentile labels while assigning them in
    # one pass. For bin counts that do not divide 100, the legacy code
    # truncated labels during assignment to its integer array.
    thresholds = np.zeros(size, dtype=int)
    if binsize > 0:
        sorted_bin = np.minimum(np.arange(size) // binsize, bins - 1)
    else:
        sorted_bin = np.full(size, bins - 1, dtype=int)
    thresholds[indices] = 100.0 * (sorted_bin + 1) / bins
    if y_group is not None:
        df = pd.DataFrame({'y_true': y_true, 'y_score': y_score, 'y_group': y_group, 'thresholds': thresholds})
    else:
        df = pd.DataFrame({'y_true': y_true, 'y_score': y_score, 'thresholds': thresholds})
    pct_df = __agg(df)

    avg_true = np.mean(y_true)
    pct_df['lift'] = np.divide(
        pct_df['cumavg_true'].to_numpy(dtype=float),
        float(avg_true),
        out=np.full(len(pct_df), np.nan),
        where=avg_true != 0,
    )
    pct_df['gain'] = np.cumsum(pct_df['capture_rate'])

    return pct_df

calc_fixed_pct

calc_fixed_pct(y_true, y_score, y_group=None, bin_edges=None, ascending=True, sample_weight=None)

Divide the score by fixed score boundaries and compute the statistics of each bin.

Parameters:

Name	Type	Description	Default
y_true		Sequence of actual sample labels; only 0/1 values are accepted.	required
y_score		Sequence of predicted probabilities.	required
y_group		Sequence of data groups. Defaults to None, i.e. no groups.	None
bin_edges		Fixed bin edges, usually taken from a benchmark dataset. Required unless sample_weight is given. The edges are sorted, and infinite edges may also be given as the strings 'inf' and '-inf'.	None
ascending		Whether to bin y_score in ascending order. Defaults to True: the percentile labels grow with the score; with False they are reversed (the interval with the lowest scores gets the largest label).	True
sample_weight		Per-sample weights aligned with y_true. When given, bin_edges and y_group are ignored (see Notes).	None

Returns:

Name	Type	Description
pct_df	DataFrame	Dataset of per-bin statistics after fixed binning. It has the same columns as the result of calc_equid_pct, including lift and gain; thresholds is the percentile label 100 * k / n_bins of the k-th interval.

Raises:

Type	Description
ValueError	If bin_edges is None and sample_weight is None.

Notes

The intervals are closed on the right, (edge_k, edge_k+1]. A score equal to the lowest edge, or outside all the edges, falls in no bin and is left out of the table, although it still counts in the totals behind proportion and capture_rate.

With sample_weight the call is delegated to the weighted Gains-table implementation, which does not use bin_edges: the result is a Gains table (columns MIN, MAX, N, N_RAW, AVG_BAD, LIFT, ...) with 10 equal-weight bins, whatever edges are passed.

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def calc_fixed_pct(y_true, y_score, y_group=None, bin_edges=None, ascending=True, sample_weight=None):
    """Divide the score by fixed score boundaries and compute the statistics of each bin.

    Parameters
    ----------
    y_true: array like
        Sequence of actual sample labels; only 0/1 values are accepted.
    y_score: array like
        Sequence of predicted probabilities.
    y_group: array like or None, default None
        Sequence of data groups. Defaults to None, i.e. no groups.
    bin_edges: array like or None, default None
        Fixed bin edges, usually taken from a benchmark dataset. Required unless ``sample_weight`` is given. The edges are
        sorted, and infinite edges may also be given as the strings 'inf' and '-inf'.
    ascending: bool, default True
        Whether to bin y_score in ascending order. Defaults to True: the percentile labels grow with the score; with False
        they are reversed (the interval with the lowest scores gets the largest label).
    sample_weight: array like or None, default None
        Per-sample weights aligned with ``y_true``. When given, ``bin_edges`` and ``y_group`` are ignored (see Notes).

    Returns
    -------
    pct_df: pandas.DataFrame
        Dataset of per-bin statistics after fixed binning. It has the same columns as the result of ``calc_equid_pct``,
        including ``lift`` and ``gain``; ``thresholds`` is the percentile label ``100 * k / n_bins`` of the k-th interval.

    Raises
    ------
    ValueError
        If ``bin_edges`` is None and ``sample_weight`` is None.

    Notes
    -----
    The intervals are closed on the right, ``(edge_k, edge_k+1]``. A score equal to the lowest edge, or outside all the
    edges, falls in no bin and is left out of the table, although it still counts in the totals behind ``proportion`` and
    ``capture_rate``.

    With ``sample_weight`` the call is delegated to the weighted Gains-table implementation, which does not use
    ``bin_edges``: the result is a Gains table (columns ``MIN``, ``MAX``, ``N``, ``N_RAW``, ``AVG_BAD``, ``LIFT``, ...)
    with 10 equal-weight bins, whatever edges are passed.
    """
    if sample_weight is not None:
        return _weighted_eval.calc_fixed_pct(
            y_true,
            y_score,
            ascending=ascending,
            sample_weight=sample_weight,
        )

    if bin_edges is None:
        raise ValueError("bin_edges cannot be None when using fixed pct bins.")

    y_true = np.array(y_true)
    y_score = np.array(y_score)
    bin_edges = sorted([np.inf if str(x).lower() == 'inf' else -np.inf if str(x).lower() == '-inf' else x for x in bin_edges])
    n_bins = len(bin_edges) - 1
    labels = [100 * (i + 1) / n_bins for i in range(n_bins)]
    if not ascending:
        labels = labels[::-1]

    thresholds = pd.cut(
        x=y_score,
        bins=bin_edges,
        right=True,
        include_lowest=False,
        labels=labels
    )

    if y_group is not None:
        df = pd.DataFrame({'y_true': y_true, 'y_score': y_score, 'y_group': y_group, 'thresholds': thresholds})
    else:
        df = pd.DataFrame({'y_true': y_true, 'y_score': y_score, 'thresholds': thresholds})

    pct_df = __agg(df)

    avg_true = np.mean(y_true)
    # With no bad samples avg_true is 0 and lift is NaN / inf (the result is unchanged); do not warn about it
    with np.errstate(divide="ignore", invalid="ignore"):
        pct_df['lift'] = [x / avg_true for x in pct_df['cumavg_true']]
    pct_df['gain'] = np.cumsum(pct_df['capture_rate'])

    return pct_df

summarize_pct

summarize_pct(pct_df, ascending=True)

Summarize the equal-frequency bins.

Parameters:

Name	Type	Description	Default
pct_df		Dataset of per-bin statistics after equal-frequency binning.	required
ascending		Score order of the rows of pct_df: True if the first row holds the lowest scores (so the last row is the top bin), False if the first row holds the highest scores.	True

Returns:

Name	Type	Description
pct_info	dict	Dictionary of equal-frequency binning summary statistics: pct_bins (number of bins), pct_interval (100 / pct_bins) and, when pct_df has an avg_true column, pct_top_avgTrue and pct_btm_avgTrue (target rate of the top and of the bottom bin). With ascending=False and a capture_rate column, pct_top_captureRate (last row) and pct_btm_captureRate (first row) are added. For an empty pct_df the first four values are NaN.

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def summarize_pct(pct_df, ascending=True):
    """Summarize the equal-frequency bins.

    Parameters
    ----------
    pct_df: pandas.DataFrame
        Dataset of per-bin statistics after equal-frequency binning.
    ascending: bool, default True
        Score order of the rows of ``pct_df``: True if the first row holds the lowest scores (so the last row is the top
        bin), False if the first row holds the highest scores.

    Returns
    -------
    pct_info: dict
        Dictionary of equal-frequency binning summary statistics: ``pct_bins`` (number of bins), ``pct_interval``
        (``100 / pct_bins``) and, when ``pct_df`` has an ``avg_true`` column, ``pct_top_avgTrue`` and ``pct_btm_avgTrue``
        (target rate of the top and of the bottom bin). With ``ascending=False`` and a ``capture_rate`` column,
        ``pct_top_captureRate`` (last row) and ``pct_btm_captureRate`` (first row) are added. For an empty ``pct_df`` the
        first four values are NaN.
    """

    if pct_df.empty:
        return {
            'pct_bins': np.nan,
            'pct_interval': np.nan,
            'pct_top_avgTrue': np.nan,
            'pct_btm_avgTrue': np.nan,
        }

    bins = pct_df.shape[0]
    interval = 100 / pct_df.shape[0]

    pct_info = {
        'pct_bins': bins,
        'pct_interval': interval,
    }

    # Precomputed gain frames historically only needed thresholds, gain, and
    # capture_rate. Keep summarize_pct usable for those frames while exposing
    # target-rate summary keys whenever the richer percentile schema is used.
    if 'avg_true' in pct_df.columns:
        top_idx, btm_idx = (bins - 1, 0) if ascending else (0, bins - 1)
        pct_info.update({
            'pct_top_avgTrue'.format(interval): pct_df['avg_true'].iloc[top_idx],
            'pct_btm_avgTrue'.format(interval): pct_df['avg_true'].iloc[btm_idx],
        })

    if not ascending and 'capture_rate' in pct_df.columns:
        # Preserve the historical descending-summary keys for direct callers.
        pct_info.update({
            'pct_top_captureRate'.format(interval): pct_df['capture_rate'].iloc[bins-1],
            'pct_btm_captureRate'.format(interval): pct_df['capture_rate'].iloc[0],
        })

    return pct_info

plot_dist_curve

plot_dist_curve(dist_dfs, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None)

Plot the score distribution curve.

Parameters:

Name	Type	Description	Default
dist_dfs		Per-bin statistics datasets (equal-width binning, same number of bins) for one or more scores, as key-value pairs in the format {name: dist_df}.	required
square_figsize		Side length of the square figure in inches. Defaults to 8.	8
fontdicts		Dictionary of font settings for the plot, in the format of the module-level presets fontdicts['main'] and fontdicts['sub'] (keys suptitle, subtitle, axislabel and legend).	fontdicts['main']
to_show		Whether to display the figure. Defaults to True.	True
save_path		File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.	None

Notes

One entry draws the proportion of each bin as bars together with the target rate of the bin (the bars are stacked by group when the table has a y_group column); several entries are overlaid, and at most three can be drawn together (the palette has three colors, a fourth raises IndexError).

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def plot_dist_curve(dist_dfs, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None):
    """Plot the score distribution curve.

    Parameters
    ----------
    dist_dfs: dict
        Per-bin statistics datasets (equal-width binning, same number of bins) for one or more scores, as key-value pairs in the format {name: dist_df}.
    square_figsize: float
        Side length of the square figure in inches. Defaults to 8.
    fontdicts: dict, default fontdicts['main']
        Dictionary of font settings for the plot, in the format of the module-level presets ``fontdicts['main']`` and
        ``fontdicts['sub']`` (keys ``suptitle``, ``subtitle``, ``axislabel`` and ``legend``).
    to_show: bool
        Whether to display the figure. Defaults to True.
    save_path: str
        File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.

    Notes
    -----
    One entry draws the proportion of each bin as bars together with the target rate of the bin (the bars are stacked by
    group when the table has a ``y_group`` column); several entries are overlaid, and at most three can be drawn together
    (the palette has three colors, a fourth raises ``IndexError``).
    """
    plt.figure(figsize=(square_figsize, square_figsize))
    plt.suptitle('Score Distribution Curve', fontsize=fontdicts['suptitle']['size'], fontweight=fontdicts['suptitle']['weight']) #, findfont=zhfont)
    ax = plt.subplot(1,1,1)
    models = list(dist_dfs.keys())
    if len(models) == 1:
        dist_df = dist_dfs[models[0]]
        if 'y_group' in dist_df.columns:
            __plot_single_stack_dist_axes(dist_df, ax, fontdicts)
        else:
            __plot_single_dist_axes(dist_df, ax, fontdicts)
    else:
        __plot_multi_dist_axes(dist_dfs, ax, fontdicts)
    if bool(save_path):
        plt.savefig(save_path, bbox_inches='tight')
    if to_show:
        plt.show()
    plt.close()

plot_cumdist_curve

plot_cumdist_curve(dist_dfs, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None)

Plot the score cumulative distribution curve.

Parameters:

Name	Type	Description	Default
dist_dfs		Per-bin statistics datasets (equal-width binning, same number of bins) for one or more scores, as key-value pairs in the format {name: dist_df}.	required
square_figsize		Side length of the square figure in inches. Defaults to 8.	8
fontdicts		Dictionary of font settings for the plot, in the format of the module-level presets fontdicts['main'] and fontdicts['sub'] (keys suptitle, subtitle, axislabel and legend).	fontdicts['main']
to_show		Whether to display the figure. Defaults to True.	True
save_path		File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.	None

Notes

One entry draws the cumulative proportion of each bin as bars together with the cumulative target rate (the bars are stacked by group when the table has a y_group column); several entries are overlaid, and at most three can be drawn together (the palette has three colors, a fourth raises IndexError).

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def plot_cumdist_curve(dist_dfs, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None):
    """Plot the score cumulative distribution curve.

    Parameters
    ----------
    dist_dfs: dict
        Per-bin statistics datasets (equal-width binning, same number of bins) for one or more scores, as key-value pairs in the format {name: dist_df}.
    square_figsize: float
        Side length of the square figure in inches. Defaults to 8.
    fontdicts: dict, default fontdicts['main']
        Dictionary of font settings for the plot, in the format of the module-level presets ``fontdicts['main']`` and
        ``fontdicts['sub']`` (keys ``suptitle``, ``subtitle``, ``axislabel`` and ``legend``).
    to_show: bool
        Whether to display the figure. Defaults to True.
    save_path: str
        File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.

    Notes
    -----
    One entry draws the cumulative proportion of each bin as bars together with the cumulative target rate (the bars are
    stacked by group when the table has a ``y_group`` column); several entries are overlaid, and at most three can be drawn
    together (the palette has three colors, a fourth raises ``IndexError``).
    """
    plt.figure(figsize=(square_figsize, square_figsize))
    plt.suptitle('Score Cumulative Distribution Curve', fontsize=fontdicts['suptitle']['size'], fontweight=fontdicts['suptitle']['weight']) #, findfont=zhfont)
    ax = plt.subplot(1,1,1)
    models = list(dist_dfs.keys())
    if len(models) == 1:
        dist_df = dist_dfs[models[0]]
        if 'y_group' in dist_df.columns:
            __plot_single_stack_cumdist_axes(dist_df, ax, fontdicts)
        else:
            __plot_single_cumdist_axes(dist_df, ax, fontdicts)
    else:
        __plot_multi_cumdist_axes(dist_dfs, ax, fontdicts)
    if bool(save_path):
        plt.savefig(save_path, bbox_inches='tight')
    if to_show:
        plt.show()
    plt.close()

plot_pct_curve

plot_pct_curve(pct_dfs, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None)

Plot the score percentile curve.

Parameters:

Name	Type	Description	Default
pct_dfs		Per-bin statistics datasets (equal-frequency binning) for one or more scores, as key-value pairs in the format {name: pct_df}.	required
square_figsize		Side length of the square figure in inches. Defaults to 8.	8
fontdicts		Dictionary of font settings for the plot, in the format of the module-level presets fontdicts['main'] and fontdicts['sub'] (keys suptitle, subtitle, axislabel and legend).	fontdicts['main']
to_show		Whether to display the figure. Defaults to True.	True
save_path		File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.	None

Notes

One entry draws the average score and the target rate of each bin against the percentile, with the overall target rate as a reference line; several entries are overlaid (target rate only), and at most three can be drawn together (the palette has three colors, a fourth raises IndexError).

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def plot_pct_curve(pct_dfs, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None):
    """Plot the score percentile curve.

    Parameters
    ----------
    pct_dfs: dict
        Per-bin statistics datasets (equal-frequency binning) for one or more scores, as key-value pairs in the format {name: pct_df}.
    square_figsize: float
        Side length of the square figure in inches. Defaults to 8.
    fontdicts: dict, default fontdicts['main']
        Dictionary of font settings for the plot, in the format of the module-level presets ``fontdicts['main']`` and
        ``fontdicts['sub']`` (keys ``suptitle``, ``subtitle``, ``axislabel`` and ``legend``).
    to_show: bool
        Whether to display the figure. Defaults to True.
    save_path: str
        File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.

    Notes
    -----
    One entry draws the average score and the target rate of each bin against the percentile, with the overall target rate
    as a reference line; several entries are overlaid (target rate only), and at most three can be drawn together (the
    palette has three colors, a fourth raises ``IndexError``).
    """
    plt.figure(figsize=(square_figsize, square_figsize))
    plt.suptitle('Score Percentile Curve', fontsize=fontdicts['suptitle']['size'], fontweight=fontdicts['suptitle']['weight']) #, findfont=zhfont)
    ax = plt.subplot(1,1,1)

    models = list(pct_dfs.keys())
    if len(models) == 1:
        pct_df = pct_dfs[models[0]]
        __plot_single_pct_axes(pct_df, ax, fontdicts)
    else:
        __plot_multi_pct_axes(pct_dfs, ax, fontdicts)

    if to_show:
        plt.show()
    if bool(save_path):
        plt.savefig(save_path, bbox_inches='tight')
    plt.close()

plot_cumpct_curve

plot_cumpct_curve(pct_dfs, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None)

Plot the cumulative score percentile curve.

Parameters:

Name	Type	Description	Default
pct_dfs		Per-bin statistics datasets (equal-frequency binning) for one or more scores, as key-value pairs in the format {name: pct_df}.	required
square_figsize		Side length of the square figure in inches. Defaults to 8.	8
fontdicts		Dictionary of font settings for the plot, in the format of the module-level presets fontdicts['main'] and fontdicts['sub'] (keys suptitle, subtitle, axislabel and legend).	fontdicts['main']
to_show		Whether to display the figure. Defaults to True.	True
save_path		File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.	None

Notes

The curves show the cumulative target rate (and, for a single entry, the cumulative average score) against the cumulative percentile. A single entry reads a thresholds_percentile column from its table, which the result of calc_equid_pct does not have (KeyError); several entries use the thresholds column, and at most three can be drawn together (the palette has three colors, a fourth raises IndexError).

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def plot_cumpct_curve(pct_dfs, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None):
    """Plot the cumulative score percentile curve.

    Parameters
    ----------
    pct_dfs: dict
        Per-bin statistics datasets (equal-frequency binning) for one or more scores, as key-value pairs in the format {name: pct_df}.
    square_figsize: float
        Side length of the square figure in inches. Defaults to 8.
    fontdicts: dict, default fontdicts['main']
        Dictionary of font settings for the plot, in the format of the module-level presets ``fontdicts['main']`` and
        ``fontdicts['sub']`` (keys ``suptitle``, ``subtitle``, ``axislabel`` and ``legend``).
    to_show: bool
        Whether to display the figure. Defaults to True.
    save_path: str
        File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.

    Notes
    -----
    The curves show the cumulative target rate (and, for a single entry, the cumulative average score) against the
    cumulative percentile. A single entry reads a ``thresholds_percentile`` column from its table, which the result of
    ``calc_equid_pct`` does not have (``KeyError``); several entries use the ``thresholds`` column, and at most three can be
    drawn together (the palette has three colors, a fourth raises ``IndexError``).
    """
    plt.figure(figsize=(square_figsize, square_figsize))
    plt.suptitle('Score Percentile Curve', fontsize=fontdicts['suptitle']['size'], fontweight=fontdicts['suptitle']['weight']) #, findfont=zhfont)
    ax = plt.subplot(1,1,1)

    models = list(pct_dfs.keys())
    if len(models) == 1:
        pct_df = pct_dfs[models[0]]
        __plot_single_cumpct_axes(pct_df, ax, fontdicts)
    else:
        __plot_multi_cumpct_axes(pct_dfs, ax, fontdicts)

    if to_show:
        plt.show()
    if bool(save_path):
        plt.savefig(save_path, bbox_inches='tight')
    plt.close()

plot_gain_curve

plot_gain_curve(pct_dfs, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None, ascending=False)

Plot the Gain curve (cumulative capture rate of the target against the cumulative percentile).

Parameters:

Name	Type	Description	Default
pct_dfs		Per-bin statistics datasets (equal-frequency binning) for one or more scores, as key-value pairs in the format {name: pct_df}.	required
square_figsize		Side length of the square figure in inches. Defaults to 8.	8
fontdicts		Dictionary of font settings for the plot, in the format of the module-level presets fontdicts['main'] and fontdicts['sub'] (keys suptitle, subtitle, axislabel and legend).	fontdicts['main']
to_show		Whether to display the figure. Defaults to True.	True
save_path		File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.	None
ascending		Whether the Gain chart accumulates by ascending score (True) or descending score (False).	False

Notes

When a table has the columns avg_score, proportion and capture_rate, its rows are re-sorted by avg_score according to ascending and the cumulative percentile and gain are recomputed from them; otherwise the thresholds and gain columns are drawn as they are. A diagonal marks random selection. Several entries are overlaid, and at most three can be drawn together (the palette has three colors, a fourth raises IndexError).

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def plot_gain_curve(pct_dfs, square_figsize=8, fontdicts=fontdicts['main'], to_show=True, save_path=None, ascending=False):
    """Plot the Gain curve (cumulative capture rate of the target against the cumulative percentile).

    Parameters
    ----------
    pct_dfs: dict
        Per-bin statistics datasets (equal-frequency binning) for one or more scores, as key-value pairs in the format {name: pct_df}.
    square_figsize: float
        Side length of the square figure in inches. Defaults to 8.
    fontdicts: dict, default fontdicts['main']
        Dictionary of font settings for the plot, in the format of the module-level presets ``fontdicts['main']`` and
        ``fontdicts['sub']`` (keys ``suptitle``, ``subtitle``, ``axislabel`` and ``legend``).
    to_show: bool
        Whether to display the figure. Defaults to True.
    save_path: str
        File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.
    ascending: bool, default False
        Whether the Gain chart accumulates by ascending score (True) or descending score (False).

    Notes
    -----
    When a table has the columns ``avg_score``, ``proportion`` and ``capture_rate``, its rows are re-sorted by ``avg_score``
    according to ``ascending`` and the cumulative percentile and gain are recomputed from them; otherwise the ``thresholds``
    and ``gain`` columns are drawn as they are. A diagonal marks random selection. Several entries are overlaid, and at
    most three can be drawn together (the palette has three colors, a fourth raises ``IndexError``).
    """
    plt.figure(figsize=(square_figsize, square_figsize))
    plt.suptitle('Gain Curve', fontsize=fontdicts['suptitle']['size'], fontweight=fontdicts['suptitle']['weight'])  #, findfont=zhfont)
    ax = plt.subplot(1,1,1)

    models = list(pct_dfs.keys())
    if len(models) == 1:
        pct_df = pct_dfs[models[0]]
        __plot_single_gain_axes(pct_df, ax, fontdicts, ascending=ascending)
    else:
        __plot_multi_gain_axes(pct_dfs, ax, fontdicts, ascending=ascending)
    ax.legend(loc=2, fontsize=fontdicts['legend']['size'])

    if to_show:
        plt.show()
    if bool(save_path):
        plt.savefig(save_path, bbox_inches='tight')
    plt.close()

evaluate_performance

evaluate_performance(datasets, dist_bins=20, pct_bins=10, square_figsize=5, fontdicts=fontdicts['sub'], to_show=True, save_path=None, gains_table=True, equal_freq=True, pct_bin_edges=None, sample_weight=None, ascending=None)

Plot the prediction performance evaluation chart of a single model.

Parameters:

Name	Type	Description	Default
datasets		Dictionary of datasets, as key-value pairs in the format {dataname: {'y_true': y_true, 'y_score': y_score}}. A dataset may also carry a 'sample_weight' entry with its own sample weights.	required
dist_bins		Number of equal-width bins. Defaults to 20.	20
pct_bins		Number of equal-frequency bins. Defaults to 10.	10
square_figsize		Side length of the square figure in inches. Defaults to 5.	5
fontdicts		Dictionary of font settings for the plot. Defaults to fontdicts['sub'].	fontdicts['sub']
to_show		Whether to display the figure. Defaults to True.	True
save_path		File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.	None
gains_table		Whether the percentile panel and the Top/Btm target rates of the result are computed from the Gains-table binning (get_gains_table) instead of the plain equal-frequency table of calc_equid_pct. It has no visible effect on the weighted path.	True
equal_freq		Equal-frequency (True) or equal-width (False) binning of the Gains table; only used when gains_table is True.	True
pct_bin_edges		Fixed score bin edges for the percentile and Gain panels, usually taken from a benchmark dataset (see calc_fixed_pct); None uses pct_bins equal-frequency bins. On the weighted path only the number of edges matters: len(pct_bin_edges) - 1 equal-weight bins are built.	None
sample_weight		Sample weights applied to every dataset that has no 'sample_weight' entry of its own; its length must match each of those datasets.	None
ascending		An explicit value controls the score direction of the Gains table, the percentile chart, and the Gain chart uniformly; None keeps the historical behavior (percentile ascending, Gain descending).	None

Returns:

Name	Type	Description
result_df	DataFrame	Dataset summarizing the model evaluation metrics, one row per dataset with the columns index (dataset name), N (number of rows, or the sum of the weights), avgTrue, avgScore, KS, AUC, Btm{p}%_TargetRate and Top{p}%_TargetRate (target rate of the lowest- and of the highest-scored bin, with p = 100 / number of bins). If any dataset has fewer than two samples, nothing is drawn and an empty DataFrame is returned.

Notes

Each dataset gets one row of four panels: ROC, KDE, percentile and Gain. Rows whose label, score or weight is not finite are dropped. With weights, the metrics are weighted, the panel titles are marked Weighted and the percentile and Gain panels are built from the equal-weight bins of the weighted Gains table.

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def evaluate_performance(datasets, dist_bins=20, pct_bins=10, square_figsize=5, fontdicts=fontdicts['sub'], to_show=True, save_path=None, gains_table = True, equal_freq = True, pct_bin_edges = None, sample_weight=None, ascending=None):
    """Plot the prediction performance evaluation chart of a single model.

    Parameters
    ----------
    datasets: dict
        Dictionary of datasets, as key-value pairs in the format {dataname: {'y_true': y_true, 'y_score': y_score}}.
        A dataset may also carry a ``'sample_weight'`` entry with its own sample weights.
    dist_bins: int
        Number of equal-width bins. Defaults to 20.
    pct_bins: int
        Number of equal-frequency bins. Defaults to 10.
    square_figsize: float
        Side length of the square figure in inches. Defaults to 5.
    fontdicts: dict
        Dictionary of font settings for the plot. Defaults to fontdicts['sub'].
    to_show: bool
        Whether to display the figure. Defaults to True.
    save_path: str
        File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.
    gains_table: bool, default True
        Whether the percentile panel and the Top/Btm target rates of the result are computed from the Gains-table binning
        (``get_gains_table``) instead of the plain equal-frequency table of ``calc_equid_pct``. It has no visible effect
        on the weighted path.
    equal_freq: bool, default True
        Equal-frequency (True) or equal-width (False) binning of the Gains table; only used when ``gains_table`` is True.
    pct_bin_edges: array like or None, default None
        Fixed score bin edges for the percentile and Gain panels, usually taken from a benchmark dataset (see
        ``calc_fixed_pct``); None uses ``pct_bins`` equal-frequency bins. On the weighted path only the number of edges
        matters: ``len(pct_bin_edges) - 1`` equal-weight bins are built.
    sample_weight: array like or None, default None
        Sample weights applied to every dataset that has no ``'sample_weight'`` entry of its own; its length must match
        each of those datasets.
    ascending: bool or None, default None
        An explicit value controls the score direction of the Gains table, the percentile chart, and the
        Gain chart uniformly; None keeps the historical behavior (percentile ascending, Gain descending).

    Returns
    -------
    result_df: pandas.DataFrame
        Dataset summarizing the model evaluation metrics, one row per dataset with the columns ``index`` (dataset name),
        ``N`` (number of rows, or the sum of the weights), ``avgTrue``, ``avgScore``, ``KS``, ``AUC``, ``Btm{p}%_TargetRate``
        and ``Top{p}%_TargetRate`` (target rate of the lowest- and of the highest-scored bin, with ``p = 100 / number of
        bins``). If any dataset has fewer than two samples, nothing is drawn and an empty DataFrame is returned.

    Notes
    -----
    Each dataset gets one row of four panels: ROC, KDE, percentile and Gain. Rows whose label, score or weight is not finite
    are dropped. With weights, the metrics are weighted, the panel titles are marked ``Weighted`` and the percentile and
    Gain panels are built from the equal-weight bins of the weighted Gains table.
    """
    if pct_bin_edges is not None:
        pct_bin_edges = list(pct_bin_edges)

    # Check that there is enough data
    for d, data_dict in datasets.items():
        if len(data_dict['y_true']) < 2:
            # Return an empty DataFrame containing the default columns
            return pd.DataFrame()

    datas = list(datasets.keys())
    nrow = len(datas)
    ncol = 4
    width = ncol * (square_figsize+1)
    height = nrow * (square_figsize+1)
    plt.figure(figsize=(width, height))
    plt.subplots_adjust(top=1-1/height, wspace=0.2, hspace=0.2)
    title = 'Model Evaluation (Row Dataset: {0})'.format(', '.join(datas)) if nrow > 1 else 'Model Evaluation'
    plt.suptitle(title, fontsize=fontdicts['suptitle']['size'], fontweight=fontdicts['suptitle']['weight']) #, findfont=zhfont)

    result = {}
    for i in range(len(datas)):
        d = datas[i]
        y_true = datasets[d]['y_true']
        y_score = datasets[d]['y_score']
        dataset_weight = datasets[d].get('sample_weight', sample_weight)
        result.update({d: __evaluate_performance(y_true, y_score, nrow, ncol, i, dist_bins, pct_bins, fontdicts, gains_table, equal_freq, pct_bin_edges, sample_weight=dataset_weight, ascending=ascending)})

    result_df = pd.DataFrame.from_dict(result, orient='index').reset_index()

    if bool(save_path):
        plt.savefig(save_path, bbox_inches='tight')

    if to_show:
        plt.show()

    plt.close('all')

    return result_df

resturct_gains

resturct_gains(gains_table)

Convert a Gains table into the percentile-table layout used by the plotting helpers.

The Gains columns are renamed (_bin_num to thresholds, MIN to min_score, MAX to max_score, N to n, PROP to proportion, N_BAD to sum_true, AVG_BAD to avg_true, AVG_SCORE to avg_score, BAD_PCT_IN_EACH_BIN to capture_rate, N_CUM_BAD to cumsum_true, LIFT to lift and CUM_BAD_PCT to gain) and the column cumavg_true is added.

Parameters:

Name	Type	Description	Default
gains_table		Gains table as returned by get_gains_table: _bin_num as an index level or a column, and the columns MIN, MAX, N, PROP, N_BAD, AVG_BAD, AVG_SCORE, BAD_PCT_IN_EACH_BIN, N_CUM_BAD, LIFT and CUM_BAD_PCT.	required

Returns:

Type	Description
DataFrame	One row per bin with the columns thresholds (the bin number), min_score, max_score, n, proportion, sum_true, avg_true, avg_score, capture_rate, cumsum_true, lift, gain and cumavg_true (cumsum_true divided by the cumulative n).

Source code in Modeling_Tool/Eval/evaluate_model.py

def resturct_gains(gains_table):
    """Convert a Gains table into the percentile-table layout used by the plotting helpers.

    The Gains columns are renamed (``_bin_num`` to ``thresholds``, ``MIN`` to ``min_score``, ``MAX`` to ``max_score``,
    ``N`` to ``n``, ``PROP`` to ``proportion``, ``N_BAD`` to ``sum_true``, ``AVG_BAD`` to ``avg_true``, ``AVG_SCORE`` to
    ``avg_score``, ``BAD_PCT_IN_EACH_BIN`` to ``capture_rate``, ``N_CUM_BAD`` to ``cumsum_true``, ``LIFT`` to ``lift`` and
    ``CUM_BAD_PCT`` to ``gain``) and the column ``cumavg_true`` is added.

    Parameters
    ----------
    gains_table: pandas.DataFrame
        Gains table as returned by ``get_gains_table``: ``_bin_num`` as an index level or a column, and the columns ``MIN``,
        ``MAX``, ``N``, ``PROP``, ``N_BAD``, ``AVG_BAD``, ``AVG_SCORE``, ``BAD_PCT_IN_EACH_BIN``, ``N_CUM_BAD``, ``LIFT`` and
        ``CUM_BAD_PCT``.

    Returns
    -------
    pandas.DataFrame
        One row per bin with the columns ``thresholds`` (the bin number), ``min_score``, ``max_score``, ``n``,
        ``proportion``, ``sum_true``, ``avg_true``, ``avg_score``, ``capture_rate``, ``cumsum_true``, ``lift``, ``gain`` and
        ``cumavg_true`` (``cumsum_true`` divided by the cumulative ``n``).
    """

    rename_dict = {"_bin_num": "thresholds",
                  "MIN": "min_score",
                  "MAX": "max_score",
                  "N": "n",
                  "PROP": 'proportion',
                  "N_BAD": "sum_true",
                  "AVG_BAD": "avg_true", 
                  "AVG_SCORE": "avg_score",
                  "BAD_PCT_IN_EACH_BIN": 'capture_rate',
                  "N_CUM_BAD": 'cumsum_true',
                  "LIFT": 'lift',
                  "CUM_BAD_PCT": 'gain'}
    gains_table = gains_table.reset_index().rename(columns = rename_dict)[rename_dict.values()]
    gains_table['cumavg_true'] = gains_table['cumsum_true']/gains_table['n'].cumsum()

    return gains_table

evaluate_distribution

evaluate_distribution(datasets, dist_bins=10, square_figsize=5, fontdicts=fontdicts['sub'], toplot=True, save_path=None)

Plot the score distribution of a single model on multiple datasets.

Parameters:

Name	Type	Description	Default
datasets		Dictionary of datasets, as key-value pairs in the format {dataname: {'y_true': y_true, 'y_score': y_score, 'y_group': y_group}}. The 'y_group' key is required in every dataset, but its value may be None (no groups).	required
dist_bins		Number of equal-width bins. Defaults to 10.	10
square_figsize		Side length of the square figure in inches. Defaults to 5.	5
fontdicts		Dictionary of font settings for the plot. Defaults to fontdicts['sub'].	fontdicts['sub']
toplot		Whether to display the figure. Defaults to True.	True
save_path		File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.	None

Returns:

Type	Description
None	Nothing is returned: the figure (one row of two panels per dataset, the score distribution and the cumulative score distribution) is displayed and/or saved.

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def evaluate_distribution(datasets, dist_bins=10, square_figsize=5, fontdicts=fontdicts['sub'], toplot=True, save_path=None):
    """Plot the score distribution of a single model on multiple datasets.

    Parameters
    ----------
    datasets: dict
        Dictionary of datasets, as key-value pairs in the format {dataname: {'y_true': y_true, 'y_score': y_score, 'y_group': y_group}}.
        The ``'y_group'`` key is required in every dataset, but its value may be None (no groups).
    dist_bins: int
        Number of equal-width bins. Defaults to 10.
    square_figsize: float
        Side length of the square figure in inches. Defaults to 5.
    fontdicts: dict
        Dictionary of font settings for the plot. Defaults to fontdicts['sub'].
    toplot: bool
        Whether to display the figure. Defaults to True.
    save_path: str
        File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.

    Returns
    -------
    None
        Nothing is returned: the figure (one row of two panels per dataset, the score distribution and the cumulative
        score distribution) is displayed and/or saved.
    """
    datas = list(datasets.keys())
    nrow = len(datas)
    ncol = 2
    width = ncol * (square_figsize+1)
    height = nrow * (square_figsize+1)
    plt.figure(figsize=(width, height))
    plt.subplots_adjust(top=1-1/height, wspace=0.2, hspace=0.2)
    if nrow > 1:
        title = 'Distribution (Row Dataset: {0})'.format(', '.join(datas))
    else:
        title = 'Distribution'
    plt.suptitle(title, fontsize=fontdicts['suptitle']['size'], fontweight=fontdicts['suptitle']['weight']) #, findfont=zhfont)
    for i in range(len(datas)):
        d = datas[i]
        y_true = datasets[d]['y_true']
        y_score = datasets[d]['y_score']
        y_group = datasets[d]['y_group']
        __evaluate_distribution(y_true, y_score, y_group, nrow, ncol, i, dist_bins, fontdicts)
    plt.tight_layout()

    if bool(save_path):
        plt.savefig(save_path)

    if toplot:
        plt.show()

    plt.close('all')

comparison_performance

comparison_performance(datasets, pct_bins=10, square_figsize=5, fontdicts=fontdicts['sub'], to_show=True, save_path=None)

Plot the prediction performance comparison chart of multiple models.

Parameters:

Name	Type	Description	Default
datasets		Dictionary of datasets, as key-value pairs in the format {dataname: {'y_true': y_true, 'y_score_dict': {modelname: y_score}}}. y_score_dict maps each model name to its sequence of predicted probabilities.	required
pct_bins		Number of equal-frequency bins. Defaults to 10.	10
square_figsize		Side length of the square figure in inches. Defaults to 5.	5
fontdicts		Dictionary of font settings for the plot. Defaults to fontdicts['sub'].	fontdicts['sub']
to_show		Whether to display the figure. Defaults to True.	True
save_path		File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.	None

Returns:

Name	Type	Description
result_df	DataFrame	Dataset summarizing the model evaluation metrics, one row per model and dataset with the columns model, KS, AUC, Btm{p}%_TargetRate, Top{p}%_TargetRate (p = 100 / pct_bins), N, avgTrue and dataset.

Notes

Each dataset gets one row of three panels: ROC, percentile target rate and cumulative target rate, with one curve per model. At most three models can be compared (the palette has three colors, a fourth raises IndexError). The comparison is unweighted: there is no weight argument.

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def comparison_performance(datasets, pct_bins=10, square_figsize=5, fontdicts=fontdicts['sub'], to_show=True, save_path=None):
    """Plot the prediction performance comparison chart of multiple models.

    Parameters
    ----------
    datasets: dict
        Dictionary of datasets, as key-value pairs in the format {dataname: {'y_true': y_true, 'y_score_dict': {modelname: y_score}}}.
        ``y_score_dict`` maps each model name to its sequence of predicted probabilities.
    pct_bins: int
        Number of equal-frequency bins. Defaults to 10.
    square_figsize: float
        Side length of the square figure in inches. Defaults to 5.
    fontdicts: dict
        Dictionary of font settings for the plot. Defaults to fontdicts['sub'].
    to_show: bool
        Whether to display the figure. Defaults to True.
    save_path: str
        File path to save the resulting figure. Defaults to None, i.e. the figure is not saved.

    Returns
    -------
    result_df: pandas.DataFrame
        Dataset summarizing the model evaluation metrics, one row per model and dataset with the columns ``model``, ``KS``,
        ``AUC``, ``Btm{p}%_TargetRate``, ``Top{p}%_TargetRate`` (``p = 100 / pct_bins``), ``N``, ``avgTrue`` and ``dataset``.

    Notes
    -----
    Each dataset gets one row of three panels: ROC, percentile target rate and cumulative target rate, with one curve per
    model. At most three models can be compared (the palette has three colors, a fourth raises ``IndexError``). The
    comparison is unweighted: there is no weight argument.
    """
    datas = list(datasets.keys())
    models = list(datasets[datas[0]]['y_score_dict'].keys())
    nrow = len(datas)
    ncol = 3
    width = ncol * (square_figsize+1)
    height = nrow * (square_figsize+1)
    plt.figure(figsize=(width, height))
    plt.subplots_adjust(top=1-1/height, wspace=0.2, hspace=0.2)
    title = title = '{0} Comparison (Row Dataset: {1})'.format(' vs '.join(models), ', '.join(datas)) if nrow > 1 else '{0} Comparison'.format(' vs '.join(models))
    plt.suptitle(title, fontsize=fontdicts['suptitle']['size'], fontweight=fontdicts['suptitle']['weight']) #, findfont=zhfont)

    result_dfs = []
    for i in range(len(datas)):
        d = datas[i]
        y_true = datasets[d]['y_true']
        y_score_dict = datasets[d]['y_score_dict']
        _result = __comparison_performance(y_true, y_score_dict, nrow, ncol, i, pct_bins, fontdicts)
        _result.loc[:, 'dataset'] = d
        result_dfs.append(_result)

    result_df = pd.concat(result_dfs)

    if bool(save_path):
        plt.savefig(save_path, bbox_inches='tight')

    if to_show:
        plt.show()

    plt.close('all')

    return result_df

calc_lift_apt

calc_lift_apt(y_true, y_score, start, stop, step, score_ascending=True, sample_weight=None)

Compute the Lift table for a given range of Lift values.

Parameters:

Name	Type	Description	Default
y_true		Sequence of actual sample labels; only 0/1 values are accepted.	required
y_score		Sequence of predicted probabilities.	required
start		Start value of the target Lift range (see Notes).	required
stop		Stop value of the target Lift range (see Notes).	required
step		Step size between two target Lift values.	required
score_ascending		Whether y_score is ascending, i.e. the larger the value, the more likely y_true=1. Defaults to True.	True
sample_weight		Per-sample weights aligned with y_true. Integer-valued weights replicate the rows (a weight of 0 drops the row) and the table described under Returns is still produced; any other (fractional) weights switch to the weighted implementation, which returns a one-dimensional array instead.	None

Returns:

Name	Type	Description
lift_df	DataFrame	Lift table with one row per reachable target Lift and the columns lift (target), lift_actual (achieved), lower_limit or upper_limit (score cut-off: the rows scoring at least lower_limit, or below upper_limit, form the group), cumsum_n, cumsum_proportion, cumsum_true and cumavg_true (size, share, bad count and bad rate of that group). With fractional sample_weight a one-dimensional numpy.ndarray is returned instead: for each target Lift of np.arange(start, stop + step / 2, step), the closest Lift available in the weighted 100-bin Gains table.

Notes

The Lift range always has 1 at one end: with start < 1 it runs from start up to 1 and stop is ignored; with start >= 1 it runs from 1 up to stop and start is ignored (so the table normally also contains the row for Lift 1). The weighted implementation uses start and stop as given and ignores score_ascending.

Source code in Modeling_Tool/Eval/evaluate_model.py

@timeit_decorator
def calc_lift_apt(y_true, y_score, start, stop, step, score_ascending=True, sample_weight=None):
    """Compute the Lift table for a given range of Lift values.

    Parameters
    ----------
    y_true: array like
        Sequence of actual sample labels; only 0/1 values are accepted.
    y_score: array like
        Sequence of predicted probabilities.
    start: numerical
        Start value of the target Lift range (see Notes).
    stop: numerical
        Stop value of the target Lift range (see Notes).
    step: numerical
        Step size between two target Lift values.
    score_ascending: bool, default True
        Whether y_score is ascending, i.e. the larger the value, the more likely y_true=1. Defaults to True.
    sample_weight: array like or None, default None
        Per-sample weights aligned with ``y_true``. Integer-valued weights replicate the rows (a weight of 0 drops the
        row) and the table described under Returns is still produced; any other (fractional) weights switch to the
        weighted implementation, which returns a one-dimensional array instead.

    Returns
    -------
    lift_df: pandas.DataFrame
        Lift table with one row per reachable target Lift and the columns ``lift`` (target), ``lift_actual`` (achieved),
        ``lower_limit`` or ``upper_limit`` (score cut-off: the rows scoring at least ``lower_limit``, or below
        ``upper_limit``, form the group), ``cumsum_n``, ``cumsum_proportion``, ``cumsum_true`` and ``cumavg_true`` (size,
        share, bad count and bad rate of that group). With fractional ``sample_weight`` a one-dimensional numpy.ndarray is
        returned instead: for each target Lift of ``np.arange(start, stop + step / 2, step)``, the closest Lift available
        in the weighted 100-bin Gains table.

    Notes
    -----
    The Lift range always has 1 at one end: with ``start < 1`` it runs from ``start`` up to 1 and ``stop`` is ignored;
    with ``start >= 1`` it runs from 1 up to ``stop`` and ``start`` is ignored (so the table normally also contains the row
    for Lift 1). The weighted implementation uses ``start`` and ``stop`` as given and ignores ``score_ascending``.
    """
    if sample_weight is not None:
        weight = np.asarray(sample_weight, dtype=float)
        int_weight = np.rint(weight).astype(int)
        if np.allclose(weight, int_weight) and np.all(int_weight >= 0):
            repeat_idx = np.repeat(np.arange(len(int_weight)), int_weight)
            y_true = np.asarray(y_true)[repeat_idx]
            y_score = np.asarray(y_score)[repeat_idx]
        else:
            return _weighted_eval.calc_lift_apt(
                y_true=y_true,
                y_score=y_score,
                start=start,
                stop=stop,
                step=step,
                sample_weight=sample_weight,
            )

    # Compute the initial number of equal-frequency bins: the larger of 200 and the length of the Lift table
    init_bins = np.max([200, int((stop - start + step) / step)])

    # Determine the direction of lift (lift_ascending) from the start and stop values
    # If ascending, stop=1; if descending, start=1
    if start < 1:
        stop = 1
    elif start >= 1:
        start = 1
    lift_ascending = start < 1
    lift_df = pd.DataFrame({'lift': np.arange(start, stop + step, step)})

    # Determine the order of the score bins from the relation between score and Y=1 (score_ascending) and the direction of lift (lift_ascending)
    # Ascending if they agree, descending if they differ
    ascending = score_ascending ==  lift_ascending
    equid_df = calc_equid_pct(y_true=y_true, y_score=y_score, y_group=None, bins=init_bins, ascending=ascending)

    # Adjust the lower and upper bin limits according to the score order, following the rule that the upper limit is exclusive
    if ascending:
        equid_df['lower_limit'] = [-np.inf, ] + list(equid_df['min_score'][1:])
        equid_df['upper_limit'] = list(equid_df['min_score'][1:]) + [np.inf, ]
    else:
        equid_df['lower_limit'] = list(equid_df['min_score'][:-1]) + [-np.inf, ]
        equid_df['upper_limit'] = [np.inf, ] + list(equid_df['min_score'][:-1])
    lift_df = pd.merge(lift_df.assign(key=1), equid_df.assign(key=1), how='left', on='key', suffixes=('', '_actual')).drop(columns=['key'])

    # Solve for the cut points according to the direction of lift (lift_ascending)
    # If ascending, take the largest value not above the lift; if descending, take the smallest value not below the lift
    if lift_ascending:
        cond = lift_df['lift'] >= lift_df['lift_actual'] 
    else:
        cond = lift_df['lift'] <= lift_df['lift_actual']
    lift_df = lift_df.loc[cond, ].reset_index(drop=True)
    lift_df = lift_df.sort_values(by=['lift', 'thresholds'], ascending=[lift_ascending, False])
    lift_df = lift_df.groupby(['lift']).first().reset_index()
    lift_df = lift_df.sort_values(by=['lift'], ascending=lift_ascending)

    # Depending on the score order, take the upper bin limit when ascending and the lower bin limit when descending
    if ascending:
        cols = ['lift', 'lift_actual', 'upper_limit', 'cumsum_n', 'cumsum_proportion', 'cumsum_true', 'cumavg_true']
    else:
        cols = ['lift', 'lift_actual', 'lower_limit', 'cumsum_n', 'cumsum_proportion', 'cumsum_true', 'cumavg_true']
    lift_df = lift_df[cols]

    return lift_df

Weighted Implementations: weighted_eval_utils

Internal helpers behind the weighted path. The public functions above delegate to this module when they receive a weight argument. They are not exported and their signatures can change without notice.

weighted_eval_utils

Native weighted evaluation helpers.

Shared implementation for public Eval APIs when sample_weight or weight_col is supplied. Unweighted callers keep using the historical implementations in evaluate_model.py and Model_Eval_Tool.py.

GainsTableCalculator

Gains-table calculator bound to one dataset.

Stores the arguments of get_gains_table and computes the table with calculate.

Parameters:

Name	Type	Description	Default
data	DataFrame or None	Dataset that holds the target, the score and, when given, the weight column. It must be set before calculate is called.	None
dep	str or None	Name of the target column (binary, 1 for bad).	None
score	str or None	Name of the score column.	None
nbins	int	Number of bins.	10
weight_col	str or None	Name of the sample weight column in data; None gives every row the weight 1.	None
weighted_binning	bool or None	Accepted for compatibility but without any effect: the bins are always equal-weight bins.	None
**kwargs		Extra keyword arguments of get_gains_table (for example ascending or spec_values), stored and passed on by calculate. Nothing is validated before calculate is called.	{}

Attributes:

Name	Type	Description
data, dep, score, nbins, weight_col, weighted_binning	object	The constructor arguments, as given.
kwargs	dict	The extra keyword arguments.

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

class GainsTableCalculator:
    """Gains-table calculator bound to one dataset.

    Stores the arguments of ``get_gains_table`` and computes the table with ``calculate``.

    Parameters
    ----------
    data : pandas.DataFrame or None, default None
        Dataset that holds the target, the score and, when given, the weight column. It must be set before ``calculate`` is
        called.
    dep : str or None, default None
        Name of the target column (binary, 1 for bad).
    score : str or None, default None
        Name of the score column.
    nbins : int, default 10
        Number of bins.
    weight_col : str or None, default None
        Name of the sample weight column in ``data``; None gives every row the weight 1.
    weighted_binning : bool or None, default None
        Accepted for compatibility but without any effect: the bins are always equal-weight bins.
    **kwargs
        Extra keyword arguments of ``get_gains_table`` (for example ``ascending`` or ``spec_values``), stored and passed on
        by ``calculate``. Nothing is validated before ``calculate`` is called.

    Attributes
    ----------
    data, dep, score, nbins, weight_col, weighted_binning : object
        The constructor arguments, as given.
    kwargs : dict
        The extra keyword arguments.
    """
    def __init__(self, data=None, dep=None, score=None, nbins=10, weight_col=None, weighted_binning=None, **kwargs):
        self.data = data
        self.dep = dep
        self.score = score
        self.nbins = nbins
        self.weight_col = weight_col
        self.weighted_binning = weighted_binning
        self.kwargs = kwargs

    def calculate(self, weight_col=None, **kwargs):
        """Compute the Gains table of the stored dataset with ``get_gains_table``.

        Parameters
        ----------
        weight_col : str or None, default None
            Weight column for this call; None keeps the ``weight_col`` given to the constructor.
        **kwargs
            Extra keyword arguments of ``get_gains_table`` for this call; they take precedence over the ones given to the
            constructor. ``nbins`` and ``weighted_binning`` must not be passed (``TypeError``: they are already set).

        Returns
        -------
        pandas.DataFrame
            The Gains table (see ``get_gains_table`` for the columns).

        Raises
        ------
        KeyError
            If the target, score or weight column is not a column of ``data``.
        ValueError
            If the weights are invalid (see ``get_gains_table``).
        """
        return get_gains_table(
            self.data,
            self.dep,
            self.score,
            nbins=self.nbins,
            weight_col=self.weight_col if weight_col is None else weight_col,
            weighted_binning=self.weighted_binning,
            **{**self.kwargs, **kwargs},
        )

calculate

calculate(weight_col=None, **kwargs)

Compute the Gains table of the stored dataset with get_gains_table.

Parameters:

Name	Type	Description	Default
weight_col	str or None	Weight column for this call; None keeps the weight_col given to the constructor.	None
**kwargs		Extra keyword arguments of get_gains_table for this call; they take precedence over the ones given to the constructor. nbins and weighted_binning must not be passed (TypeError: they are already set).	{}

Returns:

Type	Description
DataFrame	The Gains table (see get_gains_table for the columns).

Raises:

Type	Description
KeyError	If the target, score or weight column is not a column of data.
ValueError	If the weights are invalid (see get_gains_table).

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def calculate(self, weight_col=None, **kwargs):
    """Compute the Gains table of the stored dataset with ``get_gains_table``.

    Parameters
    ----------
    weight_col : str or None, default None
        Weight column for this call; None keeps the ``weight_col`` given to the constructor.
    **kwargs
        Extra keyword arguments of ``get_gains_table`` for this call; they take precedence over the ones given to the
        constructor. ``nbins`` and ``weighted_binning`` must not be passed (``TypeError``: they are already set).

    Returns
    -------
    pandas.DataFrame
        The Gains table (see ``get_gains_table`` for the columns).

    Raises
    ------
    KeyError
        If the target, score or weight column is not a column of ``data``.
    ValueError
        If the weights are invalid (see ``get_gains_table``).
    """
    return get_gains_table(
        self.data,
        self.dep,
        self.score,
        nbins=self.nbins,
        weight_col=self.weight_col if weight_col is None else weight_col,
        weighted_binning=self.weighted_binning,
        **{**self.kwargs, **kwargs},
    )

PerformanceEvaluator

Collect named datasets and summarize their performance with dataset_summary.

Parameters:

Name	Type	Description	Default
tgt_name	str or None	Name of the target column (binary, 1 for bad) in every dataset.	None
scr_name	str or None	Name of the score column in every dataset.	None
weight_col	str or None	Default sample weight column, used for the datasets added without one.	None
nbins	int	Number of bins of the Gains table behind LIFT and IV.	10
**kwargs		Stored in self.kwargs but not used.	{}

Attributes:

Name	Type	Description
tgt_name, scr_name, weight_col, nbins	object	The constructor arguments, as given.
kwargs	dict	The extra keyword arguments.
datasets	OrderedDict	Added datasets in insertion order: name mapped to the tuple (data, weight_col).

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

class PerformanceEvaluator:
    """Collect named datasets and summarize their performance with ``dataset_summary``.

    Parameters
    ----------
    tgt_name : str or None, default None
        Name of the target column (binary, 1 for bad) in every dataset.
    scr_name : str or None, default None
        Name of the score column in every dataset.
    weight_col : str or None, default None
        Default sample weight column, used for the datasets added without one.
    nbins : int, default 10
        Number of bins of the Gains table behind ``LIFT`` and ``IV``.
    **kwargs
        Stored in ``self.kwargs`` but not used.

    Attributes
    ----------
    tgt_name, scr_name, weight_col, nbins : object
        The constructor arguments, as given.
    kwargs : dict
        The extra keyword arguments.
    datasets : collections.OrderedDict
        Added datasets in insertion order: name mapped to the tuple ``(data, weight_col)``.
    """
    def __init__(self, tgt_name=None, scr_name=None, weight_col=None, nbins=10, **kwargs):
        self.tgt_name = tgt_name
        self.scr_name = scr_name
        self.weight_col = weight_col
        self.nbins = nbins
        self.kwargs = kwargs
        self.datasets = OrderedDict()

    def add_dataset(self, name, data, weight_col=None, **kwargs):
        """Register a dataset to be summarized by ``evaluate``.

        Parameters
        ----------
        name : str
            Label of the dataset. Adding a name that already exists replaces that dataset and keeps its position.
        data : pandas.DataFrame
            Dataset that holds the target and score columns (and the weight column, when used).
        weight_col : str or None, default None
            Sample weight column of this dataset; None falls back to the ``weight_col`` of ``evaluate`` and then to the
            one of the constructor.
        **kwargs
            Accepted and ignored.

        Returns
        -------
        PerformanceEvaluator
            The evaluator itself, so that calls can be chained.
        """
        self.datasets[name] = (data, weight_col)
        return self

    def evaluate(self, weight_col=None, to_show=False, display=False, **kwargs):
        """Summarize every added dataset.

        Parameters
        ----------
        weight_col : str or None, default None
            Weight column for the datasets added without their own; None falls back to the ``weight_col`` of the
            constructor (and to equal weights when that is None too).
        to_show : bool, default False
            Accepted for API compatibility; it has no effect.
        display : bool, default False
            Accepted for API compatibility; it has no effect.
        **kwargs
            Accepted and ignored.

        Returns
        -------
        pandas.DataFrame
            One row per added dataset, in insertion order, with the entries of ``dataset_summary`` as columns (``index``,
            ``dataset``, ``DATASET``, ``AUC``, ``KS``, ``LIFT``, ``IV``, ``N``, ``N_RAW``, ``avgTrue``, ``avgScore``). A
            DataFrame without rows or columns when no dataset was added.

        Raises
        ------
        KeyError
            If a target, score or weight column is missing from a dataset.
        ValueError
            If the weights of a dataset are invalid (see ``dataset_summary``).

        Notes
        -----
        The Gains table behind ``LIFT`` and ``IV`` uses ``nbins`` of the constructor, ``ascending=False`` and no special
        score values.
        """
        rows = []
        for name, (data, ds_weight_col) in self.datasets.items():
            wc = ds_weight_col or weight_col or self.weight_col
            rows.append(dataset_summary(name, data, self.tgt_name, self.scr_name, weight_col=wc, nbins=self.nbins))
        return pd.DataFrame(rows)

add_dataset

add_dataset(name, data, weight_col=None, **kwargs)

Register a dataset to be summarized by evaluate.

Parameters:

Name	Type	Description	Default
name	str	Label of the dataset. Adding a name that already exists replaces that dataset and keeps its position.	required
data	DataFrame	Dataset that holds the target and score columns (and the weight column, when used).	required
weight_col	str or None	Sample weight column of this dataset; None falls back to the weight_col of evaluate and then to the one of the constructor.	None
**kwargs		Accepted and ignored.	{}

Returns:

Type	Description
PerformanceEvaluator	The evaluator itself, so that calls can be chained.

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def add_dataset(self, name, data, weight_col=None, **kwargs):
    """Register a dataset to be summarized by ``evaluate``.

    Parameters
    ----------
    name : str
        Label of the dataset. Adding a name that already exists replaces that dataset and keeps its position.
    data : pandas.DataFrame
        Dataset that holds the target and score columns (and the weight column, when used).
    weight_col : str or None, default None
        Sample weight column of this dataset; None falls back to the ``weight_col`` of ``evaluate`` and then to the
        one of the constructor.
    **kwargs
        Accepted and ignored.

    Returns
    -------
    PerformanceEvaluator
        The evaluator itself, so that calls can be chained.
    """
    self.datasets[name] = (data, weight_col)
    return self

evaluate

evaluate(weight_col=None, to_show=False, display=False, **kwargs)

Summarize every added dataset.

Parameters:

Name	Type	Description	Default
weight_col	str or None	Weight column for the datasets added without their own; None falls back to the weight_col of the constructor (and to equal weights when that is None too).	None
to_show	bool	Accepted for API compatibility; it has no effect.	False
display	bool	Accepted for API compatibility; it has no effect.	False
**kwargs		Accepted and ignored.	{}

Returns:

Type	Description
DataFrame	One row per added dataset, in insertion order, with the entries of dataset_summary as columns (index, dataset, DATASET, AUC, KS, LIFT, IV, N, N_RAW, avgTrue, avgScore). A DataFrame without rows or columns when no dataset was added.

Raises:

Type	Description
KeyError	If a target, score or weight column is missing from a dataset.
ValueError	If the weights of a dataset are invalid (see dataset_summary).

Notes

The Gains table behind LIFT and IV uses nbins of the constructor, ascending=False and no special score values.

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def evaluate(self, weight_col=None, to_show=False, display=False, **kwargs):
    """Summarize every added dataset.

    Parameters
    ----------
    weight_col : str or None, default None
        Weight column for the datasets added without their own; None falls back to the ``weight_col`` of the
        constructor (and to equal weights when that is None too).
    to_show : bool, default False
        Accepted for API compatibility; it has no effect.
    display : bool, default False
        Accepted for API compatibility; it has no effect.
    **kwargs
        Accepted and ignored.

    Returns
    -------
    pandas.DataFrame
        One row per added dataset, in insertion order, with the entries of ``dataset_summary`` as columns (``index``,
        ``dataset``, ``DATASET``, ``AUC``, ``KS``, ``LIFT``, ``IV``, ``N``, ``N_RAW``, ``avgTrue``, ``avgScore``). A
        DataFrame without rows or columns when no dataset was added.

    Raises
    ------
    KeyError
        If a target, score or weight column is missing from a dataset.
    ValueError
        If the weights of a dataset are invalid (see ``dataset_summary``).

    Notes
    -----
    The Gains table behind ``LIFT`` and ``IV`` uses ``nbins`` of the constructor, ``ascending=False`` and no special
    score values.
    """
    rows = []
    for name, (data, ds_weight_col) in self.datasets.items():
        wc = ds_weight_col or weight_col or self.weight_col
        rows.append(dataset_summary(name, data, self.tgt_name, self.scr_name, weight_col=wc, nbins=self.nbins))
    return pd.DataFrame(rows)

resolve_weights

resolve_weights(data=None, weight_col=None, sample_weight=None, expected_len=None, wgt=None, wgt_col=None)

Resolve sample weights from a DataFrame column or an array.

Thin wrapper around Modeling_Tool.Core.sample_weight_utils.resolve_sample_weight that first maps the aliases wgt_col to weight_col and wgt to sample_weight.

Parameters:

Name	Type	Description	Default
data	DataFrame or None	Frame that holds the weight column; required when weight_col (or wgt_col) is given.	None
weight_col	str or None	Name of the weight column in data.	None
sample_weight	array - like or None	Weights given directly, one per row. It cannot be combined with weight_col.	None
expected_len	int or None	Required length of the weight vector; None skips the length check.	None
wgt	array - like or None	Alias of sample_weight, used only when sample_weight is None.	None
wgt_col	str or None	Alias of weight_col, used only when weight_col is None.	None

Returns:

Type	Description
ndarray or None	The validated 1-D float weight vector, or None when neither a column nor an array was given.

Raises:

Type	Description
ValueError	If both a column and an array are given, if a column is given without data, or if the weights are not 1-dimensional, have a length other than expected_len, contain NaN, infinity or a negative value, or sum to zero or less.
KeyError	If the weight column is not in data.

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def resolve_weights(data=None, weight_col=None, sample_weight=None, expected_len=None, wgt=None, wgt_col=None):
    """Resolve sample weights from a DataFrame column or an array.

    Thin wrapper around ``Modeling_Tool.Core.sample_weight_utils.resolve_sample_weight`` that first maps the aliases
    ``wgt_col`` to ``weight_col`` and ``wgt`` to ``sample_weight``.

    Parameters
    ----------
    data : pandas.DataFrame or None, default None
        Frame that holds the weight column; required when ``weight_col`` (or ``wgt_col``) is given.
    weight_col : str or None, default None
        Name of the weight column in ``data``.
    sample_weight : array-like or None, default None
        Weights given directly, one per row. It cannot be combined with ``weight_col``.
    expected_len : int or None, default None
        Required length of the weight vector; None skips the length check.
    wgt : array-like or None, default None
        Alias of ``sample_weight``, used only when ``sample_weight`` is None.
    wgt_col : str or None, default None
        Alias of ``weight_col``, used only when ``weight_col`` is None.

    Returns
    -------
    numpy.ndarray or None
        The validated 1-D float weight vector, or None when neither a column nor an array was given.

    Raises
    ------
    ValueError
        If both a column and an array are given, if a column is given without ``data``, or if the weights are not
        1-dimensional, have a length other than ``expected_len``, contain NaN, infinity or a negative value, or sum to
        zero or less.
    KeyError
        If the weight column is not in ``data``.
    """
    if weight_col is None:
        weight_col = wgt_col
    if sample_weight is None:
        sample_weight = wgt
    return resolve_sample_weight(
        data=data,
        weight_col=weight_col,
        sample_weight=sample_weight,
        expected_len=expected_len,
    )

safe_weighted_average

safe_weighted_average(values, weights=None)

Weighted mean that returns NaN instead of failing on empty or zero-weight input.

Parameters:

Name	Type	Description	Default
values	array - like	Values to average, converted with numpy.asarray(values, dtype=float). NaN values are not skipped: they make the result NaN.	required
weights	array - like or None	Weights aligned with values. None gives the plain unweighted mean.	None

Returns:

Type	Description
float	The weighted mean. NaN when weights is None and values is empty, or when the weights sum to zero.

Notes

The weights are not validated: negative weights are used as given, and a weights array whose length differs from values makes numpy.average raise TypeError.

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def safe_weighted_average(values, weights=None):
    """Weighted mean that returns NaN instead of failing on empty or zero-weight input.

    Parameters
    ----------
    values : array-like
        Values to average, converted with ``numpy.asarray(values, dtype=float)``. NaN values are not skipped: they make
        the result NaN.
    weights : array-like or None, default None
        Weights aligned with ``values``. None gives the plain unweighted mean.

    Returns
    -------
    float
        The weighted mean. NaN when ``weights`` is None and ``values`` is empty, or when the weights sum to zero.

    Notes
    -----
    The weights are not validated: negative weights are used as given, and a ``weights`` array whose length differs from
    ``values`` makes ``numpy.average`` raise ``TypeError``.
    """
    values = np.asarray(values, dtype=float)
    if weights is None:
        return float(np.mean(values)) if len(values) else np.nan
    weights = np.asarray(weights, dtype=float)
    total = float(np.sum(weights))
    if total == 0:
        return np.nan
    return float(np.average(values, weights=weights))

safe_auc

safe_auc(y_true, y_score, sample_weight=None)

Area under the ROC curve, or NaN when it cannot be computed.

Parameters:

Name	Type	Description	Default
y_true	array - like	Actual binary labels (0/1).	required
y_score	array - like	Predicted scores or probabilities.	required
sample_weight	array - like or None	Per-sample weights passed to sklearn.metrics.roc_auc_score; None means equal weights.	None

Returns:

Type	Description
float	The (weighted) AUC. NaN when scikit-learn raises ValueError or ZeroDivisionError, for example when y_true holds a single class or y_score contains NaN. The error is recorded with smf_logger.record_and_continue (stage weighted_eval.safe_auc) and not re-raised.

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def safe_auc(y_true, y_score, sample_weight=None):
    """Area under the ROC curve, or NaN when it cannot be computed.

    Parameters
    ----------
    y_true : array-like
        Actual binary labels (0/1).
    y_score : array-like
        Predicted scores or probabilities.
    sample_weight : array-like or None, default None
        Per-sample weights passed to ``sklearn.metrics.roc_auc_score``; None means equal weights.

    Returns
    -------
    float
        The (weighted) AUC. NaN when scikit-learn raises ``ValueError`` or ``ZeroDivisionError``, for example when
        ``y_true`` holds a single class or ``y_score`` contains NaN. The error is recorded with
        ``smf_logger.record_and_continue`` (stage ``weighted_eval.safe_auc``) and not re-raised.
    """
    try:
        return float(roc_auc_score(y_true, y_score, sample_weight=sample_weight))
    except (ValueError, ZeroDivisionError) as exc:
        smf_logger.record_and_continue("auc", exc, stage="weighted_eval.safe_auc")
        return np.nan

calc_roc

calc_roc(y_true, y_score, sample_weight=None)

Compute the statistics of the ROC curve, optionally with sample weights.

Based on sklearn.metrics.roc_curve. The rows where y_true, y_score or sample_weight is NaN or infinite are dropped first.

Parameters:

Name	Type	Description	Default
y_true	array - like	Actual binary labels (0/1), converted to float.	required
y_score	array - like	Predicted scores or probabilities, converted to float.	required
sample_weight	array - like or None	Per-sample weights aligned with y_true, used for the curve and for thresholds_percentile. They are not validated by this function. None means equal weights.	None

Returns:

Type	Description
DataFrame	One row per ROC threshold with the columns fpr, tpr, thresholds, thresholds_percentile (percentage of the rows, or of the total weight, whose score is at or below the threshold), FPR and TPR (copies of fpr and tpr) and KS (abs(tpr - fpr)). An empty DataFrame with the same columns when no row is left after the filtering.

Raises:

Type	Description
ValueError	If y_true is not binary (raised by scikit-learn).

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def calc_roc(y_true, y_score, sample_weight=None):
    """Compute the statistics of the ROC curve, optionally with sample weights.

    Based on ``sklearn.metrics.roc_curve``. The rows where ``y_true``, ``y_score`` or ``sample_weight`` is NaN or infinite
    are dropped first.

    Parameters
    ----------
    y_true : array-like
        Actual binary labels (0/1), converted to float.
    y_score : array-like
        Predicted scores or probabilities, converted to float.
    sample_weight : array-like or None, default None
        Per-sample weights aligned with ``y_true``, used for the curve and for ``thresholds_percentile``. They are not
        validated by this function. None means equal weights.

    Returns
    -------
    pandas.DataFrame
        One row per ROC threshold with the columns ``fpr``, ``tpr``, ``thresholds``, ``thresholds_percentile`` (percentage of
        the rows, or of the total weight, whose score is at or below the threshold), ``FPR`` and ``TPR`` (copies of ``fpr``
        and ``tpr``) and ``KS`` (``abs(tpr - fpr)``). An empty DataFrame with the same columns when no row is left after
        the filtering.

    Raises
    ------
    ValueError
        If ``y_true`` is not binary (raised by scikit-learn).
    """
    y_true = np.asarray(y_true, dtype=float)
    y_score = np.asarray(y_score, dtype=float)
    weight = None if sample_weight is None else np.asarray(sample_weight, dtype=float)

    mask = np.isfinite(y_true) & np.isfinite(y_score)
    if weight is not None:
        mask = mask & np.isfinite(weight)
        weight = weight[mask]
    y_true = y_true[mask]
    y_score = y_score[mask]

    if len(y_true) == 0:
        return pd.DataFrame(columns=["fpr", "tpr", "thresholds", "thresholds_percentile", "FPR", "TPR", "KS"])

    fpr, tpr, thresholds = roc_curve(y_true, y_score, sample_weight=weight)
    out = pd.DataFrame({"fpr": fpr, "tpr": tpr, "thresholds": thresholds})
    if weight is None:
        out["thresholds_percentile"] = [100 * np.mean(y_score <= x) for x in thresholds]
    else:
        total_weight = float(weight.sum()) or 1.0
        out["thresholds_percentile"] = [
            100 * float(weight[y_score <= x].sum()) / total_weight for x in thresholds
        ]
    out["FPR"] = out["fpr"]
    out["TPR"] = out["tpr"]
    out["KS"] = (out["tpr"] - out["fpr"]).abs()
    return out

calc_pr

calc_pr(y_true, y_score, sample_weight=None)

Compute the statistics of the P-R curve, optionally with sample weights.

Based on sklearn.metrics.precision_recall_curve.

Parameters:

Name	Type	Description	Default
y_true	array - like	Actual binary labels (0/1).	required
y_score	array - like	Predicted scores or probabilities.	required
sample_weight	array - like or None	Per-sample weights aligned with y_true; None means equal weights. They are not validated by this function.	None

Returns:

Type	Description
DataFrame	One row per threshold with the columns precision, recall and thresholds. The last row (precision 1, recall 0) has no threshold (NaN). Unlike the unweighted evaluate_model.calc_pr there is no thresholds_percentile column.

Notes

NaN or infinite values are not filtered out here, so scikit-learn raises ValueError when y_true or y_score contains them.

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def calc_pr(y_true, y_score, sample_weight=None):
    """Compute the statistics of the P-R curve, optionally with sample weights.

    Based on ``sklearn.metrics.precision_recall_curve``.

    Parameters
    ----------
    y_true : array-like
        Actual binary labels (0/1).
    y_score : array-like
        Predicted scores or probabilities.
    sample_weight : array-like or None, default None
        Per-sample weights aligned with ``y_true``; None means equal weights. They are not validated by this function.

    Returns
    -------
    pandas.DataFrame
        One row per threshold with the columns ``precision``, ``recall`` and ``thresholds``. The last row (precision 1,
        recall 0) has no threshold (NaN). Unlike the unweighted ``evaluate_model.calc_pr`` there is no
        ``thresholds_percentile`` column.

    Notes
    -----
    NaN or infinite values are not filtered out here, so scikit-learn raises ``ValueError`` when ``y_true`` or ``y_score``
    contains them.
    """
    precision, recall, thresholds = precision_recall_curve(
        y_true,
        y_score,
        sample_weight=sample_weight,
    )
    thresholds = np.r_[thresholds, np.nan]
    return pd.DataFrame(
        {
            "precision": precision,
            "recall": recall,
            "thresholds": thresholds,
        }
    )

rank_bins

rank_bins(score, weight, nbins, ascending=False)

Weighted equal-frequency rank bins.

ascending=False (default, legacy) sorts scores descending — bin 1 is the highest-score bucket. ascending=True sorts scores ascending so bin 1 is the lowest-score bucket, matching the unweighted gains_ascending=True convention.

Parameters:

Name	Type	Description	Default
score	array - like	Scores, converted to float. A NaN score is ranked after every finite score, whatever ascending is, so it falls in the highest-numbered bin(s).	required
weight	array - like or None	Weights aligned with score and used as given (not validated); None gives every row the weight 1. There is no default: the argument must be passed.	required
nbins	int	Number of bins (converted with int).	required
ascending	bool	False ranks the highest score first (bin 1 holds the highest scores); True ranks the lowest score first.	False

Returns:

Type	Description
ndarray	Integer bin labels from 1 to nbins, in the order of the input rows. A row gets the label ceil(cumulative weight / total weight * nbins) (clipped to [1, nbins]) computed in rank order, so every bin holds about 1 / nbins of the total weight.

Notes

Rows with the same score are ordered by their position in the input, so they can be split across two adjacent bins.

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def rank_bins(score, weight, nbins, ascending=False):
    """Weighted equal-frequency rank bins.

    ``ascending=False`` (default, legacy) sorts scores descending — bin 1 is
    the highest-score bucket. ``ascending=True`` sorts scores ascending so
    bin 1 is the lowest-score bucket, matching the unweighted
    ``gains_ascending=True`` convention.

    Parameters
    ----------
    score : array-like
        Scores, converted to float. A NaN score is ranked after every finite score, whatever ``ascending`` is, so it falls
        in the highest-numbered bin(s).
    weight : array-like or None
        Weights aligned with ``score`` and used as given (not validated); None gives every row the weight 1. There is no
        default: the argument must be passed.
    nbins : int
        Number of bins (converted with ``int``).
    ascending : bool, default False
        False ranks the highest score first (bin 1 holds the highest scores); True ranks the lowest score first.

    Returns
    -------
    numpy.ndarray
        Integer bin labels from 1 to ``nbins``, in the order of the input rows. A row gets the label
        ``ceil(cumulative weight / total weight * nbins)`` (clipped to ``[1, nbins]``) computed in rank order, so every bin
        holds about ``1 / nbins`` of the total weight.

    Notes
    -----
    Rows with the same score are ordered by their position in the input, so they can be split across two adjacent bins.
    """
    score = np.asarray(score, dtype=float)
    weight = np.ones(len(score), dtype=float) if weight is None else np.asarray(weight, dtype=float)
    sort_key = score if ascending else -score
    order = np.lexsort((np.arange(len(score)), sort_key))
    bins = np.empty(len(score), dtype=int)
    total = float(weight.sum()) or float(len(score)) or 1.0
    cum_weight = np.cumsum(weight[order])
    labels = np.ceil(cum_weight / total * int(nbins)).astype(int)
    labels = np.clip(labels, 1, int(nbins))
    bins[order] = labels
    return bins

get_gains_table

get_gains_table(data, dep, score, nbins=10, weight_col=None, weighted_binning=None, ascending=False, spec_values=None, include_missing=False, add_func=None, withSummary=False, **kwargs)

Sample-weight-aware Gains table: equal-weight score bins with their target statistics.

The rows are ranked by score (see rank_bins) and cut into nbins bins that each hold about 1 / nbins of the total weight. data is not modified.

Parameters:

Name	Type	Description	Default
data	DataFrame	Dataset that holds the target, the score and, when given, the weight column.	required
dep	str	Name of the target column; numeric, 1 for bad and 0 for good.	required
score	str	Name of the score column; converted to float.	required
nbins	int	Number of bins.	10
weight_col	str or None	Name of the sample weight column in data. None gives every row the weight 1, so N is a row count.	None
weighted_binning	bool or None	Accepted for compatibility but without any effect: the bins are always equal-weight bins.	None
ascending	bool	False puts the highest scores in bin 1; True puts the lowest scores in bin 1.	False
spec_values	list or None	Special score values (business sentinels such as -1). The rows holding one of them are taken out before the binning and reported in rows of their own (see Notes). None or an empty list means no special values.	None
include_missing	bool	Rows whose score is missing never enter the bins. False leaves them out of the table; True reports them in a row of their own (see Notes).	False
add_func	callable or None	Custom statistics function, as in the unweighted table. It receives the rows of one row of the table as a DataFrame (all the columns of data plus _bin_num and _bin_range) and returns a Series whose values become extra columns; the special and Missing rows get their values too. It sees the weight column like any other column, so a weighted statistic is up to the function.	None
withSummary	bool	Whether to add a Grand Summary row (indexed by ("Grand Summary", "")) after all the other rows. Its N, N_RAW, PERF_CNT, N_BAD, N_GOOD, AVG_BAD, AVG_GOOD and PROP cover every row of the table, special and Missing rows included; MIN, MAX, AVG_SCORE (weighted) and UNIQUE_SCORE cover the rows with a score. BAD_PCT_IN_EACH_BIN, GOOD_PCT_IN_EACH_BIN, RANK_ORDER_BUMP and IV are the sums and KS_PER_BIN and KS the maximum over the bins, WOE is the mean of the bins' WOE, AUC is the AUC, and N_CUM_BAD, N_CUM_GOOD, CUM_BAD_PCT, CUM_GOOD_PCT, LIFT and TRUE_BAD_SHIFT take the values of the unweighted summary row (the totals, 1, 1, 1 and 1). The add_func columns are NaN in it.	False
**kwargs		Accepted and ignored.	{}

Returns:

Type

Description

DataFrame

One row per bin, indexed by _bin_num and _bin_range (a MultiIndex whose two levels both hold the bin number 1 to nbins), with the columns: MIN, MAX: lowest and highest score in the bin. N: sum of the weights in the bin; N_RAW: number of rows; PERF_CNT: sum of the weights of the rows whose target is observed (rows with a missing target count in N only, as in the unweighted table). N_BAD, N_GOOD: weighted counts of bad (weight * target) and good (weight * (1 - target)) rows among the rows with an observed target. AVG_SCORE: weighted mean score of the bin (NaN when the bin has NaN scores); UNIQUE_SCORE: number of distinct scores. PROP: N divided by the total weight of the rows in the table, special (and reported missing) scores included, so PROP adds up to 1; AVG_BAD and AVG_GOOD: N_BAD and N_GOOD divided by PERF_CNT. BAD_PCT_IN_EACH_BIN, GOOD_PCT_IN_EACH_BIN: share of all bad and all good weight that falls in the bin. N_CUM_BAD, N_CUM_GOOD, CUM_BAD_PCT, CUM_GOOD_PCT: cumulative sums, from bin 1 down, of N_BAD, N_GOOD, BAD_PCT_IN_EACH_BIN and GOOD_PCT_IN_EACH_BIN. KS_PER_BIN: abs(CUM_BAD_PCT - CUM_GOOD_PCT); KS is a copy of it. LIFT: AVG_BAD divided by the overall bad rate (N_BAD over PERF_CNT of all bins). TRUE_BAD_SHIFT: relative change of AVG_BAD from the previous bin, previous / current - 1 when ascending is False and current / previous - 1 when it is True (NaN in bin 1). RANK_ORDER_BUMP: 1 when TRUE_BAD_SHIFT is negative (the bad rate is not monotonic), else 0. WOE: ln(BAD_PCT_IN_EACH_BIN / GOOD_PCT_IN_EACH_BIN), with 0 where it is infinite or undefined; IV: (BAD_PCT_IN_EACH_BIN - GOOD_PCT_IN_EACH_BIN) * WOE. AUC: weighted AUC of the score on the non-special rows with an observed target (the same value on every row), NaN when it cannot be computed.

Raises:

Type	Description
KeyError	If dep, score or weight_col is not a column of data.
ValueError	If the weights contain NaN, infinity or a negative value, or sum to zero or less.

Notes

The statistics are computed on the rows that do not hold a special score. Each special score value gets one more row after the bins, indexed by "special:<value>" in both index levels (in ascending order of the value), with the columns MIN, MAX (both the value), N, N_RAW, PERF_CNT, N_BAD, N_GOOD, AVG_SCORE, UNIQUE_SCORE (1), PROP, AVG_BAD, AVG_GOOD and AUC (the AUC of the regular rows); every other column is NaN. PROP is the share of the weight of all rows in the regular bins and in the special rows alike, so the column adds up to 1. The other shares (BAD_PCT_IN_EACH_BIN and GOOD_PCT_IN_EACH_BIN), LIFT and the overall bad rate still refer to the rows without a special score.

Rows with a missing score carry no ranking information and never enter the bins. With include_missing=True they get one row after the bins (and after the special rows), indexed by "Missing" in both index levels, with N, N_RAW, PERF_CNT, N_BAD, N_GOOD, UNIQUE_SCORE (0), PROP, AVG_BAD, AVG_GOOD and AUC; every other column is NaN. Rows with the same score can be split across two adjacent bins.

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def get_gains_table(data, dep, score, nbins=10, weight_col=None, weighted_binning=None,
                    ascending=False, spec_values=None, include_missing=False, add_func=None, withSummary=False,
                    **kwargs):
    """Sample-weight-aware Gains table: equal-weight score bins with their target statistics.

    The rows are ranked by score (see ``rank_bins``) and cut into ``nbins`` bins that each hold about ``1 / nbins`` of the
    total weight. ``data`` is not modified.

    Parameters
    ----------
    data : pandas.DataFrame
        Dataset that holds the target, the score and, when given, the weight column.
    dep : str
        Name of the target column; numeric, 1 for bad and 0 for good.
    score : str
        Name of the score column; converted to float.
    nbins : int, default 10
        Number of bins.
    weight_col : str or None, default None
        Name of the sample weight column in ``data``. None gives every row the weight 1, so ``N`` is a row count.
    weighted_binning : bool or None, default None
        Accepted for compatibility but without any effect: the bins are always equal-weight bins.
    ascending : bool, default False
        False puts the highest scores in bin 1; True puts the lowest scores in bin 1.
    spec_values : list or None, default None
        Special score values (business sentinels such as ``-1``). The rows holding one of them are taken out before the
        binning and reported in rows of their own (see Notes). None or an empty list means no special values.
    include_missing : bool, default False
        Rows whose score is missing never enter the bins. False leaves them out of the table; True reports them in a
        row of their own (see Notes).
    add_func : callable or None, default None
        Custom statistics function, as in the unweighted table. It receives the rows of one row of the table as a
        DataFrame (all the columns of ``data`` plus ``_bin_num`` and ``_bin_range``) and returns a Series whose values
        become extra columns; the special and ``Missing`` rows get their values too. It sees the weight column like any
        other column, so a weighted statistic is up to the function.
    withSummary : bool, default False
        Whether to add a ``Grand Summary`` row (indexed by ``("Grand Summary", "")``) after all the other rows. Its ``N``,
        ``N_RAW``, ``PERF_CNT``, ``N_BAD``, ``N_GOOD``, ``AVG_BAD``, ``AVG_GOOD`` and ``PROP`` cover every row of the table,
        special and ``Missing`` rows included; ``MIN``, ``MAX``, ``AVG_SCORE`` (weighted) and ``UNIQUE_SCORE`` cover the
        rows with a score. ``BAD_PCT_IN_EACH_BIN``, ``GOOD_PCT_IN_EACH_BIN``, ``RANK_ORDER_BUMP`` and ``IV`` are the sums
        and ``KS_PER_BIN`` and ``KS`` the maximum over the bins, ``WOE`` is the mean of the bins' WOE, ``AUC`` is the AUC,
        and ``N_CUM_BAD``, ``N_CUM_GOOD``, ``CUM_BAD_PCT``, ``CUM_GOOD_PCT``, ``LIFT`` and ``TRUE_BAD_SHIFT`` take the
        values of the unweighted summary row (the totals, 1, 1, 1 and 1). The ``add_func`` columns are NaN in it.
    **kwargs
        Accepted and ignored.

    Returns
    -------
    pandas.DataFrame
        One row per bin, indexed by ``_bin_num`` and ``_bin_range`` (a MultiIndex whose two levels both hold the bin number
        1 to ``nbins``), with the columns:

        - ``MIN``, ``MAX``: lowest and highest score in the bin.
        - ``N``: sum of the weights in the bin; ``N_RAW``: number of rows; ``PERF_CNT``: sum of the weights of the rows
          whose target is observed (rows with a missing target count in ``N`` only, as in the unweighted table).
        - ``N_BAD``, ``N_GOOD``: weighted counts of bad (``weight * target``) and good (``weight * (1 - target)``) rows
          among the rows with an observed target.
        - ``AVG_SCORE``: weighted mean score of the bin (NaN when the bin has NaN scores); ``UNIQUE_SCORE``: number of
          distinct scores.
        - ``PROP``: ``N`` divided by the total weight of the rows in the table, special (and reported missing) scores
          included, so ``PROP`` adds up to 1; ``AVG_BAD`` and ``AVG_GOOD``: ``N_BAD`` and ``N_GOOD`` divided by
          ``PERF_CNT``.
        - ``BAD_PCT_IN_EACH_BIN``, ``GOOD_PCT_IN_EACH_BIN``: share of all bad and all good weight that falls in the bin.
        - ``N_CUM_BAD``, ``N_CUM_GOOD``, ``CUM_BAD_PCT``, ``CUM_GOOD_PCT``: cumulative sums, from bin 1 down, of ``N_BAD``,
          ``N_GOOD``, ``BAD_PCT_IN_EACH_BIN`` and ``GOOD_PCT_IN_EACH_BIN``.
        - ``KS_PER_BIN``: ``abs(CUM_BAD_PCT - CUM_GOOD_PCT)``; ``KS`` is a copy of it.
        - ``LIFT``: ``AVG_BAD`` divided by the overall bad rate (``N_BAD`` over ``PERF_CNT`` of all bins).
        - ``TRUE_BAD_SHIFT``: relative change of ``AVG_BAD`` from the previous bin, ``previous / current - 1`` when
          ``ascending`` is False and ``current / previous - 1`` when it is True (NaN in bin 1).
        - ``RANK_ORDER_BUMP``: 1 when ``TRUE_BAD_SHIFT`` is negative (the bad rate is not monotonic), else 0.
        - ``WOE``: ``ln(BAD_PCT_IN_EACH_BIN / GOOD_PCT_IN_EACH_BIN)``, with 0 where it is infinite or undefined; ``IV``:
          ``(BAD_PCT_IN_EACH_BIN - GOOD_PCT_IN_EACH_BIN) * WOE``.
        - ``AUC``: weighted AUC of the score on the non-special rows with an observed target (the same value on every
          row), NaN when it cannot be computed.

    Raises
    ------
    KeyError
        If ``dep``, ``score`` or ``weight_col`` is not a column of ``data``.
    ValueError
        If the weights contain NaN, infinity or a negative value, or sum to zero or less.

    Notes
    -----
    The statistics are computed on the rows that do not hold a special score. Each special score value gets one more row
    after the bins, indexed by ``"special:<value>"`` in both index levels (in ascending order of the value), with the
    columns ``MIN``, ``MAX`` (both the value), ``N``, ``N_RAW``, ``PERF_CNT``, ``N_BAD``, ``N_GOOD``, ``AVG_SCORE``,
    ``UNIQUE_SCORE`` (1), ``PROP``, ``AVG_BAD``, ``AVG_GOOD`` and ``AUC`` (the AUC of the regular rows); every other column
    is NaN. ``PROP`` is the share of the weight of all rows in the regular bins and in the special rows alike, so the
    column adds up to 1. The other shares (``BAD_PCT_IN_EACH_BIN`` and ``GOOD_PCT_IN_EACH_BIN``), ``LIFT`` and the overall
    bad rate still refer to the rows without a special score.

    Rows with a missing score carry no ranking information and never enter the bins. With ``include_missing=True`` they
    get one row after the bins (and after the special rows), indexed by ``"Missing"`` in both index levels, with ``N``,
    ``N_RAW``, ``PERF_CNT``, ``N_BAD``, ``N_GOOD``, ``UNIQUE_SCORE`` (0), ``PROP``, ``AVG_BAD``, ``AVG_GOOD`` and ``AUC``;
    every other column is NaN. Rows with the same score can be split across two adjacent bins.
    """
    cols = [dep, score]
    if weight_col is not None and weight_col in data.columns:
        cols.append(weight_col)
    df = data[cols].copy()
    if add_func is not None:
        # Remember each row's position so that add_func can see all the columns of data for the rows of each table row
        df["_smf_row_pos"] = np.arange(len(df))
    # A missing score has no rank: it would be sorted to the end of the ranking and fill the last bins
    score_missing = df[score].isna()
    missing_df = df[score_missing] if bool(score_missing.any()) else None
    df = df[~score_missing]
    df, special_df = _split_special_scores(df, score, spec_values)
    weight = resolve_weights(df, weight_col=weight_col, expected_len=len(df))
    if weight is None:
        weight = np.ones(len(df), dtype=float)

    y = df[dep].astype(float).to_numpy()
    labelled = ~np.isnan(y)
    y_obs = np.where(labelled, y, 0.0)
    s = df[score].astype(float).to_numpy()
    df["_bin_num"] = rank_bins(s, weight, nbins, ascending=ascending)
    df["_bin_range"] = df["_bin_num"]
    df["_w"] = weight
    # Rows with a missing target are in N but in neither the bad nor the good counts
    df["_perf_w"] = weight * labelled
    df["_bad_w"] = weight * y_obs
    df["_good_w"] = weight * labelled * (1.0 - y_obs)
    df["_score_w"] = weight * s
    grouped = df.groupby(["_bin_num", "_bin_range"], sort=True, dropna=False)
    out = grouped.agg(
        MIN=(score, "min"),
        MAX=(score, "max"),
        N=("_w", "sum"),
        N_RAW=(dep, "size"),
        PERF_CNT=("_perf_w", "sum"),
        N_BAD=("_bad_w", "sum"),
        N_GOOD=("_good_w", "sum"),
        SCORE_W=("_score_w", "sum"),
        SCORE_COUNT=(score, "count"),
        UNIQUE_SCORE=(score, "nunique"),
    )
    out["AVG_SCORE"] = out["SCORE_W"] / out["N"].replace(0, np.nan)
    out.loc[out["SCORE_COUNT"].ne(out["N_RAW"]), "AVG_SCORE"] = np.nan
    out = out.drop(columns=["SCORE_W", "SCORE_COUNT"])
    out = out[
        [
            "MIN", "MAX", "N", "N_RAW", "PERF_CNT", "N_BAD",
            "N_GOOD", "AVG_SCORE", "UNIQUE_SCORE",
        ]
    ]

    total_perf = float(out["PERF_CNT"].sum())
    total_bad = float(out["N_BAD"].sum()) or 1.0
    total_good = float(out["N_GOOD"].sum()) or 1.0
    overall_bad_rate = float(out["N_BAD"].sum()) / total_perf if total_perf else np.nan

    # PROP is the share of the weight of ALL rows in the table, special (and reported missing) ones included,
    # so the bins and the extra rows add up to 1.
    grand_total = float(out["N"].sum())
    if special_df is not None and len(special_df):
        spec_weight = resolve_weights(special_df, weight_col=weight_col, expected_len=len(special_df))
        if spec_weight is None:
            spec_weight = np.ones(len(special_df), dtype=float)
        grand_total += float(np.sum(spec_weight))
    report_missing = include_missing and missing_df is not None
    if report_missing:
        miss_weight = resolve_weights(missing_df, weight_col=weight_col, expected_len=len(missing_df))
        if miss_weight is None:
            miss_weight = np.ones(len(missing_df), dtype=float)
        grand_total += float(np.sum(miss_weight))
    out["PROP"] = out["N"] / grand_total if grand_total else np.nan
    out["AVG_BAD"] = out["N_BAD"] / out["PERF_CNT"].replace(0, np.nan)
    out["AVG_GOOD"] = out["N_GOOD"] / out["PERF_CNT"].replace(0, np.nan)
    out["BAD_PCT_IN_EACH_BIN"] = out["N_BAD"] / total_bad
    out["GOOD_PCT_IN_EACH_BIN"] = out["N_GOOD"] / total_good
    out["N_CUM_BAD"] = out["N_BAD"].cumsum()
    out["N_CUM_GOOD"] = out["N_GOOD"].cumsum()
    out["CUM_BAD_PCT"] = out["BAD_PCT_IN_EACH_BIN"].cumsum()
    out["CUM_GOOD_PCT"] = out["GOOD_PCT_IN_EACH_BIN"].cumsum()
    out["KS_PER_BIN"] = (out["CUM_BAD_PCT"] - out["CUM_GOOD_PCT"]).abs()
    out["KS"] = out["KS_PER_BIN"]
    out["LIFT"] = out["AVG_BAD"] / overall_bad_rate if overall_bad_rate else np.nan
    out["TRUE_BAD_SHIFT"] = (
        (out["AVG_BAD"].shift(1) / out["AVG_BAD"] - 1)
        if not ascending
        else (out["AVG_BAD"] / out["AVG_BAD"].shift(1) - 1)
    )
    out["RANK_ORDER_BUMP"] = out["TRUE_BAD_SHIFT"].lt(0).astype(int)
    with np.errstate(divide="ignore", invalid="ignore"):
        out["WOE"] = np.log(out["BAD_PCT_IN_EACH_BIN"] / out["GOOD_PCT_IN_EACH_BIN"])
    out["WOE"] = out["WOE"].replace([np.inf, -np.inf], 0).fillna(0)
    out["IV"] = (out["BAD_PCT_IN_EACH_BIN"] - out["GOOD_PCT_IN_EACH_BIN"]) * out["WOE"]
    out["AUC"] = safe_auc(y[labelled], s[labelled], sample_weight=weight[labelled])

    def _extra_row(label, part, part_w, value):
        part_y = part[dep].astype(float).to_numpy()
        part_obs = ~np.isnan(part_y)
        n_w = float(np.sum(part_w))
        perf_w = float(np.sum(part_w[part_obs]))
        n_bad = float(np.sum(part_w[part_obs] * part_y[part_obs]))
        return pd.DataFrame(
            {
                "MIN": [value],
                "MAX": [value],
                "N": [n_w],
                "N_RAW": [int(len(part))],
                "PERF_CNT": [perf_w],
                "N_BAD": [n_bad],
                "N_GOOD": [perf_w - n_bad],
                "AVG_SCORE": [float(value)],
                "UNIQUE_SCORE": [1 if pd.notna(value) else 0],
                "PROP": [n_w / grand_total if grand_total else np.nan],
                "AVG_BAD": [n_bad / perf_w if perf_w else np.nan],
                "AVG_GOOD": [(perf_w - n_bad) / perf_w if perf_w else np.nan],
            },
            index=pd.MultiIndex.from_tuples([(label, label)], names=out.index.names),
        )

    bins = out
    extra_rows = []
    # The rows behind each row of the table, labelled like it, for add_func
    row_parts = [(df, df["_bin_num"].to_numpy())]
    if special_df is not None and len(special_df):
        # Special sentinel scores (e.g. -1 for all-missing rows) get their own
        # descriptive rows: never part of quantile edges, cumulative columns,
        # or ranking metrics (those stay NaN by construction).
        for value, part in special_df.groupby(score, sort=True):
            part_w = resolve_weights(part, weight_col=weight_col, expected_len=len(part))
            if part_w is None:
                part_w = np.ones(len(part), dtype=float)
            extra_rows.append(_extra_row(f"special:{value}", part, part_w, value))
            row_parts.append((part, [f"special:{value}"] * len(part)))
    if report_missing:
        extra_rows.append(_extra_row("Missing", missing_df, miss_weight, np.nan))
        row_parts.append((missing_df, ["Missing"] * len(missing_df)))
    if extra_rows:
        out = pd.concat([out] + extra_rows)
        out["AUC"] = out["AUC"].iloc[0]

    if add_func is not None:
        pieces = []
        for part, labels in row_parts:
            piece = data.iloc[part["_smf_row_pos"].to_numpy()].copy()
            piece["_bin_num"] = list(labels)
            piece["_bin_range"] = list(labels)
            pieces.append(piece)
        rows = pd.concat(pieces)
        # Explicitly select all columns (including the grouping columns), as in the unweighted table
        added = rows.groupby(["_bin_num", "_bin_range"], sort=False, dropna=False)[
            rows.columns.unique().tolist()
        ].apply(add_func)
        out = out.merge(added, left_index=True, right_index=True, how="left")

    if withSummary:
        scored = df if special_df is None else pd.concat([df, special_df])
        scored_w = resolve_weights(scored, weight_col=weight_col, expected_len=len(scored))
        if scored_w is None:
            scored_w = np.ones(len(scored), dtype=float)
        scored_s = scored[score].astype(float).to_numpy()
        n_bad, n_good, perf = float(out["N_BAD"].sum()), float(out["N_GOOD"].sum()), float(out["PERF_CNT"].sum())
        summary = {
            "MIN": float(np.min(scored_s)) if len(scored_s) else np.nan,
            "MAX": float(np.max(scored_s)) if len(scored_s) else np.nan,
            "N": float(out["N"].sum()),
            "N_RAW": int(out["N_RAW"].sum()),
            "PERF_CNT": perf,
            "N_BAD": n_bad,
            "N_GOOD": n_good,
            "AVG_SCORE": float(np.sum(scored_w * scored_s) / np.sum(scored_w)) if np.sum(scored_w) else np.nan,
            "UNIQUE_SCORE": int(scored[score].nunique()),
            "PROP": float(out["PROP"].sum()),
            "AVG_BAD": n_bad / perf if perf else np.nan,
            "AVG_GOOD": n_good / perf if perf else np.nan,
            "BAD_PCT_IN_EACH_BIN": float(bins["BAD_PCT_IN_EACH_BIN"].sum()),
            "GOOD_PCT_IN_EACH_BIN": float(bins["GOOD_PCT_IN_EACH_BIN"].sum()),
            "N_CUM_BAD": n_bad,
            "N_CUM_GOOD": n_good,
            "CUM_BAD_PCT": 1.0,
            "CUM_GOOD_PCT": 1.0,
            "KS_PER_BIN": float(bins["KS_PER_BIN"].max()),
            "KS": float(bins["KS"].max()),
            "LIFT": 1.0,
            "TRUE_BAD_SHIFT": 1.0,
            "RANK_ORDER_BUMP": int(bins["RANK_ORDER_BUMP"].sum()),
            "WOE": float(bins["WOE"].mean()),
            "IV": float(bins["IV"].sum()),
            "AUC": float(bins["AUC"].iloc[0]) if len(bins) else np.nan,
        }
        summary_row = pd.DataFrame(
            summary, index=pd.MultiIndex.from_tuples([("Grand Summary", "")], names=out.index.names)
        )
        out = pd.concat([out, summary_row])
    return out

calc_lift_apt

calc_lift_apt(y_true, y_score, start=1.0, stop=3.0, step=0.1, sample_weight=None)

Closest available Lift values for a grid of target Lifts, from a weighted 100-bin Gains table.

Parameters:

Name	Type	Description	Default
y_true	array - like	Actual binary labels (0/1), converted to float.	required
y_score	array - like	Predicted scores or probabilities, converted to float.	required
start	float	First target Lift.	1.0
stop	float	Last target Lift (included up to rounding of the grid).	3.0
step	float	Distance between two target Lifts.	0.1
sample_weight	array - like or None	Per-sample weights aligned with y_true; None means equal weights. They are validated like the weights of get_gains_table.	None

Returns:

Type	Description
ndarray	One-dimensional float array with one value per target Lift of np.arange(start, stop + step / 2, step): the LIFT of the bin of get_gains_table(..., nbins=100) (highest scores first) that is closest to that target.

Raises:

Type	Description
ValueError	If the weights are invalid (see get_gains_table).

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def calc_lift_apt(y_true, y_score, start=1.0, stop=3.0, step=0.1, sample_weight=None):
    """Closest available Lift values for a grid of target Lifts, from a weighted 100-bin Gains table.

    Parameters
    ----------
    y_true : array-like
        Actual binary labels (0/1), converted to float.
    y_score : array-like
        Predicted scores or probabilities, converted to float.
    start : float, default 1.0
        First target Lift.
    stop : float, default 3.0
        Last target Lift (included up to rounding of the grid).
    step : float, default 0.1
        Distance between two target Lifts.
    sample_weight : array-like or None, default None
        Per-sample weights aligned with ``y_true``; None means equal weights. They are validated like the weights of
        ``get_gains_table``.

    Returns
    -------
    numpy.ndarray
        One-dimensional float array with one value per target Lift of ``np.arange(start, stop + step / 2, step)``: the
        ``LIFT`` of the bin of ``get_gains_table(..., nbins=100)`` (highest scores first) that is closest to that target.

    Raises
    ------
    ValueError
        If the weights are invalid (see ``get_gains_table``).
    """
    y_true = np.asarray(y_true, dtype=float)
    y_score = np.asarray(y_score, dtype=float)
    weight = np.ones(len(y_true), dtype=float) if sample_weight is None else np.asarray(sample_weight, dtype=float)
    gains = get_gains_table(
        pd.DataFrame({"y": y_true, "s": y_score, "w": weight}),
        "y",
        "s",
        nbins=100,
        weight_col="w",
    )
    vals = []
    for target in np.arange(start, stop + step / 2.0, step):
        idx = (gains["LIFT"] - target).abs().idxmin()
        vals.append(float(gains.loc[idx, "LIFT"]))
    return np.asarray(vals)

calc_equid_dist

calc_equid_dist(y_true, y_score, bins=10, sample_weight=None, **kwargs)

Weighted Gains table over bins equal-weight score bins.

Despite its name the bins are not equal-width: the weighted implementation cuts the ranked scores into bins that each hold about 1 / bins of the total weight. It is the weighted counterpart of evaluate_model.calc_equid_dist.

Parameters:

Name	Type	Description	Default
y_true	array - like	Actual binary labels (0/1).	required
y_score	array - like	Predicted scores or probabilities, same length as y_true.	required
bins	int	Number of bins.	10
sample_weight	array - like or None	Per-sample weights aligned with y_true; None gives every row the weight 1. They are validated like the weights of get_gains_table.	None
**kwargs		Forwarded to get_gains_table: ascending, spec_values and include_missing take effect, other names are ignored. nbins and weight_col must not be passed (TypeError: they are already set).	{}

Returns:

Type	Description
DataFrame	The Gains table of get_gains_table: bins rows indexed by _bin_num and _bin_range (see get_gains_table for the columns).

Raises:

Type	Description
ValueError	If the weights are invalid (see get_gains_table).

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def calc_equid_dist(y_true, y_score, bins=10, sample_weight=None, **kwargs):
    """Weighted Gains table over ``bins`` equal-weight score bins.

    Despite its name the bins are not equal-width: the weighted implementation cuts the ranked scores into bins that each
    hold about ``1 / bins`` of the total weight. It is the weighted counterpart of ``evaluate_model.calc_equid_dist``.

    Parameters
    ----------
    y_true : array-like
        Actual binary labels (0/1).
    y_score : array-like
        Predicted scores or probabilities, same length as ``y_true``.
    bins : int, default 10
        Number of bins.
    sample_weight : array-like or None, default None
        Per-sample weights aligned with ``y_true``; None gives every row the weight 1. They are validated like the weights
        of ``get_gains_table``.
    **kwargs
        Forwarded to ``get_gains_table``: ``ascending``, ``spec_values`` and ``include_missing`` take effect, other names are ignored.
        ``nbins`` and ``weight_col`` must not be passed (``TypeError``: they are already set).

    Returns
    -------
    pandas.DataFrame
        The Gains table of ``get_gains_table``: ``bins`` rows indexed by ``_bin_num`` and ``_bin_range`` (see
        ``get_gains_table`` for the columns).

    Raises
    ------
    ValueError
        If the weights are invalid (see ``get_gains_table``).
    """
    weight = np.ones(len(y_true), dtype=float) if sample_weight is None else np.asarray(sample_weight, dtype=float)
    return get_gains_table(
        pd.DataFrame({"y": y_true, "s": y_score, "w": weight}),
        "y",
        "s",
        nbins=bins,
        weight_col="w",
        **kwargs,
    )

calc_equid_pct

calc_equid_pct(y_true, y_score, bins=10, sample_weight=None, **kwargs)

Weighted Gains table over bins equal-weight score bins.

Same as calc_equid_dist, to which it delegates: the weighted counterpart of evaluate_model.calc_equid_pct.

Parameters:

Name	Type	Description	Default
y_true	array - like	Actual binary labels (0/1).	required
y_score	array - like	Predicted scores or probabilities, same length as y_true.	required
bins	int	Number of bins.	10
sample_weight	array - like or None	Per-sample weights aligned with y_true; None gives every row the weight 1.	None
**kwargs		Forwarded to calc_equid_dist and then to get_gains_table: ascending, spec_values and include_missing take effect, other names are ignored.	{}

Returns:

Type	Description
DataFrame	The Gains table of get_gains_table: bins rows indexed by _bin_num and _bin_range (see get_gains_table for the columns).

Raises:

Type	Description
ValueError	If the weights are invalid (see get_gains_table).

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def calc_equid_pct(y_true, y_score, bins=10, sample_weight=None, **kwargs):
    """Weighted Gains table over ``bins`` equal-weight score bins.

    Same as ``calc_equid_dist``, to which it delegates: the weighted counterpart of ``evaluate_model.calc_equid_pct``.

    Parameters
    ----------
    y_true : array-like
        Actual binary labels (0/1).
    y_score : array-like
        Predicted scores or probabilities, same length as ``y_true``.
    bins : int, default 10
        Number of bins.
    sample_weight : array-like or None, default None
        Per-sample weights aligned with ``y_true``; None gives every row the weight 1.
    **kwargs
        Forwarded to ``calc_equid_dist`` and then to ``get_gains_table``: ``ascending``, ``spec_values`` and
        ``include_missing`` take effect, other names are ignored.

    Returns
    -------
    pandas.DataFrame
        The Gains table of ``get_gains_table``: ``bins`` rows indexed by ``_bin_num`` and ``_bin_range`` (see
        ``get_gains_table`` for the columns).

    Raises
    ------
    ValueError
        If the weights are invalid (see ``get_gains_table``).
    """
    return calc_equid_dist(y_true, y_score, bins=bins, sample_weight=sample_weight, **kwargs)

calc_fixed_pct

calc_fixed_pct(y_true, y_score, sample_weight=None, **kwargs)

Weighted Gains table over equal-weight score bins; the weighted counterpart of evaluate_model.calc_fixed_pct.

Fixed bin edges are not supported here: the call delegates to calc_equid_dist, so the table has 10 equal-weight bins unless bins is passed in kwargs.

Parameters:

Name	Type	Description	Default
y_true	array - like	Actual binary labels (0/1).	required
y_score	array - like	Predicted scores or probabilities, same length as y_true.	required
sample_weight	array - like or None	Per-sample weights aligned with y_true; None gives every row the weight 1.	None
**kwargs		Forwarded to calc_equid_dist: bins (number of bins, default 10), ascending and spec_values take effect; other names (such as bin_edges) are ignored.	{}

Returns:

Type	Description
DataFrame	The Gains table of get_gains_table: bins rows indexed by _bin_num and _bin_range (see get_gains_table for the columns).

Raises:

Type	Description
ValueError	If the weights are invalid (see get_gains_table).

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def calc_fixed_pct(y_true, y_score, sample_weight=None, **kwargs):
    """Weighted Gains table over equal-weight score bins; the weighted counterpart of ``evaluate_model.calc_fixed_pct``.

    Fixed bin edges are not supported here: the call delegates to ``calc_equid_dist``, so the table has 10 equal-weight
    bins unless ``bins`` is passed in ``kwargs``.

    Parameters
    ----------
    y_true : array-like
        Actual binary labels (0/1).
    y_score : array-like
        Predicted scores or probabilities, same length as ``y_true``.
    sample_weight : array-like or None, default None
        Per-sample weights aligned with ``y_true``; None gives every row the weight 1.
    **kwargs
        Forwarded to ``calc_equid_dist``: ``bins`` (number of bins, default 10), ``ascending`` and ``spec_values`` take
        effect; other names (such as ``bin_edges``) are ignored.

    Returns
    -------
    pandas.DataFrame
        The Gains table of ``get_gains_table``: ``bins`` rows indexed by ``_bin_num`` and ``_bin_range`` (see
        ``get_gains_table`` for the columns).

    Raises
    ------
    ValueError
        If the weights are invalid (see ``get_gains_table``).
    """
    return calc_equid_dist(y_true, y_score, sample_weight=sample_weight, **kwargs)

dataset_summary

dataset_summary(name, data, tgt_name, scr_name, weight_col=None, nbins=10, ascending=False, spec_values=None)

Performance summary of one dataset: AUC, KS, Lift, IV and the average target and score.

Parameters:

Name	Type	Description	Default
name	str	Label of the dataset; stored in the index, dataset and DATASET entries.	required
data	DataFrame	Dataset that holds the target, the score and, when given, the weight column.	required
tgt_name	str	Name of the target column (binary, 1 for bad).	required
scr_name	str	Name of the score column.	required
weight_col	str or None	Name of the sample weight column in data; None gives every row the weight 1.	None
nbins	int	Number of bins of the Gains table behind LIFT and IV.	10
ascending	bool	Bin order of that Gains table (see get_gains_table).	False
spec_values	list or None	Special score values (such as -1) that are left out of the ranking statistics and reported separately.	None

Returns:

Type

Description

dict

The entries index, dataset and DATASET (all equal to name), AUC, KS (maximum of abs(tpr - fpr) on the ROC curve), LIFT (largest bin Lift), IV (sum over the bins), N (sum of the weights of all rows, or the number of rows without weights), N_RAW (number of rows), avgTrue (weighted mean of the target over all rows) and avgScore (weighted mean of the score over the non-special rows). N_SPECIAL (sum of the weights) and N_SPECIAL_RAW (number of rows) are added when at least one row holds a special score.

Raises:

Type	Description
KeyError	If tgt_name, scr_name or weight_col is not a column of data.
ValueError	If the weights contain NaN, infinity or a negative value, or sum to zero or less.

Notes

AUC, KS, LIFT, IV and avgScore use the non-special rows only, because sentinel scores carry no ordering information; avgTrue, N and N_RAW cover all rows. AUC and KS are NaN when they cannot be computed (for example with a single class).

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def dataset_summary(name, data, tgt_name, scr_name, weight_col=None, nbins=10,
                    ascending=False, spec_values=None):
    """Performance summary of one dataset: AUC, KS, Lift, IV and the average target and score.

    Parameters
    ----------
    name : str
        Label of the dataset; stored in the ``index``, ``dataset`` and ``DATASET`` entries.
    data : pandas.DataFrame
        Dataset that holds the target, the score and, when given, the weight column.
    tgt_name : str
        Name of the target column (binary, 1 for bad).
    scr_name : str
        Name of the score column.
    weight_col : str or None, default None
        Name of the sample weight column in ``data``; None gives every row the weight 1.
    nbins : int, default 10
        Number of bins of the Gains table behind ``LIFT`` and ``IV``.
    ascending : bool, default False
        Bin order of that Gains table (see ``get_gains_table``).
    spec_values : list or None, default None
        Special score values (such as ``-1``) that are left out of the ranking statistics and reported separately.

    Returns
    -------
    dict
        The entries ``index``, ``dataset`` and ``DATASET`` (all equal to ``name``), ``AUC``, ``KS`` (maximum of
        ``abs(tpr - fpr)`` on the ROC curve), ``LIFT`` (largest bin Lift), ``IV`` (sum over the bins), ``N`` (sum of the
        weights of all rows, or the number of rows without weights), ``N_RAW`` (number of rows), ``avgTrue`` (weighted mean
        of the target over all rows) and ``avgScore`` (weighted mean of the score over the non-special rows). ``N_SPECIAL``
        (sum of the weights) and ``N_SPECIAL_RAW`` (number of rows) are added when at least one row holds a special score.

    Raises
    ------
    KeyError
        If ``tgt_name``, ``scr_name`` or ``weight_col`` is not a column of ``data``.
    ValueError
        If the weights contain NaN, infinity or a negative value, or sum to zero or less.

    Notes
    -----
    ``AUC``, ``KS``, ``LIFT``, ``IV`` and ``avgScore`` use the non-special rows only, because sentinel scores carry no
    ordering information; ``avgTrue``, ``N`` and ``N_RAW`` cover all rows. ``AUC`` and ``KS`` are NaN when they cannot be
    computed (for example with a single class).
    """
    weight = resolve_weights(data, weight_col=weight_col, expected_len=len(data))
    y_true = data[tgt_name].to_numpy()
    y_score = data[scr_name].to_numpy()
    # Ranking metrics (AUC/KS/avgScore) are computed on non-special rows only:
    # sentinel scores like -1 carry no ordering information.
    rank_data, special_part = _split_special_scores(data, scr_name, spec_values)
    rank_weight = resolve_weights(rank_data, weight_col=weight_col, expected_len=len(rank_data))
    rank_true = rank_data[tgt_name].to_numpy()
    rank_score = rank_data[scr_name].to_numpy()
    roc_df = calc_roc(rank_true, rank_score, sample_weight=rank_weight)
    gains = get_gains_table(
        data, tgt_name, scr_name, nbins=nbins, weight_col=weight_col,
        ascending=ascending, spec_values=spec_values,
    )
    summary = {
        "index": name,
        "dataset": name,
        "DATASET": name,
        "AUC": safe_auc(rank_true, rank_score, sample_weight=rank_weight),
        "KS": float(roc_df["KS"].max()) if "KS" in roc_df else np.nan,
        "LIFT": float(gains["LIFT"].max()) if "LIFT" in gains else np.nan,
        "IV": float(gains["IV"].sum()) if "IV" in gains else np.nan,
        "N": float(np.sum(weight)) if weight is not None else float(len(data)),
        "N_RAW": int(len(data)),
        "avgTrue": safe_weighted_average(y_true, weight),
        "avgScore": safe_weighted_average(rank_score, rank_weight),
    }
    if special_part is not None:
        spec_weight = resolve_weights(special_part, weight_col=weight_col, expected_len=len(special_part))
        summary["N_SPECIAL"] = (
            float(np.sum(spec_weight)) if spec_weight is not None else float(len(special_part))
        )
        summary["N_SPECIAL_RAW"] = int(len(special_part))
    return summary

get_perf_summary

get_perf_summary(train=None, validation=None, oot=None, tgt_name=None, scr_name=None, weight_col=None, nbins=10, ascending=False, spec_values=None, **kwargs)

Performance summary of the train, validation and OOT datasets, one row each.

Parameters:

Name	Type	Description	Default
train	DataFrame or None	Training dataset, reported in the row ins; skipped when None.	None
validation	DataFrame or None	Validation dataset, reported in the row oos; skipped when None.	None
oot	DataFrame or None	Out-of-time dataset, reported in the row oot; skipped when None.	None
tgt_name	str or None	Name of the target column (binary, 1 for bad); required as soon as a dataset is given.	None
scr_name	str or None	Name of the score column; required as soon as a dataset is given.	None
weight_col	str or None	Name of the sample weight column, which must exist in every given dataset; None gives every row the weight 1.	None
nbins	int	Number of bins of the Gains table behind LIFT and IV.	10
ascending	bool	Bin order of that Gains table (see get_gains_table).	False
spec_values	list or None	Special score values (such as -1) that are left out of the ranking statistics (see dataset_summary).	None
**kwargs		Accepted and ignored.	{}

Returns:

Type	Description
DataFrame	One row per given dataset, in the order ins, oos, oot, with the entries of dataset_summary as columns (index, dataset, DATASET, AUC, KS, LIFT, IV, N, N_RAW, avgTrue, avgScore and, when special scores occur, N_SPECIAL and N_SPECIAL_RAW). A DataFrame without rows or columns when no dataset is given.

Raises:

Type	Description
KeyError	If a target, score or weight column is missing from a given dataset.
ValueError	If the weights are invalid (see dataset_summary).

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def get_perf_summary(train=None, validation=None, oot=None, tgt_name=None, scr_name=None, weight_col=None, nbins=10,
                     ascending=False, spec_values=None, **kwargs):
    """Performance summary of the train, validation and OOT datasets, one row each.

    Parameters
    ----------
    train : pandas.DataFrame or None, default None
        Training dataset, reported in the row ``ins``; skipped when None.
    validation : pandas.DataFrame or None, default None
        Validation dataset, reported in the row ``oos``; skipped when None.
    oot : pandas.DataFrame or None, default None
        Out-of-time dataset, reported in the row ``oot``; skipped when None.
    tgt_name : str or None, default None
        Name of the target column (binary, 1 for bad); required as soon as a dataset is given.
    scr_name : str or None, default None
        Name of the score column; required as soon as a dataset is given.
    weight_col : str or None, default None
        Name of the sample weight column, which must exist in every given dataset; None gives every row the weight 1.
    nbins : int, default 10
        Number of bins of the Gains table behind ``LIFT`` and ``IV``.
    ascending : bool, default False
        Bin order of that Gains table (see ``get_gains_table``).
    spec_values : list or None, default None
        Special score values (such as ``-1``) that are left out of the ranking statistics (see ``dataset_summary``).
    **kwargs
        Accepted and ignored.

    Returns
    -------
    pandas.DataFrame
        One row per given dataset, in the order ``ins``, ``oos``, ``oot``, with the entries of ``dataset_summary`` as
        columns (``index``, ``dataset``, ``DATASET``, ``AUC``, ``KS``, ``LIFT``, ``IV``, ``N``, ``N_RAW``, ``avgTrue``,
        ``avgScore`` and, when special scores occur, ``N_SPECIAL`` and ``N_SPECIAL_RAW``). A DataFrame without rows or
        columns when no dataset is given.

    Raises
    ------
    KeyError
        If a target, score or weight column is missing from a given dataset.
    ValueError
        If the weights are invalid (see ``dataset_summary``).
    """
    rows = []
    for name, data in (("ins", train), ("oos", validation), ("oot", oot)):
        if data is not None:
            rows.append(dataset_summary(
                name, data, tgt_name, scr_name, weight_col=weight_col, nbins=nbins,
                ascending=ascending, spec_values=spec_values,
            ))
    return pd.DataFrame(rows)

evaluate_performance

evaluate_performance(datasets=None, tgt_name=None, scr_name=None, sample_weight=None, nbins=10, **kwargs)

Performance summary of several named datasets given as DataFrames or as label/score arrays.

Parameters:

Name	Type	Description	Default
datasets	dict or None	Mapping of dataset name to a payload dict. A payload is either {"data": DataFrame} (the frame must hold the tgt_name and scr_name columns; a payload whose data is None is skipped) or {"y_true": array-like, "y_score": array-like}. Either form may carry a "sample_weight" entry that overrides the sample_weight argument for that dataset. None gives an empty result.	None
tgt_name	str or None	Name of the target column (binary, 1 for bad); for array payloads, the name given to the target column of the frame built from them.	None
scr_name	str or None	Name of the score column; for array payloads, the name given to the score column of the frame built from them.	None
sample_weight	array - like or None	Default weights, one per row, for the datasets whose payload has no "sample_weight" entry. None gives every row the weight 1.	None
nbins	int	Number of bins of the Gains table behind LIFT and IV.	10
**kwargs		Accepted and ignored.	{}

Returns:

Type	Description
DataFrame	One row per dataset, in the order of datasets, with the entries of dataset_summary as columns (index, dataset, DATASET, AUC, KS, LIFT, IV, N, N_RAW, avgTrue, avgScore). A DataFrame without rows or columns when datasets is None or empty.

Raises:

Type	Description
KeyError	If an array payload lacks "y_true" or "y_score", or a frame lacks the target or score column.
ValueError	If the weights are invalid: wrong length, NaN, infinity, negative values or a sum that is not positive.

Notes

Each frame is copied and given the weight column _w (overwriting any column of that name). ascending and spec_values are not available here: the Gains table behind LIFT and IV uses ascending=False and no special values.

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def evaluate_performance(datasets=None, tgt_name=None, scr_name=None, sample_weight=None, nbins=10, **kwargs):
    """Performance summary of several named datasets given as DataFrames or as label/score arrays.

    Parameters
    ----------
    datasets : dict or None, default None
        Mapping of dataset name to a payload dict. A payload is either ``{"data": DataFrame}`` (the frame must hold the
        ``tgt_name`` and ``scr_name`` columns; a payload whose ``data`` is None is skipped) or
        ``{"y_true": array-like, "y_score": array-like}``. Either form may carry a ``"sample_weight"`` entry that
        overrides the ``sample_weight`` argument for that dataset. None gives an empty result.
    tgt_name : str or None, default None
        Name of the target column (binary, 1 for bad); for array payloads, the name given to the target column of the frame
        built from them.
    scr_name : str or None, default None
        Name of the score column; for array payloads, the name given to the score column of the frame built from them.
    sample_weight : array-like or None, default None
        Default weights, one per row, for the datasets whose payload has no ``"sample_weight"`` entry. None gives every
        row the weight 1.
    nbins : int, default 10
        Number of bins of the Gains table behind ``LIFT`` and ``IV``.
    **kwargs
        Accepted and ignored.

    Returns
    -------
    pandas.DataFrame
        One row per dataset, in the order of ``datasets``, with the entries of ``dataset_summary`` as columns (``index``,
        ``dataset``, ``DATASET``, ``AUC``, ``KS``, ``LIFT``, ``IV``, ``N``, ``N_RAW``, ``avgTrue``, ``avgScore``). A
        DataFrame without rows or columns when ``datasets`` is None or empty.

    Raises
    ------
    KeyError
        If an array payload lacks ``"y_true"`` or ``"y_score"``, or a frame lacks the target or score column.
    ValueError
        If the weights are invalid: wrong length, NaN, infinity, negative values or a sum that is not positive.

    Notes
    -----
    Each frame is copied and given the weight column ``_w`` (overwriting any column of that name). ``ascending`` and
    ``spec_values`` are not available here: the Gains table behind ``LIFT`` and ``IV`` uses ``ascending=False`` and no
    special values.
    """
    rows = []
    if datasets is None:
        return pd.DataFrame(rows)
    for name, payload in datasets.items():
        if isinstance(payload, dict) and "data" in payload:
            data = payload.get("data")
            weight = payload.get("sample_weight", sample_weight)
        else:
            data = pd.DataFrame(
                {
                    tgt_name: payload["y_true"],
                    scr_name: payload["y_score"],
                }
            )
            weight = payload.get("sample_weight", sample_weight)
        if data is None:
            continue
        tmp = data.copy()
        tmp["_w"] = np.ones(len(tmp), dtype=float) if weight is None else weight
        rows.append(dataset_summary(name, tmp, tgt_name, scr_name, weight_col="_w", nbins=nbins))
    return pd.DataFrame(rows)

cross_risk_weighted_mean

cross_risk_weighted_mean(data, agg_col, sample_weight, score_list, margin_name='Total_Avg_Risk')

Weighted-mean cross-risk table after bin columns are assigned.

Parameters:

Name	Type	Description	Default
data	DataFrame	Dataset that already holds the bin columns _bin_num1, _bin_range1 (first score) and _bin_num2, _bin_range2 (second score), and the column agg_col.	required
agg_col	str	Name of the column to average (for example the target or a risk measure); converted with pandas.to_numeric(errors="coerce"), so non-numeric entries become NaN.	required
sample_weight	array - like	Weights, one per row of data and aligned with it by position; converted to float and not validated.	required
score_list	list of str	Names of the two scores, [first, second]; used as the names of the row index levels (first) and of the column index levels (second).	required
margin_name	str	Label of the margin row and column, which hold the weighted means over a whole row, a whole column and the whole table.	"Total_Avg_Risk"

Returns:

Type	Description
DataFrame	For every cell, the sum of weight * agg_col divided by the sum of the weights: the rows are indexed by (_bin_num1, _bin_range1) and the columns by (_bin_num2, _bin_range2), each with two levels named after score_list. Cells without rows, whose weights sum to zero, or whose agg_col values are all NaN, are NaN.

Notes

A row whose agg_col value is NaN is skipped: it adds nothing to the numerator and its weight is left out of the denominator, so the result is the weighted mean of the values that exist.

Source code in Modeling_Tool/Eval/weighted_eval_utils.py

def cross_risk_weighted_mean(data, agg_col, sample_weight, score_list, margin_name="Total_Avg_Risk"):
    """Weighted-mean cross-risk table after bin columns are assigned.

    Parameters
    ----------
    data : pandas.DataFrame
        Dataset that already holds the bin columns ``_bin_num1``, ``_bin_range1`` (first score) and ``_bin_num2``,
        ``_bin_range2`` (second score), and the column ``agg_col``.
    agg_col : str
        Name of the column to average (for example the target or a risk measure); converted with
        ``pandas.to_numeric(errors="coerce")``, so non-numeric entries become NaN.
    sample_weight : array-like
        Weights, one per row of ``data`` and aligned with it by position; converted to float and not validated.
    score_list : list of str
        Names of the two scores, ``[first, second]``; used as the names of the row index levels (``first``) and of the
        column index levels (``second``).
    margin_name : str, default "Total_Avg_Risk"
        Label of the margin row and column, which hold the weighted means over a whole row, a whole column and the whole
        table.

    Returns
    -------
    pandas.DataFrame
        For every cell, the sum of ``weight * agg_col`` divided by the sum of the weights: the rows are indexed by
        (``_bin_num1``, ``_bin_range1``) and the columns by (``_bin_num2``, ``_bin_range2``), each with two levels named
        after ``score_list``. Cells without rows, whose weights sum to zero, or whose ``agg_col`` values are all NaN, are NaN.

    Notes
    -----
    A row whose ``agg_col`` value is NaN is skipped: it adds nothing to the numerator and its weight is left out of the
    denominator, so the result is the weighted mean of the values that exist.
    """
    weight = np.asarray(sample_weight, dtype=float)
    values = pd.to_numeric(data[agg_col], errors="coerce").to_numpy(dtype=float)
    valid = ~np.isnan(values)
    frame = data[["_bin_num1", "_bin_range1", "_bin_num2", "_bin_range2"]].copy()
    frame["_w"] = np.where(valid, weight, 0.0)
    frame["_wv"] = np.where(valid, values * weight, 0.0)

    numerator = pd.crosstab(
        [frame["_bin_num1"], frame["_bin_range1"]],
        [frame["_bin_num2"], frame["_bin_range2"]],
        values=frame["_wv"],
        aggfunc="sum",
        margins=True,
        margins_name=margin_name,
        rownames=[score_list[0], score_list[0]],
        colnames=[score_list[1], score_list[1]],
    )
    denominator = pd.crosstab(
        [frame["_bin_num1"], frame["_bin_range1"]],
        [frame["_bin_num2"], frame["_bin_range2"]],
        values=frame["_w"],
        aggfunc="sum",
        margins=True,
        margins_name=margin_name,
        rownames=[score_list[0], score_list[0]],
        colnames=[score_list[1], score_list[1]],
    )
    return numerator / denominator.replace(0, np.nan)