Modeling_Tool.Feature

Feature analysis layer: PSI, IV/KS insights, correlation filtering, unified feature screening, and distribution analysis.

PSICalculator, VarExtractionInsights, and CorrelationFilter are wrappers

The classes you import from Modeling_Tool or Modeling_Tool.Feature are defined in WOE_Engine_Feature_Patch. They wrap the implementations in PSI_Tool and Feature_Insights and add the arguments that reuse a fitted WOE engine: binning_engine for PSICalculator, and woe_engine, woe_binner, and woe_engine_params for VarExtractionInsights and CorrelationFilter. Every other argument and every method is the one documented for the implementation below.

PSI Population Stability: PSI_Tool

PSI_Tool

PSI (Population Stability Index) Calculator Module

This module provides functions and classes for calculating Population Stability Index (PSI) to measure the distribution drift between expected and actual datasets.

Author: Matrix Agent

PSICalculator

A class for calculating Population Stability Index (PSI) with configurable parameters.

This class encapsulates common PSI calculation parameters and provides methods for various PSI calculations including single variable, grouped, and multi-variable comparisons between datasets.

Parameters:

Name	Type	Description	Default
buckets	int	Number of bins for binning. Default is 10.	10
equal_freq	bool	Whether to use equal frequency binning. Default is True.	True
min_bin_prop	float	Minimum proportion for each bin. Default is 0.05.	0.05
content	float	Small value to avoid division by zero. Default is 1e-6.	1e-06
precision	int	Decimal precision for results. Default is 5.	5
missing_policy	(drop, include, warn_and_drop)	How NaN rows are handled. Default is "include", which routes NaN rows through a dedicated "MISSING" bin so missing-rate drift contributes to the PSI. See __init__ for the semantics of each mode.	"drop"
psi_missing_bucket_policy	(smooth_laplace, floor_1e6, exclude)	How a bin that holds rows on one side only is treated. Default is "smooth_laplace". See __init__.	"smooth_laplace"
feature_block_size	int or None	Number of variables processed per block. Default is 64. This class only validates and stores it (a positive integer or None); it has no effect on the calculation here.	64

Examples:

>>> calculator = PSICalculator(buckets=10, equal_freq=True)
>>> psi = calculator.calculate(expected_df, actual_df, 'score')

Source code in Modeling_Tool/Feature/PSI_Tool.py

class PSICalculator:
    """
    A class for calculating Population Stability Index (PSI) with configurable parameters.

    This class encapsulates common PSI calculation parameters and provides methods
    for various PSI calculations including single variable, grouped, and multi-variable
    comparisons between datasets.

    Parameters
    ----------
    buckets : int, optional
        Number of bins for binning. Default is 10.
    equal_freq : bool, optional
        Whether to use equal frequency binning. Default is True.
    min_bin_prop : float, optional
        Minimum proportion for each bin. Default is 0.05.
    content : float, optional
        Small value to avoid division by zero. Default is 1e-6.
    precision : int, optional
        Decimal precision for results. Default is 5.
    missing_policy : {"drop", "include", "warn_and_drop"}, optional
        How NaN rows are handled. Default is "include", which routes NaN rows through a dedicated "__MISSING__" bin
        so missing-rate drift contributes to the PSI. See ``__init__`` for the semantics of each mode.
    psi_missing_bucket_policy : {"smooth_laplace", "floor_1e6", "exclude"}, optional
        How a bin that holds rows on one side only is treated. Default is "smooth_laplace". See ``__init__``.
    feature_block_size : int or None, optional
        Number of variables processed per block. Default is 64. This class only validates and stores it (a positive
        integer or None); it has no effect on the calculation here.

    Examples
    --------
    >>> calculator = PSICalculator(buckets=10, equal_freq=True)
    >>> psi = calculator.calculate(expected_df, actual_df, 'score')
    """

    def __init__(
        self,
        buckets: int = 10,
        equal_freq: bool = True,
        min_bin_prop: float = 0.05,
        content: float = 1e-6,
        precision: int = 5,
        missing_policy: str = "include",
        psi_missing_bucket_policy: str = "smooth_laplace",
        feature_block_size: int | None = 64,
    ):
        """
        Initialize PSICalculator with configuration parameters.

        Parameters
        ----------
        buckets : int, optional
            Number of bins for binning. Default is 10.
        equal_freq : bool, optional
            Use equal frequency binning if True. Default is True.
        min_bin_prop : float, optional
            Minimum proportion for each bin. Default is 0.05.
        content : float, optional
            Small value to prevent division by zero. Default is 1e-6.
        precision : int, optional
            Decimal precision for rounding. Default is 5.
        missing_policy : {"drop", "include", "warn_and_drop"}, optional
            How NaN rows are handled. Default in 0.5.0 is "include", which routes
            NaN rows through a dedicated "__MISSING__" bin so missing-rate drift
            contributes to the PSI. This is the recommended production behaviour
            and became the default in 0.5.0 (previously "drop" in 0.4.2 for
            numeric backward-compat). Pass "drop" to reproduce pre-0.5.0
            numbers, or "warn_and_drop" for legacy numbers with a RuntimeWarning
            naming the NaN counts.
        psi_missing_bucket_policy : {"smooth_laplace", "floor_1e6", "exclude"}, optional
            How a bin that holds rows on one side only (for example a category that disappears) is treated.
            Default is "smooth_laplace": when such a bin exists, one is added to every bin count before the shares
            are computed. "floor_1e6" clips the bin shares at ``content`` (``1e-6`` by default), so the empty side
            counts as ``content``. "exclude" drops the one-sided bins from the sum. Any other value raises
            ``ValueError``.
        feature_block_size : int or None, optional
            Number of variables processed per block. Default is 64. This class only validates and stores it (a
            positive integer or None, otherwise ``ValueError``); it has no effect on the calculation here.

        Raises
        ------
        ValueError
            If ``missing_policy`` or ``psi_missing_bucket_policy`` is not one of the listed values, or if
            ``feature_block_size`` is not None and not a positive integer.
        """
        if missing_policy not in {"include", "drop", "warn_and_drop"}:
            raise ValueError(
                f"PSICalculator.__init__: missing_policy must be one of "
                f"'include', 'drop', 'warn_and_drop'; got {missing_policy!r}."
            )
        _validate_psi_bucket_policy(psi_missing_bucket_policy, "PSICalculator.__init__")
        if feature_block_size is not None and int(feature_block_size) <= 0:
            raise ValueError("feature_block_size must be a positive integer or None")
        self.buckets = buckets
        self.equal_freq = equal_freq
        self.min_bin_prop = min_bin_prop
        self.content = content
        self.precision = precision
        self.missing_policy = missing_policy
        self.psi_missing_bucket_policy = psi_missing_bucket_policy
        self.feature_block_size = feature_block_size

#     def _calculate_single_psi(
#         self,
#         expected_series: pd.Series,
#         actual_series: pd.Series,
#         return_details: bool = False
#     ) -> Union[float, Tuple[float, pd.DataFrame]]:
#         """
#         Calculate PSI for a single variable.

#         This method performs binning on both expected and actual series,
#         then calculates the PSI value based on the distribution difference.

#         Parameters
#         ----------
#         expected_series : pandas.Series
#             Expected/baseline data series.
#         actual_series : pandas.Series
#             Actual/comparison data series.
#         return_details : bool, optional
#             Whether to return detailed bin information. Default is False.

#         Returns
#         -------
#         float or tuple
#             If return_details is False: Returns total PSI value.
#             If return_details is True: Returns tuple of (PSI value, details DataFrame).
#         """
#         # Drop NA
#         expected_clean = expected_series.dropna()
#         actual_clean = actual_series.dropna()

#         # Bin the expected data to get breakpoints
#         expected_bins, breakpoints = quick_binning(
#             pd.DataFrame(expected_clean), 
#             expected_clean.name, 
#             labels=None, 
#             nbins=self.buckets, 
#             precision=self.precision, 
#             equal_freq=self.equal_freq, 
#             right=True, 
#             include_lowest=False,
#             min_bin_prop=self.min_bin_prop, 
#             tree_binning=False, 
#             target=None, 
#             random_state=42
#         )

#         # Bin the actual data using the same breakpoints
#         actual_bins, _ = quick_binning(
#             pd.DataFrame(actual_clean), 
#             actual_clean.name, 
#             labels=None, 
#             nbins=list(breakpoints), 
#             precision=self.precision, 
#             equal_freq=self.equal_freq, 
#             right=True, 
#             include_lowest=False,
#             min_bin_prop=self.min_bin_prop, 
#             tree_binning=False, 
#             target=None, 
#             random_state=42
#         )

#         # Get bin proportions
#         expected_percents = expected_bins.value_counts(normalize=True, sort=False)
#         actual_percents = actual_bins.value_counts(normalize=True, sort=False)

#         # Ensure both series have the same bin indices
#         all_bins = expected_percents.index.union(actual_percents.index)
#         expected_percents = expected_percents.reindex(all_bins, fill_value=self.content)
#         actual_percents = actual_percents.reindex(all_bins, fill_value=self.content)

#         # Clip to avoid division by zero
#         expected_percents = expected_percents.clip(lower=self.content)
#         actual_percents = actual_percents.clip(lower=self.content)

#         # Calculate PSI
#         psi_values = (actual_percents - expected_percents) * np.log(actual_percents / expected_percents)
#         psi_total = psi_values.sum()

#         if return_details:
#             details = pd.DataFrame({
#                 'expected_percent': expected_percents,
#                 'actual_percent': actual_percents,
#                 'psi_component': psi_values
#             })
#             return psi_total, details
#         else:
#             return psi_total

#     def calculate_psi(
#         self,
#         expected: Union[pd.DataFrame, pd.Series],
#         actual: Union[pd.DataFrame, pd.Series],
#         target_col: str,
#         group_by: Optional[Union[str, List[str]]] = None,
#         return_details: bool = False
#     ) -> Union[float, pd.DataFrame, Tuple[Dict, Dict]]:
#         """
#         Calculate PSI for a variable, optionally by groups.

#         Parameters
#         ----------
#         expected : pandas.DataFrame or pandas.Series
#             Expected/baseline data.
#         actual : pandas.DataFrame or pandas.Series
#             Actual/comparison data.
#         target_col : str
#             Column name to calculate PSI for.
#         group_by : str or list, optional
#             Column(s) to group by. Default is None (no grouping).
#         return_details : bool, optional
#             Whether to return detailed bin information. Default is False.

#         Returns
#         -------
#         float, pandas.DataFrame, or tuple
#             PSI value(s) and optionally details.
#         """
#         if group_by is not None:
#             if isinstance(group_by, str):
#                 group_by = [group_by]

#             expected_subset = expected[[target_col] + group_by].copy()
#             actual_subset = actual[[target_col] + group_by].copy()

#             expected_subset = expected_subset.copy()
#             actual_subset = actual_subset.copy()

#             expected_subset['_dataset'] = 'expected'
#             actual_subset['_dataset'] = 'actual'

#             combined = pd.concat([expected_subset, actual_subset], ignore_index=True)

#             results = {}
#             details_dict = {}

#             for group, group_data in combined.groupby(group_by):
#                 expected_group = group_data[group_data['_dataset'] == 'expected'].drop('_dataset', axis=1)
#                 actual_group = group_data[group_data['_dataset'] == 'actual'].drop('_dataset', axis=1)

#                 if expected_group.shape[0] == 0 or actual_group.shape[0] == 0:
#                     results[group] = 999999
#                     continue

#                 if return_details:
#                     psi_value, detail = self._calculate_single_psi(
#                         expected_group[target_col], 
#                         actual_group[target_col],
#                         return_details=True
#                     )
#                     results[group] = psi_value
#                     details_dict[group] = detail
#                 else:
#                     results[group] = self._calculate_single_psi(
#                         expected_group[target_col], 
#                         actual_group[target_col]
#                     )

#             if return_details:
#                 return results, details_dict
#             else:
#                 return pd.DataFrame(results, index=['psi']).T

#         else:
#             if return_details:
#                 return self._calculate_single_psi(
#                     expected[target_col], 
#                     actual[target_col],
#                     return_details=True
#                 )
#             else:
#                 return self._calculate_single_psi(
#                     expected[target_col], 
#                     actual[target_col]
#                 )

#     def calculate_within_psi(
#         self,
#         data: pd.DataFrame,
#         grp_name: str,
#         target_col: str,
#         benchmark: Optional[Any] = None,
#         return_details: bool = False,
#         benchmark_display_name: Optional[str] = None
#     ) -> Union[pd.DataFrame, Dict]:
#         """
#         Calculate PSI values within a single dataset, comparing groups to a benchmark.

#         Parameters
#         ----------
#         data : pandas.DataFrame
#             Input dataset containing all groups.
#         grp_name : str
#             Column name for grouping.
#         target_col : str
#             Column name to calculate PSI for.
#         benchmark : str or callable, optional
#             Benchmark group value or filter function. If None, uses first group.
#         return_details : bool, optional
#             Whether to return detailed bin information. Default is False.
#         benchmark_display_name : str, optional
#             Custom name for benchmark in results.

#         Returns
#         -------
#         pandas.DataFrame or dict
#             PSI results by group, or dict with 'psi' and 'details' keys.
#         """
#         if callable(benchmark):
#             benchmark_data = data[benchmark(data)]
#         else:
#             benchmark_data = data[data[grp_name] == benchmark] if benchmark is not None else data

#         obs_values = [x for x in data[grp_name].unique().tolist() if x != benchmark]

#         res_dict = {benchmark_display_name: 0} if benchmark_display_name is not None else {}
#         detail_dict = {}

#         for obs_value in obs_values:
#             obs_data = data[data[grp_name] == obs_value]

#             if return_details:
#                 psi, details = self.calculate_psi(
#                     benchmark_data, 
#                     obs_data, 
#                     target_col=target_col, 
#                     return_details=True
#                 )
#                 res_dict[obs_value] = psi
#                 detail_dict[obs_value] = details
#             else:
#                 psi = self.calculate_psi(
#                     benchmark_data, 
#                     obs_data, 
#                     target_col=target_col
#                 )
#                 res_dict[obs_value] = psi

#         fnl_res = pd.DataFrame(res_dict, index=['psi']).T\
#             .reset_index(drop=False)\
#             .rename(columns={"index": grp_name})

#         if return_details:
#             return {"psi": fnl_res, "details": detail_dict}
#         else:
#             return fnl_res

#     def calculate_psi_within_dataset(
#         self,
#         data: pd.DataFrame,
#         grp_name: str,
#         varlist: List[str],
#         benchmark: Optional[Any] = None
#     ) -> pd.DataFrame:
#         """
#         Calculate PSI for multiple variables within a dataset.

#         Parameters
#         ----------
#         data : pandas.DataFrame
#             Input dataset.
#         grp_name : str
#             Column name for grouping.
#         varlist : list
#             List of variable names to calculate PSI for.
#         benchmark : str or callable, optional
#             Benchmark group value or filter function.

#         Returns
#         -------
#         pandas.DataFrame
#             Combined PSI results for all variables.
#         """
#         fnl_psi_res = []
#         for var in tqdm(varlist):
#             single_psi = self.calculate_within_psi(
#                 data=data, 
#                 grp_name=grp_name, 
#                 benchmark=benchmark, 
#                 target_col=var
#             ).sort_values([grp_name]).reset_index(drop=True)

#             single_psi['var'] = var
#             fnl_psi_res.append(single_psi)

#         return pd.concat(fnl_psi_res)

#     def calculate_multivar_psi_two_sets(
#         self,
#         expected_df: pd.DataFrame,
#         actual_df: pd.DataFrame,
#         varlist: List[str],
#         group_by: Optional[Union[str, List[str]]] = None
#     ) -> pd.DataFrame:
#         """
#         Calculate PSI for multiple variables by comparing two datasets.

#         Parameters
#         ----------
#         expected_df : pandas.DataFrame
#             Expected/baseline dataset.
#         actual_df : pandas.DataFrame
#             Actual/comparison dataset.
#         varlist : list
#             List of variable names to calculate PSI for.
#         group_by : str or list, optional
#             Column(s) to group by.

#         Returns
#         -------
#         pandas.DataFrame
#             PSI results for all variables.
#         """
#         multi_psi_res = []
#         for var in tqdm(varlist):
#             single_psi = self.calculate_psi(
#                 expected=expected_df, 
#                 actual=actual_df, 
#                 target_col=var, 
#                 group_by=group_by
#             )
#             if group_by is None:
#                 single_psi = pd.DataFrame([single_psi], columns=['psi'])
#             single_psi['var'] = var
#             multi_psi_res.append(single_psi)

#         return pd.concat(multi_psi_res)

    def calculate(
        self,
        expected_df: pd.DataFrame,
        current_data: pd.DataFrame,
        varlist: List[str],
        group_by: Optional[str] = None,
        group_name: Optional[str] = None,
        return_details = False,
        missing_policy: Optional[str] = None,
        psi_missing_bucket_policy: Optional[str] = None,
    ) -> pd.DataFrame:
        """
        Calculate grouped PSI comparing two datasets, using expected as benchmark.

        Parameters
        ----------
        expected_df : pandas.DataFrame
            Expected/baseline dataset.
        current_data : pandas.DataFrame
            Actual/comparison dataset.
        varlist : list
            List of variable names.
        group_by : str, optional
            Column to group by in both datasets. Default is None. Legacy argument: without ``group_name`` it has no
            effect on the default result (use ``group_name`` to get PSI by group); together with ``group_name`` the
            PSI is computed per ``group_by`` value inside each ``group_name`` group.
        group_name : str, optional
            Specific group column name for multi-group calculation. Default is None. It must be a column of
            ``current_data``: every value (NaN becomes "__NULL__") defines a group that is compared with the whole
            ``expected_df``, and the bins are built once on ``expected_df``.
        return_details : bool, optional
            Whether to return detailed bin information. Default is False. If True, a dict ``{'psi': psi_df,
            'details': details_df}`` is returned, where ``details_df`` has one row per bin (and variable, and group)
            with the columns ``bin``, ``expected_count``, ``actual_count``, ``expected_percent``,
            ``actual_percent``, ``psi_component``, ``bucket_status``, ``psi_missing_bucket_policy`` (and the group
            column) and ``var``.
        missing_policy : {"drop", "include", "warn_and_drop"}, optional
            How NaN rows are handled. If None (default), falls back to the value
            configured on ``self.missing_policy``. Pass an explicit value to
            override the class-level default for this call only. See
            ``PSICalculator.__init__`` for the semantics of each mode.
        psi_missing_bucket_policy : {"smooth_laplace", "floor_1e6", "exclude"}, optional
            How a bin that holds rows on one side only is treated. If None (default), falls back to the value
            configured on ``self.psi_missing_bucket_policy``. Pass an explicit value to override it for this call
            only. See ``PSICalculator.__init__`` for the semantics of each policy.

        Returns
        -------
        pandas.DataFrame or dict
            Grouped PSI results: a DataFrame with the columns ``var`` and ``psi`` (plus the ``group_name`` column
            when it is given). If ``return_details`` is True, the dict ``{'psi': DataFrame, 'details': DataFrame}``.
        """
        effective_policy = missing_policy if missing_policy is not None else self.missing_policy
        effective_bucket_policy = (
            psi_missing_bucket_policy
            if psi_missing_bucket_policy is not None
            else self.psi_missing_bucket_policy
        )
        return calculate_multigroup_psi_two_sets(
            expected_df = expected_df,
            actual_df = current_data,
            varlist = varlist,
            group_by = group_by,
            buckets = self.buckets,
            equal_freq = self.equal_freq,
            min_bin_prop = self.min_bin_prop,
            content = self.content,
            precision = self.precision,
            group_name = group_name,
            return_details = return_details,
            missing_policy = effective_policy,
            psi_missing_bucket_policy = effective_bucket_policy,
        )

calculate

calculate(expected_df: DataFrame, current_data: DataFrame, varlist: List[str], group_by: Optional[str] = None, group_name: Optional[str] = None, return_details=False, missing_policy: Optional[str] = None, psi_missing_bucket_policy: Optional[str] = None) -> DataFrame

Calculate grouped PSI comparing two datasets, using expected as benchmark.

Parameters:

Name	Type	Description	Default
expected_df	DataFrame	Expected/baseline dataset.	required
current_data	DataFrame	Actual/comparison dataset.	required
varlist	list	List of variable names.	required
group_by	str	Column to group by in both datasets. Default is None. Legacy argument: without group_name it has no effect on the default result (use group_name to get PSI by group); together with group_name the PSI is computed per group_by value inside each group_name group.	None
group_name	str	Specific group column name for multi-group calculation. Default is None. It must be a column of current_data: every value (NaN becomes "NULL") defines a group that is compared with the whole expected_df, and the bins are built once on expected_df.	None
return_details	bool	Whether to return detailed bin information. Default is False. If True, a dict {'psi': psi_df, 'details': details_df} is returned, where details_df has one row per bin (and variable, and group) with the columns bin, expected_count, actual_count, expected_percent, actual_percent, psi_component, bucket_status, psi_missing_bucket_policy (and the group column) and var.	False
missing_policy	(drop, include, warn_and_drop)	How NaN rows are handled. If None (default), falls back to the value configured on self.missing_policy. Pass an explicit value to override the class-level default for this call only. See PSICalculator.__init__ for the semantics of each mode.	"drop"
psi_missing_bucket_policy	(smooth_laplace, floor_1e6, exclude)	How a bin that holds rows on one side only is treated. If None (default), falls back to the value configured on self.psi_missing_bucket_policy. Pass an explicit value to override it for this call only. See PSICalculator.__init__ for the semantics of each policy.	"smooth_laplace"

Returns:

Type	Description
DataFrame or dict	Grouped PSI results: a DataFrame with the columns var and psi (plus the group_name column when it is given). If return_details is True, the dict {'psi': DataFrame, 'details': DataFrame}.

Source code in Modeling_Tool/Feature/PSI_Tool.py

def calculate(
    self,
    expected_df: pd.DataFrame,
    current_data: pd.DataFrame,
    varlist: List[str],
    group_by: Optional[str] = None,
    group_name: Optional[str] = None,
    return_details = False,
    missing_policy: Optional[str] = None,
    psi_missing_bucket_policy: Optional[str] = None,
) -> pd.DataFrame:
    """
    Calculate grouped PSI comparing two datasets, using expected as benchmark.

    Parameters
    ----------
    expected_df : pandas.DataFrame
        Expected/baseline dataset.
    current_data : pandas.DataFrame
        Actual/comparison dataset.
    varlist : list
        List of variable names.
    group_by : str, optional
        Column to group by in both datasets. Default is None. Legacy argument: without ``group_name`` it has no
        effect on the default result (use ``group_name`` to get PSI by group); together with ``group_name`` the
        PSI is computed per ``group_by`` value inside each ``group_name`` group.
    group_name : str, optional
        Specific group column name for multi-group calculation. Default is None. It must be a column of
        ``current_data``: every value (NaN becomes "__NULL__") defines a group that is compared with the whole
        ``expected_df``, and the bins are built once on ``expected_df``.
    return_details : bool, optional
        Whether to return detailed bin information. Default is False. If True, a dict ``{'psi': psi_df,
        'details': details_df}`` is returned, where ``details_df`` has one row per bin (and variable, and group)
        with the columns ``bin``, ``expected_count``, ``actual_count``, ``expected_percent``,
        ``actual_percent``, ``psi_component``, ``bucket_status``, ``psi_missing_bucket_policy`` (and the group
        column) and ``var``.
    missing_policy : {"drop", "include", "warn_and_drop"}, optional
        How NaN rows are handled. If None (default), falls back to the value
        configured on ``self.missing_policy``. Pass an explicit value to
        override the class-level default for this call only. See
        ``PSICalculator.__init__`` for the semantics of each mode.
    psi_missing_bucket_policy : {"smooth_laplace", "floor_1e6", "exclude"}, optional
        How a bin that holds rows on one side only is treated. If None (default), falls back to the value
        configured on ``self.psi_missing_bucket_policy``. Pass an explicit value to override it for this call
        only. See ``PSICalculator.__init__`` for the semantics of each policy.

    Returns
    -------
    pandas.DataFrame or dict
        Grouped PSI results: a DataFrame with the columns ``var`` and ``psi`` (plus the ``group_name`` column
        when it is given). If ``return_details`` is True, the dict ``{'psi': DataFrame, 'details': DataFrame}``.
    """
    effective_policy = missing_policy if missing_policy is not None else self.missing_policy
    effective_bucket_policy = (
        psi_missing_bucket_policy
        if psi_missing_bucket_policy is not None
        else self.psi_missing_bucket_policy
    )
    return calculate_multigroup_psi_two_sets(
        expected_df = expected_df,
        actual_df = current_data,
        varlist = varlist,
        group_by = group_by,
        buckets = self.buckets,
        equal_freq = self.equal_freq,
        min_bin_prop = self.min_bin_prop,
        content = self.content,
        precision = self.precision,
        group_name = group_name,
        return_details = return_details,
        missing_policy = effective_policy,
        psi_missing_bucket_policy = effective_bucket_policy,
    )

calculate_psi

calculate_psi(expected: Union[DataFrame, Series], actual: Union[DataFrame, Series], target_col: str, buckets: int = 10, equal_freq: bool = True, group_by: Optional[Union[str, List[str]]] = None, return_details: bool = False, min_bin_prop: float = 0.05, content: float = 1e-06, precision: int = 5, missing_policy: str = 'include', psi_missing_bucket_policy: str = 'smooth_laplace') -> Union[float, DataFrame, Tuple[Dict, Dict]]

Calculate Population Stability Index (PSI) for a variable, optionally by groups.

This function computes PSI to measure the distribution shift between expected (baseline) and actual (comparison) datasets for a specified variable.

Parameters:

Name	Type	Description	Default
expected	DataFrame or Series	Expected/baseline data.	required
actual	DataFrame or Series	Actual/comparison data.	required
target_col	str	Column name to calculate PSI for.	required
buckets	int	Number of bins. Default is 10.	10
equal_freq	bool	Use equal frequency binning. Default is True.	True
group_by	str or list	Column(s) to group by for stratified PSI calculation. Default is None.	None
return_details	bool	Return detailed bin information. Default is False.	False
min_bin_prop	float	Minimum proportion for each bin. Default is 0.05.	0.05
content	float	Small value to avoid division by zero. Default is 1e-6.	1e-06
precision	int	Decimal precision. Default is 5.	5
missing_policy	(drop, include, warn_and_drop)	How NaN rows are handled. Default in 0.5.0 is "include", which routes NaN rows through a dedicated "MISSING" bin so missing-rate drift contributes to the PSI. This is the recommended production behaviour and became the default in 0.5.0 (previously "drop" in 0.4.2 for numeric backward-compat). Pass "drop" to reproduce pre-0.5.0 numbers, or "warn_and_drop" for legacy numbers with a RuntimeWarning naming the NaN counts.	"drop"
psi_missing_bucket_policy	(smooth_laplace, floor_1e6, exclude)	How a bin that holds rows on one side only is treated. Default is "smooth_laplace": when such a bin exists, one is added to every bin count before the shares are computed. "floor_1e6" clips the bin shares at content, so the empty side counts as content. "exclude" drops the one-sided bins from the sum. Any other value raises ValueError.	"smooth_laplace"

Returns:

Type

Description

float, pandas.DataFrame, or tuple

If group_by is None and return_details is False: Single PSI float value. If group_by is None and return_details is True: Tuple of (psi_value, details DataFrame), with one row per bin and the columns expected_count, actual_count, expected_percent, actual_percent, psi_component, psi_component_floor_1e6, bucket_status and psi_missing_bucket_policy. If group_by is set and return_details is False: DataFrame with PSI values per group. If group_by is set and return_details is True: Tuple of (results_dict, details_dict).

Notes

With group_by, a group that has rows in only one of the two datasets gets the placeholder PSI 999999 (and no details).

Examples:

>>> # Simple PSI calculation
>>> psi = calculate_psi(expected_df, actual_df, 'score')

>>> # PSI by groups
>>> psi_by_region = calculate_psi(expected_df, actual_df, 'score', group_by='region')

Source code in Modeling_Tool/Feature/PSI_Tool.py

def calculate_psi(
    expected: Union[pd.DataFrame, pd.Series],
    actual: Union[pd.DataFrame, pd.Series],
    target_col: str,
    buckets: int = 10,
    equal_freq: bool = True,
    group_by: Optional[Union[str, List[str]]] = None,
    return_details: bool = False,
    min_bin_prop: float = 0.05,
    content: float = 1e-6,
    precision: int = 5,
    missing_policy: str = "include",
    psi_missing_bucket_policy: str = "smooth_laplace",
) -> Union[float, pd.DataFrame, Tuple[Dict, Dict]]:
    """
    Calculate Population Stability Index (PSI) for a variable, optionally by groups.

    This function computes PSI to measure the distribution shift between expected
    (baseline) and actual (comparison) datasets for a specified variable.

    Parameters
    ----------
    expected : pandas.DataFrame or pandas.Series
        Expected/baseline data.
    actual : pandas.DataFrame or pandas.Series
        Actual/comparison data.
    target_col : str
        Column name to calculate PSI for.
    buckets : int, optional
        Number of bins. Default is 10.
    equal_freq : bool, optional
        Use equal frequency binning. Default is True.
    group_by : str or list, optional
        Column(s) to group by for stratified PSI calculation. Default is None.
    return_details : bool, optional
        Return detailed bin information. Default is False.
    min_bin_prop : float, optional
        Minimum proportion for each bin. Default is 0.05.
    content : float, optional
        Small value to avoid division by zero. Default is 1e-6.
    precision : int, optional
        Decimal precision. Default is 5.
    missing_policy : {"drop", "include", "warn_and_drop"}, optional
        How NaN rows are handled. Default in 0.5.0 is "include", which routes
        NaN rows through a dedicated "__MISSING__" bin so missing-rate drift
        contributes to the PSI. This is the recommended production behaviour
        and became the default in 0.5.0 (previously "drop" in 0.4.2 for
        numeric backward-compat). Pass "drop" to reproduce pre-0.5.0
        numbers, or "warn_and_drop" for legacy numbers with a RuntimeWarning
        naming the NaN counts.
    psi_missing_bucket_policy : {"smooth_laplace", "floor_1e6", "exclude"}, optional
        How a bin that holds rows on one side only is treated. Default is "smooth_laplace": when such a bin
        exists, one is added to every bin count before the shares are computed. "floor_1e6" clips the bin shares at
        ``content``, so the empty side counts as ``content``. "exclude" drops the one-sided bins from the sum. Any
        other value raises ``ValueError``.

    Returns
    -------
    float, pandas.DataFrame, or tuple
        - If group_by is None and return_details is False: Single PSI float value.
        - If group_by is None and return_details is True: Tuple of (psi_value, details DataFrame), with one row per
          bin and the columns ``expected_count``, ``actual_count``, ``expected_percent``, ``actual_percent``,
          ``psi_component``, ``psi_component_floor_1e6``, ``bucket_status`` and ``psi_missing_bucket_policy``.
        - If group_by is set and return_details is False: DataFrame with PSI values per group.
        - If group_by is set and return_details is True: Tuple of (results_dict, details_dict).

    Notes
    -----
    With ``group_by``, a group that has rows in only one of the two datasets gets the placeholder PSI 999999 (and no
    details).

    Examples
    --------
    >>> # Simple PSI calculation
    >>> psi = calculate_psi(expected_df, actual_df, 'score')

    >>> # PSI by groups
    >>> psi_by_region = calculate_psi(expected_df, actual_df, 'score', group_by='region')
    """
    if group_by is not None:
        if isinstance(group_by, str):
            group_by = [group_by]

        expected_subset = expected[[target_col] + group_by].copy()
        actual_subset = actual[[target_col] + group_by].copy()

        expected_subset = expected_subset.copy()
        actual_subset = actual_subset.copy()

        expected_subset['_dataset'] = 'expected'
        actual_subset['_dataset'] = 'actual'

        combined = pd.concat([expected_subset, actual_subset], ignore_index=True)

        results = {}
        details_dict = {}

        for group, group_data in combined.groupby(group_by):
            expected_group = group_data[group_data['_dataset'] == 'expected'].drop('_dataset', axis=1)
            actual_group = group_data[group_data['_dataset'] == 'actual'].drop('_dataset', axis=1)

            if expected_group.shape[0] == 0 or actual_group.shape[0] == 0:
                results[group] = 999999
                continue

            if return_details:
                psi_value, detail = _calculate_single_psi(
                    expected_group[target_col], 
                    actual_group[target_col], 
                    buckets, 
                    equal_freq,
                    True,
                    min_bin_prop,
                    content,
                    precision,
                    missing_policy,
                    psi_missing_bucket_policy,
                )
                results[group] = psi_value
                details_dict[group] = detail
            else:
                results[group] = _calculate_single_psi(
                    expected_group[target_col], 
                    actual_group[target_col], 
                    buckets, 
                    equal_freq,
                    False,
                    min_bin_prop,
                    content,
                    precision,
                    missing_policy,
                    psi_missing_bucket_policy,
                )

        if return_details:
            return results, details_dict
        else:
            return pd.DataFrame(results, index=['psi']).T

    else:
        if return_details:
            return _calculate_single_psi(
                expected[target_col], 
                actual[target_col], 
                buckets, 
                equal_freq, 
                True, 
                min_bin_prop, 
                content, 
                precision,
                missing_policy,
                psi_missing_bucket_policy,
            )
        else:
            return _calculate_single_psi(
                expected[target_col], 
                actual[target_col], 
                buckets, 
                equal_freq, 
                False, 
                min_bin_prop, 
                content, 
                precision,
                missing_policy,
                psi_missing_bucket_policy,
            )

calculate_within_psi

calculate_within_psi(data: DataFrame, grp_name: str, target_col: str, benchmark: Optional[Any] = None, equal_freq: bool = True, buckets: int = 10, return_details: bool = False, min_bin_prop: float = 0.05, content: float = 1e-06, precision: int = 5, benchmark_display_name: Optional[str] = None, missing_policy: str = 'include', psi_missing_bucket_policy: str = 'smooth_laplace') -> Union[DataFrame, Dict]

Calculate PSI values within a single dataset, comparing groups to a benchmark.

This function computes PSI between a benchmark group and all other groups in a specified column, useful for monitoring population stability over time.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input dataset containing all groups.	required
grp_name	str	Column name for grouping.	required
target_col	str	Column name to calculate PSI for.	required
benchmark	str or callable	Benchmark group value or filter function. Default is None: every group is compared with the whole dataset (the code does not pick the first group). A callable receives data and must return a boolean mask that selects the benchmark rows; every group is then compared with those rows.	None
equal_freq	bool	Use equal frequency binning. Default is True.	True
buckets	int	Number of bins. Default is 10.	10
return_details	bool	Return detailed bin information. Default is False.	False
min_bin_prop	float	Minimum proportion for each bin. Default is 0.05.	0.05
content	float	Small value to avoid division by zero. Default is 1e-6.	1e-06
precision	int	Decimal precision. Default is 5.	5
benchmark_display_name	str	Custom name for benchmark in results. Default is None. When given, the result gets an extra first row with this name and a PSI of 0; otherwise the benchmark group is not listed.	None
missing_policy	(drop, include, warn_and_drop)	How NaN rows are handled. Default in 0.5.0 is "include", which routes NaN rows through a dedicated "MISSING" bin so missing-rate drift contributes to the PSI. This is the recommended production behaviour and became the default in 0.5.0 (previously "drop" in 0.4.2 for numeric backward-compat). Pass "drop" to reproduce pre-0.5.0 numbers, or "warn_and_drop" for legacy numbers with a RuntimeWarning naming the NaN counts.	"drop"
psi_missing_bucket_policy	(smooth_laplace, floor_1e6, exclude)	How a bin that holds rows on one side only is treated. Default is "smooth_laplace": when such a bin exists, one is added to every bin count before the shares are computed. "floor_1e6" clips the bin shares at content, so the empty side counts as content. "exclude" drops the one-sided bins from the sum. Any other value raises ValueError.	"smooth_laplace"

Returns:

Type	Description
DataFrame or dict	If return_details is False: DataFrame with a column named as the grp_name argument (the group value) and the column psi, one row per group other than the benchmark. If return_details is True: Dict with 'psi' (that DataFrame) and 'details' (dict {group value: details DataFrame}, see calculate_psi).

Examples:

>>> # Compare all months against January
>>> psi_results = calculate_within_psi(data, 'month', 'score', benchmark='2024-01')

Source code in Modeling_Tool/Feature/PSI_Tool.py

def calculate_within_psi(
    data: pd.DataFrame,
    grp_name: str,
    target_col: str,
    benchmark: Optional[Any] = None,
    equal_freq: bool = True,
    buckets: int = 10,
    return_details: bool = False,
    min_bin_prop: float = 0.05,
    content: float = 1e-6,
    precision: int = 5,
    benchmark_display_name: Optional[str] = None,
    missing_policy: str = "include",
    psi_missing_bucket_policy: str = "smooth_laplace",
) -> Union[pd.DataFrame, Dict]:
    """
    Calculate PSI values within a single dataset, comparing groups to a benchmark.

    This function computes PSI between a benchmark group and all other groups
    in a specified column, useful for monitoring population stability over time.

    Parameters
    ----------
    data : pandas.DataFrame
        Input dataset containing all groups.
    grp_name : str
        Column name for grouping.
    target_col : str
        Column name to calculate PSI for.
    benchmark : str or callable, optional
        Benchmark group value or filter function. Default is None: every group is compared with the whole dataset
        (the code does not pick the first group). A callable receives ``data`` and must return a boolean mask that
        selects the benchmark rows; every group is then compared with those rows.
    equal_freq : bool, optional
        Use equal frequency binning. Default is True.
    buckets : int, optional
        Number of bins. Default is 10.
    return_details : bool, optional
        Return detailed bin information. Default is False.
    min_bin_prop : float, optional
        Minimum proportion for each bin. Default is 0.05.
    content : float, optional
        Small value to avoid division by zero. Default is 1e-6.
    precision : int, optional
        Decimal precision. Default is 5.
    benchmark_display_name : str, optional
        Custom name for benchmark in results. Default is None. When given, the result gets an extra first row
        with this name and a PSI of 0; otherwise the benchmark group is not listed.
    missing_policy : {"drop", "include", "warn_and_drop"}, optional
        How NaN rows are handled. Default in 0.5.0 is "include", which routes
        NaN rows through a dedicated "__MISSING__" bin so missing-rate drift
        contributes to the PSI. This is the recommended production behaviour
        and became the default in 0.5.0 (previously "drop" in 0.4.2 for
        numeric backward-compat). Pass "drop" to reproduce pre-0.5.0
        numbers, or "warn_and_drop" for legacy numbers with a RuntimeWarning
        naming the NaN counts.
    psi_missing_bucket_policy : {"smooth_laplace", "floor_1e6", "exclude"}, optional
        How a bin that holds rows on one side only is treated. Default is "smooth_laplace": when such a bin
        exists, one is added to every bin count before the shares are computed. "floor_1e6" clips the bin shares at
        ``content``, so the empty side counts as ``content``. "exclude" drops the one-sided bins from the sum. Any
        other value raises ``ValueError``.

    Returns
    -------
    pandas.DataFrame or dict
        If return_details is False: DataFrame with a column named as the ``grp_name`` argument (the group value)
        and the column ``psi``, one row per group other than the benchmark.
        If return_details is True: Dict with 'psi' (that DataFrame) and 'details' (dict ``{group value: details
        DataFrame}``, see ``calculate_psi``).

    Examples
    --------
    >>> # Compare all months against January
    >>> psi_results = calculate_within_psi(data, 'month', 'score', benchmark='2024-01')
    """
    if callable(benchmark):
        benchmark_data = data[benchmark(data)]
    else:
        benchmark_data = data[data[grp_name] == benchmark] if benchmark is not None else data

    obs_values = [x for x in data[grp_name].unique().tolist() if x != benchmark]

    res_dict = {benchmark_display_name: 0} if benchmark_display_name is not None else {}
    detail_dict = {}

    for obs_value in obs_values:
        obs_data = data[data[grp_name] == obs_value]

        if return_details:
            psi, details = calculate_psi(
                benchmark_data, 
                obs_data, 
                target_col=target_col, 
                buckets=buckets,
                equal_freq=equal_freq, 
                return_details=True, 
                min_bin_prop=min_bin_prop, 
                content=content, 
                precision=precision,
                missing_policy=missing_policy,
                psi_missing_bucket_policy=psi_missing_bucket_policy,
            )
            res_dict[obs_value] = psi
            detail_dict[obs_value] = details
        else:
            psi = calculate_psi(
                benchmark_data, 
                obs_data, 
                target_col=target_col, 
                buckets=buckets,
                equal_freq=equal_freq, 
                return_details=False, 
                min_bin_prop=min_bin_prop, 
                content=content, 
                precision=precision,
                missing_policy=missing_policy,
                psi_missing_bucket_policy=psi_missing_bucket_policy,
            )
            res_dict[obs_value] = psi

    fnl_res = pd.DataFrame(res_dict, index=['psi']).T\
        .reset_index(drop=False)\
        .rename(columns={"index": grp_name})

    if return_details:
        return {"psi": fnl_res, "details": detail_dict}
    else:
        return fnl_res

calculate_psi_within_dataset

calculate_psi_within_dataset(data: DataFrame, grp_name: str, varlist: List[str], benchmark: Optional[Any] = None, equal_freq: bool = True, buckets: int = 10, min_bin_prop: float = 0.05, content: float = 1e-06, precision: int = 5, missing_policy: str = 'include', psi_missing_bucket_policy: str = 'smooth_laplace') -> DataFrame

Calculate PSI for multiple variables within a dataset, comparing groups to a benchmark.

This function iterates over a list of variables and calculates PSI for each, combining results into a single DataFrame.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input dataset.	required
grp_name	str	Column name for grouping.	required
varlist	list	List of variable names to calculate PSI for.	required
benchmark	str or callable	Benchmark group value or filter function. Default is None: every group is compared with the whole dataset. A callable receives data and must return a boolean mask that selects the benchmark rows. The benchmark group itself is not listed in the result.	None
equal_freq	bool	Use equal frequency binning. Default is True.	True
buckets	int	Number of bins. Default is 10.	10
min_bin_prop	float	Minimum proportion for each bin. Default is 0.05.	0.05
content	float	Small value to avoid division by zero. Default is 1e-6.	1e-06
precision	int	Decimal precision. Default is 5.	5
missing_policy	(drop, include, warn_and_drop)	How NaN rows are handled. Default in 0.5.0 is "include", which routes NaN rows through a dedicated "MISSING" bin so missing-rate drift contributes to the PSI. This is the recommended production behaviour and became the default in 0.5.0 (previously "drop" in 0.4.2 for numeric backward-compat). Pass "drop" to reproduce pre-0.5.0 numbers, or "warn_and_drop" for legacy numbers with a RuntimeWarning naming the NaN counts.	"drop"
psi_missing_bucket_policy	(smooth_laplace, floor_1e6, exclude)	How a bin that holds rows on one side only is treated. Default is "smooth_laplace": when such a bin exists, one is added to every bin count before the shares are computed. "floor_1e6" clips the bin shares at content, so the empty side counts as content. "exclude" drops the one-sided bins from the sum. Any other value raises ValueError.	"smooth_laplace"

Returns:

Type	Description
DataFrame	Combined PSI results for all variables, sorted by group: a column named as the grp_name argument, the column psi and the column var. The rows of each variable are sorted by group.

Examples:

>>> variables = ['score', 'age', 'income']
>>> psi_df = calculate_psi_within_dataset(data, 'month', variables, benchmark='2024-01')

Source code in Modeling_Tool/Feature/PSI_Tool.py

def calculate_psi_within_dataset(
    data: pd.DataFrame,
    grp_name: str,
    varlist: List[str],
    benchmark: Optional[Any] = None,
    equal_freq: bool = True,
    buckets: int = 10,
    min_bin_prop: float = 0.05,
    content: float = 1e-6,
    precision: int = 5,
    missing_policy: str = "include",
    psi_missing_bucket_policy: str = "smooth_laplace",
) -> pd.DataFrame:
    """
    Calculate PSI for multiple variables within a dataset, comparing groups to a benchmark.

    This function iterates over a list of variables and calculates PSI for each,
    combining results into a single DataFrame.

    Parameters
    ----------
    data : pandas.DataFrame
        Input dataset.
    grp_name : str
        Column name for grouping.
    varlist : list
        List of variable names to calculate PSI for.
    benchmark : str or callable, optional
        Benchmark group value or filter function. Default is None: every group is compared with the whole dataset.
        A callable receives ``data`` and must return a boolean mask that selects the benchmark rows. The benchmark
        group itself is not listed in the result.
    equal_freq : bool, optional
        Use equal frequency binning. Default is True.
    buckets : int, optional
        Number of bins. Default is 10.
    min_bin_prop : float, optional
        Minimum proportion for each bin. Default is 0.05.
    content : float, optional
        Small value to avoid division by zero. Default is 1e-6.
    precision : int, optional
        Decimal precision. Default is 5.
    missing_policy : {"drop", "include", "warn_and_drop"}, optional
        How NaN rows are handled. Default in 0.5.0 is "include", which routes
        NaN rows through a dedicated "__MISSING__" bin so missing-rate drift
        contributes to the PSI. This is the recommended production behaviour
        and became the default in 0.5.0 (previously "drop" in 0.4.2 for
        numeric backward-compat). Pass "drop" to reproduce pre-0.5.0
        numbers, or "warn_and_drop" for legacy numbers with a RuntimeWarning
        naming the NaN counts.
    psi_missing_bucket_policy : {"smooth_laplace", "floor_1e6", "exclude"}, optional
        How a bin that holds rows on one side only is treated. Default is "smooth_laplace": when such a bin
        exists, one is added to every bin count before the shares are computed. "floor_1e6" clips the bin shares at
        ``content``, so the empty side counts as ``content``. "exclude" drops the one-sided bins from the sum. Any
        other value raises ``ValueError``.

    Returns
    -------
    pandas.DataFrame
        Combined PSI results for all variables, sorted by group: a column named as the ``grp_name`` argument, the
        column ``psi`` and the column ``var``. The rows of each variable are sorted by group.

    Examples
    --------
    >>> variables = ['score', 'age', 'income']
    >>> psi_df = calculate_psi_within_dataset(data, 'month', variables, benchmark='2024-01')
    """
    fnl_psi_res = []
    for var in tqdm(varlist):
        single_psi = calculate_within_psi(
            data=data, 
            grp_name=grp_name, 
            benchmark=benchmark, 
            target_col=var, 
            buckets=buckets, 
            equal_freq=equal_freq, 
            return_details=False, 
            min_bin_prop=min_bin_prop, 
            content=content, 
            precision=precision,
            missing_policy=missing_policy,
            psi_missing_bucket_policy=psi_missing_bucket_policy,
        ).sort_values([grp_name]).reset_index(drop=True)

        single_psi['var'] = var
        fnl_psi_res.append(single_psi)

    return pd.concat(fnl_psi_res)

calculate_multivar_psi_two_sets

calculate_multivar_psi_two_sets(expected_df: DataFrame, actual_df: DataFrame, varlist: List[str], group_by: Optional[Union[str, List[str]]] = None, buckets: int = 10, equal_freq: bool = True, min_bin_prop: float = 0.05, content: float = 1e-06, precision: int = 5, missing_policy: str = 'include', psi_missing_bucket_policy: str = 'smooth_laplace') -> DataFrame

Calculate PSI for multiple variables by comparing two different datasets.

This function computes PSI for each variable in varlist between expected and actual DataFrames.

Parameters:

Name	Type	Description	Default
expected_df	DataFrame	Expected/baseline dataset.	required
actual_df	DataFrame	Actual/comparison dataset.	required
varlist	list	List of variable names to calculate PSI for.	required
group_by	str or list	Column(s) to group by. Default is None.	None
buckets	int	Number of bins. Default is 10.	10
equal_freq	bool	Use equal frequency binning. Default is True.	True
min_bin_prop	float	Minimum proportion for each bin. Default is 0.05.	0.05
content	float	Small value to avoid division by zero. Default is 1e-6.	1e-06
precision	int	Decimal precision. Default is 5.	5
missing_policy	(drop, include, warn_and_drop)	How NaN rows are handled. Default in 0.5.0 is "include", which routes NaN rows through a dedicated "MISSING" bin so missing-rate drift contributes to the PSI. This is the recommended production behaviour and became the default in 0.5.0 (previously "drop" in 0.4.2 for numeric backward-compat). Pass "drop" to reproduce pre-0.5.0 numbers, or "warn_and_drop" for legacy numbers with a RuntimeWarning naming the NaN counts.	"drop"
psi_missing_bucket_policy	(smooth_laplace, floor_1e6, exclude)	How a bin that holds rows on one side only is treated. Default is "smooth_laplace": when such a bin exists, one is added to every bin count before the shares are computed. "floor_1e6" clips the bin shares at content, so the empty side counts as content. "exclude" drops the one-sided bins from the sum. Any other value raises ValueError.	"smooth_laplace"

Returns:

Type	Description
DataFrame	PSI results for all variables with 'var' and 'psi' columns. With group_by, the group values are in the index (one row per variable and group).

Examples:

>>> variables = ['score', 'age', 'income']
>>> psi_df = calculate_multivar_psi_two_sets(train_df, production_df, variables)

Source code in Modeling_Tool/Feature/PSI_Tool.py

def calculate_multivar_psi_two_sets(
    expected_df: pd.DataFrame,
    actual_df: pd.DataFrame,
    varlist: List[str],
    group_by: Optional[Union[str, List[str]]] = None,
    buckets: int = 10,
    equal_freq: bool = True,
    min_bin_prop: float = 0.05,
    content: float = 1e-6,
    precision: int = 5,
    missing_policy: str = "include",
    psi_missing_bucket_policy: str = "smooth_laplace",
) -> pd.DataFrame:
    """
    Calculate PSI for multiple variables by comparing two different datasets.

    This function computes PSI for each variable in varlist between expected
    and actual DataFrames.

    Parameters
    ----------
    expected_df : pandas.DataFrame
        Expected/baseline dataset.
    actual_df : pandas.DataFrame
        Actual/comparison dataset.
    varlist : list
        List of variable names to calculate PSI for.
    group_by : str or list, optional
        Column(s) to group by. Default is None.
    buckets : int, optional
        Number of bins. Default is 10.
    equal_freq : bool, optional
        Use equal frequency binning. Default is True.
    min_bin_prop : float, optional
        Minimum proportion for each bin. Default is 0.05.
    content : float, optional
        Small value to avoid division by zero. Default is 1e-6.
    precision : int, optional
        Decimal precision. Default is 5.
    missing_policy : {"drop", "include", "warn_and_drop"}, optional
        How NaN rows are handled. Default in 0.5.0 is "include", which routes
        NaN rows through a dedicated "__MISSING__" bin so missing-rate drift
        contributes to the PSI. This is the recommended production behaviour
        and became the default in 0.5.0 (previously "drop" in 0.4.2 for
        numeric backward-compat). Pass "drop" to reproduce pre-0.5.0
        numbers, or "warn_and_drop" for legacy numbers with a RuntimeWarning
        naming the NaN counts.
    psi_missing_bucket_policy : {"smooth_laplace", "floor_1e6", "exclude"}, optional
        How a bin that holds rows on one side only is treated. Default is "smooth_laplace": when such a bin
        exists, one is added to every bin count before the shares are computed. "floor_1e6" clips the bin shares at
        ``content``, so the empty side counts as ``content``. "exclude" drops the one-sided bins from the sum. Any
        other value raises ``ValueError``.

    Returns
    -------
    pandas.DataFrame
        PSI results for all variables with 'var' and 'psi' columns. With ``group_by``, the group values are in the
        index (one row per variable and group).

    Examples
    --------
    >>> variables = ['score', 'age', 'income']
    >>> psi_df = calculate_multivar_psi_two_sets(train_df, production_df, variables)
    """
    multi_psi_res = []
    for var in tqdm(varlist):
        single_psi = calculate_psi(
            expected=expected_df, 
            actual=actual_df, 
            target_col=var, 
            group_by=group_by, 
            buckets=buckets,
            equal_freq=equal_freq,
            min_bin_prop=min_bin_prop,
            content=content,
            precision=precision,
            return_details=False,
            missing_policy=missing_policy,
            psi_missing_bucket_policy=psi_missing_bucket_policy,
        )
        if group_by is None:
            single_psi = pd.DataFrame([single_psi], columns=['psi'])
        single_psi['var'] = var
        multi_psi_res.append(single_psi)

    return pd.concat(multi_psi_res)

calculate_multigroup_psi_two_sets

calculate_multigroup_psi_two_sets(expected_df: DataFrame, actual_df: DataFrame, varlist: List[str], group_by: Optional[Union[str, List[str]]] = None, buckets: int = 10, equal_freq: bool = True, min_bin_prop: float = 0.05, content: float = 1e-06, precision: int = 5, group_name: Optional[str] = None, return_details: bool = False, missing_policy: str = 'include', psi_missing_bucket_policy: str = 'smooth_laplace') -> Union[DataFrame, Dict[str, DataFrame]]

Calculate grouped PSI using expected DataFrame as benchmark, applied to actual DataFrame groups.

Parameters:

Name	Type	Description	Default
expected_df	DataFrame	Expected/baseline dataset.	required
actual_df	DataFrame	Actual/comparison dataset.	required
varlist	list	List of variable names to calculate PSI for.	required
group_by	str or list	Group column name(s); PSI is computed separately for each group. Default is None. Legacy argument: when group_name is None and return_details is False it is ignored (use group_name for PSI by group).	None
buckets	int	Number of bins. Default is 10.	10
equal_freq	bool	Whether to use equal-frequency binning. Default is True.	True
min_bin_prop	float	Minimum proportion for each bin. Default is 0.05.	0.05
content	float	Small value to avoid division by zero. Default is 1e-6.	1e-06
precision	int	Decimal precision. Default is 5.	5
group_name	str	Name of the group column for multi-group calculation. Default is None. When it is given (and group_by is None), every value of this column of actual_df (NaN becomes "NULL") is compared with the whole expected_df, and each variable is binned once on expected_df.	None
return_details	bool	Whether to return detailed bin information. Default is False. If True, return a dict {'psi': psi_df, 'details': details_df}; details_df contains the columns: ['bin', 'expected_count', 'actual_count', 'expected_percent', 'actual_percent', 'psi_component', 'bucket_status', 'psi_missing_bucket_policy', group_name, 'var'] (without group_name when it is not given; with group_name and no group_by it also holds 'psi_component_floor_1e6').	False
missing_policy	(drop, include, warn_and_drop)	How NaN rows are handled. Default in 0.5.0 is "include", which routes NaN rows through a dedicated "MISSING" bin so missing-rate drift contributes to the PSI. This is the recommended production behaviour and became the default in 0.5.0 (previously "drop" in 0.4.2 for numeric backward compatibility). Pass "drop" to reproduce pre-0.5.0 numbers, or "warn_and_drop" to keep the legacy numbers while emitting a RuntimeWarning that reports the NaN counts on both sides.	"drop"
psi_missing_bucket_policy	(smooth_laplace, floor_1e6, exclude)	How a bin that holds rows on one side only is treated. Default is "smooth_laplace": when such a bin exists, one is added to every bin count before the shares are computed. "floor_1e6" clips the bin shares at content, so the empty side counts as content. "exclude" drops the one-sided bins from the sum. Any other value raises ValueError.	"smooth_laplace"

Returns:

Type	Description
DataFrame or dict	A DataFrame with the columns var and psi (plus the group_name column when it is given), or, if return_details is True, the dict {'psi': DataFrame, 'details': DataFrame}.

Source code in Modeling_Tool/Feature/PSI_Tool.py

def calculate_multigroup_psi_two_sets(
    expected_df: pd.DataFrame,
    actual_df: pd.DataFrame,
    varlist: List[str],
    group_by: Optional[Union[str, List[str]]] = None,
    buckets: int = 10,
    equal_freq: bool = True,
    min_bin_prop: float = 0.05,
    content: float = 1e-6,
    precision: int = 5,
    group_name: Optional[str] = None,
    return_details: bool = False,
    missing_policy: str = "include",
    psi_missing_bucket_policy: str = "smooth_laplace",
) -> Union[pd.DataFrame, Dict[str, pd.DataFrame]]:
    """
    Calculate grouped PSI using expected DataFrame as benchmark, applied to actual DataFrame groups.

    Parameters
    ----------
    expected_df : pandas.DataFrame
        Expected/baseline dataset.
    actual_df : pandas.DataFrame
        Actual/comparison dataset.
    varlist : list
        List of variable names to calculate PSI for.
    group_by : str or list, optional
        Group column name(s); PSI is computed separately for each group. Default is None. Legacy argument: when
        ``group_name`` is None and ``return_details`` is False it is ignored (use ``group_name`` for PSI by group).
    buckets : int, optional
        Number of bins. Default is 10.
    equal_freq : bool, optional
        Whether to use equal-frequency binning. Default is True.
    min_bin_prop : float, optional
        Minimum proportion for each bin. Default is 0.05.
    content : float, optional
        Small value to avoid division by zero. Default is 1e-6.
    precision : int, optional
        Decimal precision. Default is 5.
    group_name : str, optional
        Name of the group column for multi-group calculation. Default is None. When it is given (and ``group_by``
        is None), every value of this column of ``actual_df`` (NaN becomes "__NULL__") is compared with the whole
        ``expected_df``, and each variable is binned once on ``expected_df``.
    return_details : bool, optional
        Whether to return detailed bin information. Default is False. If True, return a dict
        {'psi': psi_df, 'details': details_df};
        details_df contains the columns: ['bin', 'expected_count', 'actual_count', 'expected_percent',
        'actual_percent', 'psi_component', 'bucket_status', 'psi_missing_bucket_policy', group_name, 'var']
        (without ``group_name`` when it is not given; with ``group_name`` and no ``group_by`` it also holds
        'psi_component_floor_1e6').
    missing_policy : {"drop", "include", "warn_and_drop"}, optional
        How NaN rows are handled. Default in 0.5.0 is "include", which routes
        NaN rows through a dedicated "__MISSING__" bin so missing-rate drift
        contributes to the PSI. This is the recommended production behaviour
        and became the default in 0.5.0 (previously "drop" in 0.4.2 for
        numeric backward compatibility). Pass "drop" to reproduce pre-0.5.0
        numbers, or "warn_and_drop" to keep the legacy numbers while emitting
        a RuntimeWarning that reports the NaN counts on both sides.
    psi_missing_bucket_policy : {"smooth_laplace", "floor_1e6", "exclude"}, optional
        How a bin that holds rows on one side only is treated. Default is "smooth_laplace": when such a bin
        exists, one is added to every bin count before the shares are computed. "floor_1e6" clips the bin shares at
        ``content``, so the empty side counts as ``content``. "exclude" drops the one-sided bins from the sum. Any
        other value raises ``ValueError``.

    Returns
    -------
    pandas.DataFrame or dict
        A DataFrame with the columns ``var`` and ``psi`` (plus the ``group_name`` column when it is given), or, if
        ``return_details`` is True, the dict ``{'psi': DataFrame, 'details': DataFrame}``.
    """
    if group_name is not None and group_by is None:
        return _calculate_grouped_psi_fixed_reference(
            expected_df=expected_df,
            actual_df=actual_df,
            varlist=varlist,
            group_name=group_name,
            buckets=buckets,
            equal_freq=equal_freq,
            min_bin_prop=min_bin_prop,
            content=content,
            precision=precision,
            return_details=return_details,
            missing_policy=missing_policy,
            psi_missing_bucket_policy=psi_missing_bucket_policy,
        )

    if group_name is not None:
        if actual_df[group_name].isna().sum() > 0:
            actual_df = actual_df.copy()
            actual_df[group_name] = actual_df[group_name].fillna('__NULL__')

        if return_details:
            psi_records = []
            detail_records = []

            for group, group_data in actual_df.groupby(group_name):
                for var in tqdm(varlist, desc=f"Processing group {group}"):
                    psi_val, detail_df = calculate_psi(
                        expected=expected_df,
                        actual=group_data,
                        target_col=var,
                        group_by=group_by,
                        buckets=buckets,
                        equal_freq=equal_freq,
                        return_details=True,
                        min_bin_prop=min_bin_prop,
                        content=content,
                        precision=precision,
                        missing_policy=missing_policy,
                        psi_missing_bucket_policy=psi_missing_bucket_policy,
                    )

                    psi_records.append({group_name: group, 'var': var, 'psi': psi_val})

                    # ========== Normalize detail_df (core of the fix) ==========
                    detail_df = _coerce_psi_detail_frame(detail_df, group_by=group_by)
                    if detail_df is not None and not detail_df.empty:
                        if 'bin' not in detail_df.columns:
                            # Reset the index; the original index column may be named 'index' or something else
                            detail_df = detail_df.reset_index()
                            # After reset_index, the first column is the original index column
                            index_col = detail_df.columns[0]
                            detail_df = detail_df.rename(columns={index_col: 'bin'})

                        detail_df[group_name] = group
                        detail_df['var'] = var

                        required_cols = ['bin', 'expected_count', 'actual_count', 'expected_percent', 'actual_percent', 'psi_component', 'bucket_status', 'psi_missing_bucket_policy', group_name, 'var']
                        for col in required_cols:
                            if col not in detail_df.columns:
                                detail_df[col] = np.nan
                        detail_df = detail_df[required_cols]
                        detail_records.append(detail_df)
                    # ======================================================

            psi_df = pd.DataFrame(psi_records)
            details_df = pd.concat(detail_records, ignore_index=True) if detail_records else pd.DataFrame(columns=['bin', 'expected_count', 'actual_count', 'expected_percent', 'actual_percent', 'psi_component', group_name, 'var'])
            return {'psi': psi_df, 'details': details_df}

        multi_psi_res = []
        for group, group_data in actual_df.groupby(group_name, dropna=False):
            group_psi = calculate_multivar_psi_two_sets(
                expected_df=expected_df,
                actual_df=group_data,
                varlist=varlist,
                group_by=group_by,
                buckets=buckets,
                equal_freq=equal_freq,
                min_bin_prop=min_bin_prop,
                content=content,
                precision=precision,
                missing_policy=missing_policy,
                psi_missing_bucket_policy=psi_missing_bucket_policy,
            )
            group_psi[group_name] = group
            multi_psi_res.append(group_psi)
        return pd.concat(multi_psi_res, ignore_index=True) if multi_psi_res else pd.DataFrame(
            columns=["var", "psi", group_name]
        )

    else:
        # When group_name is not specified
        if return_details:
            # Supports multiple variables
            psi_records = []
            detail_records = []
            for var in tqdm(varlist, desc="Calculating PSI with details"):
                psi_val, detail_df = calculate_psi(
                    expected=expected_df,
                    actual=actual_df,
                    target_col=var,
                    group_by=group_by,
                    buckets=buckets,
                    equal_freq=equal_freq,
                    return_details=True,
                    min_bin_prop=min_bin_prop,
                    content=content,
                    precision=precision,
                    missing_policy=missing_policy,
                    psi_missing_bucket_policy=psi_missing_bucket_policy,
                )
                psi_records.append({'var': var, 'psi': psi_val})

                # Normalize detail_df
                detail_df = _coerce_psi_detail_frame(detail_df, group_by=group_by)
                if detail_df is not None and not detail_df.empty:
                    if 'bin' not in detail_df.columns:
                        detail_df = detail_df.reset_index()
                        index_col = detail_df.columns[0]
                        detail_df = detail_df.rename(columns={index_col: 'bin'})
                    detail_df['var'] = var
                    required_cols = ['bin', 'expected_count', 'actual_count', 'expected_percent', 'actual_percent', 'psi_component', 'bucket_status', 'psi_missing_bucket_policy', 'var']
                    for col in required_cols:
                        if col not in detail_df.columns:
                            detail_df[col] = np.nan
                    detail_df = detail_df[required_cols]
                    detail_records.append(detail_df)

            psi_df = pd.DataFrame(psi_records)
            details_df = pd.concat(detail_records, ignore_index=True) if detail_records else pd.DataFrame(columns=['bin', 'expected_count', 'actual_count', 'expected_percent', 'actual_percent', 'psi_component', 'var'])
            return {'psi': psi_df, 'details': details_df}

        else:

            # Details not requested: use the original batch calculation function
            group_psi = calculate_multivar_psi_two_sets(
                expected_df=expected_df, 
                actual_df=actual_df, 
                varlist=varlist, 
                group_by=None, 
                buckets=buckets, 
                equal_freq=equal_freq, 
                min_bin_prop=min_bin_prop, 
                content=content, 
                precision=precision,
                missing_policy=missing_policy,
                psi_missing_bucket_policy=psi_missing_bucket_policy,
            )

            return group_psi

Variable Insights and Correlation: Feature_Insights

Feature_Insights

Variable extraction and correlation analysis toolkit. Provides variable analysis, IV calculation, WOE plotting, and correlation filtering.

VarExtractionInsights

Variable extraction and insight analyzer.

Analyze the variables of a dataset by computing IV values and WOE bins, with support for plotting and variable screening.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input raw DataFrame.	required
dep	str	Column name of the target (dependent) variable.	required
plot_path	str	Directory in which the plots are saved.	required
nbins	int	Number of bins. Default is 10.	10
equal_freq	bool	Whether to use equal-frequency binning. Default is True.	True
min_bin_prop	float	Minimum proportion of samples in each bin. Default is 0.05.	0.05
precision	int	Numeric precision of the WOE and IV calculations. Default is 5.	5
chi2_method	bool	Whether to use chi-square binning. Default is False.	False
chi2_p	float	p-value threshold of the chi-square test. Default is 0.9.	0.9
init_equi_bins	int	Number of initial equal-frequency bins. Default is 5000.	5000
tree_binning	bool	Whether to use decision-tree binning. Default is True.	True
include_missing	bool	Whether to treat missing values as a separate bin. Default is True.	True
seed	int	Random seed. Default is 3407.	3407
missing_rate_ref	int / float	Reference value used to fill missing values. Default is -999999.	-999999
spec_values	list	List of special values. Default is an empty list.	None

Examples:

>>> analyzer = VarExtractionInsights(df, 'target', '/path/to/plots')
>>> report = analyzer.get_var_analysis_report(df, ['var1', 'var2'])

Source code in Modeling_Tool/Feature/Feature_Insights.py

class VarExtractionInsights:
    """Variable extraction and insight analyzer.

    Analyze the variables of a dataset by computing IV values and WOE bins,
    with support for plotting and variable screening.

    Parameters
    ----------
    data : pd.DataFrame
        Input raw DataFrame.
    dep : str
        Column name of the target (dependent) variable.
    plot_path : str
        Directory in which the plots are saved.
    nbins : int, optional
        Number of bins. Default is 10.
    equal_freq : bool, optional
        Whether to use equal-frequency binning. Default is True.
    min_bin_prop : float, optional
        Minimum proportion of samples in each bin. Default is 0.05.
    precision : int, optional
        Numeric precision of the WOE and IV calculations. Default is 5.
    chi2_method : bool, optional
        Whether to use chi-square binning. Default is False.
    chi2_p : float, optional
        p-value threshold of the chi-square test. Default is 0.9.
    init_equi_bins : int, optional
        Number of initial equal-frequency bins. Default is 5000.
    tree_binning : bool, optional
        Whether to use decision-tree binning. Default is True.
    include_missing : bool, optional
        Whether to treat missing values as a separate bin. Default is True.
    seed : int, optional
        Random seed. Default is 3407.
    missing_rate_ref : int/float, optional
        Reference value used to fill missing values. Default is -999999.
    spec_values : list, optional
        List of special values. Default is an empty list.

    Examples
    --------
    >>> analyzer = VarExtractionInsights(df, 'target', '/path/to/plots')
    >>> report = analyzer.get_var_analysis_report(df, ['var1', 'var2'])
    """

    def __init__(self, data, dep, plot_path,
                 nbins=10, equal_freq=True, min_bin_prop=0.05, precision=5, chi2_method=False, chi2_p=0.9,
                 init_equi_bins=5000, tree_binning=True, include_missing=True, seed=3407, missing_rate_ref=-999999, spec_values=None):
        """Initialize the variable extraction and insight analyzer.

        Parameters
        ----------
        data : pd.DataFrame
            Input raw DataFrame.
        dep : str
            Column name of the target (dependent) variable.
        plot_path : str
            Directory in which the plots are saved.
        nbins : int, optional
            Number of bins.
        equal_freq : bool, optional
            Whether to use equal-frequency binning.
        min_bin_prop : float, optional
            Minimum proportion of samples in each bin.
        precision : int, optional
            Numeric precision of the WOE and IV calculations.
        chi2_method : bool, optional
            Whether to use chi-square binning.
        chi2_p : float, optional
            p-value threshold of the chi-square test.
        init_equi_bins : int, optional
            Number of initial equal-frequency bins.
        tree_binning : bool, optional
            Whether to use decision-tree binning.
        include_missing : bool, optional
            Whether to treat missing values as a separate bin.
        seed : int, optional
            Random seed.
        missing_rate_ref : int/float, optional
            Reference value used to fill missing values.
        spec_values : list, optional
            List of special values.
        """
        self.data = data
        self.dep = dep
        self.plot_path = plot_path

        self.nbins = nbins
        self.equal_freq = equal_freq
        self.min_bin_prop = min_bin_prop
        self.precision = precision
        self.chi2_method = chi2_method
        self.chi2_p = chi2_p
        self.init_equi_bins = init_equi_bins
        self.tree_binning = tree_binning
        self.include_missing = include_missing
        self.seed = seed
        self.missing_rate_ref = missing_rate_ref
        self.spec_values = spec_values if spec_values is not None else []
        self.failed_variables = []

    @staticmethod
    def remove_folder(file_path):
        """Delete the specified folder.

        Recursively delete the folder at the given path together with all of its
        contents; do nothing (silently) if the folder does not exist.

        Parameters
        ----------
        file_path : str
            Path of the folder to delete.

        Examples
        --------
        >>> VarExtractionInsights.remove_folder('/path/to/folder')
        """
        import shutil
        try:
            shutil.rmtree(file_path)
        except Exception:
            pass

    def get_var_analysis_report(self, data, varlist, dep=None, iv_cut=0.01):
        """Generate the variable analysis report.

        Compute the IV, KS, Lift, and other metrics for the given list of variables
        and return the analysis summary of the variables that meet the IV threshold.

        Parameters
        ----------
        data : pd.DataFrame
            Input raw DataFrame.
        varlist : list
            List of variable names to analyze.
        dep : str, optional
            Name of the target column. Default is None (use the dep given at initialization).
        iv_cut : float, optional
            IV screening threshold. Default is 0.01.

        Returns
        -------
        pd.DataFrame
            Summary table of the variable analysis results, with the columns:
            - var: variable name
            - n_all: total number of samples
            - n: number of non-missing samples
            - ks_in_gains: KS statistic
            - lift_in_gains: Lift value
            - iv: IV value
            - n_bump: number of bins
            - missing_rate: missing rate
            - min, mean, max: descriptive statistics
            - n_bins: number of bins

        Examples
        --------
        >>> analyzer = VarExtractionInsights(df, 'target', '/path/to/plots')
        >>> report = analyzer.get_var_analysis_report(df, ['var1', 'var2'])
        """
        if dep is None:
            dep = self.dep
        self.failed_variables = []

        from Modeling_Tool.Eval.Model_Eval_Tool import get_gains_table

        iv_info_res = []
        for var in tqdm(varlist):
            if data[var].nunique() > 1:
                try:
                    attr_iv = get_gains_table(
                        data=data,
                        dep=self.dep,
                        nbins=self.nbins,
                        precision=self.precision,
                        min_bin_prop=self.min_bin_prop,
                        include_missing=self.include_missing,
                        score=var,
                        equal_freq=self.equal_freq,
                        chi2_method=self.chi2_method,
                        chi2_p=self.chi2_p,
                        init_equi_bins=self.init_equi_bins,
                        fillna=self.missing_rate_ref,
                        spec_values=self.spec_values,
                        retSummary=True,
                        tree_binning=self.tree_binning,
                        random_state=self.seed,
                        ascending=True,
                    )

                    attr_iv['var'] = var
                    iv_info_res.append(attr_iv)

                except (TypeError, ValueError, KeyError, ZeroDivisionError) as exc:
                    row = smf_logger.record_and_continue(var, exc, stage="feature_insights")
                    self.failed_variables.append((row["feature"], row["exception_type"]))
                    continue

        if self.failed_variables:
            failed = [name for name, _ in self.failed_variables]
            warnings.warn(
                f"{len(failed)}/{len(varlist)} variables failed insight computation: "
                f"{failed[:10]}{'...' if len(failed) > 10 else ''} — see smf_logger for details",
                UserWarning,
                stacklevel=2,
            )

        output_cols = [
            'var', 'n_all', 'n', 'ks_in_gains', 'lift_in_gains', 'iv',
            'n_bump', 'missing_rate', 'min', 'mean', 'max', 'n_bins'
        ]
        if not iv_info_res:
            return pd.DataFrame(columns=output_cols)

        iv_info_res = pd.concat(iv_info_res).sort_values("IV", ascending=False)

        high_iv_summary = iv_info_res.query(f"IV >= {iv_cut}").round(4)
        high_iv_varlist = high_iv_summary['var'].tolist()

        if len(high_iv_varlist) == 0:
            logger.info(f"WARNING: No variable with IV >= {iv_cut}")

        means = proc_means_for_screening(
            data, high_iv_varlist, spec_missing_value=self.missing_rate_ref,
        )

        fnl_summary = high_iv_summary.merge(
            means,
            left_on=['var'],
            right_on=['attribute'],
            how='left'
        )
        fnl_summary.columns = [x.lower() for x in fnl_summary.columns]
        fnl_summary = fnl_summary[output_cols]

        return fnl_summary

    def plot_woe(self, data, varlist, plot_group=None, plot_dirname="var_analysis_plot", plot_path=None):
        """Plot the WOE distribution charts.

        Compute the WOE values for the given list of variables, plot their
        distribution charts, and save them to the specified directory.

        Parameters
        ----------
        data : pd.DataFrame
            Input raw DataFrame.
        varlist : list
            List of variable names to plot.
        plot_group : str, optional
            Name of the grouping variable. Default is None.
        plot_dirname : str, optional
            Name of the subdirectory in which the plots are saved. Default is "var_analysis_plot".
        plot_path : str, optional
            Root directory in which the plots are saved. Default is None (use the plot_path given at initialization).

        Returns
        -------
        None

        Examples
        --------
        >>> analyzer = VarExtractionInsights(df, 'target', '/path/to/plots')
        >>> analyzer.plot_woe(df, ['var1', 'var2'])
        """

        if plot_path is None:
            plot_path = self.plot_path

        from Modeling_Tool.WOE.WOE_Master import WOE_Master

        # Fill Missing Value.
        drv_fillna = data.copy()
        drv_fillna[varlist] = drv_fillna[varlist].fillna(self.missing_rate_ref)

        woe_master = WOE_Master(
            train_data=drv_fillna,
            varlist=varlist,
            dep=self.dep,
            graph_save_dir=plot_path
        )

        woe_master.fit(
            nbins=self.nbins,
            equal_freq=self.equal_freq,
            min_bin_prop=self.min_bin_prop,
            precision=self.precision,
            chi2_config=(self.init_equi_bins, self.chi2_p) if self.chi2_method else None,
            tree_binning_seed=self.seed if self.tree_binning else None,
            include_missing=self.include_missing,
            spec_values=self.spec_values
        )

        train_woe = woe_master.transform(drv_fillna)
        woe_master.plot_bivar_graph(train_woe, group=plot_group, dirname=plot_dirname)

remove_folder staticmethod

remove_folder(file_path)

Delete the specified folder.

Recursively delete the folder at the given path together with all of its contents; do nothing (silently) if the folder does not exist.

Parameters:

Name	Type	Description	Default
file_path	str	Path of the folder to delete.	required

Examples:

>>> VarExtractionInsights.remove_folder('/path/to/folder')

Source code in Modeling_Tool/Feature/Feature_Insights.py

@staticmethod
def remove_folder(file_path):
    """Delete the specified folder.

    Recursively delete the folder at the given path together with all of its
    contents; do nothing (silently) if the folder does not exist.

    Parameters
    ----------
    file_path : str
        Path of the folder to delete.

    Examples
    --------
    >>> VarExtractionInsights.remove_folder('/path/to/folder')
    """
    import shutil
    try:
        shutil.rmtree(file_path)
    except Exception:
        pass

get_var_analysis_report

get_var_analysis_report(data, varlist, dep=None, iv_cut=0.01)

Generate the variable analysis report.

Compute the IV, KS, Lift, and other metrics for the given list of variables and return the analysis summary of the variables that meet the IV threshold.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input raw DataFrame.	required
varlist	list	List of variable names to analyze.	required
dep	str	Name of the target column. Default is None (use the dep given at initialization).	None
iv_cut	float	IV screening threshold. Default is 0.01.	0.01

Returns:

Type	Description
DataFrame	Summary table of the variable analysis results, with the columns: - var: variable name - n_all: total number of samples - n: number of non-missing samples - ks_in_gains: KS statistic - lift_in_gains: Lift value - iv: IV value - n_bump: number of bins - missing_rate: missing rate - min, mean, max: descriptive statistics - n_bins: number of bins

Examples:

>>> analyzer = VarExtractionInsights(df, 'target', '/path/to/plots')
>>> report = analyzer.get_var_analysis_report(df, ['var1', 'var2'])

Source code in Modeling_Tool/Feature/Feature_Insights.py

def get_var_analysis_report(self, data, varlist, dep=None, iv_cut=0.01):
    """Generate the variable analysis report.

    Compute the IV, KS, Lift, and other metrics for the given list of variables
    and return the analysis summary of the variables that meet the IV threshold.

    Parameters
    ----------
    data : pd.DataFrame
        Input raw DataFrame.
    varlist : list
        List of variable names to analyze.
    dep : str, optional
        Name of the target column. Default is None (use the dep given at initialization).
    iv_cut : float, optional
        IV screening threshold. Default is 0.01.

    Returns
    -------
    pd.DataFrame
        Summary table of the variable analysis results, with the columns:
        - var: variable name
        - n_all: total number of samples
        - n: number of non-missing samples
        - ks_in_gains: KS statistic
        - lift_in_gains: Lift value
        - iv: IV value
        - n_bump: number of bins
        - missing_rate: missing rate
        - min, mean, max: descriptive statistics
        - n_bins: number of bins

    Examples
    --------
    >>> analyzer = VarExtractionInsights(df, 'target', '/path/to/plots')
    >>> report = analyzer.get_var_analysis_report(df, ['var1', 'var2'])
    """
    if dep is None:
        dep = self.dep
    self.failed_variables = []

    from Modeling_Tool.Eval.Model_Eval_Tool import get_gains_table

    iv_info_res = []
    for var in tqdm(varlist):
        if data[var].nunique() > 1:
            try:
                attr_iv = get_gains_table(
                    data=data,
                    dep=self.dep,
                    nbins=self.nbins,
                    precision=self.precision,
                    min_bin_prop=self.min_bin_prop,
                    include_missing=self.include_missing,
                    score=var,
                    equal_freq=self.equal_freq,
                    chi2_method=self.chi2_method,
                    chi2_p=self.chi2_p,
                    init_equi_bins=self.init_equi_bins,
                    fillna=self.missing_rate_ref,
                    spec_values=self.spec_values,
                    retSummary=True,
                    tree_binning=self.tree_binning,
                    random_state=self.seed,
                    ascending=True,
                )

                attr_iv['var'] = var
                iv_info_res.append(attr_iv)

            except (TypeError, ValueError, KeyError, ZeroDivisionError) as exc:
                row = smf_logger.record_and_continue(var, exc, stage="feature_insights")
                self.failed_variables.append((row["feature"], row["exception_type"]))
                continue

    if self.failed_variables:
        failed = [name for name, _ in self.failed_variables]
        warnings.warn(
            f"{len(failed)}/{len(varlist)} variables failed insight computation: "
            f"{failed[:10]}{'...' if len(failed) > 10 else ''} — see smf_logger for details",
            UserWarning,
            stacklevel=2,
        )

    output_cols = [
        'var', 'n_all', 'n', 'ks_in_gains', 'lift_in_gains', 'iv',
        'n_bump', 'missing_rate', 'min', 'mean', 'max', 'n_bins'
    ]
    if not iv_info_res:
        return pd.DataFrame(columns=output_cols)

    iv_info_res = pd.concat(iv_info_res).sort_values("IV", ascending=False)

    high_iv_summary = iv_info_res.query(f"IV >= {iv_cut}").round(4)
    high_iv_varlist = high_iv_summary['var'].tolist()

    if len(high_iv_varlist) == 0:
        logger.info(f"WARNING: No variable with IV >= {iv_cut}")

    means = proc_means_for_screening(
        data, high_iv_varlist, spec_missing_value=self.missing_rate_ref,
    )

    fnl_summary = high_iv_summary.merge(
        means,
        left_on=['var'],
        right_on=['attribute'],
        how='left'
    )
    fnl_summary.columns = [x.lower() for x in fnl_summary.columns]
    fnl_summary = fnl_summary[output_cols]

    return fnl_summary

plot_woe

plot_woe(data, varlist, plot_group=None, plot_dirname='var_analysis_plot', plot_path=None)

Plot the WOE distribution charts.

Compute the WOE values for the given list of variables, plot their distribution charts, and save them to the specified directory.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input raw DataFrame.	required
varlist	list	List of variable names to plot.	required
plot_group	str	Name of the grouping variable. Default is None.	None
plot_dirname	str	Name of the subdirectory in which the plots are saved. Default is "var_analysis_plot".	'var_analysis_plot'
plot_path	str	Root directory in which the plots are saved. Default is None (use the plot_path given at initialization).	None

Returns:

Type	Description
None

Examples:

>>> analyzer = VarExtractionInsights(df, 'target', '/path/to/plots')
>>> analyzer.plot_woe(df, ['var1', 'var2'])

Source code in Modeling_Tool/Feature/Feature_Insights.py

def plot_woe(self, data, varlist, plot_group=None, plot_dirname="var_analysis_plot", plot_path=None):
    """Plot the WOE distribution charts.

    Compute the WOE values for the given list of variables, plot their
    distribution charts, and save them to the specified directory.

    Parameters
    ----------
    data : pd.DataFrame
        Input raw DataFrame.
    varlist : list
        List of variable names to plot.
    plot_group : str, optional
        Name of the grouping variable. Default is None.
    plot_dirname : str, optional
        Name of the subdirectory in which the plots are saved. Default is "var_analysis_plot".
    plot_path : str, optional
        Root directory in which the plots are saved. Default is None (use the plot_path given at initialization).

    Returns
    -------
    None

    Examples
    --------
    >>> analyzer = VarExtractionInsights(df, 'target', '/path/to/plots')
    >>> analyzer.plot_woe(df, ['var1', 'var2'])
    """

    if plot_path is None:
        plot_path = self.plot_path

    from Modeling_Tool.WOE.WOE_Master import WOE_Master

    # Fill Missing Value.
    drv_fillna = data.copy()
    drv_fillna[varlist] = drv_fillna[varlist].fillna(self.missing_rate_ref)

    woe_master = WOE_Master(
        train_data=drv_fillna,
        varlist=varlist,
        dep=self.dep,
        graph_save_dir=plot_path
    )

    woe_master.fit(
        nbins=self.nbins,
        equal_freq=self.equal_freq,
        min_bin_prop=self.min_bin_prop,
        precision=self.precision,
        chi2_config=(self.init_equi_bins, self.chi2_p) if self.chi2_method else None,
        tree_binning_seed=self.seed if self.tree_binning else None,
        include_missing=self.include_missing,
        spec_values=self.spec_values
    )

    train_woe = woe_master.transform(drv_fillna)
    woe_master.plot_bivar_graph(train_woe, group=plot_group, dirname=plot_dirname)

CorrelationFilter

Correlation filter analyzer.

Provide screening and removal of highly correlated variables based on correlation analysis, with IV comparison and iterative filtering.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input raw DataFrame.	required
dep	str	Column name of the target (dependent) variable.	required
corr_cutpoint	float	Correlation coefficient threshold; variable pairs exceeding it are flagged as highly correlated. Default is 0.8.	0.8
method	str	Method used to compute the correlation coefficients; one of 'pearson', 'spearman', 'kendall'. Default is 'pearson'.	'pearson'
tree_binning	bool	Whether the IV / KS that decides between correlated variables is computed on decision-tree bins (passed on to VarExtractionInsights).	False
chi2_method	bool	Whether that IV / KS is computed on chi-square merged bins.	False
seed	int	Random seed of the binning behind that IV / KS.	42
chi2_p	float	p-value threshold of the chi-square merging.	0.999
init_equi_bins	int	Number of initial equal-frequency bins for the chi-square binning.	1000
missing_rate_ref	int or float	Value that fills missing values before binning.	-9999999
spec_values	list	Special values (for example -1 or 999 for "no record") that get rows of their own in the IV / KS calculation of VarExtractionInsights instead of being mixed into the numeric bins, which decides the winner of each correlated group. The correlation itself is still computed on the raw values.	[]
base_metric	(iv, ks)	Metric (case-insensitive) compared inside a group of correlated variables: the variable with the highest value is kept and the others are removed.	"iv"

Examples:

>>> filter_analyzer = CorrelationFilter(df, 'target')
>>> keep_vars = filter_analyzer.remove_highly_correlated(['var1', 'var2'])

Source code in Modeling_Tool/Feature/Feature_Insights.py

class CorrelationFilter:
    """Correlation filter analyzer.

    Provide screening and removal of highly correlated variables based on
    correlation analysis, with IV comparison and iterative filtering.

    Parameters
    ----------
    data : pd.DataFrame
        Input raw DataFrame.
    dep : str
        Column name of the target (dependent) variable.
    corr_cutpoint : float, optional
        Correlation coefficient threshold; variable pairs exceeding it are flagged as highly correlated. Default is 0.8.
    method : str, optional
        Method used to compute the correlation coefficients; one of 'pearson', 'spearman', 'kendall'. Default is 'pearson'.
    tree_binning : bool, default False
        Whether the IV / KS that decides between correlated variables is computed on decision-tree bins (passed on to
        ``VarExtractionInsights``).
    chi2_method : bool, default False
        Whether that IV / KS is computed on chi-square merged bins.
    seed : int, default 42
        Random seed of the binning behind that IV / KS.
    chi2_p : float, default 0.999
        p-value threshold of the chi-square merging.
    init_equi_bins : int, default 1000
        Number of initial equal-frequency bins for the chi-square binning.
    missing_rate_ref : int or float, default -9999999
        Value that fills missing values before binning.
    spec_values : list, default []
        Special values (for example ``-1`` or ``999`` for "no record") that get rows of their own in the IV / KS
        calculation of ``VarExtractionInsights`` instead of being mixed into the numeric bins, which decides the
        winner of each correlated group. The correlation itself is still computed on the raw values.
    base_metric : {"iv", "ks"}, default "iv"
        Metric (case-insensitive) compared inside a group of correlated variables: the variable with the highest value
        is kept and the others are removed.

    Examples
    --------
    >>> filter_analyzer = CorrelationFilter(df, 'target')
    >>> keep_vars = filter_analyzer.remove_highly_correlated(['var1', 'var2'])
    """

    def __init__(self, data, dep, corr_cutpoint=0.8, method='pearson', tree_binning=False, chi2_method=False, seed = 42, chi2_p =0.999, init_equi_bins = 1000, 
                 missing_rate_ref = -9999999, spec_values = [], base_metric = 'iv'):
        """Initialize the correlation filter analyzer (the parameters are described in the class docstring)."""
        self.data = data
        self.dep = dep
        self.corr_cutpoint = corr_cutpoint
        self.method = method
        self.tree_binning = tree_binning
        self.chi2_method = chi2_method
        self.seed = seed
        self.chi2_p = chi2_p
        self.init_equi_bins = init_equi_bins
        self.missing_rate_ref = missing_rate_ref
        self.spec_values = spec_values
        self.base_metric = base_metric

        self.correlated_dict = {}
        self.filtered_varlist = []
        self._corr_matrix_cache = None
        self._corr_matrix_excluded = set()
        self._metric_summary_cache = None
        self._correlation_decision_trace = []

    def _corr_matrix_frame(self, varlist):
        # 0.7.0-R1: numeric-subset correlation base. A pure-numeric varlist is
        # byte-identical to the legacy self.data[varlist].corr() path. Non-numeric
        # cols (raw Pearson is undefined) are excluded and tracked in
        # self._corr_matrix_excluded; _high_corr_pairs reindexes them back as NaN
        # rows/cols so they survive the corr stage instead of crashing the float
        # cast. At most one raw-value UserWarning per matrix build (no WOE binner).
        non_numeric = [c for c in varlist if not pd.api.types.is_numeric_dtype(self.data[c])]
        if not non_numeric:
            self._corr_matrix_excluded = set()
            return self.data[varlist]
        self._corr_matrix_excluded = set(non_numeric)
        warnings.warn(
            f"raw-value correlation skips {len(non_numeric)} non-numeric "
            f"feature(s) {non_numeric[:5]}; they are kept through the corr "
            f"stage. Set corr_use_woe_bins=True to correlate categorical "
            f"features via their WOE encoding.",
            UserWarning,
            stacklevel=2,
        )
        selected = [c for c in varlist if c not in self._corr_matrix_excluded]
        return self.data[selected]

    def _high_corr_pairs(self, varlist):
        if (
            self._corr_matrix_cache is None
            or not (set(varlist) <= set(self._corr_matrix_cache.columns) | self._corr_matrix_excluded)
        ):
            self._corr_matrix_cache = self._corr_matrix_frame(varlist).corr(method=self.method)
        matrix = self._corr_matrix_cache.reindex(index=varlist, columns=varlist)
        values = matrix.to_numpy(dtype=float)
        row_idx, col_idx = np.triu_indices(len(varlist), k=1)
        pair_values = values[row_idx, col_idx]
        keep = np.isfinite(pair_values) & (np.abs(pair_values) > self.corr_cutpoint)
        names = np.asarray(varlist, dtype=object)
        return pd.DataFrame(
            {
                "VAR1": names[row_idx[keep]],
                "VAR2": names[col_idx[keep]],
                "CORR": pair_values[keep],
            },
            columns=["VAR1", "VAR2", "CORR"],
        )

    def _metric_summary(self, varlist):
        cached_vars = (
            set(self._metric_summary_cache["var"])
            if self._metric_summary_cache is not None and "var" in self._metric_summary_cache
            else set()
        )
        if not set(varlist).issubset(cached_vars):
            insights = VarExtractionInsights(
                data=self.data,
                dep=self.dep,
                plot_path=None,
                nbins=10,
                equal_freq=True,
                min_bin_prop=0.05,
                precision=5,
                chi2_method=self.chi2_method,
                chi2_p=self.chi2_p,
                init_equi_bins=self.init_equi_bins,
                tree_binning=self.tree_binning,
                include_missing=True,
                seed=self.seed,
                missing_rate_ref=self.missing_rate_ref,
                spec_values=self.spec_values,
            )
            self._metric_summary_cache = insights.get_var_analysis_report(
                data=self.data,
                varlist=varlist,
                dep=self.dep,
                iv_cut=0,
            )
        return self._metric_summary_cache

    def filter_single_iteration(self, varlist):
        """Filter highly correlated variables in a single iteration.

        Run one pass of correlation filtering over the variable list, keeping the variable with the highest IV.

        Parameters
        ----------
        varlist : list
            List of variable names to screen.

        Returns
        -------
        list
            List of variable names retained after screening.
        """
        base_metric = self.base_metric.lower()

        name_mapping = {
            "iv": "iv",
            "ks": "ks_in_gains"
        }

        high_corr_var = self._high_corr_pairs(varlist)

        if len(high_corr_var) == 0:
            return varlist

        base_varlist = high_corr_var['VAR1'].drop_duplicates().tolist()

        correlated_dict = self.correlated_dict
        selected_varlist = []
        removed_varlist = []
        for var in tqdm(base_varlist):
            if var not in set(removed_varlist + selected_varlist):                
                single_var_corr = high_corr_var.loc[high_corr_var["VAR1"].eq(var)]
                correlated_list = [var] + single_var_corr['VAR2'].drop_duplicates().tolist()

                metric_summary = self._metric_summary(varlist)
                fnl_summary = metric_summary[
                    metric_summary["var"].isin(correlated_list)
                ].copy()
                if fnl_summary.empty:
                    continue
                fnl_selected_var = fnl_summary.sort_values(
                    [name_mapping[base_metric]],
                    ascending=False,
                )["var"].iloc[0]

                if fnl_selected_var not in selected_varlist:
                    selected_varlist.append(fnl_selected_var)

                newly_removed = [
                    x for x in correlated_list
                    if x != fnl_selected_var and x not in removed_varlist
                ]
                metric_map = fnl_summary.set_index("var")[name_mapping[base_metric]].to_dict()
                positions = {name: idx for idx, name in enumerate(varlist)}
                for dropped_var in newly_removed:
                    decision_pair = [fnl_selected_var, dropped_var]
                    decision_pair.sort(key=positions.__getitem__)
                    var_a, var_b = decision_pair
                    corr_value = self._corr_matrix_cache.loc[var_a, var_b]
                    self._correlation_decision_trace.append({
                        "var_a": var_a,
                        "var_b": var_b,
                        "corr": float(corr_value),
                        "iv_a": float(metric_map.get(var_a, 0.0)),
                        "iv_b": float(metric_map.get(var_b, 0.0)),
                        "kept": fnl_selected_var,
                        "dropped": dropped_var,
                    })
                removed_varlist += newly_removed

                if var not in correlated_dict:
                    correlated_dict[var] = {}
                    correlated_dict[var]['corr'] = single_var_corr
                    correlated_dict[var]['gains'] = fnl_summary
                else:
                    correlated_dict[var]['corr'] = pd.concat([correlated_dict[var]['corr'], single_var_corr]).drop_duplicates()
                    correlated_dict[var]['gains'] = pd.concat([correlated_dict[var]['gains'], fnl_summary]).drop_duplicates()

        other_varlist = [x for x in varlist if x not in (selected_varlist + removed_varlist)]
        fnl_keep_varlist = selected_varlist + other_varlist

        self.correlated_dict = correlated_dict

        return fnl_keep_varlist

    def remove_highly_correlated(self, varlist, max_iterations=10):
        """Iteratively remove highly correlated variables.

        Run correlation filtering repeatedly until no variable is removed or the maximum number of iterations is reached.

        Parameters
        ----------
        varlist : list
            List of variable names to screen.
        max_iterations : int, optional
            Maximum number of iterations. Default is 10.

        Returns
        -------
        list
            List of variable names finally retained.

        Examples
        --------
        >>> filter_analyzer = CorrelationFilter(df, 'target')
        >>> keep_vars = filter_analyzer.remove_highly_correlated(['var1', 'var2', 'var3'])
        """
        self._correlation_decision_trace = []
        self._corr_matrix_cache = self._corr_matrix_frame(varlist).corr(method=self.method)
        self._metric_summary_cache = None
        self._metric_summary(varlist)
        last_keep_list = self.filter_single_iteration(varlist)

        for i in range(1, max_iterations):
            fnl_keep_list = self.filter_single_iteration(last_keep_list)

            removed_vars = [x for x in last_keep_list if x not in fnl_keep_list]
            self.filtered_varlist.append(removed_vars)
            if len(removed_vars) == 0:
                break

            last_keep_list = fnl_keep_list

        self.filtered_varlist = [x for x in varlist if x not in last_keep_list]
        return last_keep_list


    @staticmethod
    def calculate_vif(df):
        """Compute the variance inflation factor (VIF).

        Detect multicollinearity by returning the VIF of each variable.
        The larger the VIF, the more severe the collinearity; a VIF > 10 usually indicates severe collinearity.

        Parameters
        ----------
        df : pd.DataFrame
            DataFrame containing the independent variables.

        Returns
        -------
        pd.DataFrame
            DataFrame with the following columns:
            - index: variable name
            - VIF: variance inflation factor value

        Examples
        --------
        >>> vif_result = calculate_vif(X_train)
        >>> high_vif_vars = vif_result[vif_result['VIF'] > 10]['index'].tolist()
        """
        from statsmodels.stats.outliers_influence import variance_inflation_factor

        vif = pd.DataFrame()
        vif['index'] = df.columns
        vif['VIF'] = [variance_inflation_factor(df.values, i) for i in range(df.shape[1])]
        return vif

filter_single_iteration

filter_single_iteration(varlist)

Filter highly correlated variables in a single iteration.

Run one pass of correlation filtering over the variable list, keeping the variable with the highest IV.

Parameters:

Name	Type	Description	Default
varlist	list	List of variable names to screen.	required

Returns:

Type	Description
list	List of variable names retained after screening.

Source code in Modeling_Tool/Feature/Feature_Insights.py

def filter_single_iteration(self, varlist):
    """Filter highly correlated variables in a single iteration.

    Run one pass of correlation filtering over the variable list, keeping the variable with the highest IV.

    Parameters
    ----------
    varlist : list
        List of variable names to screen.

    Returns
    -------
    list
        List of variable names retained after screening.
    """
    base_metric = self.base_metric.lower()

    name_mapping = {
        "iv": "iv",
        "ks": "ks_in_gains"
    }

    high_corr_var = self._high_corr_pairs(varlist)

    if len(high_corr_var) == 0:
        return varlist

    base_varlist = high_corr_var['VAR1'].drop_duplicates().tolist()

    correlated_dict = self.correlated_dict
    selected_varlist = []
    removed_varlist = []
    for var in tqdm(base_varlist):
        if var not in set(removed_varlist + selected_varlist):                
            single_var_corr = high_corr_var.loc[high_corr_var["VAR1"].eq(var)]
            correlated_list = [var] + single_var_corr['VAR2'].drop_duplicates().tolist()

            metric_summary = self._metric_summary(varlist)
            fnl_summary = metric_summary[
                metric_summary["var"].isin(correlated_list)
            ].copy()
            if fnl_summary.empty:
                continue
            fnl_selected_var = fnl_summary.sort_values(
                [name_mapping[base_metric]],
                ascending=False,
            )["var"].iloc[0]

            if fnl_selected_var not in selected_varlist:
                selected_varlist.append(fnl_selected_var)

            newly_removed = [
                x for x in correlated_list
                if x != fnl_selected_var and x not in removed_varlist
            ]
            metric_map = fnl_summary.set_index("var")[name_mapping[base_metric]].to_dict()
            positions = {name: idx for idx, name in enumerate(varlist)}
            for dropped_var in newly_removed:
                decision_pair = [fnl_selected_var, dropped_var]
                decision_pair.sort(key=positions.__getitem__)
                var_a, var_b = decision_pair
                corr_value = self._corr_matrix_cache.loc[var_a, var_b]
                self._correlation_decision_trace.append({
                    "var_a": var_a,
                    "var_b": var_b,
                    "corr": float(corr_value),
                    "iv_a": float(metric_map.get(var_a, 0.0)),
                    "iv_b": float(metric_map.get(var_b, 0.0)),
                    "kept": fnl_selected_var,
                    "dropped": dropped_var,
                })
            removed_varlist += newly_removed

            if var not in correlated_dict:
                correlated_dict[var] = {}
                correlated_dict[var]['corr'] = single_var_corr
                correlated_dict[var]['gains'] = fnl_summary
            else:
                correlated_dict[var]['corr'] = pd.concat([correlated_dict[var]['corr'], single_var_corr]).drop_duplicates()
                correlated_dict[var]['gains'] = pd.concat([correlated_dict[var]['gains'], fnl_summary]).drop_duplicates()

    other_varlist = [x for x in varlist if x not in (selected_varlist + removed_varlist)]
    fnl_keep_varlist = selected_varlist + other_varlist

    self.correlated_dict = correlated_dict

    return fnl_keep_varlist

remove_highly_correlated

remove_highly_correlated(varlist, max_iterations=10)

Iteratively remove highly correlated variables.

Run correlation filtering repeatedly until no variable is removed or the maximum number of iterations is reached.

Parameters:

Name	Type	Description	Default
varlist	list	List of variable names to screen.	required
max_iterations	int	Maximum number of iterations. Default is 10.	10

Returns:

Type	Description
list	List of variable names finally retained.

Examples:

>>> filter_analyzer = CorrelationFilter(df, 'target')
>>> keep_vars = filter_analyzer.remove_highly_correlated(['var1', 'var2', 'var3'])

Source code in Modeling_Tool/Feature/Feature_Insights.py

def remove_highly_correlated(self, varlist, max_iterations=10):
    """Iteratively remove highly correlated variables.

    Run correlation filtering repeatedly until no variable is removed or the maximum number of iterations is reached.

    Parameters
    ----------
    varlist : list
        List of variable names to screen.
    max_iterations : int, optional
        Maximum number of iterations. Default is 10.

    Returns
    -------
    list
        List of variable names finally retained.

    Examples
    --------
    >>> filter_analyzer = CorrelationFilter(df, 'target')
    >>> keep_vars = filter_analyzer.remove_highly_correlated(['var1', 'var2', 'var3'])
    """
    self._correlation_decision_trace = []
    self._corr_matrix_cache = self._corr_matrix_frame(varlist).corr(method=self.method)
    self._metric_summary_cache = None
    self._metric_summary(varlist)
    last_keep_list = self.filter_single_iteration(varlist)

    for i in range(1, max_iterations):
        fnl_keep_list = self.filter_single_iteration(last_keep_list)

        removed_vars = [x for x in last_keep_list if x not in fnl_keep_list]
        self.filtered_varlist.append(removed_vars)
        if len(removed_vars) == 0:
            break

        last_keep_list = fnl_keep_list

    self.filtered_varlist = [x for x in varlist if x not in last_keep_list]
    return last_keep_list

calculate_vif staticmethod

calculate_vif(df)

Compute the variance inflation factor (VIF).

Detect multicollinearity by returning the VIF of each variable. The larger the VIF, the more severe the collinearity; a VIF > 10 usually indicates severe collinearity.

Parameters:

Name	Type	Description	Default
df	DataFrame	DataFrame containing the independent variables.	required

Returns:

Type	Description
DataFrame	DataFrame with the following columns: - index: variable name - VIF: variance inflation factor value

Examples:

>>> vif_result = calculate_vif(X_train)
>>> high_vif_vars = vif_result[vif_result['VIF'] > 10]['index'].tolist()

Source code in Modeling_Tool/Feature/Feature_Insights.py

@staticmethod
def calculate_vif(df):
    """Compute the variance inflation factor (VIF).

    Detect multicollinearity by returning the VIF of each variable.
    The larger the VIF, the more severe the collinearity; a VIF > 10 usually indicates severe collinearity.

    Parameters
    ----------
    df : pd.DataFrame
        DataFrame containing the independent variables.

    Returns
    -------
    pd.DataFrame
        DataFrame with the following columns:
        - index: variable name
        - VIF: variance inflation factor value

    Examples
    --------
    >>> vif_result = calculate_vif(X_train)
    >>> high_vif_vars = vif_result[vif_result['VIF'] > 10]['index'].tolist()
    """
    from statsmodels.stats.outliers_influence import variance_inflation_factor

    vif = pd.DataFrame()
    vif['index'] = df.columns
    vif['VIF'] = [variance_inflation_factor(df.values, i) for i in range(df.shape[1])]
    return vif

var_corr_filter

var_corr_filter(data, varlist, corr_cutpoint=0.8, method='pearson')

Screen for highly correlated variable pairs.

Compute the correlation coefficients between variables and return the list of highly correlated variable pairs whose correlation exceeds the threshold.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input DataFrame.	required
varlist	list	List of variable names to screen.	required
corr_cutpoint	float	Correlation coefficient threshold. Default is 0.8.	0.8
method	str	Method used to compute the correlation coefficients; one of 'pearson', 'spearman', 'kendall'. Default is 'pearson'.	'pearson'

Returns:

Type	Description
DataFrame	DataFrame of the highly correlated variable pairs, with the columns: - VAR1: variable 1 - VAR2: variable 2 - CORR: correlation coefficient

Examples:

>>> high_corr = var_corr_filter(df, ['var1', 'var2', 'var3'], corr_cutpoint=0.8)

Source code in Modeling_Tool/Feature/Feature_Insights.py

def var_corr_filter(data, varlist, corr_cutpoint=0.8, method='pearson'):
    """Screen for highly correlated variable pairs.

    Compute the correlation coefficients between variables and return the list of
    highly correlated variable pairs whose correlation exceeds the threshold.

    Parameters
    ----------
    data : pd.DataFrame
        Input DataFrame.
    varlist : list
        List of variable names to screen.
    corr_cutpoint : float, optional
        Correlation coefficient threshold. Default is 0.8.
    method : str, optional
        Method used to compute the correlation coefficients; one of 'pearson', 'spearman', 'kendall'.
        Default is 'pearson'.

    Returns
    -------
    pd.DataFrame
        DataFrame of the highly correlated variable pairs, with the columns:
        - VAR1: variable 1
        - VAR2: variable 2
        - CORR: correlation coefficient

    Examples
    --------
    >>> high_corr = var_corr_filter(df, ['var1', 'var2', 'var3'], corr_cutpoint=0.8)
    """
    import numpy as np

    corr_matrix = data[varlist].corr(method=method)

    values = corr_matrix.to_numpy(dtype=float)
    row_idx, col_idx = np.triu_indices(len(varlist), k=1)
    pair_values = values[row_idx, col_idx]
    keep = np.isfinite(pair_values) & (np.abs(pair_values) > corr_cutpoint)
    return pd.DataFrame(
        {
            "VAR1": np.asarray(varlist, dtype=object)[row_idx[keep]],
            "VAR2": np.asarray(varlist, dtype=object)[col_idx[keep]],
            "CORR": pair_values[keep],
        },
        columns=["VAR1", "VAR2", "CORR"],
    )

WOE-Engine-Aware Classes: WOE_Engine_Feature_Patch

The signatures of the three exported classes. Their methods and arguments behave as described in PSI_Tool and Feature_Insights above.

WOE_Engine_Feature_Patch

WOE-engine-aware wrappers for feature screening tools.

The original Feature modules are compiled in release builds, so mutating their class objects at import time is fragile. These wrappers delegate to the original classes for legacy behavior and only take over when a fitted WOE engine is explicitly supplied.

PSICalculator

Population Stability Index (PSI) calculator that can bin with a fitted WOE engine.

Without binning_engine every call is delegated to Modeling_Tool.Feature.PSI_Tool.PSICalculator: each variable is binned on the expected sample (equal-frequency or equal-width bins) and the current sample is cut with the same edges. With a binning_engine the PSI is computed on the bins of that engine instead (its WOE value is the bin label), so monitoring uses the same mapping as the model.

Parameters:

Name	Type	Description	Default
buckets	int	Number of bins of the default binning. It has no effect when binning_engine is given.	10
equal_freq	bool	Use equal-frequency bins (False gives equal-width bins). It has no effect when binning_engine is given.	True
min_bin_prop	float	Minimum proportion for each bin; it caps the number of bins at max(5, 1 / min_bin_prop). It has no effect when binning_engine is given.	0.05
content	float	Floor of a bin share, so that the logarithm stays finite (see psi_missing_bucket_policy).	1e-06
precision	int	Decimals of the bin edges of the default binning. With binning_engine it is the number of decimals the PSI values are rounded to.	5
binning_engine	(WOE_Master, MonotoneWOEBinner, WOEEngineAdapter or None)	Fitted WOE engine whose bins define the PSI. None keeps the default binning. An object of any other type raises TypeError.	None
missing_policy	('include', 'drop', 'warn_and_drop')	How NaN rows are handled by the default binning: "include" gives them a bin of their own ("__MISSING__"), so a change of the missing rate moves the PSI; "drop" ignores them; "warn_and_drop" drops them and emits a RuntimeWarning with the NaN counts. It has no effect when binning_engine is given (the engine decides how missing values are binned).	"include"
psi_missing_bucket_policy	('smooth_laplace', 'floor_1e6', 'exclude')	How a bin that holds rows on one side only is treated. "smooth_laplace" adds one to every bin count when such a bin exists; "floor_1e6" clips the bin shares at content; "exclude" drops the one-sided bins from the sum. Any other value raises ValueError.	"smooth_laplace"
feature_block_size	int or None	Number of variables that the engine transforms per block (used only with binning_engine). None processes all variables at once. A value that is not a positive integer raises ValueError.	64

Attributes:

Name	Type	Description
buckets, equal_freq, min_bin_prop, content, precision, missing_policy, psi_missing_bucket_policy, feature_block_size		The constructor values.
binning_engine	(WOE_Master, MonotoneWOEBinner, WOEEngineAdapter or None)	The engine as passed to the constructor.

Examples:

>>> psi = PSICalculator(buckets=10, binning_engine=woe).calculate(train_df, oot_df, features)

Source code in Modeling_Tool/Feature/WOE_Engine_Feature_Patch.py

class PSICalculator:
    """Population Stability Index (PSI) calculator that can bin with a fitted WOE engine.

    Without ``binning_engine`` every call is delegated to ``Modeling_Tool.Feature.PSI_Tool.PSICalculator``: each
    variable is binned on the expected sample (equal-frequency or equal-width bins) and the current sample is
    cut with the same edges. With a ``binning_engine`` the PSI is computed on the bins of that engine instead
    (its WOE value is the bin label), so monitoring uses the same mapping as the model.

    Parameters
    ----------
    buckets : int, default 10
        Number of bins of the default binning. It has no effect when ``binning_engine`` is given.
    equal_freq : bool, default True
        Use equal-frequency bins (False gives equal-width bins). It has no effect when ``binning_engine`` is given.
    min_bin_prop : float, default 0.05
        Minimum proportion for each bin; it caps the number of bins at ``max(5, 1 / min_bin_prop)``. It has no
        effect when ``binning_engine`` is given.
    content : float, default 1e-06
        Floor of a bin share, so that the logarithm stays finite (see ``psi_missing_bucket_policy``).
    precision : int, default 5
        Decimals of the bin edges of the default binning. With ``binning_engine`` it is the number of decimals
        the PSI values are rounded to.
    binning_engine : WOE_Master, MonotoneWOEBinner, WOEEngineAdapter or None, default None
        Fitted WOE engine whose bins define the PSI. ``None`` keeps the default binning. An object of any other type
        raises ``TypeError``.
    missing_policy : {"include", "drop", "warn_and_drop"}, default "include"
        How NaN rows are handled by the default binning: ``"include"`` gives them a bin of their own
        (``"__MISSING__"``), so a change of the missing rate moves the PSI; ``"drop"`` ignores them;
        ``"warn_and_drop"`` drops them and emits a ``RuntimeWarning`` with the NaN counts. It has no effect when
        ``binning_engine`` is given (the engine decides how missing values are binned).
    psi_missing_bucket_policy : {"smooth_laplace", "floor_1e6", "exclude"}, default "smooth_laplace"
        How a bin that holds rows on one side only is treated. ``"smooth_laplace"`` adds one to every bin count
        when such a bin exists; ``"floor_1e6"`` clips the bin shares at ``content``; ``"exclude"`` drops the
        one-sided bins from the sum. Any other value raises ``ValueError``.
    feature_block_size : int or None, default 64
        Number of variables that the engine transforms per block (used only with ``binning_engine``). ``None``
        processes all variables at once. A value that is not a positive integer raises ``ValueError``.

    Attributes
    ----------
    buckets, equal_freq, min_bin_prop, content, precision, missing_policy, psi_missing_bucket_policy, feature_block_size
        The constructor values.
    binning_engine : WOE_Master, MonotoneWOEBinner, WOEEngineAdapter or None
        The engine as passed to the constructor.

    Examples
    --------
    >>> psi = PSICalculator(buckets=10, binning_engine=woe).calculate(train_df, oot_df, features)
    """

    def __init__(
        self,
        buckets: int = 10,
        equal_freq: bool = True,
        min_bin_prop: float = 0.05,
        content: float = 1e-6,
        precision: int = 5,
        binning_engine=None,
        missing_policy: str = "include",
        psi_missing_bucket_policy: str = "smooth_laplace",
        feature_block_size: int | None = 64,
    ):
        _validate_psi_bucket_policy(psi_missing_bucket_policy, "Feature.PSICalculator.__init__")
        self._base = _BasePSICalculator(
            buckets=buckets,
            equal_freq=equal_freq,
            min_bin_prop=min_bin_prop,
            content=content,
            precision=precision,
            missing_policy=missing_policy,
            psi_missing_bucket_policy=psi_missing_bucket_policy,
            feature_block_size=feature_block_size,
        )
        self.buckets = buckets
        self.equal_freq = equal_freq
        self.min_bin_prop = min_bin_prop
        self.content = content
        self.precision = precision
        self.missing_policy = missing_policy
        self.psi_missing_bucket_policy = psi_missing_bucket_policy
        if feature_block_size is not None and int(feature_block_size) <= 0:
            raise ValueError("feature_block_size must be a positive integer or None")
        self.feature_block_size = feature_block_size
        self.binning_engine = binning_engine
        self._woe_engine_adapter = as_woe_engine(binning_engine) if binning_engine is not None else None

    def __getattr__(self, name):
        return getattr(self._base, name)

    def _prepare_woe_bins(self, expected_df, current_data, varlist):
        adapter = self._woe_engine_adapter
        if adapter is None:
            return None
        return (
            adapter.assign_bins_frame(
                expected_df,
                varlist,
                feature_block_size=self.feature_block_size,
            ),
            adapter.assign_bins_frame(
                current_data,
                varlist,
                feature_block_size=self.feature_block_size,
            ),
        )

    @staticmethod
    def _group_codes(current_data, resolved_group):
        if resolved_group is None:
            return np.zeros(len(current_data), dtype=np.int64), [(None, {})]

        group_cols = [resolved_group] if isinstance(resolved_group, str) else list(resolved_group)
        group_frame = current_data[group_cols].copy()
        for col in group_cols:
            group_frame[col] = group_frame[col].where(group_frame[col].notna(), "__NULL__")
        group_key = group_cols[0] if len(group_cols) == 1 else group_cols
        grouped = group_frame.groupby(group_key, dropna=False, sort=True)
        codes = grouped.ngroup().to_numpy(dtype=np.int64)
        keys = list(grouped.size().index)
        groups = []
        for key in keys:
            values = key if isinstance(key, tuple) else (key,)
            groups.append((key, dict(zip(group_cols, values))))
        return codes, groups

    def _calculate_prebinned(
        self,
        expected_bins,
        current_bins,
        current_data,
        varlist,
        resolved_group,
        return_details,
        bucket_policy,
    ):
        group_codes, groups = self._group_codes(current_data, resolved_group)
        n_groups = len(groups)
        rows = []
        detail = {}

        for var in varlist:
            expected_values = expected_bins[var].to_numpy(dtype=object)
            current_values = current_bins[var].to_numpy(dtype=object)
            joined = np.concatenate([expected_values, current_values])
            bin_codes, categories = pd.factorize(joined, sort=False)
            expected_codes = bin_codes[: len(expected_values)]
            current_codes = bin_codes[len(expected_values) :]
            n_bins = len(categories)
            expected_count_values = np.bincount(expected_codes, minlength=n_bins)
            current_count_values = np.bincount(
                group_codes * n_bins + current_codes,
                minlength=n_groups * n_bins,
            ).reshape(n_groups, n_bins)
            expected_count = pd.Series(expected_count_values, index=categories)

            for group_idx, (group_value, group_info) in enumerate(groups):
                current_count = pd.Series(current_count_values[group_idx], index=categories)
                _, _, psi_values, _ = _psi_distributions_from_counts(
                    expected_count,
                    current_count,
                    float(expected_count_values.sum()),
                    float(current_count_values[group_idx].sum()),
                    content=self.content,
                    policy=bucket_policy,
                )
                row = {"var": var, "psi": round(float(psi_values.sum()), self.precision)}
                row.update(group_info)
                rows.append(row)
                if return_details:
                    detail_key = var if resolved_group is None else (var, group_value)
                    exp_total = max(float(expected_count_values.sum()), 1.0)
                    cur_total = max(float(current_count_values[group_idx].sum()), 1.0)
                    detail[detail_key] = {
                        "expected_bins": (expected_count / exp_total).sort_values(ascending=False),
                        "current_bins": (current_count / cur_total).sort_values(ascending=False),
                    }
        result = pd.DataFrame(rows)
        return (result, detail) if return_details else result

    def calculate(
        self,
        expected_df,
        current_data,
        varlist,
        group_by=None,
        group_name=None,
        return_details=False,
        missing_policy=None,
        psi_missing_bucket_policy=None,
    ):
        """Calculate the PSI of each variable between ``expected_df`` and ``current_data``.

        Parameters
        ----------
        expected_df : pandas.DataFrame
            Expected (baseline) sample. The default binning builds the bins on it.
        current_data : pandas.DataFrame
            Current sample that is compared with ``expected_df``.
        varlist : list of str
            Variables to compute the PSI for. With a ``binning_engine`` every variable must be one the engine was
            fitted on, otherwise ``KeyError`` is raised.
        group_by : str, list of str or None, default None
            Legacy grouping argument. With the default binning it has no effect unless ``group_name`` is also given
            or ``return_details`` is True; use ``group_name`` to get PSI by group. With a ``binning_engine`` it acts
            as ``group_name`` when ``group_name`` is None.
        group_name : str, list of str or None, default None
            Column(s) of ``current_data`` whose values define groups. Every group is compared with the whole
            ``expected_df`` and the bins are built once, on ``expected_df``. A NaN group value becomes ``"__NULL__"``.
            A list of columns is supported only with a ``binning_engine``.
        return_details : bool, default False
            Whether to return the bin-level details together with the PSI table (see Returns).
        missing_policy : {"include", "drop", "warn_and_drop"} or None, default None
            How NaN rows are handled by the default binning. ``None`` uses the value of the constructor. It has no
            effect when a ``binning_engine`` is used.
        psi_missing_bucket_policy : {"smooth_laplace", "floor_1e6", "exclude"} or None, default None
            How a bin that holds rows on one side only is treated. ``None`` uses the value of the constructor.

        Returns
        -------
        pandas.DataFrame, dict or tuple
            The PSI table has one row per variable (and per group value) with the columns ``var`` and ``psi`` plus
            the group column(s). With ``return_details=True``:

            - default binning: the dict ``{'psi': PSI table, 'details': DataFrame}``, where ``details`` has one row
              per bin with the columns ``bin``, ``expected_count``, ``actual_count``, ``expected_percent``,
              ``actual_percent``, ``psi_component``, ``bucket_status``, ``psi_missing_bucket_policy``, the group
              column when ``group_name`` is given, and ``var``;
            - ``binning_engine``: the tuple ``(PSI table, details)``, where ``details`` maps ``var`` (or
              ``(var, group value)`` when grouped) to ``{'expected_bins': Series, 'current_bins': Series}``, the
              share of rows per bin sorted in descending order.

        Raises
        ------
        ValueError
            If ``psi_missing_bucket_policy`` is not a valid value (or, with the default binning, ``missing_policy``).
        KeyError
            With a ``binning_engine``, if a variable of ``varlist`` was not fitted by the engine.

        Notes
        -----
        With a ``binning_engine`` the PSI values are rounded to ``precision`` decimals. The default binning does not
        round them.
        """
        adapter = self._woe_engine_adapter
        effective_missing_policy = missing_policy if missing_policy is not None else self.missing_policy
        effective_bucket_policy = (
            psi_missing_bucket_policy
            if psi_missing_bucket_policy is not None
            else self.psi_missing_bucket_policy
        )
        if adapter is None:
            return self._base.calculate(
                expected_df,
                current_data,
                varlist,
                group_by,
                group_name,
                return_details,
                missing_policy=effective_missing_policy,
                psi_missing_bucket_policy=effective_bucket_policy,
            )

        resolved_group = group_name if group_name is not None else group_by
        expected_bins, current_bins = self._prepare_woe_bins(expected_df, current_data, varlist)
        return self._calculate_prebinned(
            expected_bins,
            current_bins,
            current_data,
            varlist,
            resolved_group,
            return_details,
            effective_bucket_policy,
        )

buckets instance-attribute

buckets = buckets

equal_freq instance-attribute

equal_freq = equal_freq

min_bin_prop instance-attribute

min_bin_prop = min_bin_prop

content instance-attribute

content = content

precision instance-attribute

precision = precision

missing_policy instance-attribute

missing_policy = missing_policy

psi_missing_bucket_policy instance-attribute

psi_missing_bucket_policy = psi_missing_bucket_policy

feature_block_size instance-attribute

feature_block_size = feature_block_size

binning_engine instance-attribute

binning_engine = binning_engine

calculate

calculate(expected_df, current_data, varlist, group_by=None, group_name=None, return_details=False, missing_policy=None, psi_missing_bucket_policy=None)

Calculate the PSI of each variable between expected_df and current_data.

Parameters:

Name	Type	Description	Default
expected_df	DataFrame	Expected (baseline) sample. The default binning builds the bins on it.	required
current_data	DataFrame	Current sample that is compared with expected_df.	required
varlist	list of str	Variables to compute the PSI for. With a binning_engine every variable must be one the engine was fitted on, otherwise KeyError is raised.	required
group_by	str, list of str or None	Legacy grouping argument. With the default binning it has no effect unless group_name is also given or return_details is True; use group_name to get PSI by group. With a binning_engine it acts as group_name when group_name is None.	None
group_name	str, list of str or None	Column(s) of current_data whose values define groups. Every group is compared with the whole expected_df and the bins are built once, on expected_df. A NaN group value becomes "__NULL__". A list of columns is supported only with a binning_engine.	None
return_details	bool	Whether to return the bin-level details together with the PSI table (see Returns).	False
missing_policy	('include', 'drop', 'warn_and_drop')	How NaN rows are handled by the default binning. None uses the value of the constructor. It has no effect when a binning_engine is used.	"include"
psi_missing_bucket_policy	('smooth_laplace', 'floor_1e6', 'exclude')	How a bin that holds rows on one side only is treated. None uses the value of the constructor.	"smooth_laplace"

Returns:

Type

Description

(DataFrame, dict or tuple)

The PSI table has one row per variable (and per group value) with the columns var and psi plus the group column(s). With return_details=True: default binning: the dict {'psi': PSI table, 'details': DataFrame}, where details has one row per bin with the columns bin, expected_count, actual_count, expected_percent, actual_percent, psi_component, bucket_status, psi_missing_bucket_policy, the group column when group_name is given, and var; binning_engine: the tuple (PSI table, details), where details maps var (or (var, group value) when grouped) to {'expected_bins': Series, 'current_bins': Series}, the share of rows per bin sorted in descending order.

Raises:

Type	Description
ValueError	If psi_missing_bucket_policy is not a valid value (or, with the default binning, missing_policy).
KeyError	With a binning_engine, if a variable of varlist was not fitted by the engine.

Notes

With a binning_engine the PSI values are rounded to precision decimals. The default binning does not round them.

Source code in Modeling_Tool/Feature/WOE_Engine_Feature_Patch.py

def calculate(
    self,
    expected_df,
    current_data,
    varlist,
    group_by=None,
    group_name=None,
    return_details=False,
    missing_policy=None,
    psi_missing_bucket_policy=None,
):
    """Calculate the PSI of each variable between ``expected_df`` and ``current_data``.

    Parameters
    ----------
    expected_df : pandas.DataFrame
        Expected (baseline) sample. The default binning builds the bins on it.
    current_data : pandas.DataFrame
        Current sample that is compared with ``expected_df``.
    varlist : list of str
        Variables to compute the PSI for. With a ``binning_engine`` every variable must be one the engine was
        fitted on, otherwise ``KeyError`` is raised.
    group_by : str, list of str or None, default None
        Legacy grouping argument. With the default binning it has no effect unless ``group_name`` is also given
        or ``return_details`` is True; use ``group_name`` to get PSI by group. With a ``binning_engine`` it acts
        as ``group_name`` when ``group_name`` is None.
    group_name : str, list of str or None, default None
        Column(s) of ``current_data`` whose values define groups. Every group is compared with the whole
        ``expected_df`` and the bins are built once, on ``expected_df``. A NaN group value becomes ``"__NULL__"``.
        A list of columns is supported only with a ``binning_engine``.
    return_details : bool, default False
        Whether to return the bin-level details together with the PSI table (see Returns).
    missing_policy : {"include", "drop", "warn_and_drop"} or None, default None
        How NaN rows are handled by the default binning. ``None`` uses the value of the constructor. It has no
        effect when a ``binning_engine`` is used.
    psi_missing_bucket_policy : {"smooth_laplace", "floor_1e6", "exclude"} or None, default None
        How a bin that holds rows on one side only is treated. ``None`` uses the value of the constructor.

    Returns
    -------
    pandas.DataFrame, dict or tuple
        The PSI table has one row per variable (and per group value) with the columns ``var`` and ``psi`` plus
        the group column(s). With ``return_details=True``:

        - default binning: the dict ``{'psi': PSI table, 'details': DataFrame}``, where ``details`` has one row
          per bin with the columns ``bin``, ``expected_count``, ``actual_count``, ``expected_percent``,
          ``actual_percent``, ``psi_component``, ``bucket_status``, ``psi_missing_bucket_policy``, the group
          column when ``group_name`` is given, and ``var``;
        - ``binning_engine``: the tuple ``(PSI table, details)``, where ``details`` maps ``var`` (or
          ``(var, group value)`` when grouped) to ``{'expected_bins': Series, 'current_bins': Series}``, the
          share of rows per bin sorted in descending order.

    Raises
    ------
    ValueError
        If ``psi_missing_bucket_policy`` is not a valid value (or, with the default binning, ``missing_policy``).
    KeyError
        With a ``binning_engine``, if a variable of ``varlist`` was not fitted by the engine.

    Notes
    -----
    With a ``binning_engine`` the PSI values are rounded to ``precision`` decimals. The default binning does not
    round them.
    """
    adapter = self._woe_engine_adapter
    effective_missing_policy = missing_policy if missing_policy is not None else self.missing_policy
    effective_bucket_policy = (
        psi_missing_bucket_policy
        if psi_missing_bucket_policy is not None
        else self.psi_missing_bucket_policy
    )
    if adapter is None:
        return self._base.calculate(
            expected_df,
            current_data,
            varlist,
            group_by,
            group_name,
            return_details,
            missing_policy=effective_missing_policy,
            psi_missing_bucket_policy=effective_bucket_policy,
        )

    resolved_group = group_name if group_name is not None else group_by
    expected_bins, current_bins = self._prepare_woe_bins(expected_df, current_data, varlist)
    return self._calculate_prebinned(
        expected_bins,
        current_bins,
        current_data,
        varlist,
        resolved_group,
        return_details,
        effective_bucket_policy,
    )

VarExtractionInsights

Variable insight analyzer: IV, KS and lift per variable, and bivariate WOE charts.

Without an engine the calls are delegated to Modeling_Tool.Feature.Feature_Insights.VarExtractionInsights, which bins every variable by its own settings (decision-tree bins by default). With woe_binner (a fitted WOE_Master or MonotoneWOEBinner) or with woe_engine="monotone", the bins and the IV come from that engine instead.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input data. It is stored on the instance, and every method takes its own data argument.	required
dep	str	Name of the target column (0/1).	required
plot_path	str or None	Root folder of the charts written by plot_woe. Pass None if you do not plot. get_var_analysis_report writes no files.	required
nbins	int	Number of bins per variable.	10
equal_freq	bool	Use equal-frequency bins (False gives equal-width bins).	True
min_bin_prop	float	Minimum proportion of rows per bin.	0.05
precision	int	Decimals of the bin edges.	5
chi2_method	bool	Merge an initial fine binning with a chi-square test.	False
chi2_p	float	Confidence level of the chi-square merge.	0.9
init_equi_bins	int	Number of fine bins before the chi-square merge.	5000
tree_binning	bool	Place the bin edges with a decision tree on the target (supervised bins).	True
include_missing	bool	Missing values form their own bin.	True
seed	int	Random seed of the tree binning.	3407
missing_rate_ref	int or float	Sentinel for missing values. It fills NaN in the default binning and is treated as missing in the missing_rate, min, mean and max columns of the report.	-999999
spec_values	list or None	Special values that get their own bins. None is stored as an empty list.	None
woe_engine	str	"monotone" makes the object fit its own MonotoneWOEBinner on first use and keep it in woe_binner. Any other value behaves like "master" (no self-fitted engine).	"master"
woe_binner	(WOE_Master, MonotoneWOEBinner or None)	A fitted engine. It is used whenever it is given, whatever woe_engine says: bins and IV then come from it, and nbins, equal_freq, min_bin_prop, precision, chi2_method, chi2_p, init_equi_bins, tree_binning, include_missing, seed and spec_values are ignored by the report. An object of another type raises TypeError when a method uses it.	None
woe_engine_params	dict or None	Constructor arguments of the self-fitted MonotoneWOEBinner (woe_engine="monotone" without woe_binner). The key fit_params is passed on to its fit. chi2_method, chi2_p, init_equi_bins and spec_values also feed that fit. None is stored as an empty dict.	None
screen_iv_convention	('guarded', 'raw')	How the IV on the bins of a WOE engine treats a class-pure bin (a bin without bad rows or without good rows, whose WOE ln((bad_share + 1e-6) / (good_share + 1e-6)) comes from the 1e-6 alone). "guarded" leaves such bins out of the IV sum (their rows stay in the bad and good totals, so every other bin keeps its WOE and IV; a special-value bin that the engine smoothed with sv_woe_smoothing="laplace" counts with its WOE); "raw" counts every bin, the IV up to 0.9.1. The default binning (no engine) is not affected. Any other value raises ValueError.	"guarded"

Attributes:

Name	Type	Description
data, dep, plot_path, nbins, equal_freq, min_bin_prop, precision, chi2_method, chi2_p, init_equi_bins, tree_binning, include_missing, seed, missing_rate_ref, spec_values, woe_engine, woe_engine_params, screen_iv_convention		The constructor values.
woe_binner	(WOE_Master, MonotoneWOEBinner or None)	The engine of the constructor, or the self-fitted MonotoneWOEBinner after the first call that needed it.
failed_variables	list of tuple	(variable, exception type name) pairs of the variables that the last get_var_analysis_report call could not analyze.
iv_sv_pure_bins	DataFrame	The class-pure special-value bins found by the last get_var_analysis_report call on the bins of an engine, one row per bin with the columns var, special_value, bin_label, n (rows), bad, good, bad_rate, woe and iv (the bin's contribution) and low_count (n < 30). A bin whose rows are all missing counts as one (special_value NaN). Empty (with these columns) when there is none or no engine is used. It is a list to review, for example as candidate rules; nothing acts on it.

Examples:

>>> insights = VarExtractionInsights(train_df, "bad_flag", None, woe_binner=woe)
>>> report = insights.get_var_analysis_report(train_df, features, iv_cut=0)

Source code in Modeling_Tool/Feature/WOE_Engine_Feature_Patch.py

class VarExtractionInsights:
    """Variable insight analyzer: IV, KS and lift per variable, and bivariate WOE charts.

    Without an engine the calls are delegated to ``Modeling_Tool.Feature.Feature_Insights.VarExtractionInsights``, which
    bins every variable by its own settings (decision-tree bins by default). With ``woe_binner`` (a fitted ``WOE_Master``
    or ``MonotoneWOEBinner``) or with ``woe_engine="monotone"``, the bins and the IV come from that engine instead.

    Parameters
    ----------
    data : pandas.DataFrame
        Input data. It is stored on the instance, and every method takes its own ``data`` argument.
    dep : str
        Name of the target column (0/1).
    plot_path : str or None
        Root folder of the charts written by ``plot_woe``. Pass ``None`` if you do not plot.
        ``get_var_analysis_report`` writes no files.
    nbins : int, default 10
        Number of bins per variable.
    equal_freq : bool, default True
        Use equal-frequency bins (False gives equal-width bins).
    min_bin_prop : float, default 0.05
        Minimum proportion of rows per bin.
    precision : int, default 5
        Decimals of the bin edges.
    chi2_method : bool, default False
        Merge an initial fine binning with a chi-square test.
    chi2_p : float, default 0.9
        Confidence level of the chi-square merge.
    init_equi_bins : int, default 5000
        Number of fine bins before the chi-square merge.
    tree_binning : bool, default True
        Place the bin edges with a decision tree on the target (supervised bins).
    include_missing : bool, default True
        Missing values form their own bin.
    seed : int, default 3407
        Random seed of the tree binning.
    missing_rate_ref : int or float, default -999999
        Sentinel for missing values. It fills ``NaN`` in the default binning and is treated as missing in the
        ``missing_rate``, ``min``, ``mean`` and ``max`` columns of the report.
    spec_values : list or None, default None
        Special values that get their own bins. ``None`` is stored as an empty list.
    woe_engine : str, default "master"
        ``"monotone"`` makes the object fit its own ``MonotoneWOEBinner`` on first use and keep it in
        ``woe_binner``. Any other value behaves like ``"master"`` (no self-fitted engine).
    woe_binner : WOE_Master, MonotoneWOEBinner or None, default None
        A fitted engine. It is used whenever it is given, whatever ``woe_engine`` says: bins and IV then come from
        it, and ``nbins``, ``equal_freq``, ``min_bin_prop``, ``precision``, ``chi2_method``, ``chi2_p``,
        ``init_equi_bins``, ``tree_binning``, ``include_missing``, ``seed`` and ``spec_values`` are ignored by the
        report. An object of another type raises ``TypeError`` when a method uses it.
    woe_engine_params : dict or None, default None
        Constructor arguments of the self-fitted ``MonotoneWOEBinner`` (``woe_engine="monotone"`` without
        ``woe_binner``). The key ``fit_params`` is passed on to its ``fit``. ``chi2_method``, ``chi2_p``,
        ``init_equi_bins`` and ``spec_values`` also feed that fit. ``None`` is stored as an empty dict.
    screen_iv_convention : {"guarded", "raw"}, default "guarded"
        How the IV on the bins of a WOE engine treats a class-pure bin (a bin without bad rows or without good rows,
        whose WOE ``ln((bad_share + 1e-6) / (good_share + 1e-6))`` comes from the ``1e-6`` alone). ``"guarded"`` leaves
        such bins out of the IV sum (their rows stay in the bad and good totals, so every other bin keeps its WOE and
        IV; a special-value bin that the engine smoothed with ``sv_woe_smoothing="laplace"`` counts with its WOE);
        ``"raw"`` counts every bin, the IV up to 0.9.1. The default binning (no engine) is not affected. Any other
        value raises ``ValueError``.

    Attributes
    ----------
    data, dep, plot_path, nbins, equal_freq, min_bin_prop, precision, chi2_method, chi2_p, init_equi_bins, tree_binning, include_missing, seed, missing_rate_ref, spec_values, woe_engine, woe_engine_params, screen_iv_convention
        The constructor values.
    woe_binner : WOE_Master, MonotoneWOEBinner or None
        The engine of the constructor, or the self-fitted ``MonotoneWOEBinner`` after the first call that needed it.
    failed_variables : list of tuple
        ``(variable, exception type name)`` pairs of the variables that the last ``get_var_analysis_report`` call
        could not analyze.
    iv_sv_pure_bins : pandas.DataFrame
        The class-pure special-value bins found by the last ``get_var_analysis_report`` call on the bins of an engine,
        one row per bin with the columns ``var``, ``special_value``, ``bin_label``, ``n`` (rows), ``bad``, ``good``,
        ``bad_rate``, ``woe`` and ``iv`` (the bin's contribution) and ``low_count`` (``n < 30``). A bin whose rows are
        all missing counts as one (``special_value`` NaN). Empty (with these columns) when there is none or no engine
        is used. It is a list to review, for example as candidate rules; nothing acts on it.

    Examples
    --------
    >>> insights = VarExtractionInsights(train_df, "bad_flag", None, woe_binner=woe)
    >>> report = insights.get_var_analysis_report(train_df, features, iv_cut=0)
    """

    def __init__(
        self,
        data,
        dep,
        plot_path,
        nbins=10,
        equal_freq=True,
        min_bin_prop=0.05,
        precision=5,
        chi2_method=False,
        chi2_p=0.9,
        init_equi_bins=5000,
        tree_binning=True,
        include_missing=True,
        seed=3407,
        missing_rate_ref=-999999,
        spec_values=None,
        woe_engine="master",
        woe_binner=None,
        woe_engine_params=None,
        screen_iv_convention="guarded",
    ):
        self._base = _BaseVarExtractionInsights(
            data, dep, plot_path, nbins, equal_freq, min_bin_prop, precision,
            chi2_method, chi2_p, init_equi_bins, tree_binning, include_missing,
            seed, missing_rate_ref, spec_values,
        )
        self.data = data
        self.dep = dep
        self.plot_path = plot_path
        self.nbins = nbins
        self.equal_freq = equal_freq
        self.min_bin_prop = min_bin_prop
        self.precision = precision
        self.chi2_method = chi2_method
        self.chi2_p = chi2_p
        self.init_equi_bins = init_equi_bins
        self.tree_binning = tree_binning
        self.include_missing = include_missing
        self.seed = seed
        self.missing_rate_ref = missing_rate_ref
        self.spec_values = spec_values if spec_values is not None else []
        self.woe_engine = woe_engine
        self.woe_binner = woe_binner
        self.woe_engine_params = woe_engine_params or {}
        self.screen_iv_convention = check_screen_iv_convention(screen_iv_convention)
        self.failed_variables = []
        self.iv_sv_pure_bins = sv_pure_frame([])

    def __getattr__(self, name):
        return getattr(self._base, name)

    @staticmethod
    def remove_folder(file_path):
        """Delete the specified folder.

        Recursively delete the folder at the given path together with all of its
        contents; do nothing (silently) if the folder does not exist.

        Parameters
        ----------
        file_path : str
            Path of the folder to delete.
        """
        return _BaseVarExtractionInsights.remove_folder(file_path)

    def get_var_analysis_report(self, data, varlist, dep=None, iv_cut=0.01):
        """Compute IV, KS and lift of each variable and return those that reach ``iv_cut``.

        Parameters
        ----------
        data : pandas.DataFrame
            Data that holds the variables and the target column.
        varlist : list of str
            Variables to analyze.
        dep : str or None, default None
            Name of the target column. ``None`` uses the ``dep`` of the constructor. The argument is honored only
            when a WOE engine supplies the bins (``woe_binner``, or ``woe_engine="monotone"``): the default binning
            always uses the ``dep`` of the constructor.
        iv_cut : float, default 0.01
            IV threshold. Variables with ``iv < iv_cut`` are left out of the report; pass 0 to keep them all.

        Returns
        -------
        pandas.DataFrame
            One row per reported variable with the columns ``var``, ``n_all`` (rows), ``n`` (non-missing rows),
            ``ks_in_gains``, ``lift_in_gains`` (largest bin lift), ``iv`` (rounded to 4 decimals), ``n_bump``,
            ``n_bins``, ``missing_rate``, ``min``, ``mean`` and ``max``. ``n_bump`` counts the rank-order breaks
            between the bins of the default binning, and equals the bin count when a WOE engine is used. The
            default binning sorts the rows by ``iv``, descending; with a WOE engine they follow ``varlist``. The
            frame is empty (with these columns) when no variable could be analyzed. With a WOE engine ``iv`` follows
            ``screen_iv_convention`` (``iv_cut`` is applied to it), and the report adds ``iv_raw`` (every bin
            counted), ``iv_sv_pure`` (the IV of the class-pure bins) and ``n_sv_pure`` (their rows); the class-pure
            special-value bins are listed in ``iv_sv_pure_bins``.

        Warns
        -----
        UserWarning
            If some variables could not be analyzed (they are listed in ``failed_variables``).

        Notes
        -----
        A constant column is skipped silently. A variable that cannot be binned is left out and recorded in
        ``failed_variables`` as ``(name, exception type)``. With a WOE engine, a variable that is missing from
        ``data`` is skipped silently, and a variable that the engine was not fitted on is recorded as failed (a
        self-fitted ``MonotoneWOEBinner`` is fitted once, on the ``varlist`` of the first call). The default binning
        handles numeric variables only.
        """
        if dep is None:
            dep = self.dep
        self.failed_variables = []
        self.iv_sv_pure_bins = sv_pure_frame([])
        adapter = as_woe_engine(self.woe_binner) if self.woe_binner is not None else _make_monotone_adapter(self, data, varlist)
        if adapter is None:
            result = self._base.get_var_analysis_report(data, varlist, dep, iv_cut)
            self.failed_variables = getattr(self._base, "failed_variables", [])
            return result
        sv_pure_rows: list[dict] = []
        result = _screening_summary_from_engine(
            data,
            varlist,
            dep,
            adapter,
            iv_cut,
            self.missing_rate_ref,
            failed_variables=self.failed_variables,
            screen_iv_convention=getattr(self, "screen_iv_convention", "guarded"),
            sv_pure_rows=sv_pure_rows,
        )
        self.iv_sv_pure_bins = sv_pure_frame(sv_pure_rows)
        if self.failed_variables:
            failed = [name for name, _ in self.failed_variables]
            warnings.warn(
                f"{len(failed)}/{len(varlist)} variables failed WOE-engine insight computation: "
                f"{failed[:10]}{'...' if len(failed) > 10 else ''} — see smf_logger for details",
                UserWarning,
                stacklevel=2,
            )
        return result

    def plot_woe(self, data, varlist, plot_group=None, plot_dirname="var_analysis_plot", plot_path=None):
        """Plot the bivariate WOE charts of the variables and save them as PNG files.

        Parameters
        ----------
        data : pandas.DataFrame
            Data that holds the variables and the target column (and ``plot_group`` when it is given).
        varlist : list of str
            Variables to plot. When the object holds a ``MonotoneWOEBinner`` (given as ``woe_binner`` or
            self-fitted), every variable that the binner was fitted on is plotted and ``varlist`` is ignored.
        plot_group : str or None, default None
            Name of a grouping column. A chart per variable split by this column is written as well
            (``<var>_<plot_group>.png``). With a ``MonotoneWOEBinner`` the charts of the groups
            (``<var>_by_<plot_group>.png``) replace the ungrouped ones. The group values must be strings in the
            default plot (numeric values raise ``TypeError``).
        plot_dirname : str, default "var_analysis_plot"
            Subdirectory of ``plot_path`` that receives the charts. The folders are created if they do not exist.
        plot_path : str or None, default None
            Root folder of the charts. ``None`` uses the ``plot_path`` of the constructor.

        Returns
        -------
        None
            Saves the images directly.

        Raises
        ------
        TypeError
            If no plot path is available (``plot_path`` and the constructor's ``plot_path`` are both None) while the
            default binning is used. With a ``MonotoneWOEBinner`` the method returns without plotting instead.

        Notes
        -----
        Only a ``MonotoneWOEBinner`` plots from its own bins. Without an engine, or with a ``WOE_Master`` as
        ``woe_binner``, a new ``WOE_Master`` is fitted with the binning settings of the constructor (missing values
        are first filled with ``missing_rate_ref``) and the charts are drawn from it.
        """
        adapter = as_woe_engine(self.woe_binner) if self.woe_binner is not None else _make_monotone_adapter(self, data, varlist)
        if adapter is None:
            return self._base.plot_woe(data, varlist, plot_group, plot_dirname, plot_path)
        if plot_path is None:
            plot_path = self.plot_path
        if plot_path is None:
            return None
        if adapter.get_engine_name() == "monotone" and hasattr(adapter.engine, "plot_woe_graph"):
            import os
            graph_path = os.path.join(plot_path, plot_dirname)
            adapter.engine.plot_woe_graph(graph_path, group_name=plot_group, _df_for_group=data if plot_group else None)
            return None
        return self._base.plot_woe(data, varlist, plot_group, plot_dirname, plot_path)

data instance-attribute

data = data

dep instance-attribute

dep = dep

plot_path instance-attribute

plot_path = plot_path

nbins instance-attribute

nbins = nbins

equal_freq instance-attribute

equal_freq = equal_freq

min_bin_prop instance-attribute

min_bin_prop = min_bin_prop

precision instance-attribute

precision = precision

chi2_method instance-attribute

chi2_method = chi2_method

chi2_p instance-attribute

chi2_p = chi2_p

init_equi_bins instance-attribute

init_equi_bins = init_equi_bins

tree_binning instance-attribute

tree_binning = tree_binning

include_missing instance-attribute

include_missing = include_missing

seed instance-attribute

seed = seed

missing_rate_ref instance-attribute

missing_rate_ref = missing_rate_ref

spec_values instance-attribute

spec_values = spec_values if spec_values is not None else []

woe_engine instance-attribute

woe_engine = woe_engine

woe_binner instance-attribute

woe_binner = woe_binner

woe_engine_params instance-attribute

woe_engine_params = woe_engine_params or {}

screen_iv_convention instance-attribute

screen_iv_convention = check_screen_iv_convention(screen_iv_convention)

failed_variables instance-attribute

failed_variables = []

iv_sv_pure_bins instance-attribute

iv_sv_pure_bins = sv_pure_frame([])

remove_folder staticmethod

remove_folder(file_path)

Delete the specified folder.

Recursively delete the folder at the given path together with all of its contents; do nothing (silently) if the folder does not exist.

Parameters:

Name	Type	Description	Default
file_path	str	Path of the folder to delete.	required

Source code in Modeling_Tool/Feature/WOE_Engine_Feature_Patch.py

@staticmethod
def remove_folder(file_path):
    """Delete the specified folder.

    Recursively delete the folder at the given path together with all of its
    contents; do nothing (silently) if the folder does not exist.

    Parameters
    ----------
    file_path : str
        Path of the folder to delete.
    """
    return _BaseVarExtractionInsights.remove_folder(file_path)

get_var_analysis_report

get_var_analysis_report(data, varlist, dep=None, iv_cut=0.01)

Compute IV, KS and lift of each variable and return those that reach iv_cut.

Parameters:

Name	Type	Description	Default
data	DataFrame	Data that holds the variables and the target column.	required
varlist	list of str	Variables to analyze.	required
dep	str or None	Name of the target column. None uses the dep of the constructor. The argument is honored only when a WOE engine supplies the bins (woe_binner, or woe_engine="monotone"): the default binning always uses the dep of the constructor.	None
iv_cut	float	IV threshold. Variables with iv < iv_cut are left out of the report; pass 0 to keep them all.	0.01

Returns:

Type

Description

DataFrame

One row per reported variable with the columns var, n_all (rows), n (non-missing rows), ks_in_gains, lift_in_gains (largest bin lift), iv (rounded to 4 decimals), n_bump, n_bins, missing_rate, min, mean and max. n_bump counts the rank-order breaks between the bins of the default binning, and equals the bin count when a WOE engine is used. The default binning sorts the rows by iv, descending; with a WOE engine they follow varlist. The frame is empty (with these columns) when no variable could be analyzed. With a WOE engine iv follows screen_iv_convention (iv_cut is applied to it), and the report adds iv_raw (every bin counted), iv_sv_pure (the IV of the class-pure bins) and n_sv_pure (their rows); the class-pure special-value bins are listed in iv_sv_pure_bins.

Warns:

Type	Description
UserWarning	If some variables could not be analyzed (they are listed in failed_variables).

Notes

A constant column is skipped silently. A variable that cannot be binned is left out and recorded in failed_variables as (name, exception type). With a WOE engine, a variable that is missing from data is skipped silently, and a variable that the engine was not fitted on is recorded as failed (a self-fitted MonotoneWOEBinner is fitted once, on the varlist of the first call). The default binning handles numeric variables only.

Source code in Modeling_Tool/Feature/WOE_Engine_Feature_Patch.py

def get_var_analysis_report(self, data, varlist, dep=None, iv_cut=0.01):
    """Compute IV, KS and lift of each variable and return those that reach ``iv_cut``.

    Parameters
    ----------
    data : pandas.DataFrame
        Data that holds the variables and the target column.
    varlist : list of str
        Variables to analyze.
    dep : str or None, default None
        Name of the target column. ``None`` uses the ``dep`` of the constructor. The argument is honored only
        when a WOE engine supplies the bins (``woe_binner``, or ``woe_engine="monotone"``): the default binning
        always uses the ``dep`` of the constructor.
    iv_cut : float, default 0.01
        IV threshold. Variables with ``iv < iv_cut`` are left out of the report; pass 0 to keep them all.

    Returns
    -------
    pandas.DataFrame
        One row per reported variable with the columns ``var``, ``n_all`` (rows), ``n`` (non-missing rows),
        ``ks_in_gains``, ``lift_in_gains`` (largest bin lift), ``iv`` (rounded to 4 decimals), ``n_bump``,
        ``n_bins``, ``missing_rate``, ``min``, ``mean`` and ``max``. ``n_bump`` counts the rank-order breaks
        between the bins of the default binning, and equals the bin count when a WOE engine is used. The
        default binning sorts the rows by ``iv``, descending; with a WOE engine they follow ``varlist``. The
        frame is empty (with these columns) when no variable could be analyzed. With a WOE engine ``iv`` follows
        ``screen_iv_convention`` (``iv_cut`` is applied to it), and the report adds ``iv_raw`` (every bin
        counted), ``iv_sv_pure`` (the IV of the class-pure bins) and ``n_sv_pure`` (their rows); the class-pure
        special-value bins are listed in ``iv_sv_pure_bins``.

    Warns
    -----
    UserWarning
        If some variables could not be analyzed (they are listed in ``failed_variables``).

    Notes
    -----
    A constant column is skipped silently. A variable that cannot be binned is left out and recorded in
    ``failed_variables`` as ``(name, exception type)``. With a WOE engine, a variable that is missing from
    ``data`` is skipped silently, and a variable that the engine was not fitted on is recorded as failed (a
    self-fitted ``MonotoneWOEBinner`` is fitted once, on the ``varlist`` of the first call). The default binning
    handles numeric variables only.
    """
    if dep is None:
        dep = self.dep
    self.failed_variables = []
    self.iv_sv_pure_bins = sv_pure_frame([])
    adapter = as_woe_engine(self.woe_binner) if self.woe_binner is not None else _make_monotone_adapter(self, data, varlist)
    if adapter is None:
        result = self._base.get_var_analysis_report(data, varlist, dep, iv_cut)
        self.failed_variables = getattr(self._base, "failed_variables", [])
        return result
    sv_pure_rows: list[dict] = []
    result = _screening_summary_from_engine(
        data,
        varlist,
        dep,
        adapter,
        iv_cut,
        self.missing_rate_ref,
        failed_variables=self.failed_variables,
        screen_iv_convention=getattr(self, "screen_iv_convention", "guarded"),
        sv_pure_rows=sv_pure_rows,
    )
    self.iv_sv_pure_bins = sv_pure_frame(sv_pure_rows)
    if self.failed_variables:
        failed = [name for name, _ in self.failed_variables]
        warnings.warn(
            f"{len(failed)}/{len(varlist)} variables failed WOE-engine insight computation: "
            f"{failed[:10]}{'...' if len(failed) > 10 else ''} — see smf_logger for details",
            UserWarning,
            stacklevel=2,
        )
    return result

plot_woe

plot_woe(data, varlist, plot_group=None, plot_dirname='var_analysis_plot', plot_path=None)

Plot the bivariate WOE charts of the variables and save them as PNG files.

Parameters:

Name	Type	Description	Default
data	DataFrame	Data that holds the variables and the target column (and plot_group when it is given).	required
varlist	list of str	Variables to plot. When the object holds a MonotoneWOEBinner (given as woe_binner or self-fitted), every variable that the binner was fitted on is plotted and varlist is ignored.	required
plot_group	str or None	Name of a grouping column. A chart per variable split by this column is written as well (<var>_<plot_group>.png). With a MonotoneWOEBinner the charts of the groups (<var>_by_<plot_group>.png) replace the ungrouped ones. The group values must be strings in the default plot (numeric values raise TypeError).	None
plot_dirname	str	Subdirectory of plot_path that receives the charts. The folders are created if they do not exist.	"var_analysis_plot"
plot_path	str or None	Root folder of the charts. None uses the plot_path of the constructor.	None

Returns:

Type	Description
None	Saves the images directly.

Raises:

Type	Description
TypeError	If no plot path is available (plot_path and the constructor's plot_path are both None) while the default binning is used. With a MonotoneWOEBinner the method returns without plotting instead.

Notes

Only a MonotoneWOEBinner plots from its own bins. Without an engine, or with a WOE_Master as woe_binner, a new WOE_Master is fitted with the binning settings of the constructor (missing values are first filled with missing_rate_ref) and the charts are drawn from it.

Source code in Modeling_Tool/Feature/WOE_Engine_Feature_Patch.py

def plot_woe(self, data, varlist, plot_group=None, plot_dirname="var_analysis_plot", plot_path=None):
    """Plot the bivariate WOE charts of the variables and save them as PNG files.

    Parameters
    ----------
    data : pandas.DataFrame
        Data that holds the variables and the target column (and ``plot_group`` when it is given).
    varlist : list of str
        Variables to plot. When the object holds a ``MonotoneWOEBinner`` (given as ``woe_binner`` or
        self-fitted), every variable that the binner was fitted on is plotted and ``varlist`` is ignored.
    plot_group : str or None, default None
        Name of a grouping column. A chart per variable split by this column is written as well
        (``<var>_<plot_group>.png``). With a ``MonotoneWOEBinner`` the charts of the groups
        (``<var>_by_<plot_group>.png``) replace the ungrouped ones. The group values must be strings in the
        default plot (numeric values raise ``TypeError``).
    plot_dirname : str, default "var_analysis_plot"
        Subdirectory of ``plot_path`` that receives the charts. The folders are created if they do not exist.
    plot_path : str or None, default None
        Root folder of the charts. ``None`` uses the ``plot_path`` of the constructor.

    Returns
    -------
    None
        Saves the images directly.

    Raises
    ------
    TypeError
        If no plot path is available (``plot_path`` and the constructor's ``plot_path`` are both None) while the
        default binning is used. With a ``MonotoneWOEBinner`` the method returns without plotting instead.

    Notes
    -----
    Only a ``MonotoneWOEBinner`` plots from its own bins. Without an engine, or with a ``WOE_Master`` as
    ``woe_binner``, a new ``WOE_Master`` is fitted with the binning settings of the constructor (missing values
    are first filled with ``missing_rate_ref``) and the charts are drawn from it.
    """
    adapter = as_woe_engine(self.woe_binner) if self.woe_binner is not None else _make_monotone_adapter(self, data, varlist)
    if adapter is None:
        return self._base.plot_woe(data, varlist, plot_group, plot_dirname, plot_path)
    if plot_path is None:
        plot_path = self.plot_path
    if plot_path is None:
        return None
    if adapter.get_engine_name() == "monotone" and hasattr(adapter.engine, "plot_woe_graph"):
        import os
        graph_path = os.path.join(plot_path, plot_dirname)
        adapter.engine.plot_woe_graph(graph_path, group_name=plot_group, _df_for_group=data if plot_group else None)
        return None
    return self._base.plot_woe(data, varlist, plot_group, plot_dirname, plot_path)

CorrelationFilter

Correlation filter: drop the weaker variable of each highly correlated pair.

Pairs whose absolute correlation exceeds corr_cutpoint are found, and from each group of correlated variables the one with the highest IV (or KS) is kept. Without an engine the calls are delegated to Modeling_Tool.Feature.Feature_Insights.CorrelationFilter; with woe_binner (a fitted WOE_Master or MonotoneWOEBinner) or woe_engine="monotone", the IV and KS come from that engine.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input data (raw values). The correlation is computed on the raw values of the numeric variables.	required
dep	str	Name of the target column (0/1).	required
corr_cutpoint	float	Correlation threshold: a pair is correlated when abs(corr) > corr_cutpoint.	0.8
method	str	Correlation method, "pearson", "spearman" or "kendall", passed to DataFrame.corr.	"pearson"
tree_binning	bool	Use decision-tree bins in the default IV and KS computation. Ignored when an engine supplies the bins.	False
chi2_method	bool	Merge an initial fine binning with a chi-square test in the default computation. With woe_engine="monotone" it also feeds the self-fitted binner.	False
seed	int	Random seed of the tree binning. Ignored when an engine supplies the bins.	42
chi2_p	float	Confidence level of the chi-square merge.	0.999
init_equi_bins	int	Number of fine bins before the chi-square merge.	1000
missing_rate_ref	int or float	Sentinel for missing values in the default IV and KS computation. Ignored when an engine supplies the bins.	-9999999
spec_values	list	Special values (for example -1 or 999 for "no record") that get bins or rows of their own in the IV and KS calculation, with the default computation and with an engine alike. The correlation itself is still computed on the raw values.	[]
base_metric	('iv', 'ks')	Metric that decides which variable of a correlated group survives. Any other value raises KeyError when a pair is filtered.	"iv"
woe_engine	str	"monotone" without woe_binner fits a MonotoneWOEBinner on data. Any other value behaves like "master" (the default IV and KS computation).	"master"
woe_binner	(WOE_Master, MonotoneWOEBinner or None)	A fitted engine that supplies the IV and KS. It is used whenever it is given, and it also encodes the non-numeric variables for the correlation. An object of another type raises TypeError when it is used.	None
woe_engine_params	dict or None	Constructor arguments of the self-fitted MonotoneWOEBinner; the key fit_params is passed on to its fit. None is stored as an empty dict.	None
screen_iv_convention	('guarded', 'raw')	How the IV from an engine's bins treats a class-pure bin: "guarded" leaves it out of the IV sum, "raw" counts every bin (the IV up to 0.9.1). See VarExtractionInsights. The default computation (no engine) is not affected. Any other value raises ValueError.	"guarded"

Attributes:

Name	Type	Description
data, dep, corr_cutpoint, method, tree_binning, chi2_method, seed, chi2_p, init_equi_bins, missing_rate_ref, spec_values, base_metric, woe_engine, woe_binner, woe_engine_params, screen_iv_convention		The constructor values.
correlated_dict	dict	For each anchor variable of a correlated group, {"corr": pairs DataFrame, "gains": metric table}.
filtered_varlist	list	The variables removed by remove_highly_correlated.

Examples:

>>> corr_filter = CorrelationFilter(train_df, "bad_flag", corr_cutpoint=0.7, woe_binner=woe)
>>> keep_vars = corr_filter.remove_highly_correlated(features)

Source code in Modeling_Tool/Feature/WOE_Engine_Feature_Patch.py

class CorrelationFilter:
    """Correlation filter: drop the weaker variable of each highly correlated pair.

    Pairs whose absolute correlation exceeds ``corr_cutpoint`` are found, and from each group of correlated variables
    the one with the highest IV (or KS) is kept. Without an engine the calls are delegated to
    ``Modeling_Tool.Feature.Feature_Insights.CorrelationFilter``; with ``woe_binner`` (a fitted ``WOE_Master`` or
    ``MonotoneWOEBinner``) or ``woe_engine="monotone"``, the IV and KS come from that engine.

    Parameters
    ----------
    data : pandas.DataFrame
        Input data (raw values). The correlation is computed on the raw values of the numeric variables.
    dep : str
        Name of the target column (0/1).
    corr_cutpoint : float, default 0.8
        Correlation threshold: a pair is correlated when ``abs(corr) > corr_cutpoint``.
    method : str, default "pearson"
        Correlation method, ``"pearson"``, ``"spearman"`` or ``"kendall"``, passed to ``DataFrame.corr``.
    tree_binning : bool, default False
        Use decision-tree bins in the default IV and KS computation. Ignored when an engine supplies the bins.
    chi2_method : bool, default False
        Merge an initial fine binning with a chi-square test in the default computation. With
        ``woe_engine="monotone"`` it also feeds the self-fitted binner.
    seed : int, default 42
        Random seed of the tree binning. Ignored when an engine supplies the bins.
    chi2_p : float, default 0.999
        Confidence level of the chi-square merge.
    init_equi_bins : int, default 1000
        Number of fine bins before the chi-square merge.
    missing_rate_ref : int or float, default -9999999
        Sentinel for missing values in the default IV and KS computation. Ignored when an engine supplies the bins.
    spec_values : list, default []
        Special values (for example ``-1`` or ``999`` for "no record") that get bins or rows of their own in the IV and
        KS calculation, with the default computation and with an engine alike. The correlation itself is still
        computed on the raw values.
    base_metric : {"iv", "ks"}, default "iv"
        Metric that decides which variable of a correlated group survives. Any other value raises ``KeyError`` when
        a pair is filtered.
    woe_engine : str, default "master"
        ``"monotone"`` without ``woe_binner`` fits a ``MonotoneWOEBinner`` on ``data``. Any other value behaves like
        ``"master"`` (the default IV and KS computation).
    woe_binner : WOE_Master, MonotoneWOEBinner or None, default None
        A fitted engine that supplies the IV and KS. It is used whenever it is given, and it also encodes the
        non-numeric variables for the correlation. An object of another type raises ``TypeError`` when it is used.
    woe_engine_params : dict or None, default None
        Constructor arguments of the self-fitted ``MonotoneWOEBinner``; the key ``fit_params`` is passed on to its
        ``fit``. ``None`` is stored as an empty dict.
    screen_iv_convention : {"guarded", "raw"}, default "guarded"
        How the IV from an engine's bins treats a class-pure bin: ``"guarded"`` leaves it out of the IV sum,
        ``"raw"`` counts every bin (the IV up to 0.9.1). See ``VarExtractionInsights``. The default computation (no
        engine) is not affected. Any other value raises ``ValueError``.

    Attributes
    ----------
    data, dep, corr_cutpoint, method, tree_binning, chi2_method, seed, chi2_p, init_equi_bins, missing_rate_ref, spec_values, base_metric, woe_engine, woe_binner, woe_engine_params, screen_iv_convention
        The constructor values.
    correlated_dict : dict
        For each anchor variable of a correlated group, ``{"corr": pairs DataFrame, "gains": metric table}``.
    filtered_varlist : list
        The variables removed by ``remove_highly_correlated``.

    Examples
    --------
    >>> corr_filter = CorrelationFilter(train_df, "bad_flag", corr_cutpoint=0.7, woe_binner=woe)
    >>> keep_vars = corr_filter.remove_highly_correlated(features)
    """

    def __init__(
        self,
        data,
        dep,
        corr_cutpoint=0.8,
        method="pearson",
        tree_binning=False,
        chi2_method=False,
        seed=42,
        chi2_p=0.999,
        init_equi_bins=1000,
        missing_rate_ref=-9999999,
        spec_values=[],
        base_metric="iv",
        woe_engine="master",
        woe_binner=None,
        woe_engine_params=None,
        screen_iv_convention="guarded",
    ):
        self._base = _BaseCorrelationFilter(
            data, dep, corr_cutpoint, method, tree_binning, chi2_method, seed,
            chi2_p, init_equi_bins, missing_rate_ref, spec_values, base_metric,
        )
        self.data = data
        self.dep = dep
        self.corr_cutpoint = corr_cutpoint
        self.method = method
        self.tree_binning = tree_binning
        self.chi2_method = chi2_method
        self.seed = seed
        self.chi2_p = chi2_p
        self.init_equi_bins = init_equi_bins
        self.missing_rate_ref = missing_rate_ref
        self.spec_values = spec_values
        self.base_metric = base_metric
        self.woe_engine = woe_engine
        self.woe_binner = woe_binner
        self.woe_engine_params = woe_engine_params or {}
        self.screen_iv_convention = check_screen_iv_convention(screen_iv_convention)
        self.correlated_dict = {}
        self.filtered_varlist = []
        self._corr_matrix_cache = None
        self._corr_matrix_excluded = set()
        self._metric_summary_cache = None
        self._correlation_decision_trace = []

    def _corr_matrix_frame(self, varlist):
        # 0.7.0-R1: mixed correlation base. Numeric cols stay raw so a
        # pure-numeric varlist is byte-identical to the legacy .corr() path;
        # non-numeric cols are WOE-encoded via the screening binner
        # (corr_use_woe_bins semantics) and renamed back to raw names. Cols that
        # cannot be encoded go to self._corr_matrix_excluded: they leave the
        # matrix and survive the corr stage as NaN rows/cols once _high_corr_pairs
        # reindexes them back in (same as the weighted raw precedent). At most one
        # UserWarning per matrix build.
        non_numeric = [c for c in varlist if not pd.api.types.is_numeric_dtype(self.data[c])]
        if not non_numeric:
            self._corr_matrix_excluded = set()
            return self.data[varlist]

        encoded = {}
        excluded = set()
        if self.woe_binner is not None:
            suffix = str(getattr(self.woe_binner, "woe_suffix", "_woe"))
            try:
                adapter = as_woe_engine(self.woe_binner, woe_suffix=suffix)
            except TypeError as exc:
                raise TypeError(
                    "corr_use_woe_bins=True routed categorical feature(s) through "
                    "the screening WOE engine at the corr stage, but the supplied "
                    "woe_binner is an Unsupported WOE engine. Expected WOE_Master, "
                    "MonotoneWOEBinner, or WOEEngineAdapter."
                ) from exc
            suffix = adapter.woe_suffix
            try:
                tx = adapter.transform(self.data.copy(), varlist=non_numeric, suffix=suffix)
            except (TypeError, ValueError, KeyError, AttributeError, np.linalg.LinAlgError):
                tx = None
            for name in non_numeric:
                column = f"{name}{suffix}"
                if tx is not None and column in tx.columns and pd.api.types.is_numeric_dtype(tx[column]):
                    encoded[name] = tx[column].to_numpy()
                else:
                    excluded.add(name)
            if excluded:
                excluded_list = [c for c in non_numeric if c in excluded]
                warnings.warn(
                    f"corr_use_woe_bins=True could not WOE-encode {len(excluded)} "
                    f"feature(s) {excluded_list[:5]}; they are excluded from the "
                    f"correlation matrix and kept through the corr stage. Refit the "
                    f"screening WOE engine to cover them.",
                    UserWarning,
                    stacklevel=2,
                )
        else:
            excluded = set(non_numeric)
            warnings.warn(
                f"raw-value correlation skips {len(non_numeric)} non-numeric "
                f"feature(s) {non_numeric[:5]}; they are kept through the corr "
                f"stage. Set corr_use_woe_bins=True to correlate categorical "
                f"features via their WOE encoding.",
                UserWarning,
                stacklevel=2,
            )

        self._corr_matrix_excluded = excluded
        selected = [c for c in varlist if c not in excluded]
        frame = self.data[selected].copy()
        for name, values in encoded.items():
            frame[name] = values
        return frame

    def _high_corr_pairs(self, varlist):
        if (
            self._corr_matrix_cache is None
            or not (set(varlist) <= set(self._corr_matrix_cache.columns) | self._corr_matrix_excluded)
        ):
            self._corr_matrix_cache = self._corr_matrix_frame(varlist).corr(method=self.method)
        matrix = self._corr_matrix_cache.reindex(index=varlist, columns=varlist)
        values = matrix.to_numpy(dtype=float)
        row_idx, col_idx = np.triu_indices(len(varlist), k=1)
        pair_values = values[row_idx, col_idx]
        keep = np.isfinite(pair_values) & (np.abs(pair_values) > self.corr_cutpoint)
        names = np.asarray(varlist, dtype=object)
        return pd.DataFrame(
            {
                "VAR1": names[row_idx[keep]],
                "VAR2": names[col_idx[keep]],
                "CORR": pair_values[keep],
            },
            columns=["VAR1", "VAR2", "CORR"],
        )

    def _metric_summary(self, varlist):
        cached_vars = (
            set(self._metric_summary_cache["var"])
            if self._metric_summary_cache is not None and "var" in self._metric_summary_cache
            else set()
        )
        if not set(varlist).issubset(cached_vars):
            insights = VarExtractionInsights(
                self.data,
                self.dep,
                None,
                chi2_method=self.chi2_method,
                chi2_p=self.chi2_p,
                init_equi_bins=self.init_equi_bins,
                tree_binning=self.tree_binning,
                seed=self.seed,
                missing_rate_ref=self.missing_rate_ref,
                spec_values=self.spec_values,
                woe_engine=self.woe_engine,
                woe_binner=self.woe_binner,
                woe_engine_params=self.woe_engine_params,
                screen_iv_convention=getattr(self, "screen_iv_convention", "guarded"),
            )
            self._metric_summary_cache = insights.get_var_analysis_report(
                self.data,
                varlist,
                dep=self.dep,
                iv_cut=0,
            )
        return self._metric_summary_cache

    def __getattr__(self, name):
        return getattr(self._base, name)

    @staticmethod
    def calculate_vif(df):
        """Compute the variance inflation factor (VIF) of each column.

        Parameters
        ----------
        df : pandas.DataFrame
            DataFrame that holds the independent variables (numeric columns without missing values).

        Returns
        -------
        pandas.DataFrame
            One row per column of ``df`` with the columns ``index`` (the variable name) and ``VIF``. A VIF above 10
            usually signals serious collinearity.

        Notes
        -----
        It needs the optional ``statsmodels`` package. There is no NaN handling and no weights.
        """
        return _BaseCorrelationFilter.calculate_vif(df)

    def _sync_base_state(self):
        self.correlated_dict = getattr(self._base, "correlated_dict", {})
        self.filtered_varlist = getattr(self._base, "filtered_varlist", [])
        self._corr_matrix_cache = getattr(self._base, "_corr_matrix_cache", None)
        self._corr_matrix_excluded = getattr(self._base, "_corr_matrix_excluded", set())
        self._metric_summary_cache = getattr(self._base, "_metric_summary_cache", None)
        self._correlation_decision_trace = getattr(
            self._base, "_correlation_decision_trace", [],
        )

    def filter_single_iteration(self, varlist):
        """Run one pass of the correlation filter over ``varlist``.

        From each group of highly correlated variables the one with the highest ``base_metric`` is kept and the
        others are removed.

        Parameters
        ----------
        varlist : list of str
            Variables to screen. They must be columns of ``data``.

        Returns
        -------
        list of str
            The retained variables: the winner of each correlated group first (in the order of discovery), followed
            by the variables that are not part of any highly correlated pair in their original order. ``varlist``
            itself is returned when no pair exceeds ``corr_cutpoint``.

        Notes
        -----
        The call updates ``correlated_dict`` and the internal decision trace. Non-numeric variables are encoded with
        the WOE engine for the correlation when one is available; otherwise they are skipped with a warning and
        kept.
        """
        if self.woe_binner is None and self.woe_engine == "master":
            result = self._base.filter_single_iteration(varlist)
            self._sync_base_state()
            return result

        name_mapping = {"iv": "iv", "ks": "ks_in_gains"}
        high_corr_var = self._high_corr_pairs(varlist)
        if len(high_corr_var) == 0:
            return varlist

        selected_varlist = []
        removed_varlist = []
        for var in tqdm(high_corr_var["VAR1"].drop_duplicates().tolist()):
            if var in set(removed_varlist + selected_varlist):
                continue
            single_var_corr = high_corr_var.loc[high_corr_var["VAR1"].eq(var)]
            correlated_list = [var] + single_var_corr["VAR2"].drop_duplicates().tolist()
            metric_summary = self._metric_summary(varlist)
            summary = metric_summary[metric_summary["var"].isin(correlated_list)].copy()
            if summary.empty:
                continue
            selected = summary.sort_values([name_mapping[self.base_metric.lower()]], ascending=False)["var"].iloc[0]
            if selected not in selected_varlist:
                selected_varlist.append(selected)
            newly_removed = [
                x for x in correlated_list
                if x != selected and x not in removed_varlist
            ]
            metric_name = name_mapping[self.base_metric.lower()]
            metric_map = summary.set_index("var")[metric_name].to_dict()
            positions = {name: idx for idx, name in enumerate(varlist)}
            for dropped_var in newly_removed:
                decision_pair = [selected, dropped_var]
                decision_pair.sort(key=positions.__getitem__)
                var_a, var_b = decision_pair
                corr_value = self._corr_matrix_cache.loc[var_a, var_b]
                self._correlation_decision_trace.append({
                    "var_a": var_a,
                    "var_b": var_b,
                    "corr": float(corr_value),
                    "iv_a": float(metric_map.get(var_a, 0.0)),
                    "iv_b": float(metric_map.get(var_b, 0.0)),
                    "kept": selected,
                    "dropped": dropped_var,
                })
            removed_varlist += newly_removed
            self.correlated_dict[var] = {"corr": single_var_corr, "gains": summary}

        return selected_varlist + [x for x in varlist if x not in (selected_varlist + removed_varlist)]

    def remove_highly_correlated(self, varlist, max_iterations=10):
        """Iteratively remove highly correlated variables.

        Run the correlation filter repeatedly until no variable is removed or the maximum number of iterations is
        reached.

        Parameters
        ----------
        varlist : list of str
            Variables to screen. They must be columns of ``data``.
        max_iterations : int, default 10
            Maximum number of filtering passes.

        Returns
        -------
        list of str
            The variables finally retained, with the winner of each correlated group first.

        Notes
        -----
        ``filtered_varlist`` is set to the removed variables, and ``correlated_dict`` holds the correlated pairs and
        the metric table behind each decision.
        """
        if self.woe_binner is None and self.woe_engine == "master":
            result = self._base.remove_highly_correlated(varlist, max_iterations)
            self._sync_base_state()
            return result

        self._correlation_decision_trace = []
        self._corr_matrix_cache = self._corr_matrix_frame(varlist).corr(method=self.method)
        self._metric_summary_cache = None
        self._metric_summary(varlist)
        last_keep_list = self.filter_single_iteration(varlist)
        for _ in range(1, max_iterations):
            keep_list = self.filter_single_iteration(last_keep_list)
            removed_vars = [x for x in last_keep_list if x not in keep_list]
            self.filtered_varlist.append(removed_vars)
            if len(removed_vars) == 0:
                break
            last_keep_list = keep_list
        self.filtered_varlist = [x for x in varlist if x not in last_keep_list]
        return last_keep_list

data instance-attribute

data = data

dep instance-attribute

dep = dep

corr_cutpoint instance-attribute

corr_cutpoint = corr_cutpoint

method instance-attribute

method = method

tree_binning instance-attribute

tree_binning = tree_binning

chi2_method instance-attribute

chi2_method = chi2_method

seed instance-attribute

seed = seed

chi2_p instance-attribute

chi2_p = chi2_p

init_equi_bins instance-attribute

init_equi_bins = init_equi_bins

missing_rate_ref instance-attribute

missing_rate_ref = missing_rate_ref

spec_values instance-attribute

spec_values = spec_values

base_metric instance-attribute

base_metric = base_metric

woe_engine instance-attribute

woe_engine = woe_engine

woe_binner instance-attribute

woe_binner = woe_binner

woe_engine_params instance-attribute

woe_engine_params = woe_engine_params or {}

screen_iv_convention instance-attribute

screen_iv_convention = check_screen_iv_convention(screen_iv_convention)

correlated_dict instance-attribute

correlated_dict = {}

filtered_varlist instance-attribute

filtered_varlist = []

calculate_vif staticmethod

calculate_vif(df)

Compute the variance inflation factor (VIF) of each column.

Parameters:

Name	Type	Description	Default
df	DataFrame	DataFrame that holds the independent variables (numeric columns without missing values).	required

Returns:

Type	Description
DataFrame	One row per column of df with the columns index (the variable name) and VIF. A VIF above 10 usually signals serious collinearity.

Notes

It needs the optional statsmodels package. There is no NaN handling and no weights.

Source code in Modeling_Tool/Feature/WOE_Engine_Feature_Patch.py

@staticmethod
def calculate_vif(df):
    """Compute the variance inflation factor (VIF) of each column.

    Parameters
    ----------
    df : pandas.DataFrame
        DataFrame that holds the independent variables (numeric columns without missing values).

    Returns
    -------
    pandas.DataFrame
        One row per column of ``df`` with the columns ``index`` (the variable name) and ``VIF``. A VIF above 10
        usually signals serious collinearity.

    Notes
    -----
    It needs the optional ``statsmodels`` package. There is no NaN handling and no weights.
    """
    return _BaseCorrelationFilter.calculate_vif(df)

filter_single_iteration

filter_single_iteration(varlist)

Run one pass of the correlation filter over varlist.

From each group of highly correlated variables the one with the highest base_metric is kept and the others are removed.

Parameters:

Name	Type	Description	Default
varlist	list of str	Variables to screen. They must be columns of data.	required

Returns:

Type	Description
list of str	The retained variables: the winner of each correlated group first (in the order of discovery), followed by the variables that are not part of any highly correlated pair in their original order. varlist itself is returned when no pair exceeds corr_cutpoint.

Notes

The call updates correlated_dict and the internal decision trace. Non-numeric variables are encoded with the WOE engine for the correlation when one is available; otherwise they are skipped with a warning and kept.

Source code in Modeling_Tool/Feature/WOE_Engine_Feature_Patch.py

def filter_single_iteration(self, varlist):
    """Run one pass of the correlation filter over ``varlist``.

    From each group of highly correlated variables the one with the highest ``base_metric`` is kept and the
    others are removed.

    Parameters
    ----------
    varlist : list of str
        Variables to screen. They must be columns of ``data``.

    Returns
    -------
    list of str
        The retained variables: the winner of each correlated group first (in the order of discovery), followed
        by the variables that are not part of any highly correlated pair in their original order. ``varlist``
        itself is returned when no pair exceeds ``corr_cutpoint``.

    Notes
    -----
    The call updates ``correlated_dict`` and the internal decision trace. Non-numeric variables are encoded with
    the WOE engine for the correlation when one is available; otherwise they are skipped with a warning and
    kept.
    """
    if self.woe_binner is None and self.woe_engine == "master":
        result = self._base.filter_single_iteration(varlist)
        self._sync_base_state()
        return result

    name_mapping = {"iv": "iv", "ks": "ks_in_gains"}
    high_corr_var = self._high_corr_pairs(varlist)
    if len(high_corr_var) == 0:
        return varlist

    selected_varlist = []
    removed_varlist = []
    for var in tqdm(high_corr_var["VAR1"].drop_duplicates().tolist()):
        if var in set(removed_varlist + selected_varlist):
            continue
        single_var_corr = high_corr_var.loc[high_corr_var["VAR1"].eq(var)]
        correlated_list = [var] + single_var_corr["VAR2"].drop_duplicates().tolist()
        metric_summary = self._metric_summary(varlist)
        summary = metric_summary[metric_summary["var"].isin(correlated_list)].copy()
        if summary.empty:
            continue
        selected = summary.sort_values([name_mapping[self.base_metric.lower()]], ascending=False)["var"].iloc[0]
        if selected not in selected_varlist:
            selected_varlist.append(selected)
        newly_removed = [
            x for x in correlated_list
            if x != selected and x not in removed_varlist
        ]
        metric_name = name_mapping[self.base_metric.lower()]
        metric_map = summary.set_index("var")[metric_name].to_dict()
        positions = {name: idx for idx, name in enumerate(varlist)}
        for dropped_var in newly_removed:
            decision_pair = [selected, dropped_var]
            decision_pair.sort(key=positions.__getitem__)
            var_a, var_b = decision_pair
            corr_value = self._corr_matrix_cache.loc[var_a, var_b]
            self._correlation_decision_trace.append({
                "var_a": var_a,
                "var_b": var_b,
                "corr": float(corr_value),
                "iv_a": float(metric_map.get(var_a, 0.0)),
                "iv_b": float(metric_map.get(var_b, 0.0)),
                "kept": selected,
                "dropped": dropped_var,
            })
        removed_varlist += newly_removed
        self.correlated_dict[var] = {"corr": single_var_corr, "gains": summary}

    return selected_varlist + [x for x in varlist if x not in (selected_varlist + removed_varlist)]

remove_highly_correlated

remove_highly_correlated(varlist, max_iterations=10)

Iteratively remove highly correlated variables.

Run the correlation filter repeatedly until no variable is removed or the maximum number of iterations is reached.

Parameters:

Name	Type	Description	Default
varlist	list of str	Variables to screen. They must be columns of data.	required
max_iterations	int	Maximum number of filtering passes.	10

Returns:

Type	Description
list of str	The variables finally retained, with the winner of each correlated group first.

Notes

filtered_varlist is set to the removed variables, and correlated_dict holds the correlated pairs and the metric table behind each decision.

Source code in Modeling_Tool/Feature/WOE_Engine_Feature_Patch.py

def remove_highly_correlated(self, varlist, max_iterations=10):
    """Iteratively remove highly correlated variables.

    Run the correlation filter repeatedly until no variable is removed or the maximum number of iterations is
    reached.

    Parameters
    ----------
    varlist : list of str
        Variables to screen. They must be columns of ``data``.
    max_iterations : int, default 10
        Maximum number of filtering passes.

    Returns
    -------
    list of str
        The variables finally retained, with the winner of each correlated group first.

    Notes
    -----
    ``filtered_varlist`` is set to the removed variables, and ``correlated_dict`` holds the correlated pairs and
    the metric table behind each decision.
    """
    if self.woe_binner is None and self.woe_engine == "master":
        result = self._base.remove_highly_correlated(varlist, max_iterations)
        self._sync_base_state()
        return result

    self._correlation_decision_trace = []
    self._corr_matrix_cache = self._corr_matrix_frame(varlist).corr(method=self.method)
    self._metric_summary_cache = None
    self._metric_summary(varlist)
    last_keep_list = self.filter_single_iteration(varlist)
    for _ in range(1, max_iterations):
        keep_list = self.filter_single_iteration(last_keep_list)
        removed_vars = [x for x in last_keep_list if x not in keep_list]
        self.filtered_varlist.append(removed_vars)
        if len(removed_vars) == 0:
            break
        last_keep_list = keep_list
    self.filtered_varlist = [x for x in varlist if x not in last_keep_list]
    return last_keep_list

Unified Feature Screening: Feature_Screen

Feature_Screen

Unified feature screening: PSI -> IV -> correlation dedup.

FeatureScreenConfig dataclass

Settings of :func:feature_screen: the missing-rate, PSI, IV and correlation stages and the optional gates.

The stages run in the order missing rate, PSI, IV, correlation, then the post-selection gates (VIF, group stability, multi-target, truncation). A stage whose threshold is None or whose switch is off is skipped, and every gate is off by default. CreditModelPipeline builds this object from its feature_selection dict with :func:screen_config_from_mapping.

Parameters:

Name	Type	Description	Default
psi_enabled	bool	Whether to run the PSI stage, which compares INS with the splits in psi_compare_splits.	True
psi_threshold	float	A feature is kept when its largest PSI over the compared splits is strictly below this value.	0.2
psi_compare_splits	list of str	Splits compared with "ins": "oos", "oot" or both. A listed split that has no rows is skipped.	['oos']
psi_buckets	int	Number of equal-frequency buckets of the default PSI binning. Only unweighted runs that do not bin PSI on WOE bins read it; a weighted run without WOE bins takes its PSI bins from iv_bins and min_bin_prop.	10
psi_use_woe_bins	bool	Compute PSI on the bins of the screening WOE engine (prefit_woe_engine, or one fitted from woe_engine) instead of the default binning. A weighted run that has an engine always bins PSI with it.	False
iv_enabled	bool	Whether to run the IV stage.	True
iv_threshold	float	Lower IV limit: a feature is kept when its IV is at least this value.	0.02
iv_upper_threshold	float or None	Upper IV limit (gate G02, a leakage guard), on by default since 0.9.3: features whose IV is above it are dropped and recorded in dropped_detail with the reason "iv_above_upper". The test never uses the guarded iv, which leaves class-pure bins out and so understates exactly the near-perfect separators the limit targets. It uses an IV whose zero-count cells are floored at content: for numeric features on equal-frequency bins (weighted on weighted runs) on every path, so it does not depend on how the WOE engine merged its bins. On WOE bins under screen_iv_convention="guarded" it uses the larger of that value and iv_raw (every WOE bin counted), and iv_raw alone where the floored IV cannot be computed (non-numeric features); under "raw" and without WOE bins the test is that of 0.9.2. A feature that would not exceed the limit without its class-pure special-value bins (a special value that holds only bads can be a label leak) is dropped as well, with the reason "iv_above_upper_sv_pure" so that a human can review it. Every such drop warns ("possible target leakage; check the field's time point (PIT) and source before training"; since 0.9.3) and is a hard drop that no empty-stage fallback brings back (see on_empty_stage). None disables the limit (the default up to 0.9.2).	2.0
iv_bins	int	Maximum number of bins of the default IV binning: decision-tree leaves on unweighted runs, weighted equal-frequency bins on weighted runs (which also derive their PSI bins from it).	10
iv_min_bin_prop	float	Minimum share of rows per bin in the default IV binning of unweighted runs. Weighted runs read min_bin_prop instead.	0.05
iv_equal_freq	bool	Must stay True: the IV binning cannot switch equal-frequency bins off (unweighted runs use decision-tree bins and weighted runs weighted equal-frequency bins), so any other value raises ValueError when the config is built. Use iv_bins and iv_min_bin_prop to tune the bins, or iv_use_woe_bins to take them from the WOE engine.	True
iv_use_woe_bins	bool	Compute IV on the bins of the screening WOE engine instead of the default binning. A weighted run that has an engine always bins IV with it.	False
corr_enabled	bool	Whether to run the correlation stage.	True
corr_threshold	float	Two features are redundant when the absolute value of their correlation is above this value; of such a group the feature with the higher IV is kept.	0.75
corr_max_iterations	int	Maximum number of filtering rounds of the correlation stage.	10
corr_use_woe_bins	bool	Let the screening WOE engine take part in the correlation stage: non-numeric features are WOE-encoded so that they enter the correlation matrix (otherwise they are skipped with a warning and kept), and on unweighted runs the engine's bins also give the IV that decides between two correlated features.	False
corr_method	('pearson', 'spearman', 'kendall')	Correlation coefficient of the correlation stage. Weighted runs with non-constant weights support Pearson only (ValueError otherwise).	"pearson"
corr_base_metric	('iv', 'ks')	Metric that decides which of two correlated features is kept: the higher value wins.	"iv"
corr_nan_policy	('pairwise', 'median_fill', 'raise')	Missing values in the weighted correlation matrix: "pairwise" correlates each pair on the rows where both values exist, "median_fill" fills them with the weighted median and "raise" raises ValueError. Weighted runs only; unweighted runs, and weighted runs with constant weights and "pairwise", use CorrelationFilter.	"pairwise"
corr_block_size	int	Number of feature columns per block of the weighted pairwise correlation matrix. A smaller value lowers peak memory without changing the result; it must be positive. Weighted runs only.	256
on_empty_stage	('keep_all_warn', 'keep_all', 'raise')	What to do when a stage would drop every feature (each fallback adds a <stage>_fallback row to the summary). "keep_all_warn" (the name is kept for compatibility; since 0.9.3 it does not keep all) never brings back a hard drop, a feature that the IV upper limit dropped (dropped_detail reason "iv_above_upper" or "iv_above_upper_sv_pure"). From the IV stage on (the IV stage and the VIF, group-stability and multi-target gates) it keeps the fallback_top_k other features with the highest screening IV (the iv column, never iv_raw), in that order, warns with their names and IVs, and marks them in dropped_detail with the reason "fallback_kept"; the missing-rate and PSI stages, which run before any IV exists, keep every feature as before. When only hard drops are left it raises EmptyStageError with their names and reasons; their leak warnings have been issued before. "keep_all" keeps every feature, hard drops included: the behavior up to 0.9.3, which can let a leak back in, so use it only to compare with old results. "raise" raises EmptyStageError (a ValueError).	"keep_all_warn"
missing_rate_threshold	float or None	Maximum missing rate on INS: features with a higher missing rate are dropped before the PSI stage. None skips the stage, and missing_rate_table and missing_rate_dropped then stay empty.	None
missing_rate_ref	float or int	Value that counts as missing in addition to NaN when the missing rate is computed.	-999999
woe_engine	str	Engine fitted when a *_use_woe_bins flag is set and no prefit_woe_engine is given: "monotone" (case-insensitive) fits a MonotoneWOEBinner, any other value fits a WOE_Master.	"equal_freq"
woe_fit_query	str or None	DataFrame.query expression that selects the INS rows used to fit the screening engine. A query that fails is silently ignored and all INS rows are used.	None
woe_params	dict	Keyword arguments of WOE_Master.fit for the WOE_Master engine; the keys woe_suffix and missing_ref_value go to the WOE_Master constructor instead. The monotone engine ignores it.	{'nbins': 10, 'equal_freq': True, 'min_bin_prop': 0.05}
monotone_woe_params	dict	Arguments of the MonotoneWOEBinner engine: its tuning keys (for example n_init_bins, min_bin_size, min_n_bins, special_values; the columns are supplied by the screen) plus the fit keys chi2_binning, chi2_p, chi2_init_size and n_jobs. Any other key is silently dropped. The WOE_Master engine ignores it.	{'n_init_bins': 20, 'min_bin_size': 0.03, 'min_n_bins': 2}
categorical_features	list of str or None	Features that a self-fitted monotone engine treats as categorical (only names among the screened features count). The WOE_Master engine ignores it.	None
plot_path	str or None	Directory for the IV-stage charts. Charts are written to <plot_path>/overall only when plot_outputs is also True, and only by unweighted runs.	None
plot_outputs	bool	Whether to draw the WOE charts of the features that enter the IV stage (needs plot_path; unweighted runs only).	False
content	float	Floor for bin shares that avoids division by zero and log(0): in the weighted PSI and in the zero-cell-floored IV behind iv_upper_threshold.	1e-6
precision	int	Number of decimals to which the weighted equal-frequency bin edges are rounded. Read only by weighted runs that do not use WOE bins.	5
min_bin_prop	float	Minimum bin share of the weighted equal-frequency bins: it caps their number at int(1 / min_bin_prop). Read only by weighted runs that do not use WOE bins; unweighted IV reads iv_min_bin_prop.	0.05
monthly_iv_min	float or None	Gate G03 (group stability): minimum IV that a feature must reach in every group of the evidence (for example every month); a feature whose lowest group IV is below it is dropped. Needs selection_evidence. None switches the check off.	None
monthly_iv_cv_max	float or None	G03: maximum coefficient of variation (population standard deviation divided by the mean) of a feature's group IVs; features above it are dropped, and a zero mean IV counts as an infinite coefficient. Needs selection_evidence.	None
direction_consistency_min	float or None	G03: minimum share of a feature's groups (those with a non-zero direction) whose direction, the sign of the association with the target, equals the most common one. Needs selection_evidence.	None
min_group_n	int or None	G03: minimum number of rows for a group to count. None (or 0) uses the default of the evidence. A feature with fewer than two eligible groups is handled by insufficient_group_policy.	None
insufficient_group_policy	('keep_warn', 'drop', 'raise')	G03: what to do with a feature that has fewer than two eligible groups: keep it with a warning, drop it, or raise ValueError.	"keep_warn"
target_rules	('all', 'any', 'min_pass_count')	Gate G04 (multi-target): a feature must pass its per-target IV range and direction check on all targets, on any target, or on at least min_pass_count targets. Needs selection_evidence. None switches the gate off; any other value raises ValueError.	"all"
min_pass_count	int or None	G04: number of targets a feature must pass when target_rules="min_pass_count"; None counts as 1.	None
per_target_iv_range	tuple or dict or None	G04: allowed IV range (low, high) for every target, or {target: (low, high)} per target; a None bound is open and a target missing from the dict is not range-checked. None applies no IV range.	None
direction_reference_target	str or None	G04: target whose direction the other targets must not contradict; a target with a different non-zero direction for a feature counts as failed for that feature.	None
max_selected_features	int or None	Gate G05 (truncation): keep at most this many features, ranked by ranking_metric (highest first). It runs after every other gate, never backfills, and records the cut features in dropped_detail with the reason "max_selected_features". None sets no cap.	None
min_selected_features	int or None	G05: when fewer features survive the gates, only a warning and a truncation_fallback row in the summary are produced; nothing is added back.	None
ranking_metric	str	G05: metric that ranks the features for truncation. Only "iv" exists; any other value raises ValueError when the cap actually cuts features.	"iv"
tie_breaker	str	G05: how features with equal ranking values are ordered when the cap cuts between them. The only rule is ascending feature name, so "name" (or None) is the only accepted value; any other raises ValueError when the config is built.	"name"
vif_enabled	bool	Gate G06: after the correlation stage, repeatedly drop the feature with the highest VIF until no VIF is above vif_threshold or only vif_min_features features remain. Needs the optional statsmodels package (ImportError otherwise).	False
vif_threshold	float	G06: the feature with the highest VIF is dropped while that VIF is above this value; an infinite VIF is always dropped.	10.0
vif_min_features	int	G06: the gate stops dropping when this many features are left, and is skipped (summary note "skipped_at_floor") when it starts with this many or fewer.	2
vif_tie_break_metric	str	G06: metric that decides among features with an equal VIF; the one with the lower value is dropped first. Only "iv" is supported, any other value raises ValueError when the gate is enabled.	"iv"
vif_use_woe_bins	bool	G06: False computes the VIF on the raw numeric columns (non-numeric survivors stay out of the VIF matrix, are kept in the selection and are reported with a warning). True computes it on the WOE-encoded INS view, so categorical features take part; this needs a screening WOE engine (fitted when none is given), and when the screen fits that engine, categorical_features additionally requires woe_engine="monotone" (ValueError otherwise).	False
screen_iv_convention	('guarded', 'raw')	How the IV on the bins of a WOE engine (iv_use_woe_bins, or any weighted run with an engine) treats a class-pure bin, a bin without bads or without goods whose WOE ln((bad_share + 1e-6) / (good_share + 1e-6)) comes from the 1e-6 alone. "guarded" leaves such bins (special-value bins, and ordinary bins that min_n_bins kept) out of the IV sum, the iv_guard convention of MonotoneWOEBinner's by-group IV: their rows stay in the bad and good totals, so every other bin keeps its WOE and IV; a special-value bin that the binner smoothed (sv_woe_smoothing="laplace") counts with its finite WOE. Purity is judged on the quantities the IV uses (weighted sums on weighted runs, counts otherwise), so unit weights flag the same bins. The IV threshold, the correlation stage and the ranking of the gates use that IV. "raw" counts every bin, which reproduces 0.9.1. iv_table reports both (iv and iv_raw) with iv_sv_pure and n_sv_pure, and stage_tables["iv_sv_pure_bins"] lists the class-pure special-value bins. The paths without WOE bins are not affected. Any other value raises ValueError.	"guarded"
iv_gap_ratio_warn	float	Leak warning (0.9.3): a feature with iv > 0 and iv_raw / iv at or above this value gets iv_gap_flag=True in iv_table and a UserWarning ("possible leakage or unresolved class-pure bins; review before training"). An iv at or below 1e-12 counts as zero and only iv_gap_abs_warn applies. The rule only warns and flags; it never drops a feature, and the flag is set also for a feature that the IV stage drops. Under screen_iv_convention="raw", and without WOE bins, iv equals iv_raw and nothing is flagged. Must be above 1 (inf switches the rule off; ValueError otherwise).	2.0
iv_gap_abs_warn	float	Leak warning (0.9.3): flag and warn as above when iv_raw - iv is at or above this value. Must be positive (inf switches the rule off; ValueError otherwise).	0.5
fallback_top_k	int	Number of features that the empty-stage fallback keeps under on_empty_stage="keep_all_warn" (0.9.3): the ones with the highest screening IV among those the stage dropped, leak-gate drops excluded. A whole number of at least 1 (ValueError otherwise).	5

Notes

The G03 and G04 settings (monthly_iv_min, monthly_iv_cv_max, direction_consistency_min, target_rules) need group or per-target evidence that only FeatureValidationPipeline builds. Setting any of them where no selection_evidence is passed (feature_screen without it, or feature_screen_from_dataframe, which has no such argument) raises ValueError; inside CreditModelPipeline the error is caught and the whole screening is skipped (see its feature_selection setting).

Source code in Modeling_Tool/Feature/Feature_Screen.py

@dataclass
class FeatureScreenConfig:
    """Settings of :func:`feature_screen`: the missing-rate, PSI, IV and correlation stages and the optional gates.

    The stages run in the order missing rate, PSI, IV, correlation, then the post-selection gates (VIF, group stability,
    multi-target, truncation). A stage whose threshold is ``None`` or whose switch is off is skipped, and every gate is
    off by default. ``CreditModelPipeline`` builds this object from its ``feature_selection`` dict with
    :func:`screen_config_from_mapping`.

    Parameters
    ----------
    psi_enabled : bool, default True
        Whether to run the PSI stage, which compares INS with the splits in ``psi_compare_splits``.
    psi_threshold : float, default 0.2
        A feature is kept when its largest PSI over the compared splits is strictly below this value.
    psi_compare_splits : list of str, default ['oos']
        Splits compared with ``"ins"``: ``"oos"``, ``"oot"`` or both. A listed split that has no rows is skipped.
    psi_buckets : int, default 10
        Number of equal-frequency buckets of the default PSI binning. Only unweighted runs that do not bin PSI on WOE
        bins read it; a weighted run without WOE bins takes its PSI bins from ``iv_bins`` and ``min_bin_prop``.
    psi_use_woe_bins : bool, default False
        Compute PSI on the bins of the screening WOE engine (``prefit_woe_engine``, or one fitted from ``woe_engine``)
        instead of the default binning. A weighted run that has an engine always bins PSI with it.
    iv_enabled : bool, default True
        Whether to run the IV stage.
    iv_threshold : float, default 0.02
        Lower IV limit: a feature is kept when its IV is at least this value.
    iv_upper_threshold : float or None, default 2.0
        Upper IV limit (gate G02, a leakage guard), on by default since 0.9.3: features whose IV is above it are dropped
        and recorded in ``dropped_detail`` with the reason ``"iv_above_upper"``. The test never uses the guarded ``iv``,
        which leaves class-pure bins out and so understates exactly the near-perfect separators the limit targets. It
        uses an IV whose zero-count cells are floored at ``content``: for numeric features on equal-frequency bins
        (weighted on weighted runs) on every path, so it does not depend on how the WOE engine merged its bins. On WOE
        bins under ``screen_iv_convention="guarded"`` it uses the larger of that value and ``iv_raw`` (every WOE bin
        counted), and ``iv_raw`` alone where the floored IV cannot be computed (non-numeric features); under
        ``"raw"`` and without WOE bins the test is that of 0.9.2. A feature that would not exceed the limit without its
        class-pure special-value bins (a special value that holds only bads can be a label leak) is dropped as well,
        with the reason ``"iv_above_upper_sv_pure"`` so that a human can review it. Every such drop warns ("possible
        target leakage; check the field's time point (PIT) and source before training"; since 0.9.3) and is a hard drop
        that no empty-stage fallback brings back (see ``on_empty_stage``). ``None`` disables the limit (the default up
        to 0.9.2).
    iv_bins : int, default 10
        Maximum number of bins of the default IV binning: decision-tree leaves on unweighted runs, weighted
        equal-frequency bins on weighted runs (which also derive their PSI bins from it).
    iv_min_bin_prop : float, default 0.05
        Minimum share of rows per bin in the default IV binning of unweighted runs. Weighted runs read ``min_bin_prop``
        instead.
    iv_equal_freq : bool, default True
        Must stay True: the IV binning cannot switch equal-frequency bins off (unweighted runs use decision-tree bins and
        weighted runs weighted equal-frequency bins), so any other value raises ``ValueError`` when the config is built.
        Use ``iv_bins`` and ``iv_min_bin_prop`` to tune the bins, or ``iv_use_woe_bins`` to take them from the WOE engine.
    iv_use_woe_bins : bool, default False
        Compute IV on the bins of the screening WOE engine instead of the default binning. A weighted run that has an
        engine always bins IV with it.
    corr_enabled : bool, default True
        Whether to run the correlation stage.
    corr_threshold : float, default 0.75
        Two features are redundant when the absolute value of their correlation is above this value; of such a group
        the feature with the higher IV is kept.
    corr_max_iterations : int, default 10
        Maximum number of filtering rounds of the correlation stage.
    corr_use_woe_bins : bool, default False
        Let the screening WOE engine take part in the correlation stage: non-numeric features are WOE-encoded so that
        they enter the correlation matrix (otherwise they are skipped with a warning and kept), and on unweighted runs
        the engine's bins also give the IV that decides between two correlated features.
    corr_method : {"pearson", "spearman", "kendall"}, default "pearson"
        Correlation coefficient of the correlation stage. Weighted runs with non-constant weights support Pearson only
        (``ValueError`` otherwise).
    corr_base_metric : {"iv", "ks"}, default "iv"
        Metric that decides which of two correlated features is kept: the higher value wins.
    corr_nan_policy : {"pairwise", "median_fill", "raise"}, default "pairwise"
        Missing values in the weighted correlation matrix: ``"pairwise"`` correlates each pair on the rows where both
        values exist, ``"median_fill"`` fills them with the weighted median and ``"raise"`` raises ``ValueError``.
        Weighted runs only; unweighted runs, and weighted runs with constant weights and ``"pairwise"``, use
        ``CorrelationFilter``.
    corr_block_size : int, default 256
        Number of feature columns per block of the weighted pairwise correlation matrix. A smaller value lowers peak
        memory without changing the result; it must be positive. Weighted runs only.
    on_empty_stage : {"keep_all_warn", "keep_all", "raise"}, default "keep_all_warn"
        What to do when a stage would drop every feature (each fallback adds a ``<stage>_fallback`` row to the
        summary). ``"keep_all_warn"`` (the name is kept for compatibility; since 0.9.3 it does not keep all) never
        brings back a hard drop, a feature that the IV upper limit dropped (``dropped_detail`` reason
        ``"iv_above_upper"`` or ``"iv_above_upper_sv_pure"``). From the IV stage on (the IV stage and the VIF,
        group-stability and multi-target gates) it keeps the ``fallback_top_k`` other features with the highest
        screening IV (the ``iv`` column, never ``iv_raw``), in that order, warns with their names and IVs, and marks
        them in ``dropped_detail`` with the reason ``"fallback_kept"``; the missing-rate and PSI stages, which run
        before any IV exists, keep every feature as before. When only hard drops are left it raises
        ``EmptyStageError`` with their names and reasons; their leak warnings have been issued before. ``"keep_all"``
        keeps every feature, hard drops included: the behavior up to 0.9.3, which can let a leak back in, so use it
        only to compare with old results. ``"raise"`` raises ``EmptyStageError`` (a ``ValueError``).
    missing_rate_threshold : float or None, default None
        Maximum missing rate on INS: features with a higher missing rate are dropped before the PSI stage. ``None``
        skips the stage, and ``missing_rate_table`` and ``missing_rate_dropped`` then stay empty.
    missing_rate_ref : float or int, default -999999
        Value that counts as missing in addition to ``NaN`` when the missing rate is computed.
    woe_engine : str, default "equal_freq"
        Engine fitted when a ``*_use_woe_bins`` flag is set and no ``prefit_woe_engine`` is given: ``"monotone"``
        (case-insensitive) fits a ``MonotoneWOEBinner``, any other value fits a ``WOE_Master``.
    woe_fit_query : str or None, default None
        ``DataFrame.query`` expression that selects the INS rows used to fit the screening engine. A query that fails is
        silently ignored and all INS rows are used.
    woe_params : dict, default {'nbins': 10, 'equal_freq': True, 'min_bin_prop': 0.05}
        Keyword arguments of ``WOE_Master.fit`` for the ``WOE_Master`` engine; the keys ``woe_suffix`` and
        ``missing_ref_value`` go to the ``WOE_Master`` constructor instead. The monotone engine ignores it.
    monotone_woe_params : dict, default {'n_init_bins': 20, 'min_bin_size': 0.03, 'min_n_bins': 2}
        Arguments of the ``MonotoneWOEBinner`` engine: its tuning keys (for example ``n_init_bins``, ``min_bin_size``,
        ``min_n_bins``, ``special_values``; the columns are supplied by the screen) plus the fit keys ``chi2_binning``,
        ``chi2_p``, ``chi2_init_size`` and ``n_jobs``. Any other key is silently dropped. The ``WOE_Master`` engine
        ignores it.
    categorical_features : list of str or None, default None
        Features that a self-fitted monotone engine treats as categorical (only names among the screened features
        count). The ``WOE_Master`` engine ignores it.
    plot_path : str or None, default None
        Directory for the IV-stage charts. Charts are written to ``<plot_path>/overall`` only when ``plot_outputs`` is
        also True, and only by unweighted runs.
    plot_outputs : bool, default False
        Whether to draw the WOE charts of the features that enter the IV stage (needs ``plot_path``; unweighted runs
        only).
    content : float, default 1e-6
        Floor for bin shares that avoids division by zero and ``log(0)``: in the weighted PSI and in the zero-cell-floored
        IV behind ``iv_upper_threshold``.
    precision : int, default 5
        Number of decimals to which the weighted equal-frequency bin edges are rounded. Read only by weighted runs
        that do not use WOE bins.
    min_bin_prop : float, default 0.05
        Minimum bin share of the weighted equal-frequency bins: it caps their number at ``int(1 / min_bin_prop)``. Read
        only by weighted runs that do not use WOE bins; unweighted IV reads ``iv_min_bin_prop``.
    monthly_iv_min : float or None, default None
        Gate G03 (group stability): minimum IV that a feature must reach in every group of the evidence (for example
        every month); a feature whose lowest group IV is below it is dropped. Needs ``selection_evidence``. ``None``
        switches the check off.
    monthly_iv_cv_max : float or None, default None
        G03: maximum coefficient of variation (population standard deviation divided by the mean) of a feature's group
        IVs; features above it are dropped, and a zero mean IV counts as an infinite coefficient. Needs
        ``selection_evidence``.
    direction_consistency_min : float or None, default None
        G03: minimum share of a feature's groups (those with a non-zero direction) whose direction, the sign of the
        association with the target, equals the most common one. Needs ``selection_evidence``.
    min_group_n : int or None, default None
        G03: minimum number of rows for a group to count. ``None`` (or 0) uses the default of the evidence. A feature
        with fewer than two eligible groups is handled by ``insufficient_group_policy``.
    insufficient_group_policy : {"keep_warn", "drop", "raise"}, default "keep_warn"
        G03: what to do with a feature that has fewer than two eligible groups: keep it with a warning, drop it, or
        raise ``ValueError``.
    target_rules : {"all", "any", "min_pass_count"} or None, default None
        Gate G04 (multi-target): a feature must pass its per-target IV range and direction check on all targets, on any
        target, or on at least ``min_pass_count`` targets. Needs ``selection_evidence``. ``None`` switches the gate off;
        any other value raises ``ValueError``.
    min_pass_count : int or None, default None
        G04: number of targets a feature must pass when ``target_rules="min_pass_count"``; ``None`` counts as 1.
    per_target_iv_range : tuple or dict or None, default None
        G04: allowed IV range ``(low, high)`` for every target, or ``{target: (low, high)}`` per target; a ``None`` bound
        is open and a target missing from the dict is not range-checked. ``None`` applies no IV range.
    direction_reference_target : str or None, default None
        G04: target whose direction the other targets must not contradict; a target with a different non-zero direction
        for a feature counts as failed for that feature.
    max_selected_features : int or None, default None
        Gate G05 (truncation): keep at most this many features, ranked by ``ranking_metric`` (highest first). It runs
        after every other gate, never backfills, and records the cut features in ``dropped_detail`` with the reason
        ``"max_selected_features"``. ``None`` sets no cap.
    min_selected_features : int or None, default None
        G05: when fewer features survive the gates, only a warning and a ``truncation_fallback`` row in the summary are
        produced; nothing is added back.
    ranking_metric : str, default "iv"
        G05: metric that ranks the features for truncation. Only ``"iv"`` exists; any other value raises ``ValueError``
        when the cap actually cuts features.
    tie_breaker : str, default "name"
        G05: how features with equal ranking values are ordered when the cap cuts between them. The only rule is ascending
        feature name, so ``"name"`` (or None) is the only accepted value; any other raises ``ValueError`` when the config is
        built.
    vif_enabled : bool, default False
        Gate G06: after the correlation stage, repeatedly drop the feature with the highest VIF until no VIF is above
        ``vif_threshold`` or only ``vif_min_features`` features remain. Needs the optional ``statsmodels`` package
        (``ImportError`` otherwise).
    vif_threshold : float, default 10.0
        G06: the feature with the highest VIF is dropped while that VIF is above this value; an infinite VIF is always
        dropped.
    vif_min_features : int, default 2
        G06: the gate stops dropping when this many features are left, and is skipped (summary note
        ``"skipped_at_floor"``) when it starts with this many or fewer.
    vif_tie_break_metric : str, default "iv"
        G06: metric that decides among features with an equal VIF; the one with the lower value is dropped first. Only
        ``"iv"`` is supported, any other value raises ``ValueError`` when the gate is enabled.
    vif_use_woe_bins : bool, default False
        G06: ``False`` computes the VIF on the raw numeric columns (non-numeric survivors stay out of the VIF matrix,
        are kept in the selection and are reported with a warning). ``True`` computes it on the WOE-encoded INS view, so
        categorical features take part; this needs a screening WOE engine (fitted when none is given), and when the
        screen fits that engine, ``categorical_features`` additionally requires ``woe_engine="monotone"`` (``ValueError``
        otherwise).
    screen_iv_convention : {"guarded", "raw"}, default "guarded"
        How the IV on the bins of a WOE engine (``iv_use_woe_bins``, or any weighted run with an engine) treats a
        class-pure bin, a bin without bads or without goods whose WOE ``ln((bad_share + 1e-6) / (good_share + 1e-6))``
        comes from the ``1e-6`` alone. ``"guarded"`` leaves such bins (special-value bins, and ordinary bins that
        ``min_n_bins`` kept) out of the IV sum, the ``iv_guard`` convention of ``MonotoneWOEBinner``'s by-group IV:
        their rows stay in the bad and good totals, so every other bin keeps its WOE and IV; a special-value bin that
        the binner smoothed (``sv_woe_smoothing="laplace"``) counts with its finite WOE. Purity is judged on the
        quantities the IV uses (weighted sums on weighted runs, counts otherwise), so unit weights flag the same bins.
        The IV threshold, the correlation stage and the ranking of the gates use that IV. ``"raw"`` counts every bin,
        which reproduces 0.9.1. ``iv_table`` reports both (``iv`` and ``iv_raw``) with ``iv_sv_pure`` and
        ``n_sv_pure``, and ``stage_tables["iv_sv_pure_bins"]`` lists the class-pure special-value bins. The paths
        without WOE bins are not affected. Any other value raises ``ValueError``.
    iv_gap_ratio_warn : float, default 2.0
        Leak warning (0.9.3): a feature with ``iv > 0`` and ``iv_raw / iv`` at or above this value gets
        ``iv_gap_flag=True`` in ``iv_table`` and a ``UserWarning`` ("possible leakage or unresolved class-pure bins;
        review before training"). An ``iv`` at or below ``1e-12`` counts as zero and only ``iv_gap_abs_warn`` applies.
        The rule only warns and flags; it never drops a feature, and the flag is set also for a feature that the IV
        stage drops. Under ``screen_iv_convention="raw"``, and without WOE bins, ``iv`` equals ``iv_raw`` and nothing is
        flagged. Must be above 1 (``inf`` switches the rule off; ``ValueError`` otherwise).
    iv_gap_abs_warn : float, default 0.5
        Leak warning (0.9.3): flag and warn as above when ``iv_raw - iv`` is at or above this value. Must be positive
        (``inf`` switches the rule off; ``ValueError`` otherwise).
    fallback_top_k : int, default 5
        Number of features that the empty-stage fallback keeps under ``on_empty_stage="keep_all_warn"`` (0.9.3): the
        ones with the highest screening IV among those the stage dropped, leak-gate drops excluded. A whole number of
        at least 1 (``ValueError`` otherwise).

    Notes
    -----
    The G03 and G04 settings (``monthly_iv_min``, ``monthly_iv_cv_max``, ``direction_consistency_min``,
    ``target_rules``) need group or per-target evidence that only ``FeatureValidationPipeline`` builds. Setting any of
    them where no ``selection_evidence`` is passed (``feature_screen`` without it, or ``feature_screen_from_dataframe``,
    which has no such argument) raises ``ValueError``; inside ``CreditModelPipeline`` the error is caught and the whole
    screening is skipped (see its ``feature_selection`` setting).
    """

    psi_enabled: bool = True
    psi_threshold: float = 0.2
    psi_compare_splits: list[str] = field(default_factory=lambda: ["oos"])
    psi_buckets: int = 10
    psi_use_woe_bins: bool = False
    iv_enabled: bool = True
    iv_threshold: float = 0.02
    iv_upper_threshold: float | None = DEFAULT_IV_UPPER_THRESHOLD
    iv_bins: int = 10
    iv_min_bin_prop: float = 0.05
    iv_equal_freq: bool = True
    iv_use_woe_bins: bool = False
    corr_enabled: bool = True
    corr_threshold: float = 0.75
    corr_max_iterations: int = 10
    corr_use_woe_bins: bool = False
    corr_method: str = "pearson"
    corr_base_metric: str = "iv"
    corr_nan_policy: Literal["pairwise", "median_fill", "raise"] = "pairwise"
    corr_block_size: int = 256
    on_empty_stage: Literal["keep_all_warn", "keep_all", "raise"] = "keep_all_warn"
    missing_rate_threshold: float | None = None
    missing_rate_ref: float | int = -999999
    woe_engine: str = "equal_freq"
    woe_fit_query: str | None = None
    woe_params: dict[str, Any] = field(
        default_factory=lambda: {"nbins": 10, "equal_freq": True, "min_bin_prop": 0.05}
    )
    monotone_woe_params: dict[str, Any] = field(
        default_factory=lambda: {"n_init_bins": 20, "min_bin_size": 0.03, "min_n_bins": 2}
    )
    categorical_features: list[str] | None = None
    plot_path: str | None = None
    plot_outputs: bool = False
    content: float = 1e-6
    precision: int = 5
    min_bin_prop: float = 0.05
    # --- Post-corr selection gates (G03/G04/G05/G06); all legacy-off ------
    # G03 group stability (requires SelectionEvidence — FVP only)
    monthly_iv_min: float | None = None
    monthly_iv_cv_max: float | None = None
    direction_consistency_min: float | None = None
    min_group_n: int | None = None
    insufficient_group_policy: Literal["keep_warn", "drop", "raise"] = "keep_warn"
    # G04 multi-target joint gate (requires SelectionEvidence — FVP only)
    target_rules: Literal["all", "any", "min_pass_count"] | None = None
    min_pass_count: int | None = None
    per_target_iv_range: Any = None  # (low, high) or {target: (low, high)}
    direction_reference_target: str | None = None
    # G05 hard top-N truncation
    max_selected_features: int | None = None
    min_selected_features: int | None = None
    ranking_metric: str = "iv"
    tie_breaker: str = "name"
    # G06 VIF gate (needs the optional statsmodels extra)
    vif_enabled: bool = False
    vif_threshold: float = 10.0
    vif_min_features: int = 2
    vif_tie_break_metric: str = "iv"
    # G06-2: False (legacy) computes VIF on raw values with non-numeric
    # survivors excluded+warned (raw string columns crash statsmodels);
    # True computes VIF on the WOE-encoded INS view (all-numeric, matches
    # the LR design-matrix collinearity semantics).
    vif_use_woe_bins: bool = False
    # 0.9.2: class-pure bins of the IV on WOE bins ("guarded" leaves them out of the sum; "raw" is 0.9.1)
    screen_iv_convention: Literal["guarded", "raw"] = "guarded"
    # 0.9.3: warn about (and flag) features whose iv_raw is far above iv; never drops
    iv_gap_ratio_warn: float = DEFAULT_IV_GAP_RATIO_WARN
    iv_gap_abs_warn: float = DEFAULT_IV_GAP_ABS_WARN
    # 0.9.3: number of soft drops that the empty-stage fallback keeps (never a leak-gate drop)
    fallback_top_k: int = DEFAULT_FALLBACK_TOP_K

    def __post_init__(self) -> None:
        _check_iv_equal_freq(self.iv_equal_freq)
        _check_tie_breaker(self.tie_breaker)
        check_screen_iv_convention(self.screen_iv_convention)
        _check_iv_gap_thresholds(self.iv_gap_ratio_warn, self.iv_gap_abs_warn)
        self.psi_compare_splits = _normalize_psi_compare_splits(self.psi_compare_splits)
        self.corr_method = str(self.corr_method).strip().lower()
        if self.corr_method not in ("pearson", "spearman", "kendall"):
            raise ValueError(f"corr_method must be 'pearson', 'spearman' or 'kendall'; got {self.corr_method!r}")
        self.corr_base_metric = str(self.corr_base_metric).strip().lower()
        if self.corr_base_metric not in ("iv", "ks"):
            raise ValueError(f"corr_base_metric must be 'iv' or 'ks'; got {self.corr_base_metric!r}")
        _check_on_empty_stage(self.on_empty_stage)
        _check_fallback_top_k(self.fallback_top_k)

screen_config_from_mapping

screen_config_from_mapping(mapping: Mapping[str, Any] | None, *, woe_engine: str | None = None, woe_fit_query: str | None = None, woe_params: Mapping[str, Any] | None = None, monotone_woe_params: Mapping[str, Any] | None = None, plot_path: str | None = None, plot_outputs: bool = False, screen_iv_convention: str | None = None) -> FeatureScreenConfig

Build FeatureScreenConfig from a CM-style feature_selection dict.

Parameters:

Name	Type	Description	Default
mapping	Mapping[str, Any] or None	feature_selection settings keyed by the field names of FeatureScreenConfig. None is treated as an empty mapping. Missing keys take the config defaults (psi_buckets defaults to iv_bins, and min_bin_prop to iv_min_bin_prop), unknown keys are ignored, and each value is cast with bool, int, float, str or list as its field requires. iv_nbins is accepted as an alias of iv_bins and wins when both are given. plot_path and plot_outputs are never read from it.	required
woe_engine	str or None	Value of woe_engine. None (or an empty string) falls back to mapping["woe_engine"] and then to "equal_freq".	None
woe_fit_query	str or None	Value of woe_fit_query. None falls back to mapping["woe_fit_query"].	None
woe_params	Mapping[str, Any] or None	Overrides of woe_params, merged key by key over the config default. None or an empty mapping falls back to mapping["woe_params"].	None
monotone_woe_params	Mapping[str, Any] or None	Overrides of monotone_woe_params, merged key by key over the config default. None or an empty mapping falls back to mapping["monotone_woe_params"].	None
plot_path	str or None	Value of plot_path.	None
plot_outputs	bool	Value of plot_outputs.	False
screen_iv_convention	str or None	Value of screen_iv_convention. None falls back to mapping["screen_iv_convention"] and then to "guarded".	None

Returns:

Type	Description
FeatureScreenConfig	The configuration; its values are not validated here.

Source code in Modeling_Tool/Feature/Feature_Screen.py

def screen_config_from_mapping(
    mapping: Mapping[str, Any] | None,
    *,
    woe_engine: str | None = None,
    woe_fit_query: str | None = None,
    woe_params: Mapping[str, Any] | None = None,
    monotone_woe_params: Mapping[str, Any] | None = None,
    plot_path: str | None = None,
    plot_outputs: bool = False,
    screen_iv_convention: str | None = None,
) -> FeatureScreenConfig:
    """Build ``FeatureScreenConfig`` from a CM-style ``feature_selection`` dict.

    Parameters
    ----------
    mapping : Mapping[str, Any] or None
        ``feature_selection`` settings keyed by the field names of ``FeatureScreenConfig``. ``None`` is treated as an
        empty mapping. Missing keys take the config defaults (``psi_buckets`` defaults to ``iv_bins``, and
        ``min_bin_prop`` to ``iv_min_bin_prop``), unknown keys are ignored, and each value is cast with ``bool``,
        ``int``, ``float``, ``str`` or ``list`` as its field requires. ``iv_nbins`` is accepted as an alias of
        ``iv_bins`` and wins when both are given. ``plot_path`` and ``plot_outputs`` are never read from it.
    woe_engine : str or None, default None
        Value of ``woe_engine``. ``None`` (or an empty string) falls back to ``mapping["woe_engine"]`` and then to
        ``"equal_freq"``.
    woe_fit_query : str or None, default None
        Value of ``woe_fit_query``. ``None`` falls back to ``mapping["woe_fit_query"]``.
    woe_params : Mapping[str, Any] or None, default None
        Overrides of ``woe_params``, merged key by key over the config default. ``None`` or an empty mapping falls back
        to ``mapping["woe_params"]``.
    monotone_woe_params : Mapping[str, Any] or None, default None
        Overrides of ``monotone_woe_params``, merged key by key over the config default. ``None`` or an empty mapping
        falls back to ``mapping["monotone_woe_params"]``.
    plot_path : str or None, default None
        Value of ``plot_path``.
    plot_outputs : bool, default False
        Value of ``plot_outputs``.
    screen_iv_convention : str or None, default None
        Value of ``screen_iv_convention``. ``None`` falls back to ``mapping["screen_iv_convention"]`` and then to
        ``"guarded"``.

    Returns
    -------
    FeatureScreenConfig
        The configuration; its values are not validated here.
    """
    cfg = dict(mapping or {})
    iv_nbins = int(cfg.get("iv_nbins", cfg.get("iv_bins", 10)))
    return FeatureScreenConfig(
        psi_enabled=bool(cfg.get("psi_enabled", True)),
        psi_threshold=float(cfg.get("psi_threshold", 0.2)),
        psi_compare_splits=cfg.get("psi_compare_splits", ["oos"]),
        psi_buckets=int(cfg.get("psi_buckets", iv_nbins)),
        psi_use_woe_bins=bool(cfg.get("psi_use_woe_bins", False)),
        iv_enabled=bool(cfg.get("iv_enabled", True)),
        iv_threshold=float(cfg.get("iv_threshold", 0.02)),
        # a missing key takes the default (2.0 since 0.9.3); an explicit None switches the limit off
        iv_upper_threshold=(
            DEFAULT_IV_UPPER_THRESHOLD if "iv_upper_threshold" not in cfg
            else (float(cfg["iv_upper_threshold"]) if cfg["iv_upper_threshold"] is not None else None)
        ),
        iv_bins=iv_nbins,
        iv_min_bin_prop=float(cfg.get("iv_min_bin_prop", 0.05)),
        iv_equal_freq=bool(cfg.get("iv_equal_freq", True)),
        iv_use_woe_bins=bool(cfg.get("iv_use_woe_bins", False)),
        corr_enabled=bool(cfg.get("corr_enabled", True)),
        corr_threshold=float(cfg.get("corr_threshold", 0.75)),
        corr_max_iterations=int(cfg.get("corr_max_iterations", 10)),
        corr_use_woe_bins=bool(cfg.get("corr_use_woe_bins", False)),
        corr_method=str(cfg.get("corr_method", "pearson")),
        corr_base_metric=str(cfg.get("corr_base_metric", "iv")),
        corr_nan_policy=str(cfg.get("corr_nan_policy", "pairwise")),  # type: ignore[arg-type]
        corr_block_size=int(cfg.get("corr_block_size", 256)),
        on_empty_stage=str(cfg.get("on_empty_stage", "keep_all_warn")),  # type: ignore[arg-type]
        missing_rate_threshold=cfg.get("missing_rate_threshold"),
        missing_rate_ref=cfg.get("missing_rate_ref", -999999),
        woe_engine=str(woe_engine or cfg.get("woe_engine", "equal_freq")),
        woe_fit_query=woe_fit_query if woe_fit_query is not None else cfg.get("woe_fit_query"),
        woe_params=dict(FeatureScreenConfig().woe_params | dict(woe_params or cfg.get("woe_params") or {})),
        monotone_woe_params=dict(
            FeatureScreenConfig().monotone_woe_params | dict(monotone_woe_params or cfg.get("monotone_woe_params") or {})
        ),
        categorical_features=list(cfg["categorical_features"]) if cfg.get("categorical_features") else None,
        plot_path=plot_path,
        plot_outputs=plot_outputs,
        content=float(cfg.get("content", 1e-6)),
        precision=int(cfg.get("precision", 5)),
        min_bin_prop=float(cfg.get("min_bin_prop", cfg.get("iv_min_bin_prop", 0.05))),
        monthly_iv_min=cfg.get("monthly_iv_min"),
        monthly_iv_cv_max=cfg.get("monthly_iv_cv_max"),
        direction_consistency_min=cfg.get("direction_consistency_min"),
        min_group_n=cfg.get("min_group_n"),
        insufficient_group_policy=str(cfg.get("insufficient_group_policy", "keep_warn")),  # type: ignore[arg-type]
        target_rules=cfg.get("target_rules"),
        min_pass_count=cfg.get("min_pass_count"),
        per_target_iv_range=cfg.get("per_target_iv_range"),
        direction_reference_target=cfg.get("direction_reference_target"),
        max_selected_features=cfg.get("max_selected_features"),
        min_selected_features=cfg.get("min_selected_features"),
        ranking_metric=str(cfg.get("ranking_metric", "iv")),
        tie_breaker=str(cfg.get("tie_breaker") or "name"),
        vif_enabled=bool(cfg.get("vif_enabled", False)),
        vif_threshold=float(cfg.get("vif_threshold", 10.0)),
        vif_min_features=int(cfg.get("vif_min_features", 2)),
        vif_tie_break_metric=str(cfg.get("vif_tie_break_metric", "iv")),
        vif_use_woe_bins=bool(cfg.get("vif_use_woe_bins", False)),
        screen_iv_convention=str(  # type: ignore[arg-type]
            screen_iv_convention if screen_iv_convention is not None else cfg.get("screen_iv_convention", "guarded")
        ),
        iv_gap_ratio_warn=float(cfg.get("iv_gap_ratio_warn", DEFAULT_IV_GAP_RATIO_WARN)),
        iv_gap_abs_warn=float(cfg.get("iv_gap_abs_warn", DEFAULT_IV_GAP_ABS_WARN)),
        fallback_top_k=cfg.get("fallback_top_k", DEFAULT_FALLBACK_TOP_K),
    )

fit_screening_woe_engine

fit_screening_woe_engine(train: DataFrame, features: list[str], target_col: str, *, woe_engine: str = 'monotone', woe_fit_query: str | None = None, woe_params: Mapping[str, Any] | None = None, monotone_woe_params: Mapping[str, Any] | None = None, categorical_features: list[str] | None = None) -> Any

Fit a WOE engine on INS for screening steps that reuse bin boundaries.

Parameters:

Name	Type	Description	Default
train	DataFrame	INS frame to fit on. It is not modified (a copy is fitted).	required
features	list of str	Features to bin.	required
target_col	str	Binary target column (1 = bad).	required
woe_engine	str	"monotone" (case-insensitive) fits a MonotoneWOEBinner; any other value fits a WOE_Master.	"monotone"
woe_fit_query	str or None	DataFrame.query expression that selects the rows of train used for the fit. A query that fails, or that selects no row, raises ValueError.	None
woe_params	Mapping[str, Any] or None	Keyword arguments of WOE_Master.fit (WOE_Master engine only); the keys woe_suffix and missing_ref_value are passed to the WOE_Master constructor instead. None uses the fit defaults.	None
monotone_woe_params	Mapping[str, Any] or None	Arguments of the monotone engine (monotone only): the tuning keys of the MonotoneWOEBinner constructor (for example n_init_bins, min_bin_size, min_n_bins, special_values; the columns are supplied by this function) and the fit keys chi2_binning, chi2_p, chi2_init_size and n_jobs. Any other key is silently dropped. Without a special_values key the legacy sentinel -999999 is declared a special value when a numeric feature holds it.	None
categorical_features	list of str or None	Names among features that the monotone engine fits as categorical; the other features are fitted as numeric. The WOE_Master engine ignores it.	None

Returns:

Type	Description
MonotoneWOEBinner or WOE_Master	The fitted engine, ready to be passed to feature_screen as prefit_woe_engine.

Source code in Modeling_Tool/Feature/Feature_Screen.py

def fit_screening_woe_engine(
    train: pd.DataFrame,
    features: list[str],
    target_col: str,
    *,
    woe_engine: str = "monotone",
    woe_fit_query: str | None = None,
    woe_params: Mapping[str, Any] | None = None,
    monotone_woe_params: Mapping[str, Any] | None = None,
    categorical_features: list[str] | None = None,
) -> Any:
    """Fit a WOE engine on INS for screening steps that reuse bin boundaries.

    Parameters
    ----------
    train : pandas.DataFrame
        INS frame to fit on. It is not modified (a copy is fitted).
    features : list of str
        Features to bin.
    target_col : str
        Binary target column (1 = bad).
    woe_engine : str, default "monotone"
        ``"monotone"`` (case-insensitive) fits a ``MonotoneWOEBinner``; any other value fits a ``WOE_Master``.
    woe_fit_query : str or None, default None
        ``DataFrame.query`` expression that selects the rows of ``train`` used for the fit. A query that fails, or that
        selects no row, raises ``ValueError``.
    woe_params : Mapping[str, Any] or None, default None
        Keyword arguments of ``WOE_Master.fit`` (``WOE_Master`` engine only); the keys ``woe_suffix`` and
        ``missing_ref_value`` are passed to the ``WOE_Master`` constructor instead. ``None`` uses the ``fit`` defaults.
    monotone_woe_params : Mapping[str, Any] or None, default None
        Arguments of the monotone engine (monotone only): the tuning keys of the ``MonotoneWOEBinner`` constructor (for
        example ``n_init_bins``, ``min_bin_size``, ``min_n_bins``, ``special_values``; the columns are supplied by this
        function) and the ``fit`` keys ``chi2_binning``, ``chi2_p``, ``chi2_init_size`` and ``n_jobs``. Any other key is
        silently dropped. Without a ``special_values`` key the legacy sentinel ``-999999`` is declared a special value
        when a numeric feature holds it.
    categorical_features : list of str or None, default None
        Names among ``features`` that the monotone engine fits as categorical; the other features are fitted as numeric.
        The ``WOE_Master`` engine ignores it.

    Returns
    -------
    MonotoneWOEBinner or WOE_Master
        The fitted engine, ready to be passed to ``feature_screen`` as ``prefit_woe_engine``.
    """
    from Modeling_Tool.Pipeline._common import apply_woe_fit_query

    fit_ins, _ = apply_woe_fit_query(train, woe_fit_query, target=target_col)
    # A screening WOE fit is an internal diagnostic artifact.  In particular,
    # WOE_Master can append transformed columns while fitting/transforming, so
    # never let an opt-in screen stage mutate the caller's INS frame.
    fit_ins = fit_ins.copy()
    if woe_engine.lower() == "monotone":
        from Modeling_Tool import MonotoneWOEBinner

        from Modeling_Tool.Pipeline._common import as_list, quiet_default_sentinel, with_default_special_values

        categorical = [col for col in as_list(categorical_features) if col in features]
        numeric = [col for col in features if col not in set(categorical)]
        params = with_default_special_values(monotone_woe_params, fit_ins, numeric)
        init_params = {k: v for k, v in params.items() if k in _MONOTONE_INIT_KEYS}
        fit_params = {k: v for k, v in params.items() if k in _MONOTONE_FIT_KEYS}
        binner = MonotoneWOEBinner(
            feature_cols=numeric,
            target_col=target_col,
            cate_feats=categorical,
            **init_params,
        )
        with quiet_default_sentinel(monotone_woe_params):
            binner.fit(fit_ins, **fit_params)
        return binner

    from Modeling_Tool import WOE_Master

    master_params = dict(woe_params or {})
    woe_suffix = master_params.pop("woe_suffix", "_woe")
    missing_ref_value = master_params.pop("missing_ref_value", SMF_MISSING_BIN)
    master = WOE_Master(
        train_data=fit_ins,
        varlist=features,
        dep=target_col,
        graph_save_dir=None,
        woe_suffix=woe_suffix,
        missing_ref_value=missing_ref_value,
    )
    master.fit(**master_params)
    return master

feature_screen

feature_screen(splits: dict[str, DataFrame], feature_cols: list[str], target_col: str, *, weight_col: str | None = None, config: FeatureScreenConfig | None = None, prefit_woe_engine: Any | None = None, selection_evidence: Any | None = None) -> FeatureScreenResult

Run PSI -> IV -> correlation screening on pre-split INS/OOS/OOT frames.

The stages run in the order missing rate, PSI, IV, correlation and then the optional gates (VIF, group stability, multi-target, truncation); a stage that is not configured is skipped. Bins and IV always come from splits["ins"].

Parameters:

Name	Type	Description	Default
splits	dict of str to pandas.DataFrame	Frames keyed "ins", "oos" and "oot". All three keys are required: pass an empty frame for a split you do not have (df.iloc[0:0]), because a missing key raises KeyError.	required
feature_cols	list of str	Candidate features, columns of the INS frame.	required
target_col	str	Binary target column (1 = bad), present in the INS frame.	required
weight_col	str or None	Sample-weight column, present in the INS frame and in each compared OOS or OOT frame (finite, non-negative, with a positive sum; ValueError or KeyError otherwise). None runs the unweighted screen.	None
config	FeatureScreenConfig or None	Screening settings. None uses FeatureScreenConfig().	None
prefit_woe_engine	(WOE_Master, MonotoneWOEBinner or WOEEngineAdapter or None)	A fitted WOE engine. On an unweighted run each stage uses it only when its *_use_woe_bins flag is set, and without any flag it is merely recorded on the result (a monotone binner or adapter as woe_engine; a WOE_Master is never attached, only its woe_table in woe_engine_meta). A weighted run that has an engine always bins PSI and IV with it. None fits an engine only when a *_use_woe_bins flag needs one.	None
selection_evidence	SelectionEvidence or None	Evidence for the group-stability (G03) and multi-target (G04) gates. Only FeatureValidationPipeline builds it; without it, setting a G03 or G04 threshold raises ValueError.	None

Returns:

Type	Description
FeatureScreenResult	The surviving features (selected_features) with the stage tables and the audit trail; FeatureScreenResult is the same class as WeightedScreenResult.

Raises:

Type	Description
KeyError	If splits lacks one of the three keys, or a frame that needs the weights lacks weight_col.
ValueError	If the weights are invalid, if a G03 or G04 threshold is set without selection_evidence, or if a gate setting is invalid (target_rules, ranking_metric, vif_tie_break_metric).
EmptyStageError	A ValueError: a stage would drop every feature and on_empty_stage="raise", or (under the default "keep_all_warn") every feature it would drop is a leak-gate drop, which no fallback may revive.
ImportError	If vif_enabled is True and statsmodels is not installed.

Warns:

Type	Description
UserWarning	For each feature that the IV upper limit drops (possible target leakage), for each feature with a large gap between iv_raw and iv (see FeatureScreenConfig.iv_gap_ratio_warn), and for each empty-stage fallback.

Notes

Four code paths exist. An unweighted run without WOE bins uses PSICalculator, VarExtractionInsights (decision-tree IV bins) and CorrelationFilter. A weighted run without WOE bins uses weighted equal-frequency bins and needs numeric features. When a *_use_woe_bins flag is set or a prefit_woe_engine is given, an unweighted run switches only the flagged stages to the bins of the WOE engine, while a weighted run bins PSI and IV with the engine. corr_dropped is filled by weighted runs only.

Source code in Modeling_Tool/Feature/Feature_Screen.py

def feature_screen(
    splits: dict[str, pd.DataFrame],
    feature_cols: list[str],
    target_col: str,
    *,
    weight_col: str | None = None,
    config: FeatureScreenConfig | None = None,
    prefit_woe_engine: Any | None = None,
    selection_evidence: Any | None = None,
) -> FeatureScreenResult:
    """Run PSI -> IV -> correlation screening on pre-split INS/OOS/OOT frames.

    The stages run in the order missing rate, PSI, IV, correlation and then the optional gates (VIF, group stability,
    multi-target, truncation); a stage that is not configured is skipped. Bins and IV always come from ``splits["ins"]``.

    Parameters
    ----------
    splits : dict of str to pandas.DataFrame
        Frames keyed ``"ins"``, ``"oos"`` and ``"oot"``. All three keys are required: pass an empty frame for a split
        you do not have (``df.iloc[0:0]``), because a missing key raises ``KeyError``.
    feature_cols : list of str
        Candidate features, columns of the INS frame.
    target_col : str
        Binary target column (1 = bad), present in the INS frame.
    weight_col : str or None, default None
        Sample-weight column, present in the INS frame and in each compared OOS or OOT frame (finite, non-negative, with
        a positive sum; ``ValueError`` or ``KeyError`` otherwise). ``None`` runs the unweighted screen.
    config : FeatureScreenConfig or None, default None
        Screening settings. ``None`` uses ``FeatureScreenConfig()``.
    prefit_woe_engine : WOE_Master, MonotoneWOEBinner or WOEEngineAdapter or None, default None
        A fitted WOE engine. On an unweighted run each stage uses it only when its ``*_use_woe_bins`` flag is set, and
        without any flag it is merely recorded on the result (a monotone binner or adapter as ``woe_engine``; a
        ``WOE_Master`` is never attached, only its ``woe_table`` in ``woe_engine_meta``). A weighted run that has an
        engine always bins PSI and IV with it. ``None`` fits an engine only when a ``*_use_woe_bins`` flag needs one.
    selection_evidence : SelectionEvidence or None, default None
        Evidence for the group-stability (G03) and multi-target (G04) gates. Only ``FeatureValidationPipeline`` builds
        it; without it, setting a G03 or G04 threshold raises ``ValueError``.

    Returns
    -------
    FeatureScreenResult
        The surviving features (``selected_features``) with the stage tables and the audit trail; ``FeatureScreenResult``
        is the same class as ``WeightedScreenResult``.

    Raises
    ------
    KeyError
        If ``splits`` lacks one of the three keys, or a frame that needs the weights lacks ``weight_col``.
    ValueError
        If the weights are invalid, if a G03 or G04 threshold is set without ``selection_evidence``, or if a gate setting
        is invalid (``target_rules``, ``ranking_metric``, ``vif_tie_break_metric``).
    EmptyStageError
        A ``ValueError``: a stage would drop every feature and ``on_empty_stage="raise"``, or (under the default
        ``"keep_all_warn"``) every feature it would drop is a leak-gate drop, which no fallback may revive.
    ImportError
        If ``vif_enabled`` is True and ``statsmodels`` is not installed.

    Warns
    -----
    UserWarning
        For each feature that the IV upper limit drops (possible target leakage), for each feature with a large gap
        between ``iv_raw`` and ``iv`` (see ``FeatureScreenConfig.iv_gap_ratio_warn``), and for each empty-stage
        fallback.

    Notes
    -----
    Four code paths exist. An unweighted run without WOE bins uses ``PSICalculator``, ``VarExtractionInsights``
    (decision-tree IV bins) and ``CorrelationFilter``. A weighted run without WOE bins uses weighted equal-frequency
    bins and needs numeric features. When a ``*_use_woe_bins`` flag is set or a ``prefit_woe_engine`` is given, an
    unweighted run switches only the flagged stages to the bins of the WOE engine, while a weighted run bins PSI and
    IV with the engine. ``corr_dropped`` is filled by weighted runs only.
    """
    cfg = config or FeatureScreenConfig()
    use_woe_bins = _needs_woe_bins(cfg)

    if cfg.psi_enabled:
        empty = [name for name in cfg.psi_compare_splits if name in splits and len(splits[name]) == 0]
        if empty:
            warnings.warn(
                f"feature_screen: psi_compare_splits {empty} has no rows in this run, so the PSI stage does not "
                "compare INS with it" + (" and checks nothing" if len(empty) == len(cfg.psi_compare_splits) else ""),
                UserWarning,
                stacklevel=2,
            )

    if weight_col is not None and target_col in splits["ins"].columns:
        # Rows without a target cannot enter an IV: the weighted path cast the target to float and the NaN made every
        # IV 0, which dropped all features (or kept them all through ``keep_all_warn``). The unweighted path skips them.
        observed = splits["ins"][target_col].notna()
        if not bool(observed.all()):
            splits = {**splits, "ins": splits["ins"][observed]}

    # The stages fill this list in place. When a stage raises after the IV stage, the hard drops found so far travel
    # on the exception (``smf_hard_dropped``), so that a caller that keeps going, such as ``CreditModelPipeline``,
    # cannot bring them back (0.9.3).
    dropped_rows: list[dict] = []
    try:
        return _dispatch_feature_screen(
            splits, feature_cols, target_col, weight_col, cfg, use_woe_bins,
            prefit_woe_engine=prefit_woe_engine, selection_evidence=selection_evidence, dropped_rows=dropped_rows,
        )
    except EmptyStageError:
        raise
    except Exception as exc:
        hard = hard_dropped_features(dropped_rows)
        if hard:
            try:
                exc.smf_hard_dropped = hard  # type: ignore[attr-defined]
            except AttributeError:  # an exception type without an instance dict
                pass
        raise

feature_screen_from_dataframe

feature_screen_from_dataframe(data: DataFrame, feature_cols: list[str], target_col: str, split_col: str, *, weight_col: str | None = None, config: FeatureScreenConfig | None = None, prefit_woe_engine: Any | None = None) -> FeatureScreenResult

Convenience wrapper that resolves INS/OOS/OOT from a tagged dataframe.

The rows are assigned to the splits by an exact match of split_col with "ins", "oos" and "oot" (rows with any other value are left out), and the screen then runs as in :func:feature_screen.

Parameters:

Name	Type	Description	Default
data	DataFrame	Frame that holds the features, the target and the split column. It is not modified (the splits are copies).	required
feature_cols	list of str	Candidate features. target_col, split_col and weight_col are removed from the list, and names that are not columns of data are silently ignored.	required
target_col	str	Binary target column (1 = bad).	required
split_col	str	Column whose values "ins", "oos" and "oot" mark the splits (case-sensitive). At least one row must be "ins"; an absent OOS or OOT split gives an empty frame.	required
weight_col	str or None	Sample-weight column of data. None runs the unweighted screen.	None
config	FeatureScreenConfig or None	Screening settings. None uses FeatureScreenConfig().	None
prefit_woe_engine	(WOE_Master, MonotoneWOEBinner or WOEEngineAdapter or None)	A fitted WOE engine, handled as in :func:feature_screen.	None

Returns:

Type	Description
FeatureScreenResult	The result of :func:feature_screen.

Raises:

Type	Description
KeyError	If split_col or weight_col is not a column of data.
ValueError	If no row has split_col == "ins", plus the errors of :func:feature_screen.

Notes

This wrapper has no selection_evidence argument, so a config with a G03 or G04 threshold raises ValueError.

Source code in Modeling_Tool/Feature/Feature_Screen.py

def feature_screen_from_dataframe(
    data: pd.DataFrame,
    feature_cols: list[str],
    target_col: str,
    split_col: str,
    *,
    weight_col: str | None = None,
    config: FeatureScreenConfig | None = None,
    prefit_woe_engine: Any | None = None,
) -> FeatureScreenResult:
    """Convenience wrapper that resolves INS/OOS/OOT from a tagged dataframe.

    The rows are assigned to the splits by an exact match of ``split_col`` with ``"ins"``, ``"oos"`` and ``"oot"`` (rows
    with any other value are left out), and the screen then runs as in :func:`feature_screen`.

    Parameters
    ----------
    data : pandas.DataFrame
        Frame that holds the features, the target and the split column. It is not modified (the splits are copies).
    feature_cols : list of str
        Candidate features. ``target_col``, ``split_col`` and ``weight_col`` are removed from the list, and names that are
        not columns of ``data`` are silently ignored.
    target_col : str
        Binary target column (1 = bad).
    split_col : str
        Column whose values ``"ins"``, ``"oos"`` and ``"oot"`` mark the splits (case-sensitive). At least one row must be
        ``"ins"``; an absent OOS or OOT split gives an empty frame.
    weight_col : str or None, default None
        Sample-weight column of ``data``. ``None`` runs the unweighted screen.
    config : FeatureScreenConfig or None, default None
        Screening settings. ``None`` uses ``FeatureScreenConfig()``.
    prefit_woe_engine : WOE_Master, MonotoneWOEBinner or WOEEngineAdapter or None, default None
        A fitted WOE engine, handled as in :func:`feature_screen`.

    Returns
    -------
    FeatureScreenResult
        The result of :func:`feature_screen`.

    Raises
    ------
    KeyError
        If ``split_col`` or ``weight_col`` is not a column of ``data``.
    ValueError
        If no row has ``split_col == "ins"``, plus the errors of :func:`feature_screen`.

    Notes
    -----
    This wrapper has no ``selection_evidence`` argument, so a ``config`` with a G03 or G04 threshold raises
    ``ValueError``.
    """
    exclude = {target_col, split_col}
    if weight_col:
        exclude.add(weight_col)
    features = [c for c in feature_cols if c not in exclude and c in data.columns]
    splits = _resolve_splits(data, split_col)
    if weight_col is not None and weight_col not in data.columns:
        raise KeyError(f"Missing weight_col {weight_col!r}")
    return feature_screen(
        splits,
        features,
        target_col,
        weight_col=weight_col,
        config=config,
        prefit_woe_engine=prefit_woe_engine,
    )

Weighted Feature Screening: Weighted_Screen

Weighted_Screen

Weighted feature screening: PSI -> IV -> correlation dedup.

WeightedScreenResult dataclass

Result of :func:weighted_feature_screen and of feature_screen (also exported as FeatureScreenResult).

Parameters:

Name	Type	Description	Default
selected_features	list of str	The features that passed every stage.	required
iv_table	DataFrame	IV of the features that entered the IV stage, with the columns var, iv_weighted, n_bins, missing_rate, iv, iv_raw, iv_sv_pure and n_sv_pure. iv_weighted and iv hold the IV that the stage tests (unweighted runs store the plain IV). On WOE bins, iv follows screen_iv_convention (since 0.9.2 the class-pure bins are left out of the sum by default), iv_raw counts every bin (the IV up to 0.9.1), iv_sv_pure is the IV of the class-pure bins and n_sv_pure their rows. The paths without WOE bins have their own class-pure handling: there iv_raw equals iv and iv_sv_pure / n_sv_pure are NaN. iv_gap_flag (0.9.3) is True for a feature whose iv_raw is far above iv (iv_gap_ratio_warn / iv_gap_abs_warn; possible leakage or unresolved class-pure bins), also when the IV stage dropped it. The frame is empty when the IV stage is off.	required
psi_table	DataFrame	PSI of the features that entered the PSI stage, with the columns var, psi_ins_oos, psi_ins_oot and psi_max (the larger of the two). The frame is empty when the PSI stage is off.	required
corr_dropped	DataFrame	One row per feature that the correlation stage removed, with the columns var_a, var_b, corr, iv_a, iv_b, kept and dropped. Weighted runs fill it; unweighted runs leave it empty even when the stage removed features (compare selected_features with the features that entered the stage).	required
summary	DataFrame	One row per stage (initial, missing_rate, psi, iv, corr, the gates that ran, final) with the columns stage, n_in, n_out, threshold and weight_col. A stage that would have dropped every feature adds a <stage>_fallback row (n_out is the number of features the fallback kept), and some gates add audit columns such as note.	required
missing_rate_table	DataFrame	Missing rate on INS of every feature that entered the missing-rate stage, with the columns var and missing_rate. Empty when missing_rate_threshold is None.	empty frame
missing_rate_dropped	DataFrame	The rows of missing_rate_table above the threshold, which the stage dropped (columns var and missing_rate).	empty frame
woe_engine	object or None	The WOE engine that the screen fitted or was given. A MonotoneWOEBinner (or adapter) is attached as it is; a WOE_Master is never attached because it holds the training frame (its woe_table is in woe_engine_meta). None when no engine was used.	None
woe_engine_meta	dict	Metadata of the engine: engine_name, fit_features, target_col, weight_col, screen_stages and engine_attached, plus woe_table (or woe_table_error) for a WOE_Master. Empty when no engine was used.	{}
dropped_detail	DataFrame	Per-feature drop evidence of the IV upper limit and of the gates, with the columns var, stage, metric, value, threshold and reason. The IV upper limit records "iv_above_upper", or "iv_above_upper_sv_pure" when the feature would not exceed the limit without its class-pure special-value bins (it is dropped all the same); these leak-gate drops are hard drops that no fallback brings back. A feature that an empty-stage fallback kept has the reason "fallback_kept" (on its own row of the stage, or on a new row with metric="iv", its IV and the stage threshold). Empty when no such row exists.	empty frame
stage_tables	dict of str to pandas.DataFrame	Evidence frames of the gates that ran, keyed vif, group_stability, group_stability_violations, multi_target and truncation. When the IV stage on WOE bins finds class-pure special-value bins, the key iv_sv_pure_bins lists them: var, special_value, bin_label, n (rows), bad, good (weighted sums on a weighted run), bad_rate, woe (the WOE the binning gives the bin), iv (its contribution to iv_raw) and low_count (n < 30). The list is for review (for example as candidate rules); nothing acts on it.	{}

Source code in Modeling_Tool/Feature/Weighted_Screen.py

@dataclass
class WeightedScreenResult:
    """Result of :func:`weighted_feature_screen` and of ``feature_screen`` (also exported as ``FeatureScreenResult``).

    Parameters
    ----------
    selected_features : list of str
        The features that passed every stage.
    iv_table : pandas.DataFrame
        IV of the features that entered the IV stage, with the columns ``var``, ``iv_weighted``, ``n_bins``,
        ``missing_rate``, ``iv``, ``iv_raw``, ``iv_sv_pure`` and ``n_sv_pure``. ``iv_weighted`` and ``iv`` hold the IV
        that the stage tests (unweighted runs store the plain IV). On WOE bins, ``iv`` follows
        ``screen_iv_convention`` (since 0.9.2 the class-pure bins are left out of the sum by default), ``iv_raw`` counts
        every bin (the IV up to 0.9.1), ``iv_sv_pure`` is the IV of the class-pure bins and ``n_sv_pure`` their rows. The
        paths without WOE bins have their own class-pure handling: there ``iv_raw`` equals ``iv`` and ``iv_sv_pure`` /
        ``n_sv_pure`` are NaN. ``iv_gap_flag`` (0.9.3) is True for a feature whose ``iv_raw`` is far above ``iv``
        (``iv_gap_ratio_warn`` / ``iv_gap_abs_warn``; possible leakage or unresolved class-pure bins), also when the
        IV stage dropped it. The frame is empty when the IV stage is off.
    psi_table : pandas.DataFrame
        PSI of the features that entered the PSI stage, with the columns ``var``, ``psi_ins_oos``, ``psi_ins_oot`` and
        ``psi_max`` (the larger of the two). The frame is empty when the PSI stage is off.
    corr_dropped : pandas.DataFrame
        One row per feature that the correlation stage removed, with the columns ``var_a``, ``var_b``, ``corr``,
        ``iv_a``, ``iv_b``, ``kept`` and ``dropped``. Weighted runs fill it; unweighted runs leave it empty even when
        the stage removed features (compare ``selected_features`` with the features that entered the stage).
    summary : pandas.DataFrame
        One row per stage (``initial``, ``missing_rate``, ``psi``, ``iv``, ``corr``, the gates that ran, ``final``) with
        the columns ``stage``, ``n_in``, ``n_out``, ``threshold`` and ``weight_col``. A stage that would have dropped
        every feature adds a ``<stage>_fallback`` row (``n_out`` is the number of features the fallback kept), and some
        gates add audit columns such as ``note``.
    missing_rate_table : pandas.DataFrame, default empty frame
        Missing rate on INS of every feature that entered the missing-rate stage, with the columns ``var`` and
        ``missing_rate``. Empty when ``missing_rate_threshold`` is ``None``.
    missing_rate_dropped : pandas.DataFrame, default empty frame
        The rows of ``missing_rate_table`` above the threshold, which the stage dropped (columns ``var`` and
        ``missing_rate``).
    woe_engine : object or None, default None
        The WOE engine that the screen fitted or was given. A ``MonotoneWOEBinner`` (or adapter) is attached as it is;
        a ``WOE_Master`` is never attached because it holds the training frame (its ``woe_table`` is in
        ``woe_engine_meta``). ``None`` when no engine was used.
    woe_engine_meta : dict, default {}
        Metadata of the engine: ``engine_name``, ``fit_features``, ``target_col``, ``weight_col``, ``screen_stages`` and
        ``engine_attached``, plus ``woe_table`` (or ``woe_table_error``) for a ``WOE_Master``. Empty when no engine was
        used.
    dropped_detail : pandas.DataFrame, default empty frame
        Per-feature drop evidence of the IV upper limit and of the gates, with the columns ``var``, ``stage``,
        ``metric``, ``value``, ``threshold`` and ``reason``. The IV upper limit records ``"iv_above_upper"``, or
        ``"iv_above_upper_sv_pure"`` when the feature would not exceed the limit without its class-pure special-value
        bins (it is dropped all the same); these leak-gate drops are hard drops that no fallback brings back. A feature
        that an empty-stage fallback kept has the reason ``"fallback_kept"`` (on its own row of the stage, or on a new
        row with ``metric="iv"``, its IV and the stage threshold). Empty when no such row exists.
    stage_tables : dict of str to pandas.DataFrame, default {}
        Evidence frames of the gates that ran, keyed ``vif``, ``group_stability``, ``group_stability_violations``,
        ``multi_target`` and ``truncation``. When the IV stage on WOE bins finds class-pure special-value bins, the key
        ``iv_sv_pure_bins`` lists them: ``var``, ``special_value``, ``bin_label``, ``n`` (rows), ``bad``, ``good``
        (weighted sums on a weighted run), ``bad_rate``, ``woe`` (the WOE the binning gives the bin), ``iv`` (its
        contribution to ``iv_raw``) and ``low_count`` (``n < 30``). The list is for review (for example as candidate
        rules); nothing acts on it.
    """

    selected_features: list[str]
    iv_table: pd.DataFrame
    psi_table: pd.DataFrame
    corr_dropped: pd.DataFrame
    summary: pd.DataFrame
    missing_rate_table: pd.DataFrame = field(
        default_factory=lambda: pd.DataFrame(columns=_MISSING_RATE_COLS),
    )
    missing_rate_dropped: pd.DataFrame = field(
        default_factory=lambda: pd.DataFrame(columns=_MISSING_RATE_COLS),
    )
    # G00: the WOE engine fitted (or reused) by the screen, with enough
    # metadata to rebuild the CM reuse contract. Monotone engines are attached
    # as-is (they hold no training frame); WOE_Master engines are never
    # attached — only their woe_table travels (see woe_engine_meta).
    woe_engine: Any | None = None
    woe_engine_meta: dict[str, Any] = field(default_factory=dict)
    # Per-feature drop evidence across all gate stages.
    dropped_detail: pd.DataFrame = field(
        default_factory=lambda: pd.DataFrame(columns=_DROPPED_DETAIL_COLS),
    )
    # Evidence frames per gate stage (vif / group_stability / multi_target ...).
    stage_tables: dict[str, pd.DataFrame] = field(default_factory=dict)

EmptyStageError

Bases: ValueError

A screening stage would drop every feature, and the screening must not go on.

Raised when on_empty_stage is "raise", and (since 0.9.3) under the default "keep_all_warn" when every feature the stage would drop is a hard drop (a leak-gate drop, see HARD_DROP_REASONS), which no fallback may bring back. It is a ValueError (what the stage raised before), but a distinct type so that a caller which catches screening failures, such as CreditModelPipeline, can tell this deliberate stop from an unexpected failure.

Source code in Modeling_Tool/Feature/Weighted_Screen.py

class EmptyStageError(ValueError):
    """A screening stage would drop every feature, and the screening must not go on.

    Raised when ``on_empty_stage`` is ``"raise"``, and (since 0.9.3) under the default ``"keep_all_warn"`` when every
    feature the stage would drop is a hard drop (a leak-gate drop, see ``HARD_DROP_REASONS``), which no fallback may
    bring back. It is a ``ValueError`` (what the stage raised before), but a distinct type so that a caller which catches
    screening failures, such as ``CreditModelPipeline``, can tell this deliberate stop from an unexpected failure.
    """

hard_dropped_features

hard_dropped_features(dropped_detail: Any) -> dict[str, str]

The hard drops of a screening: the features that the leak gate dropped, which no empty-stage fallback brings back.

Parameters:

Name	Type	Description	Default
dropped_detail	pandas.DataFrame or list of dict or None	The dropped_detail frame of a screening result, or the list of row dicts that the stages fill (keys var and reason). None or an empty frame gives an empty dict.	required

Returns:

Type	Description
dict of str to str	Feature -> reason of every row whose reason is in HARD_DROP_REASONS.

Source code in Modeling_Tool/Feature/Weighted_Screen.py

def hard_dropped_features(dropped_detail: Any) -> dict[str, str]:
    """The hard drops of a screening: the features that the leak gate dropped, which no empty-stage fallback brings back.

    Parameters
    ----------
    dropped_detail : pandas.DataFrame or list of dict or None
        The ``dropped_detail`` frame of a screening result, or the list of row dicts that the stages fill (keys ``var``
        and ``reason``). ``None`` or an empty frame gives an empty dict.

    Returns
    -------
    dict of str to str
        Feature -> reason of every row whose reason is in ``HARD_DROP_REASONS``.
    """
    if dropped_detail is None:
        return {}
    if isinstance(dropped_detail, pd.DataFrame):
        if dropped_detail.empty or not {"var", "reason"} <= set(dropped_detail.columns):
            return {}
        rows = dropped_detail[["var", "reason"]].to_dict("records")
    else:
        rows = list(dropped_detail)
    return {str(row["var"]): str(row["reason"]) for row in rows if row.get("reason") in HARD_DROP_REASONS}

weighted_feature_screen

weighted_feature_screen(data: DataFrame, feature_cols: list[str], target_col: str, split_col: str, weight_col: str | None = None, psi_enabled: bool = True, psi_threshold: float = 0.2, psi_compare_splits: list[str] | None = None, iv_enabled: bool = True, iv_threshold: float = 0.02, corr_enabled: bool = True, corr_threshold: float = 0.75, iv_bins: int = 10, min_bin_prop: float = 0.05, iv_min_bin_prop: float | None = None, corr_max_iterations: int = 10, content: float = 1e-06, precision: int = 5, psi_buckets: int | None = None, plot_path: str | None = None, plot_outputs: bool = False, iv_equal_freq: bool = True, psi_use_woe_bins: bool = False, iv_use_woe_bins: bool = False, corr_use_woe_bins: bool = False, corr_method: str = 'pearson', corr_base_metric: str = 'iv', woe_engine: str = 'equal_freq', woe_fit_query: str | None = None, woe_params: dict[str, Any] | None = None, monotone_woe_params: dict[str, Any] | None = None, prefit_woe_engine: Any | None = None, corr_nan_policy: Literal['pairwise', 'median_fill', 'raise'] = 'pairwise', corr_block_size: int = 256, on_empty_stage: Literal['keep_all_warn', 'keep_all', 'raise'] = 'keep_all_warn', screen_iv_convention: Literal['guarded', 'raw'] = 'guarded', iv_upper_threshold: float | None = DEFAULT_IV_UPPER_THRESHOLD, iv_gap_ratio_warn: float = DEFAULT_IV_GAP_RATIO_WARN, iv_gap_abs_warn: float = DEFAULT_IV_GAP_ABS_WARN, fallback_top_k: int = DEFAULT_FALLBACK_TOP_K) -> WeightedScreenResult

Run PSI -> IV -> correlation screening with optional sample weights.

Bin edges are fit on ins only. When weight_col is None the legacy unweighted tools are used for backward-compatible results with CreditModelPipeline (IV uses tree binning via VarExtractionInsights). Weighted screening uses equal-frequency bins on weighted quantiles.

The function builds a FeatureScreenConfig from its arguments and runs feature_screen_from_dataframe; a *_use_woe_bins flag or a prefit_woe_engine makes the screen bin with a WOE engine instead.

Parameters:

Name	Type	Description	Default
data	DataFrame	Frame that holds the features, the target and the split column. It is not modified.	required
feature_cols	list of str	Candidate features. target_col, split_col and weight_col are removed from the list, and names that are not columns of data are silently ignored.	required
target_col	str	Binary target column (1 = bad).	required
split_col	str	Column whose values "ins", "oos" and "oot" mark the splits (case-sensitive). At least one row must be "ins".	required
weight_col	str or None	Sample-weight column of data (finite, non-negative, positive sum). None runs the unweighted screen with the legacy tools (unless a *_use_woe_bins flag or prefit_woe_engine selects the WOE-engine bins).	None
psi_enabled	bool	Whether to run the PSI stage.	True
psi_threshold	float	A feature is kept when its largest PSI over the compared splits is strictly below this value.	0.2
psi_compare_splits	list of str or None	Splits compared with "ins": "oos", "oot" or both. None means ["oos", "oot"] (the default of FeatureScreenConfig is ["oos"]). A listed split without rows is skipped.	None
iv_enabled	bool	Whether to run the IV stage.	True
iv_threshold	float	Features with an IV below this value are dropped.	0.02
corr_enabled	bool	Whether to run the correlation stage.	True
corr_threshold	float	Absolute correlation above which two features are redundant; of such a group the feature with the higher IV is kept.	0.75
iv_bins	int	Maximum number of IV bins (decision-tree leaves on unweighted runs, weighted equal-frequency bins on weighted runs). A weighted run that does not use WOE bins also derives its PSI bins from it.	10
min_bin_prop	float	Minimum bin share of the weighted equal-frequency bins (it caps their number at int(1 / min_bin_prop)). It is also the default of iv_min_bin_prop.	0.05
iv_min_bin_prop	float or None	Minimum bin share of the IV binning of unweighted runs. None uses min_bin_prop.	None
corr_max_iterations	int	Maximum number of filtering rounds of the correlation stage.	10
content	float	Floor for bin shares that avoids division by zero and log(0) in the weighted PSI.	1e-6
precision	int	Number of decimals to which the weighted equal-frequency bin edges are rounded.	5
psi_buckets	int or None	Number of equal-frequency buckets of the default PSI binning of unweighted runs. None uses iv_bins.	None
plot_path	str or None	Directory for the IV-stage charts; used only by unweighted runs together with plot_outputs.	None
plot_outputs	bool	Whether to draw the WOE charts of the features that enter the IV stage into <plot_path>/overall (unweighted runs only).	False
iv_equal_freq	bool	Must stay True: the IV binning cannot switch equal-frequency bins off (unweighted runs use decision-tree bins and weighted runs weighted equal-frequency bins), so any other value raises ValueError.	True
psi_use_woe_bins	bool	Compute PSI on the bins of the screening WOE engine instead of the default binning.	False
iv_use_woe_bins	bool	Compute IV on the bins of the screening WOE engine instead of the default binning.	False
corr_use_woe_bins	bool	Let the screening WOE engine take part in the correlation stage (WOE encoding of non-numeric features, and the IV that decides between correlated features on unweighted runs).	False
corr_method	('pearson', 'spearman', 'kendall')	Correlation coefficient of the correlation stage (Pearson only with non-constant weights).	"pearson"
corr_base_metric	('iv', 'ks')	Metric that decides which of two correlated features is kept.	"iv"
woe_engine	str	Engine fitted when a *_use_woe_bins flag is set and no prefit_woe_engine is given: "monotone" fits a MonotoneWOEBinner, any other value a WOE_Master.	"equal_freq"
woe_fit_query	str or None	DataFrame.query expression that selects the INS rows used to fit the screening engine; a failing query is silently ignored.	None
woe_params	dict or None	Keyword arguments of WOE_Master.fit for the WOE_Master engine. None means {}, which gives the fit defaults.	None
monotone_woe_params	dict or None	Arguments of the MonotoneWOEBinner engine (constructor keys and the fit keys chi2_binning, chi2_p, chi2_init_size and n_jobs). None means {}, which gives the binner defaults.	None
prefit_woe_engine	(WOE_Master, MonotoneWOEBinner or WOEEngineAdapter or None)	A fitted WOE engine. On an unweighted run a stage uses it only when its *_use_woe_bins flag is set; a weighted run that has an engine always bins PSI and IV with it.	None
corr_nan_policy	('pairwise', 'median_fill', 'raise')	Missing values in the weighted correlation matrix: "pairwise" uses the rows where both values exist, "median_fill" fills with the weighted median, "raise" raises ValueError. Weighted runs only.	"pairwise"
corr_block_size	int	Number of feature columns per block of the weighted pairwise correlation matrix; it only limits memory and must be positive. Weighted runs only.	256
on_empty_stage	('keep_all_warn', 'keep_all', 'raise')	What to do when a stage would drop every feature. "keep_all_warn" never brings back a feature that the IV upper limit dropped (a possible leak); from the IV stage on it keeps the fallback_top_k other features with the highest screening IV and warns, and it raises EmptyStageError when only such leak drops are left (0.9.3). "keep_all" keeps every feature, leak drops included (the behavior up to 0.9.3, only for comparing with old results); "raise" raises EmptyStageError (a ValueError). See FeatureScreenConfig.on_empty_stage.	"keep_all_warn"
screen_iv_convention	('guarded', 'raw')	How the IV on WOE bins treats class-pure bins: "guarded" leaves them out of the IV sum, "raw" counts every bin (the numbers of 0.9.1). See FeatureScreenConfig.screen_iv_convention.	"guarded"
iv_upper_threshold	float or None	Upper IV limit (gate G02, a leakage guard). The test uses the zero-cell-floored IV and, on WOE bins under the guarded convention, the larger of it and iv_raw, never the guarded iv. None switches it off (the behavior up to 0.9.2, where it was not exposed here). See FeatureScreenConfig.iv_upper_threshold.	2.0
iv_gap_ratio_warn	float	Warn about a feature, and set its iv_gap_flag, when iv_raw / iv reaches this value (with iv > 0). See FeatureScreenConfig.iv_gap_ratio_warn.	2.0
iv_gap_abs_warn	float	Warn about a feature, and set its iv_gap_flag, when iv_raw - iv reaches this value. See FeatureScreenConfig.iv_gap_abs_warn.	0.5
fallback_top_k	int	Number of features that the empty-stage fallback keeps under on_empty_stage="keep_all_warn" (0.9.3). See FeatureScreenConfig.fallback_top_k.	5

Returns:

Type	Description
WeightedScreenResult	The surviving features with the stage tables and the audit trail.

Raises:

Type	Description
KeyError	If split_col or weight_col is not a column of data.
ValueError	If no row has split_col == "ins", if the weights are invalid, if iv_equal_freq is False, or when a stage would drop every feature and on_empty_stage="raise" or only leak drops are left (EmptyStageError).

Warns:

Type	Description
UserWarning	For each feature whose iv_raw is far above its iv (see iv_gap_ratio_warn and iv_gap_abs_warn): possible leakage or unresolved class-pure bins. For each feature that the IV upper limit drops: possible target leakage (0.9.3). For each empty-stage fallback, naming the features it kept.

Notes

missing_rate_threshold, categorical_features and the gate settings (VIF, group stability, multi-target, truncation) are not exposed here; use feature_screen with a FeatureScreenConfig for them. A weighted run without WOE bins needs numeric features.

Source code in Modeling_Tool/Feature/Weighted_Screen.py

def weighted_feature_screen(
    data: pd.DataFrame,
    feature_cols: list[str],
    target_col: str,
    split_col: str,
    weight_col: str | None = None,
    psi_enabled: bool = True,
    psi_threshold: float = 0.2,
    psi_compare_splits: list[str] | None = None,
    iv_enabled: bool = True,
    iv_threshold: float = 0.02,
    corr_enabled: bool = True,
    corr_threshold: float = 0.75,
    iv_bins: int = 10,
    min_bin_prop: float = 0.05,
    iv_min_bin_prop: float | None = None,
    corr_max_iterations: int = 10,
    content: float = 1e-6,
    precision: int = 5,
    psi_buckets: int | None = None,
    plot_path: str | None = None,
    plot_outputs: bool = False,
    iv_equal_freq: bool = True,
    psi_use_woe_bins: bool = False,
    iv_use_woe_bins: bool = False,
    corr_use_woe_bins: bool = False,
    corr_method: str = "pearson",
    corr_base_metric: str = "iv",
    woe_engine: str = "equal_freq",
    woe_fit_query: str | None = None,
    woe_params: dict[str, Any] | None = None,
    monotone_woe_params: dict[str, Any] | None = None,
    prefit_woe_engine: Any | None = None,
    corr_nan_policy: Literal["pairwise", "median_fill", "raise"] = "pairwise",
    corr_block_size: int = 256,
    on_empty_stage: Literal["keep_all_warn", "keep_all", "raise"] = "keep_all_warn",
    screen_iv_convention: Literal["guarded", "raw"] = "guarded",
    iv_upper_threshold: float | None = DEFAULT_IV_UPPER_THRESHOLD,
    iv_gap_ratio_warn: float = DEFAULT_IV_GAP_RATIO_WARN,
    iv_gap_abs_warn: float = DEFAULT_IV_GAP_ABS_WARN,
    fallback_top_k: int = DEFAULT_FALLBACK_TOP_K,
) -> WeightedScreenResult:
    """Run PSI -> IV -> correlation screening with optional sample weights.

    Bin edges are fit on ``ins`` only. When ``weight_col`` is None the legacy
    unweighted tools are used for backward-compatible results with
    ``CreditModelPipeline`` (IV uses tree binning via ``VarExtractionInsights``).
    Weighted screening uses equal-frequency bins on weighted quantiles.

    The function builds a ``FeatureScreenConfig`` from its arguments and runs ``feature_screen_from_dataframe``; a
    ``*_use_woe_bins`` flag or a ``prefit_woe_engine`` makes the screen bin with a WOE engine instead.

    Parameters
    ----------
    data : pandas.DataFrame
        Frame that holds the features, the target and the split column. It is not modified.
    feature_cols : list of str
        Candidate features. ``target_col``, ``split_col`` and ``weight_col`` are removed from the list, and names that are
        not columns of ``data`` are silently ignored.
    target_col : str
        Binary target column (1 = bad).
    split_col : str
        Column whose values ``"ins"``, ``"oos"`` and ``"oot"`` mark the splits (case-sensitive). At least one row must be
        ``"ins"``.
    weight_col : str or None, default None
        Sample-weight column of ``data`` (finite, non-negative, positive sum). ``None`` runs the unweighted screen with
        the legacy tools (unless a ``*_use_woe_bins`` flag or ``prefit_woe_engine`` selects the WOE-engine bins).
    psi_enabled : bool, default True
        Whether to run the PSI stage.
    psi_threshold : float, default 0.2
        A feature is kept when its largest PSI over the compared splits is strictly below this value.
    psi_compare_splits : list of str or None, default None
        Splits compared with ``"ins"``: ``"oos"``, ``"oot"`` or both. ``None`` means ``["oos", "oot"]`` (the default of
        ``FeatureScreenConfig`` is ``["oos"]``). A listed split without rows is skipped.
    iv_enabled : bool, default True
        Whether to run the IV stage.
    iv_threshold : float, default 0.02
        Features with an IV below this value are dropped.
    corr_enabled : bool, default True
        Whether to run the correlation stage.
    corr_threshold : float, default 0.75
        Absolute correlation above which two features are redundant; of such a group the feature with the higher IV is
        kept.
    iv_bins : int, default 10
        Maximum number of IV bins (decision-tree leaves on unweighted runs, weighted equal-frequency bins on weighted
        runs). A weighted run that does not use WOE bins also derives its PSI bins from it.
    min_bin_prop : float, default 0.05
        Minimum bin share of the weighted equal-frequency bins (it caps their number at ``int(1 / min_bin_prop)``). It
        is also the default of ``iv_min_bin_prop``.
    iv_min_bin_prop : float or None, default None
        Minimum bin share of the IV binning of unweighted runs. ``None`` uses ``min_bin_prop``.
    corr_max_iterations : int, default 10
        Maximum number of filtering rounds of the correlation stage.
    content : float, default 1e-6
        Floor for bin shares that avoids division by zero and ``log(0)`` in the weighted PSI.
    precision : int, default 5
        Number of decimals to which the weighted equal-frequency bin edges are rounded.
    psi_buckets : int or None, default None
        Number of equal-frequency buckets of the default PSI binning of unweighted runs. ``None`` uses ``iv_bins``.
    plot_path : str or None, default None
        Directory for the IV-stage charts; used only by unweighted runs together with ``plot_outputs``.
    plot_outputs : bool, default False
        Whether to draw the WOE charts of the features that enter the IV stage into ``<plot_path>/overall``
        (unweighted runs only).
    iv_equal_freq : bool, default True
        Must stay True: the IV binning cannot switch equal-frequency bins off (unweighted runs use decision-tree bins and
        weighted runs weighted equal-frequency bins), so any other value raises ``ValueError``.
    psi_use_woe_bins : bool, default False
        Compute PSI on the bins of the screening WOE engine instead of the default binning.
    iv_use_woe_bins : bool, default False
        Compute IV on the bins of the screening WOE engine instead of the default binning.
    corr_use_woe_bins : bool, default False
        Let the screening WOE engine take part in the correlation stage (WOE encoding of non-numeric features, and the
        IV that decides between correlated features on unweighted runs).
    corr_method : {"pearson", "spearman", "kendall"}, default "pearson"
        Correlation coefficient of the correlation stage (Pearson only with non-constant weights).
    corr_base_metric : {"iv", "ks"}, default "iv"
        Metric that decides which of two correlated features is kept.
    woe_engine : str, default "equal_freq"
        Engine fitted when a ``*_use_woe_bins`` flag is set and no ``prefit_woe_engine`` is given: ``"monotone"`` fits a
        ``MonotoneWOEBinner``, any other value a ``WOE_Master``.
    woe_fit_query : str or None, default None
        ``DataFrame.query`` expression that selects the INS rows used to fit the screening engine; a failing query is
        silently ignored.
    woe_params : dict or None, default None
        Keyword arguments of ``WOE_Master.fit`` for the ``WOE_Master`` engine. ``None`` means ``{}``, which gives the
        ``fit`` defaults.
    monotone_woe_params : dict or None, default None
        Arguments of the ``MonotoneWOEBinner`` engine (constructor keys and the fit keys ``chi2_binning``, ``chi2_p``,
        ``chi2_init_size`` and ``n_jobs``). ``None`` means ``{}``, which gives the binner defaults.
    prefit_woe_engine : WOE_Master, MonotoneWOEBinner or WOEEngineAdapter or None, default None
        A fitted WOE engine. On an unweighted run a stage uses it only when its ``*_use_woe_bins`` flag is set; a weighted
        run that has an engine always bins PSI and IV with it.
    corr_nan_policy : {"pairwise", "median_fill", "raise"}, default "pairwise"
        Missing values in the weighted correlation matrix: ``"pairwise"`` uses the rows where both values exist,
        ``"median_fill"`` fills with the weighted median, ``"raise"`` raises ``ValueError``. Weighted runs only.
    corr_block_size : int, default 256
        Number of feature columns per block of the weighted pairwise correlation matrix; it only limits memory and must
        be positive. Weighted runs only.
    on_empty_stage : {"keep_all_warn", "keep_all", "raise"}, default "keep_all_warn"
        What to do when a stage would drop every feature. ``"keep_all_warn"`` never brings back a feature that the IV
        upper limit dropped (a possible leak); from the IV stage on it keeps the ``fallback_top_k`` other features with
        the highest screening IV and warns, and it raises ``EmptyStageError`` when only such leak drops are left (0.9.3).
        ``"keep_all"`` keeps every feature, leak drops included (the behavior up to 0.9.3, only for comparing with old
        results); ``"raise"`` raises ``EmptyStageError`` (a ``ValueError``). See ``FeatureScreenConfig.on_empty_stage``.
    screen_iv_convention : {"guarded", "raw"}, default "guarded"
        How the IV on WOE bins treats class-pure bins: ``"guarded"`` leaves them out of the IV sum, ``"raw"`` counts
        every bin (the numbers of 0.9.1). See ``FeatureScreenConfig.screen_iv_convention``.
    iv_upper_threshold : float or None, default 2.0
        Upper IV limit (gate G02, a leakage guard). The test uses the zero-cell-floored IV and, on WOE bins under the
        guarded convention, the larger of it and ``iv_raw``, never the guarded ``iv``. ``None`` switches it off (the
        behavior up to 0.9.2, where it was not exposed here). See ``FeatureScreenConfig.iv_upper_threshold``.
    iv_gap_ratio_warn : float, default 2.0
        Warn about a feature, and set its ``iv_gap_flag``, when ``iv_raw / iv`` reaches this value (with ``iv > 0``).
        See ``FeatureScreenConfig.iv_gap_ratio_warn``.
    iv_gap_abs_warn : float, default 0.5
        Warn about a feature, and set its ``iv_gap_flag``, when ``iv_raw - iv`` reaches this value. See
        ``FeatureScreenConfig.iv_gap_abs_warn``.
    fallback_top_k : int, default 5
        Number of features that the empty-stage fallback keeps under ``on_empty_stage="keep_all_warn"`` (0.9.3). See
        ``FeatureScreenConfig.fallback_top_k``.

    Returns
    -------
    WeightedScreenResult
        The surviving features with the stage tables and the audit trail.

    Raises
    ------
    KeyError
        If ``split_col`` or ``weight_col`` is not a column of ``data``.
    ValueError
        If no row has ``split_col == "ins"``, if the weights are invalid, if ``iv_equal_freq`` is False, or when a stage
        would drop every feature and ``on_empty_stage="raise"`` or only leak drops are left (``EmptyStageError``).

    Warns
    -----
    UserWarning
        For each feature whose ``iv_raw`` is far above its ``iv`` (see ``iv_gap_ratio_warn`` and ``iv_gap_abs_warn``):
        possible leakage or unresolved class-pure bins. For each feature that the IV upper limit drops: possible target
        leakage (0.9.3). For each empty-stage fallback, naming the features it kept.

    Notes
    -----
    ``missing_rate_threshold``, ``categorical_features`` and the gate settings (VIF, group stability, multi-target,
    truncation) are not exposed here; use ``feature_screen`` with a ``FeatureScreenConfig`` for them. A weighted run
    without WOE bins needs numeric features.
    """
    from .Feature_Screen import FeatureScreenConfig, feature_screen_from_dataframe

    if iv_min_bin_prop is None:
        iv_min_bin_prop = min_bin_prop
    if psi_buckets is None:
        psi_buckets = iv_bins
    if psi_compare_splits is None:
        psi_compare_splits = ["oos", "oot"]

    config = FeatureScreenConfig(
        psi_enabled=psi_enabled,
        psi_threshold=psi_threshold,
        psi_compare_splits=psi_compare_splits,
        psi_buckets=psi_buckets,
        psi_use_woe_bins=psi_use_woe_bins,
        iv_enabled=iv_enabled,
        iv_threshold=iv_threshold,
        iv_bins=iv_bins,
        iv_min_bin_prop=iv_min_bin_prop,
        iv_equal_freq=iv_equal_freq,
        iv_use_woe_bins=iv_use_woe_bins,
        corr_enabled=corr_enabled,
        corr_threshold=corr_threshold,
        corr_max_iterations=corr_max_iterations,
        corr_use_woe_bins=corr_use_woe_bins,
        corr_method=corr_method,
        corr_base_metric=corr_base_metric,
        woe_engine=woe_engine,
        woe_fit_query=woe_fit_query,
        woe_params=dict(woe_params or {}),
        monotone_woe_params=dict(monotone_woe_params or {}),
        plot_path=plot_path,
        plot_outputs=plot_outputs,
        content=content,
        precision=precision,
        min_bin_prop=min_bin_prop,
        corr_nan_policy=corr_nan_policy,
        corr_block_size=corr_block_size,
        on_empty_stage=on_empty_stage,
        screen_iv_convention=screen_iv_convention,
        iv_upper_threshold=iv_upper_threshold,
        iv_gap_ratio_warn=iv_gap_ratio_warn,
        iv_gap_abs_warn=iv_gap_abs_warn,
        fallback_top_k=fallback_top_k,
    )
    return feature_screen_from_dataframe(
        data,
        feature_cols,
        target_col,
        split_col,
        weight_col=weight_col,
        config=config,
        prefit_woe_engine=prefit_woe_engine,
    )

Distribution Analysis: Distribution_Tool

Distribution_Tool

Data processing and analysis toolkit. Provides grouped statistics, distribution analysis, and visualization.

proc_means

Proc Means by Group.

Compute descriptive statistics of numeric variables by grouping variables, including the mean, quantiles, and missing rate.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input raw DataFrame.	required
varlist	list	List of numeric variable names to summarize. Non-numeric variables are accepted too: they get N, UNIQUE, TOP and FREQ instead of the numeric statistics.	required
groupby	list	List of grouping variable names (a single name as a string is accepted; an empty list or None means no grouping).	required
spec_missing_value	any	Special value to be treated as missing. Default is None.	None
feature_block_size	int or None	Number of variables described per block, which limits the memory use on wide tables. Default is 128. None describes all variables in one block. A value that is not a positive integer raises ValueError.	128

Notes

The special value is replaced by NaN in the whole DataFrame (a copy), including the grouping columns: rows whose group value equals spec_missing_value fall out of the groups. The input DataFrame itself is not modified.

Examples:

>>> pm = proc_means(df, ['age', 'score'], ['gender'])
>>> result = pm()

Source code in Modeling_Tool/Feature/Distribution_Tool.py

class proc_means:
    """ Proc Means by Group.

    Compute descriptive statistics of numeric variables by grouping variables, including the mean, quantiles, and missing rate.

    Parameters
    ----------
    data : pd.DataFrame
        Input raw DataFrame.
    varlist : list
        List of numeric variable names to summarize. Non-numeric variables are accepted too: they get ``N``,
        ``UNIQUE``, ``TOP`` and ``FREQ`` instead of the numeric statistics.
    groupby : list
        List of grouping variable names (a single name as a string is accepted; an empty list or None means no
        grouping).
    spec_missing_value : any, optional
        Special value to be treated as missing. Default is None.
    feature_block_size : int or None, optional
        Number of variables described per block, which limits the memory use on wide tables. Default is 128. None
        describes all variables in one block. A value that is not a positive integer raises ``ValueError``.

    Notes
    -----
    The special value is replaced by NaN in the whole DataFrame (a copy), including the grouping columns: rows whose
    group value equals ``spec_missing_value`` fall out of the groups. The input DataFrame itself is not modified.

    Examples
    --------
    >>> pm = proc_means(df, ['age', 'score'], ['gender'])
    >>> result = pm()
    """

    def __init__(
        self,
        data,
        varlist,
        groupby,
        spec_missing_value=None,
        feature_block_size=128,
    ):
        """Initialize the proc_means object.

        Parameters
        ----------
        data : pd.DataFrame
            Input raw DataFrame.
        varlist : list
            List of numeric variable names to summarize.
        groupby : list
            List of grouping variable names (a single name as a string is accepted; an empty list or None means no
            grouping).
        spec_missing_value : any, optional
            Special value to be treated as missing.
        feature_block_size : int or None, optional
            Number of variables described per block, which limits the memory use on wide tables. Default is 128. None
            describes all variables in one block.

        Raises
        ------
        ValueError
            If ``feature_block_size`` is not None and is not a positive integer.
        """
        self.data = data
        self.varlist = varlist
        self.groupby = [groupby] if isinstance(groupby, str) else list(groupby or [])
        self.spec_missing_value = spec_missing_value
        if feature_block_size is not None and int(feature_block_size) <= 0:
            raise ValueError("feature_block_size must be a positive integer or None")
        self.feature_block_size = feature_block_size

    def treat_spec_missing(self):
        """Handle the special missing value.

        Replace the value given by self.spec_missing_value with np.nan so that the statistics are computed correctly.

        Returns
        -------
        pd.DataFrame
            DataFrame after the missing values have been handled.
        """
        if self.spec_missing_value is not None:
            self.data = self.data.replace(self.spec_missing_value, np.nan)
        return self.data

    def group_means(self, q=None):
        """Compute descriptive statistics by group.

        Compute descriptive statistics of the given variables by group, including the count, mean,
        standard deviation, minimum, maximum, and custom quantiles.

        Parameters
        ----------
        q : list, optional
            List of quantiles. Default is [0.05, 0.15, 0.25, 0.5, 0.75, 0.95, 0.99].

        Returns
        -------
        pd.DataFrame
            DataFrame containing the descriptive statistics.
        """
        if q is None:
            q = [0.05, 0.15, 0.25, 0.5, 0.75, 0.95, 0.99]

        block_size = self.feature_block_size or max(len(self.varlist), 1)
        frames = []
        numeric_vars = [
            var for var in self.varlist if pd.api.types.is_numeric_dtype(self.data[var])
        ]
        categorical_vars = [var for var in self.varlist if var not in numeric_vars]

        for start in range(0, len(numeric_vars), block_size):
            block = numeric_vars[start : start + block_size]
            if self.groupby:
                described = self.data.groupby(self.groupby, sort=True)[block].describe(
                    percentiles=q
                )
                try:
                    part = described.stack(level=0, future_stack=True)
                except TypeError:  # pandas < 2.1
                    part = described.stack(level=0)
                part.index = part.index.set_names(self.groupby + ["attribute"])
            else:
                part = self.data[block].describe(percentiles=q).T
                part.index.name = "attribute"
            frames.append(part)

        for start in range(0, len(categorical_vars), block_size):
            block = categorical_vars[start : start + block_size]
            long_block = self.data[self.groupby + block].melt(
                id_vars=self.groupby,
                value_vars=block,
                var_name="attribute",
                value_name="value",
            )
            described = long_block.groupby(
                self.groupby + ["attribute"],
                dropna=False,
                sort=True,
            ).describe(percentiles=q)
            frames.append(described.droplevel(level=0, axis=1))
        if not frames:
            return pd.DataFrame()
        return pd.concat(frames, axis=0).sort_index()

    def group_sum(self):
        """Compute the number of samples in each group.

        Count the observations (total number of samples) in each combination of groups.

        Returns
        -------
        pd.DataFrame
            Aggregated result containing the number of samples in each group.
        """
        if not self.groupby:
            index = pd.Index(sorted(self.varlist), name="attribute")
            return pd.DataFrame({"sum_all": len(self.data)}, index=index)

        group_sizes = self.data.groupby(self.groupby, sort=True).size()
        index = pd.MultiIndex.from_tuples(
            [
                (*((group,) if len(self.groupby) == 1 else tuple(group)), attribute)
                for group in group_sizes.index
                for attribute in sorted(self.varlist)
            ],
            names=self.groupby + ["attribute"],
        )
        values = np.repeat(group_sizes.to_numpy(), len(self.varlist))
        return pd.DataFrame({"sum_all": values}, index=index)

    def __call__(self, q=None):
        """Run the complete grouped statistical analysis.

        Compute the grouped statistics, including the sample count, N_ALL, mean, standard deviation, quantiles, and missing rate.

        Parameters
        ----------
        q : list, optional
            List of quantiles. Default is [0.05, 0.15, 0.25, 0.5, 0.75, 0.95, 0.99].

        Returns
        -------
        pd.DataFrame
            Complete grouped statistics report, containing N, N_ALL, the quantiles, and the missing rate.
        """
        if q is None:
            q = [0.05, 0.15, 0.25, 0.5, 0.75, 0.95, 0.99]

        self.data = self.treat_spec_missing()
        sum_total = self.group_sum()
        means = self.group_means(q=q)
        res_fnl = sum_total.merge(means, left_index=True, right_index=True)
        res_fnl["missing_rate"] = 1 - res_fnl["count"] / res_fnl["sum_all"]
        from ..Core.utils import quantile_rename_map

        res_fnl = res_fnl.rename(columns=quantile_rename_map(res_fnl.columns))
        res_fnl = res_fnl.rename(columns={"count": "N", "sum_all": "N_ALL"})
        res_fnl.columns = [x.upper() for x in res_fnl.columns]
        return res_fnl

treat_spec_missing

treat_spec_missing()

Handle the special missing value.

Replace the value given by self.spec_missing_value with np.nan so that the statistics are computed correctly.

Returns:

Type	Description
DataFrame	DataFrame after the missing values have been handled.

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def treat_spec_missing(self):
    """Handle the special missing value.

    Replace the value given by self.spec_missing_value with np.nan so that the statistics are computed correctly.

    Returns
    -------
    pd.DataFrame
        DataFrame after the missing values have been handled.
    """
    if self.spec_missing_value is not None:
        self.data = self.data.replace(self.spec_missing_value, np.nan)
    return self.data

group_means

group_means(q=None)

Compute descriptive statistics by group.

Compute descriptive statistics of the given variables by group, including the count, mean, standard deviation, minimum, maximum, and custom quantiles.

Parameters:

Name	Type	Description	Default
q	list	List of quantiles. Default is [0.05, 0.15, 0.25, 0.5, 0.75, 0.95, 0.99].	None

Returns:

Type	Description
DataFrame	DataFrame containing the descriptive statistics.

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def group_means(self, q=None):
    """Compute descriptive statistics by group.

    Compute descriptive statistics of the given variables by group, including the count, mean,
    standard deviation, minimum, maximum, and custom quantiles.

    Parameters
    ----------
    q : list, optional
        List of quantiles. Default is [0.05, 0.15, 0.25, 0.5, 0.75, 0.95, 0.99].

    Returns
    -------
    pd.DataFrame
        DataFrame containing the descriptive statistics.
    """
    if q is None:
        q = [0.05, 0.15, 0.25, 0.5, 0.75, 0.95, 0.99]

    block_size = self.feature_block_size or max(len(self.varlist), 1)
    frames = []
    numeric_vars = [
        var for var in self.varlist if pd.api.types.is_numeric_dtype(self.data[var])
    ]
    categorical_vars = [var for var in self.varlist if var not in numeric_vars]

    for start in range(0, len(numeric_vars), block_size):
        block = numeric_vars[start : start + block_size]
        if self.groupby:
            described = self.data.groupby(self.groupby, sort=True)[block].describe(
                percentiles=q
            )
            try:
                part = described.stack(level=0, future_stack=True)
            except TypeError:  # pandas < 2.1
                part = described.stack(level=0)
            part.index = part.index.set_names(self.groupby + ["attribute"])
        else:
            part = self.data[block].describe(percentiles=q).T
            part.index.name = "attribute"
        frames.append(part)

    for start in range(0, len(categorical_vars), block_size):
        block = categorical_vars[start : start + block_size]
        long_block = self.data[self.groupby + block].melt(
            id_vars=self.groupby,
            value_vars=block,
            var_name="attribute",
            value_name="value",
        )
        described = long_block.groupby(
            self.groupby + ["attribute"],
            dropna=False,
            sort=True,
        ).describe(percentiles=q)
        frames.append(described.droplevel(level=0, axis=1))
    if not frames:
        return pd.DataFrame()
    return pd.concat(frames, axis=0).sort_index()

group_sum

group_sum()

Compute the number of samples in each group.

Count the observations (total number of samples) in each combination of groups.

Returns:

Type	Description
DataFrame	Aggregated result containing the number of samples in each group.

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def group_sum(self):
    """Compute the number of samples in each group.

    Count the observations (total number of samples) in each combination of groups.

    Returns
    -------
    pd.DataFrame
        Aggregated result containing the number of samples in each group.
    """
    if not self.groupby:
        index = pd.Index(sorted(self.varlist), name="attribute")
        return pd.DataFrame({"sum_all": len(self.data)}, index=index)

    group_sizes = self.data.groupby(self.groupby, sort=True).size()
    index = pd.MultiIndex.from_tuples(
        [
            (*((group,) if len(self.groupby) == 1 else tuple(group)), attribute)
            for group in group_sizes.index
            for attribute in sorted(self.varlist)
        ],
        names=self.groupby + ["attribute"],
    )
    values = np.repeat(group_sizes.to_numpy(), len(self.varlist))
    return pd.DataFrame({"sum_all": values}, index=index)

DistributionShiftAnalyzer

Distribution shift analyzer.

Analyze how the distribution of variables shifts between groups, assessing the difference by comparing the proportion of observations in each group that exceed the outlier threshold of the benchmark group.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input raw DataFrame.	required
grp_name	str	Name of the grouping variable.	required
benchmark_value	any	Group value of the benchmark group, used to determine the outlier threshold.	required

Examples:

>>> analyzer = DistributionShiftAnalyzer(df, 'gender', 'Male')
>>> result = analyzer.analyze(['age', 'score'])

Source code in Modeling_Tool/Feature/Distribution_Tool.py

class DistributionShiftAnalyzer:
    """Distribution shift analyzer.

    Analyze how the distribution of variables shifts between groups, assessing the difference
    by comparing the proportion of observations in each group that exceed the outlier
    threshold of the benchmark group.

    Parameters
    ----------
    data : pd.DataFrame
        Input raw DataFrame.
    grp_name : str
        Name of the grouping variable.
    benchmark_value : any
        Group value of the benchmark group, used to determine the outlier threshold.

    Examples
    --------
    >>> analyzer = DistributionShiftAnalyzer(df, 'gender', 'Male')
    >>> result = analyzer.analyze(['age', 'score'])
    """

    def __init__(self, data, grp_name, benchmark_value):
        """Initialize the distribution shift analyzer.

        Parameters
        ----------
        data : pd.DataFrame
            Input raw DataFrame.
        grp_name : str
            Name of the grouping variable.
        benchmark_value : any
            Group value of the benchmark group.
        """
        self.data = data
        self.grp_name = grp_name
        self.benchmark_value = benchmark_value

    def analyze_single_var(self, var, outlier_value=0.99):
        """Analyze the distribution shift of a single variable.

        Compute, for each group, the proportion of observations that exceed the given quantile threshold of the benchmark group.

        Parameters
        ----------
        var : str
            Name of the variable to analyze.
        outlier_value : float, optional
            Quantile used to determine the outlier threshold. Default is 0.99.

        Returns
        -------
        dict
            Keys are the group values; values are the proportions of observations exceeding the threshold.
        """
        means_rpt = proc_means_by_grp(
            self.data, [var], [self.grp_name],
            spec_missing_value=None, q=[outlier_value]
        )

        outlier_name = "Q" + format(round(float(outlier_value) * 100, 10), "g")
        outlier_threshold = means_rpt[
            means_rpt[self.grp_name] == self.benchmark_value
        ][outlier_name].iloc[0]

        res_dict = {}
        for group, group_data in self.data.groupby(self.grp_name):
            cnt = group_data[group_data[var] > outlier_threshold].shape[0]
            prop = round(cnt / group_data.shape[0], 4)
            res_dict[group] = prop
        return res_dict

    def analyze(self, varlist, outlier_value=0.99):
        """Analyze the distribution shift of multiple variables.

        For each variable in the list, compute the proportion of observations in each group
        that exceed the threshold of the benchmark group, and return all results as a DataFrame.

        Parameters
        ----------
        varlist : list
            List of variable names to analyze.
        outlier_value : float, optional
            Quantile used to determine the outlier threshold. Default is 0.99.

        Returns
        -------
        pd.DataFrame
            Rows are indexed by variable name and columns are the group values; each cell is the proportion of observations exceeding the threshold.

        Examples
        --------
        >>> analyzer = DistributionShiftAnalyzer(df, 'gender', 'Male')
        >>> result = analyzer.analyze(['age', 'score'])
        """
        res_dict = {}
        for var in varlist:
            res = self.analyze_single_var(var=var, outlier_value=outlier_value)
            res_dict[var] = res
        return pd.DataFrame(res_dict).T

analyze_single_var

analyze_single_var(var, outlier_value=0.99)

Analyze the distribution shift of a single variable.

Compute, for each group, the proportion of observations that exceed the given quantile threshold of the benchmark group.

Parameters:

Name	Type	Description	Default
var	str	Name of the variable to analyze.	required
outlier_value	float	Quantile used to determine the outlier threshold. Default is 0.99.	0.99

Returns:

Type	Description
dict	Keys are the group values; values are the proportions of observations exceeding the threshold.

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def analyze_single_var(self, var, outlier_value=0.99):
    """Analyze the distribution shift of a single variable.

    Compute, for each group, the proportion of observations that exceed the given quantile threshold of the benchmark group.

    Parameters
    ----------
    var : str
        Name of the variable to analyze.
    outlier_value : float, optional
        Quantile used to determine the outlier threshold. Default is 0.99.

    Returns
    -------
    dict
        Keys are the group values; values are the proportions of observations exceeding the threshold.
    """
    means_rpt = proc_means_by_grp(
        self.data, [var], [self.grp_name],
        spec_missing_value=None, q=[outlier_value]
    )

    outlier_name = "Q" + format(round(float(outlier_value) * 100, 10), "g")
    outlier_threshold = means_rpt[
        means_rpt[self.grp_name] == self.benchmark_value
    ][outlier_name].iloc[0]

    res_dict = {}
    for group, group_data in self.data.groupby(self.grp_name):
        cnt = group_data[group_data[var] > outlier_threshold].shape[0]
        prop = round(cnt / group_data.shape[0], 4)
        res_dict[group] = prop
    return res_dict

analyze

analyze(varlist, outlier_value=0.99)

Analyze the distribution shift of multiple variables.

For each variable in the list, compute the proportion of observations in each group that exceed the threshold of the benchmark group, and return all results as a DataFrame.

Parameters:

Name	Type	Description	Default
varlist	list	List of variable names to analyze.	required
outlier_value	float	Quantile used to determine the outlier threshold. Default is 0.99.	0.99

Returns:

Type	Description
DataFrame	Rows are indexed by variable name and columns are the group values; each cell is the proportion of observations exceeding the threshold.

Examples:

>>> analyzer = DistributionShiftAnalyzer(df, 'gender', 'Male')
>>> result = analyzer.analyze(['age', 'score'])

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def analyze(self, varlist, outlier_value=0.99):
    """Analyze the distribution shift of multiple variables.

    For each variable in the list, compute the proportion of observations in each group
    that exceed the threshold of the benchmark group, and return all results as a DataFrame.

    Parameters
    ----------
    varlist : list
        List of variable names to analyze.
    outlier_value : float, optional
        Quantile used to determine the outlier threshold. Default is 0.99.

    Returns
    -------
    pd.DataFrame
        Rows are indexed by variable name and columns are the group values; each cell is the proportion of observations exceeding the threshold.

    Examples
    --------
    >>> analyzer = DistributionShiftAnalyzer(df, 'gender', 'Male')
    >>> result = analyzer.analyze(['age', 'score'])
    """
    res_dict = {}
    for var in varlist:
        res = self.analyze_single_var(var=var, outlier_value=outlier_value)
        res_dict[var] = res
    return pd.DataFrame(res_dict).T

DistributionPlotter

Distribution plotter.

Visualize the distribution of a numeric variable in several ways; supports kernel density plots, histograms, and rug plots.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input DataFrame.	required
score	str	Name of the variable whose distribution is plotted.	required

Examples:

>>> plotter = DistributionPlotter(df, 'age')
>>> plotter.plot(method='kdeplot', title='Age Distribution')

Source code in Modeling_Tool/Feature/Distribution_Tool.py

class DistributionPlotter:
    """Distribution plotter.

    Visualize the distribution of a numeric variable in several ways; supports kernel density plots, histograms, and rug plots.

    Parameters
    ----------
    data : pd.DataFrame
        Input DataFrame.
    score : str
        Name of the variable whose distribution is plotted.

    Examples
    --------
    >>> plotter = DistributionPlotter(df, 'age')
    >>> plotter.plot(method='kdeplot', title='Age Distribution')
    """

    def __init__(self, data, score):
        """Initialize the distribution plotter.

        Parameters
        ----------
        data : pd.DataFrame
            Input DataFrame.
        score : str
            Name of the variable whose distribution is plotted.
        """
        self.data = data
        self.score = score
        self.plot_series = data[score]

    def plot_rugplot(self, figsize=(15, 15), title="Distribution Plot"):
        """Plot a rug plot.

        Overlay a rug plot on a kernel density estimate plot to show the density of the data distribution.

        Parameters
        ----------
        figsize : tuple, optional
            Figure size. Default is (15, 15).
        title : str, optional
            Figure title. Default is "Distribution Plot".
        """
        plt.figure(figsize=figsize)
        sns.kdeplot(self.plot_series, color='purple')
        sns.rugplot(self.plot_series, color='purple')
        plt.title(title)
        plt.xlabel(self.score)
        plt.ylabel('Density')

    def plot_kdeplot(self, figsize=(15, 15), title="Distribution Plot"):
        """Plot a kernel density estimate.

        Show the data distribution with a filled kernel density estimate plot.

        Parameters
        ----------
        figsize : tuple, optional
            Figure size. Default is (15, 15).
        title : str, optional
            Figure title. Default is "Distribution Plot".
        """
        plt.figure(figsize=figsize)
        sns.kdeplot(self.plot_series, fill=True, color='orange')
        plt.title(title)
        plt.xlabel(self.score)
        plt.ylabel('Density')

    def plot_displot(self, figsize=(15, 15), title="Distribution Plot", nbins=10):
        """Plot a distribution histogram.

        Plot a histogram with a kernel density estimate to show the data distribution.

        Parameters
        ----------
        figsize : tuple, optional
            Figure size. Default is (15, 15).
        title : str, optional
            Figure title. Default is "Distribution Plot".
        nbins : int, optional
            Number of histogram bins. Default is 10.
        """
        plt.figure(figsize=figsize)
        sns.displot(self.plot_series, kde=True, bins=nbins)
        plt.title(title)
        plt.xlabel(self.score)
        plt.ylabel('Density')

    def plot(self, method='displot', title="Distribution Plot", figsize=(15, 15), nbins=10):
        """Plot the distribution chart.

        Plot the distribution of the variable with the specified method.

        Parameters
        ----------
        method : str, optional
            Plotting method; one of 'rugplot', 'kdeplot' or 'displot'. Default is 'displot'.
        title : str, optional
            Figure title. Default is "Distribution Plot".
        figsize : tuple, optional
            Figure size. Default is (15, 15).
        nbins : int, optional
            Number of histogram bins (only used by the displot method). Default is 10.

        Raises
        ------
        ValueError
            Raised when an unsupported plotting method is specified.

        Examples
        --------
        >>> plotter = DistributionPlotter(df, 'age')
        >>> plotter.plot(method='kdeplot', title='Age Distribution')
        """
        if method == 'rugplot':
            self.plot_rugplot(figsize=figsize, title=title)
        elif method == 'kdeplot':
            self.plot_kdeplot(figsize=figsize, title=title)
        elif method == 'displot':
            self.plot_displot(figsize=figsize, title=title, nbins=nbins)
        else:
            raise ValueError(f"Unsupported method: {method}. Choose from 'rugplot', 'kdeplot', 'displot'.")

plot_rugplot

plot_rugplot(figsize=(15, 15), title='Distribution Plot')

Plot a rug plot.

Overlay a rug plot on a kernel density estimate plot to show the density of the data distribution.

Parameters:

Name	Type	Description	Default
figsize	tuple	Figure size. Default is (15, 15).	(15, 15)
title	str	Figure title. Default is "Distribution Plot".	'Distribution Plot'

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def plot_rugplot(self, figsize=(15, 15), title="Distribution Plot"):
    """Plot a rug plot.

    Overlay a rug plot on a kernel density estimate plot to show the density of the data distribution.

    Parameters
    ----------
    figsize : tuple, optional
        Figure size. Default is (15, 15).
    title : str, optional
        Figure title. Default is "Distribution Plot".
    """
    plt.figure(figsize=figsize)
    sns.kdeplot(self.plot_series, color='purple')
    sns.rugplot(self.plot_series, color='purple')
    plt.title(title)
    plt.xlabel(self.score)
    plt.ylabel('Density')

plot_kdeplot

plot_kdeplot(figsize=(15, 15), title='Distribution Plot')

Plot a kernel density estimate.

Show the data distribution with a filled kernel density estimate plot.

Parameters:

Name	Type	Description	Default
figsize	tuple	Figure size. Default is (15, 15).	(15, 15)
title	str	Figure title. Default is "Distribution Plot".	'Distribution Plot'

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def plot_kdeplot(self, figsize=(15, 15), title="Distribution Plot"):
    """Plot a kernel density estimate.

    Show the data distribution with a filled kernel density estimate plot.

    Parameters
    ----------
    figsize : tuple, optional
        Figure size. Default is (15, 15).
    title : str, optional
        Figure title. Default is "Distribution Plot".
    """
    plt.figure(figsize=figsize)
    sns.kdeplot(self.plot_series, fill=True, color='orange')
    plt.title(title)
    plt.xlabel(self.score)
    plt.ylabel('Density')

plot_displot

plot_displot(figsize=(15, 15), title='Distribution Plot', nbins=10)

Plot a distribution histogram.

Plot a histogram with a kernel density estimate to show the data distribution.

Parameters:

Name	Type	Description	Default
figsize	tuple	Figure size. Default is (15, 15).	(15, 15)
title	str	Figure title. Default is "Distribution Plot".	'Distribution Plot'
nbins	int	Number of histogram bins. Default is 10.	10

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def plot_displot(self, figsize=(15, 15), title="Distribution Plot", nbins=10):
    """Plot a distribution histogram.

    Plot a histogram with a kernel density estimate to show the data distribution.

    Parameters
    ----------
    figsize : tuple, optional
        Figure size. Default is (15, 15).
    title : str, optional
        Figure title. Default is "Distribution Plot".
    nbins : int, optional
        Number of histogram bins. Default is 10.
    """
    plt.figure(figsize=figsize)
    sns.displot(self.plot_series, kde=True, bins=nbins)
    plt.title(title)
    plt.xlabel(self.score)
    plt.ylabel('Density')

plot

plot(method='displot', title='Distribution Plot', figsize=(15, 15), nbins=10)

Plot the distribution chart.

Plot the distribution of the variable with the specified method.

Parameters:

Name	Type	Description	Default
method	str	Plotting method; one of 'rugplot', 'kdeplot' or 'displot'. Default is 'displot'.	'displot'
title	str	Figure title. Default is "Distribution Plot".	'Distribution Plot'
figsize	tuple	Figure size. Default is (15, 15).	(15, 15)
nbins	int	Number of histogram bins (only used by the displot method). Default is 10.	10

Raises:

Type	Description
ValueError	Raised when an unsupported plotting method is specified.

Examples:

>>> plotter = DistributionPlotter(df, 'age')
>>> plotter.plot(method='kdeplot', title='Age Distribution')

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def plot(self, method='displot', title="Distribution Plot", figsize=(15, 15), nbins=10):
    """Plot the distribution chart.

    Plot the distribution of the variable with the specified method.

    Parameters
    ----------
    method : str, optional
        Plotting method; one of 'rugplot', 'kdeplot' or 'displot'. Default is 'displot'.
    title : str, optional
        Figure title. Default is "Distribution Plot".
    figsize : tuple, optional
        Figure size. Default is (15, 15).
    nbins : int, optional
        Number of histogram bins (only used by the displot method). Default is 10.

    Raises
    ------
    ValueError
        Raised when an unsupported plotting method is specified.

    Examples
    --------
    >>> plotter = DistributionPlotter(df, 'age')
    >>> plotter.plot(method='kdeplot', title='Age Distribution')
    """
    if method == 'rugplot':
        self.plot_rugplot(figsize=figsize, title=title)
    elif method == 'kdeplot':
        self.plot_kdeplot(figsize=figsize, title=title)
    elif method == 'displot':
        self.plot_displot(figsize=figsize, title=title, nbins=nbins)
    else:
        raise ValueError(f"Unsupported method: {method}. Choose from 'rugplot', 'kdeplot', 'displot'.")

proc_means_by_grp

proc_means_by_grp(data, varlist, groupby=None, spec_missing_value=None, q=None, feature_block_size=128)

Compute the variable statistics report by group.

Compute descriptive statistics of the given variables by group and return a report containing the sample count, mean, quantiles, and missing rate. The computation is delegated to the proc_means class.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input raw DataFrame.	required
varlist	list	List of numeric variable names to summarize.	required
groupby	list	List of grouping variable names (a single name as a string is accepted). Default is None, which means no grouping (the same as an empty list).	None
spec_missing_value	any	Special value to be treated as missing. Default is None. It is replaced by NaN in the whole DataFrame, including the grouping columns, so rows whose group value equals it fall out of the groups.	None
q	list	List of quantiles. Default is [0.05, 0.15, 0.25, 0.5, 0.75, 0.95, 0.99].	None
feature_block_size	int or None	Number of variables described per block, which limits the memory use on wide tables. Default is 128. None describes all variables in one block. A value that is not a positive integer raises ValueError.	128

Returns:

Type	Description
DataFrame	Grouped statistics report containing the descriptive statistics of each variable: one row per group value and variable, with the grouping columns, attribute (the variable name), N_ALL, N, MEAN, STD, MIN, one Q<percent> column per quantile (Q5, Q15, ...), MAX and MISSING_RATE (non-numeric variables get UNIQUE, TOP and FREQ instead of the numeric statistics).

Examples:

>>> result = proc_means_by_grp(df, ['age', 'score'], ['gender'])

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def proc_means_by_grp(
    data,
    varlist,
    groupby=None,
    spec_missing_value=None,
    q=None,
    feature_block_size=128,
):
    """Compute the variable statistics report by group.

    Compute descriptive statistics of the given variables by group and return a report containing
    the sample count, mean, quantiles, and missing rate.
    The computation is delegated to the proc_means class.

    Parameters
    ----------
    data : pd.DataFrame
        Input raw DataFrame.
    varlist : list
        List of numeric variable names to summarize.
    groupby : list, optional
        List of grouping variable names (a single name as a string is accepted). Default is None, which means no
        grouping (the same as an empty list).
    spec_missing_value : any, optional
        Special value to be treated as missing. Default is None. It is replaced by NaN in the whole DataFrame,
        including the grouping columns, so rows whose group value equals it fall out of the groups.
    q : list, optional
        List of quantiles. Default is [0.05, 0.15, 0.25, 0.5, 0.75, 0.95, 0.99].
    feature_block_size : int or None, optional
        Number of variables described per block, which limits the memory use on wide tables. Default is 128. None
        describes all variables in one block. A value that is not a positive integer raises ``ValueError``.

    Returns
    -------
    pd.DataFrame
        Grouped statistics report containing the descriptive statistics of each variable: one row per group value
        and variable, with the grouping columns, ``attribute`` (the variable name), ``N_ALL``, ``N``, ``MEAN``, ``STD``,
        ``MIN``, one ``Q<percent>`` column per quantile (``Q5``, ``Q15``, ...), ``MAX`` and ``MISSING_RATE``
        (non-numeric variables get ``UNIQUE``, ``TOP`` and ``FREQ`` instead of the numeric statistics).

    Examples
    --------
    >>> result = proc_means_by_grp(df, ['age', 'score'], ['gender'])
    """
    if groupby is None:
        groupby = []
    if q is None:
        q = [0.05, 0.15, 0.25, 0.5, 0.75, 0.95, 0.99]

    means = proc_means(
        data,
        varlist,
        groupby=groupby,
        spec_missing_value=spec_missing_value,
        feature_block_size=feature_block_size,
    )
    means_rpt = means(q=q)
    means_rpt = means_rpt.reset_index(drop=False)

    return means_rpt

proc_means_for_screening

proc_means_for_screening(data, varlist, spec_missing_value=None, q=None, feature_block_size=128)

Build a uniform means table for IV screening reports.

proc_means_by_grp returns object stats (unique/top/freq) when numeric and categorical columns are melted together. This helper splits by dtype so MIN, MEAN, and MAX are always present (NaN for non-numeric features).

Parameters:

Name	Type	Description	Default
data	DataFrame	Input raw DataFrame.	required
varlist	list	List of variable names to summarize. A name that is not a column of data is dropped silently. An empty list returns an empty table.	required
spec_missing_value	any	Special value to be treated as missing. Default is None.	None
q	list	List of quantiles passed to proc_means_by_grp. Default is None, which uses its default quantiles.	None
feature_block_size	int or None	Number of variables described per block. Default is 128. None describes all variables in one block.	128

Returns:

Type	Description
DataFrame	One row per variable with the columns attribute, N_ALL, N, MISSING_RATE, MIN, MEAN and MAX (NaN in MIN, MEAN and MAX for non-numeric variables). The table has no rows when no variable is found.

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def proc_means_for_screening(
    data,
    varlist,
    spec_missing_value=None,
    q=None,
    feature_block_size=128,
):
    """Build a uniform means table for IV screening reports.

    ``proc_means_by_grp`` returns object stats (unique/top/freq) when numeric and
    categorical columns are melted together.  This helper splits by dtype so MIN,
    MEAN, and MAX are always present (NaN for non-numeric features).

    Parameters
    ----------
    data : pd.DataFrame
        Input raw DataFrame.
    varlist : list
        List of variable names to summarize. A name that is not a column of ``data`` is dropped silently. An empty
        list returns an empty table.
    spec_missing_value : any, optional
        Special value to be treated as missing. Default is None.
    q : list, optional
        List of quantiles passed to ``proc_means_by_grp``. Default is None, which uses its default quantiles.
    feature_block_size : int or None, optional
        Number of variables described per block. Default is 128. None describes all variables in one block.

    Returns
    -------
    pd.DataFrame
        One row per variable with the columns ``attribute``, ``N_ALL``, ``N``, ``MISSING_RATE``, ``MIN``, ``MEAN``
        and ``MAX`` (NaN in ``MIN``, ``MEAN`` and ``MAX`` for non-numeric variables). The table has no rows when
        no variable is found.
    """
    if not varlist:
        return pd.DataFrame(columns=_SCREENING_MEANS_COLS)

    numeric = [v for v in varlist if v in data.columns and pd.api.types.is_numeric_dtype(data[v])]
    other = [v for v in varlist if v not in numeric and v in data.columns]
    frames = []

    if numeric:
        frames.append(
            proc_means_by_grp(
                data,
                numeric,
                spec_missing_value=spec_missing_value,
                q=q,
                feature_block_size=feature_block_size,
            )
        )

    for var in other:
        part = proc_means_by_grp(
            data,
            [var],
            spec_missing_value=spec_missing_value,
            q=q,
            feature_block_size=feature_block_size,
        )
        for col in ("MIN", "MEAN", "MAX"):
            if col not in part.columns:
                part[col] = np.nan
        frames.append(part)

    if not frames:
        return pd.DataFrame(columns=_SCREENING_MEANS_COLS)

    out = pd.concat(frames, ignore_index=True)
    return out[_SCREENING_MEANS_COLS]

get_distribution_shift_single_var

get_distribution_shift_single_var(data, var, grp_name, benchmark_value, outlier_value=0.99)

Compute the distribution shift of a single variable.

Analyze, for the given variable, the proportion of observations in each group that exceed the outlier threshold of the benchmark group.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input raw DataFrame.	required
var	str	Name of the variable to analyze.	required
grp_name	str	Name of the grouping variable.	required
benchmark_value	any	Group value of the benchmark group.	required
outlier_value	float	Quantile used to determine the outlier threshold. Default is 0.99.	0.99

Returns:

Type	Description
dict	Keys are the group values; values are the proportions of observations exceeding the threshold.

Examples:

>>> result = get_distribution_shift_single_var(df, 'age', 'gender', 'Male')

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def get_distribution_shift_single_var(data, var, grp_name, benchmark_value, outlier_value=0.99):
    """Compute the distribution shift of a single variable.

    Analyze, for the given variable, the proportion of observations in each group that exceed the outlier threshold of the benchmark group.

    Parameters
    ----------
    data : pd.DataFrame
        Input raw DataFrame.
    var : str
        Name of the variable to analyze.
    grp_name : str
        Name of the grouping variable.
    benchmark_value : any
        Group value of the benchmark group.
    outlier_value : float, optional
        Quantile used to determine the outlier threshold. Default is 0.99.

    Returns
    -------
    dict
        Keys are the group values; values are the proportions of observations exceeding the threshold.

    Examples
    --------
    >>> result = get_distribution_shift_single_var(df, 'age', 'gender', 'Male')
    """
    analyzer = DistributionShiftAnalyzer(data, grp_name, benchmark_value)
    return analyzer.analyze_single_var(var, outlier_value)

get_distribution_shift

get_distribution_shift(data, varlist, grp_name, benchmark_value, outlier_value=0.99)

Compute the distribution shift of multiple variables.

For each variable in the list, analyze the proportion of observations in each group that exceed the outlier threshold of the benchmark group, and return the transposed DataFrame containing all results.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input raw DataFrame.	required
varlist	list	List of variable names to analyze.	required
grp_name	str	Name of the grouping variable.	required
benchmark_value	any	Group value of the benchmark group.	required
outlier_value	float	Quantile used to determine the outlier threshold. Default is 0.99.	0.99

Returns:

Type	Description
DataFrame	Rows are indexed by variable name and columns are the group values; each cell is the proportion of observations exceeding the threshold.

Examples:

>>> result = get_distribution_shift(df, ['age', 'score'], 'gender', 'Male')

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def get_distribution_shift(data, varlist, grp_name, benchmark_value, outlier_value=0.99):
    """Compute the distribution shift of multiple variables.

    For each variable in the list, analyze the proportion of observations in each group that exceed
    the outlier threshold of the benchmark group, and return the transposed DataFrame containing all results.

    Parameters
    ----------
    data : pd.DataFrame
        Input raw DataFrame.
    varlist : list
        List of variable names to analyze.
    grp_name : str
        Name of the grouping variable.
    benchmark_value : any
        Group value of the benchmark group.
    outlier_value : float, optional
        Quantile used to determine the outlier threshold. Default is 0.99.

    Returns
    -------
    pd.DataFrame
        Rows are indexed by variable name and columns are the group values; each cell is the proportion of observations exceeding the threshold.

    Examples
    --------
    >>> result = get_distribution_shift(df, ['age', 'score'], 'gender', 'Male')
    """
    analyzer = DistributionShiftAnalyzer(data, grp_name, benchmark_value)
    return analyzer.analyze(varlist, outlier_value)

plot_distribution

plot_distribution(data, score, method='displot', title='Distribution Plot', figsize=(15, 15), nbins=10)

Plot the distribution chart of a variable.

Plot the distribution of the variable with the specified method; supports kernel density estimate, histogram, and rug plots.

Parameters:

Name	Type	Description	Default
data	DataFrame	Input DataFrame.	required
score	str	Name of the variable whose distribution is plotted.	required
method	str	Plotting method; one of 'rugplot', 'kdeplot' or 'displot'. Default is 'displot'.	'displot'
title	str	Figure title. Default is "Distribution Plot".	'Distribution Plot'
figsize	tuple	Figure size. Default is (15, 15).	(15, 15)
nbins	int	Number of histogram bins (only used by the displot method). Default is 10.	10

Returns:

Type	Description
None	The plot is displayed directly.

Examples:

>>> plot_distribution(df, 'age', method='kdeplot')
>>> plot_distribution(df, 'score', method='displot', nbins=20)

Source code in Modeling_Tool/Feature/Distribution_Tool.py

def plot_distribution(data, score, method='displot', title="Distribution Plot", figsize=(15, 15), nbins=10):
    """Plot the distribution chart of a variable.

    Plot the distribution of the variable with the specified method; supports kernel density estimate, histogram, and rug plots.

    Parameters
    ----------
    data : pd.DataFrame
        Input DataFrame.
    score : str
        Name of the variable whose distribution is plotted.
    method : str, optional
        Plotting method; one of 'rugplot', 'kdeplot' or 'displot'. Default is 'displot'.
    title : str, optional
        Figure title. Default is "Distribution Plot".
    figsize : tuple, optional
        Figure size. Default is (15, 15).
    nbins : int, optional
        Number of histogram bins (only used by the displot method). Default is 10.

    Returns
    -------
    None
        The plot is displayed directly.

    Examples
    --------
    >>> plot_distribution(df, 'age', method='kdeplot')
    >>> plot_distribution(df, 'score', method='displot', nbins=20)
    """
    plotter = DistributionPlotter(data, score)
    plotter.plot(method=method, title=title, figsize=figsize, nbins=nbins)

ODPS-Side Distribution Analysis: ODPS_Distribution_Tool

ODPS_Distribution_Tool

MaxCompute-backed descriptive statistics.

The public proc_means_odps helper keeps the aggregation in MaxCompute and downloads only the compact aggregate result. PyODPS is imported lazily so the regular Feature package remains usable without the optional ODPS dependency.

proc_means_odps

proc_means_odps(input_table_name: str, skip_cols: list[str] | None = None, select_cols: list[str] | None = None, batch_size: int = 50, group: str | list[str] | None = None, *, q: list[float] | None = None, quantile_method: Literal['approx', 'exact'] = 'approx', percentile_accuracy: int = 10000, where_clause: str | None = None, spec_missing_value: Any | list[Any] | dict[str, Any] | None = None, include_missing_group: bool = False, sqlrunner: Any | None = None, output_csv: str | Path | None = None, output_table_name: str | None = None, output_table_mode: Literal['overwrite', 'append'] | None = None) -> DataFrame

Calculate numeric descriptive statistics directly in MaxCompute.

batch_size is the number of feature columns included in one aggregate SQL statement. It does not split or download source-table rows.

Parameters:

Name	Type	Description	Default
input_table_name	str	Table to analyze: table, project.table or project.schema.table, made of letters, digits and underscores only (ValueError otherwise).	required
skip_cols	list of str or None	Columns excluded from the analyzed variables. They win over select_cols and must exist in the table; a single name may be passed as a string. Names are matched case-insensitively.	None
select_cols	list of str or None	Columns to analyze. None analyzes every numeric ordinary column (string and partition columns are skipped, and a numeric target column is analyzed too, so list it in skip_cols). A non-numeric column raises ValueError.	None
batch_size	int	Number of variables per aggregate SQL statement; each batch scans the table once. Must be a positive integer.	50
group	str or list of str or None	Grouping column(s). None gives one global row per variable. Group columns are never analyzed as variables, come first in the result, and the rows are sorted by group values and then by variable.	None
q	list of float or None	Quantile points in [0, 1], strictly increasing. None means [0.05, 0.15, 0.25, 0.5, 0.75, 0.95, 0.99]. The result column of a point is named Q<int(q * 100)> (0.05 gives Q5), so every point must map to a distinct whole percent.	None
quantile_method	('approx', 'exact')	"approx" uses MaxCompute PERCENTILE_APPROX (suited to large tables); "exact" uses PERCENTILE_CONT (linear interpolation as in pandas), which can need much more compute on very large tables or many groups. Case-insensitive.	"approx"
percentile_accuracy	int	Accuracy argument of PERCENTILE_APPROX; used only with quantile_method="approx". Must be a positive integer.	10000
where_clause	str or None	One SQL condition, wrapped in parentheses and combined with AND (useful for partition pruning). It must be non-empty and must not contain ;.	None
spec_missing_value	number, list of numbers, dict or None	Numeric sentinel value(s) turned into NULL before aggregation, so they count neither in N nor in the statistics. A number or a list applies to every analyzed column; a {column: number or list} dict applies per column, and its keys must be analyzed columns. Non-numeric or non-finite values raise ValueError.	None
include_missing_group	bool	Whether to keep the groups in which a group column is NULL. By default such rows are left out, as in a pandas groupby.	False
sqlrunner	ODPSRunner or None	Runner to reuse; it must expose the ODPS client as sqlrunner.o and provide run_sql. None creates an ODPSRunner from the environment (needs the optional pyODPS package).	None
output_csv	str or Path or None	Path of a CSV file for the result (without the index); missing parent folders are created. None writes no file.	None
output_table_name	str or None	MaxCompute table that receives the result; it must differ from input_table_name. None writes no table.	None
output_table_mode	('overwrite', 'append')	Required with output_table_name and invalid without it. "overwrite" creates the table or replaces it atomically with a schema inferred from the result; "append" needs an existing unpartitioned table whose column names, order and types match the result (ValueError otherwise).	"overwrite"

Returns:

Type	Description
DataFrame	One row per variable (and group): the group columns, then attribute, N_ALL (rows), N (valid values), MEAN, STD (sample standard deviation, NaN when N < 2), MIN, the quantile columns, MAX and MISSING_RATE (1 - N / N_ALL).

Raises:

Type	Description
ValueError	If an argument is invalid (table or column names, batch_size, q, where_clause, spec_missing_value, output settings), if no numeric variable remains, or if an "append" target does not match.
TypeError	If sqlrunner does not provide the ODPS client or the methods the function needs.
ImportError	If sqlrunner is None and the optional pyODPS package is not installed.
RuntimeError	If a batch fails; the message names the batch and its first and last variable, and neither the CSV nor the result table is written.

Notes

Only numeric variables are analyzed: no UNIQUE, TOP or FREQ is computed for categorical columns.

Source code in Modeling_Tool/Feature/ODPS_Distribution_Tool.py

def proc_means_odps(
    input_table_name: str,
    skip_cols: list[str] | None = None,
    select_cols: list[str] | None = None,
    batch_size: int = 50,
    group: str | list[str] | None = None,
    *,
    q: list[float] | None = None,
    quantile_method: Literal["approx", "exact"] = "approx",
    percentile_accuracy: int = 10000,
    where_clause: str | None = None,
    spec_missing_value: Any | list[Any] | dict[str, Any] | None = None,
    include_missing_group: bool = False,
    sqlrunner: Any | None = None,
    output_csv: str | Path | None = None,
    output_table_name: str | None = None,
    output_table_mode: Literal["overwrite", "append"] | None = None,
) -> pd.DataFrame:
    """Calculate numeric descriptive statistics directly in MaxCompute.

    ``batch_size`` is the number of feature columns included in one aggregate
    SQL statement.  It does not split or download source-table rows.

    Parameters
    ----------
    input_table_name : str
        Table to analyze: ``table``, ``project.table`` or ``project.schema.table``, made of letters, digits and
        underscores only (``ValueError`` otherwise).
    skip_cols : list of str or None, default None
        Columns excluded from the analyzed variables. They win over ``select_cols`` and must exist in the table; a single
        name may be passed as a string. Names are matched case-insensitively.
    select_cols : list of str or None, default None
        Columns to analyze. ``None`` analyzes every numeric ordinary column (string and partition columns are skipped,
        and a numeric target column is analyzed too, so list it in ``skip_cols``). A non-numeric column raises
        ``ValueError``.
    batch_size : int, default 50
        Number of variables per aggregate SQL statement; each batch scans the table once. Must be a positive integer.
    group : str or list of str or None, default None
        Grouping column(s). ``None`` gives one global row per variable. Group columns are never analyzed as variables,
        come first in the result, and the rows are sorted by group values and then by variable.
    q : list of float or None, default None
        Quantile points in ``[0, 1]``, strictly increasing. ``None`` means ``[0.05, 0.15, 0.25, 0.5, 0.75, 0.95, 0.99]``.
        The result column of a point is named ``Q<int(q * 100)>`` (``0.05`` gives ``Q5``), so every point must map to a
        distinct whole percent.
    quantile_method : {"approx", "exact"}, default "approx"
        ``"approx"`` uses MaxCompute ``PERCENTILE_APPROX`` (suited to large tables); ``"exact"`` uses
        ``PERCENTILE_CONT`` (linear interpolation as in pandas), which can need much more compute on very large tables
        or many groups. Case-insensitive.
    percentile_accuracy : int, default 10000
        Accuracy argument of ``PERCENTILE_APPROX``; used only with ``quantile_method="approx"``. Must be a positive
        integer.
    where_clause : str or None, default None
        One SQL condition, wrapped in parentheses and combined with ``AND`` (useful for partition pruning). It must be
        non-empty and must not contain ``;``.
    spec_missing_value : number, list of numbers, dict or None, default None
        Numeric sentinel value(s) turned into NULL before aggregation, so they count neither in ``N`` nor in the
        statistics. A number or a list applies to every analyzed column; a ``{column: number or list}`` dict applies per
        column, and its keys must be analyzed columns. Non-numeric or non-finite values raise ``ValueError``.
    include_missing_group : bool, default False
        Whether to keep the groups in which a group column is NULL. By default such rows are left out, as in a pandas
        ``groupby``.
    sqlrunner : ODPSRunner or None, default None
        Runner to reuse; it must expose the ODPS client as ``sqlrunner.o`` and provide ``run_sql``. ``None`` creates an
        ``ODPSRunner`` from the environment (needs the optional pyODPS package).
    output_csv : str or pathlib.Path or None, default None
        Path of a CSV file for the result (without the index); missing parent folders are created. ``None`` writes no
        file.
    output_table_name : str or None, default None
        MaxCompute table that receives the result; it must differ from ``input_table_name``. ``None`` writes no table.
    output_table_mode : {"overwrite", "append"} or None, default None
        Required with ``output_table_name`` and invalid without it. ``"overwrite"`` creates the table or replaces it
        atomically with a schema inferred from the result; ``"append"`` needs an existing unpartitioned table whose
        column names, order and types match the result (``ValueError`` otherwise).

    Returns
    -------
    pandas.DataFrame
        One row per variable (and group): the group columns, then ``attribute``, ``N_ALL`` (rows), ``N`` (valid values),
        ``MEAN``, ``STD`` (sample standard deviation, ``NaN`` when ``N < 2``), ``MIN``, the quantile columns, ``MAX`` and
        ``MISSING_RATE`` (``1 - N / N_ALL``).

    Raises
    ------
    ValueError
        If an argument is invalid (table or column names, ``batch_size``, ``q``, ``where_clause``, ``spec_missing_value``,
        output settings), if no numeric variable remains, or if an ``"append"`` target does not match.
    TypeError
        If ``sqlrunner`` does not provide the ODPS client or the methods the function needs.
    ImportError
        If ``sqlrunner`` is None and the optional pyODPS package is not installed.
    RuntimeError
        If a batch fails; the message names the batch and its first and last variable, and neither the CSV nor the result
        table is written.

    Notes
    -----
    Only numeric variables are analyzed: no ``UNIQUE``, ``TOP`` or ``FREQ`` is computed for categorical columns.
    """
    input_table_name = _validate_table_name(input_table_name, "input_table_name")
    if isinstance(batch_size, bool) or not isinstance(batch_size, int) or batch_size <= 0:
        raise ValueError("batch_size must be a positive integer.")
    method = str(quantile_method).lower()
    if method not in {"approx", "exact"}:
        raise ValueError("quantile_method must be 'approx' or 'exact'.")
    if (
        isinstance(percentile_accuracy, bool)
        or not isinstance(percentile_accuracy, int)
        or percentile_accuracy <= 0
    ):
        raise ValueError("percentile_accuracy must be a positive integer.")
    if where_clause is not None:
        if not isinstance(where_clause, str) or not where_clause.strip():
            raise ValueError("where_clause must be a non-empty SQL condition or None.")
        if ";" in where_clause:
            raise ValueError("where_clause must contain one condition and cannot contain ';'.")

    if output_table_name is None and output_table_mode is not None:
        raise ValueError("output_table_mode requires output_table_name.")
    if output_table_name is not None:
        output_table_name = _validate_table_name(output_table_name, "output_table_name")
        if output_table_mode not in {"overwrite", "append"}:
            raise ValueError(
                "output_table_mode is required with output_table_name and must be "
                "'overwrite' or 'append'."
            )
        if _tables_may_match(input_table_name, output_table_name):
            raise ValueError("output_table_name must not refer to input_table_name.")

    quantiles, quantile_labels = _normalize_quantiles(q)
    active_runner = sqlrunner if sqlrunner is not None else _create_default_runner()
    client = getattr(active_runner, "o", None)
    if client is None or not callable(getattr(client, "get_table", None)):
        raise TypeError("sqlrunner must expose an ODPS client as sqlrunner.o.get_table().")
    if not callable(getattr(active_runner, "run_sql", None)):
        raise TypeError("sqlrunner must provide run_sql().")

    table = client.get_table(input_table_name)
    features, group_cols, schema_by_lower = _resolve_columns(
        table.schema,
        select_cols=select_cols,
        skip_cols=skip_cols,
        group=group,
    )
    special_values = _resolve_special_values(
        spec_missing_value,
        features=features,
        schema_by_lower=schema_by_lower,
    )

    n_batches = math.ceil(len(features) / batch_size)
    frames = []
    for batch_idx in range(n_batches):
        start = batch_idx * batch_size
        batch_features = features[start : start + batch_size]
        sql = _build_batch_sql(
            input_table_name=input_table_name,
            features=batch_features,
            group_cols=group_cols,
            quantiles=quantiles,
            quantile_method=method,
            percentile_accuracy=percentile_accuracy,
            where_clause=where_clause,
            include_missing_group=include_missing_group,
            special_values=special_values,
        )
        logger.info(
            "proc_means_odps batch %s/%s: %s features (%s ... %s)",
            batch_idx + 1,
            n_batches,
            len(batch_features),
            batch_features[0],
            batch_features[-1],
        )
        try:
            raw = active_runner.run_sql(sql, to_df=True)
            frames.append(
                _reshape_batch_result(
                    raw,
                    features=batch_features,
                    group_cols=group_cols,
                    quantile_labels=quantile_labels,
                )
            )
        except Exception as exc:
            message = str(exc).splitlines()[0] if str(exc) else exc.__class__.__name__
            raise RuntimeError(
                f"proc_means_odps batch {batch_idx + 1}/{n_batches} failed for "
                f"features {batch_features[0]!r}..{batch_features[-1]!r}: {message}"
            ) from exc

    output_columns = group_cols + [
        "attribute",
        "N_ALL",
        "N",
        "MEAN",
        "STD",
        "MIN",
        *quantile_labels,
        "MAX",
        "MISSING_RATE",
    ]
    result = (
        pd.concat(frames, ignore_index=True)
        if frames
        else pd.DataFrame(columns=output_columns)
    )
    result = _sort_result(result[output_columns], features=features, group_cols=group_cols)

    if output_csv is not None:
        csv_path = Path(output_csv)
        csv_path.parent.mkdir(parents=True, exist_ok=True)
        result.to_csv(csv_path, index=False)
    if output_table_name is not None:
        _write_odps_output(
            active_runner,
            result,
            output_table_name=output_table_name,
            output_table_mode=str(output_table_mode),
        )
    return result