Skip to content

Reject Inference and Distribution Adaptation

Modeling samples usually come only from approved customers, which introduces selection bias. SuperModelingFactory provides two kinds of tools to correct for it: reject inference and sample weighting.

1. Reject Inference — RejectInferenceFactory

Factory Method

from Modeling_Tool import RejectInferenceFactory

inferrer = RejectInferenceFactory.create(
    method="parceling",
    target_col="bad_flag",
    score_col="prob",
)
df_combined = inferrer.infer(
    df_approved=approved_df,
    df_rejected=rejected_df,
    score_col="prob",
)

Supported Inference Methods

Method When to use Idea
hard_cutoff The old model has a clear score threshold Rejected samples with a score above the threshold are labeled bad
fuzzy Transition near the threshold Assign different bad probabilities by score band
parceling Bin mapping Bin rejected samples by score and assign labels by the bin's bad rate
simple_augment Quick approximation Assign labels to rejected samples by the overall bad rate

Usage Examples by Method

from Modeling_Tool import ParcelingInferrer

inferrer = ParcelingInferrer(target_col="bad_flag", score_col="prob", n_parcels=10)
df_combined = inferrer.infer(approved_df, rejected_df, score_col="prob")

from Modeling_Tool import HardCutoffInferrer

inferrer = HardCutoffInferrer(
    target_col="bad_flag",
    score_col="prob",
    cutoff=0.5,
)
df_combined = inferrer.infer(approved_df, rejected_df, score_col="prob")

from Modeling_Tool import FuzzyAugmentInferrer

inferrer = FuzzyAugmentInferrer(
    target_col="bad_flag",
    score_col="prob",
    weight_factor=0.8,
)
df_combined = inferrer.infer(approved_df, rejected_df, score_col="prob")

from Modeling_Tool import SimpleAugmentInferrer

inferrer = SimpleAugmentInferrer(target_col="bad_flag", score_col="prob")
df_combined = inferrer.infer(approved_df, rejected_df, score_col="prob")

Custom Inferrers

Subclass the abstract class RejectInferrer (infer(self, df_approved, df_rejected, score_col=None)):

import pandas as pd
from Modeling_Tool import RejectInferrer

class MyInferrer(RejectInferrer):
    def infer(self, df_approved, df_rejected, score_col=None):
        # Custom logic
        score_col = score_col or self.score_col
        df_rejected = df_rejected.copy()
        df_rejected[self.target_col] = 0   # default: everyone is good
        df_rejected.loc[df_rejected[score_col] > 0.3, self.target_col] = 1
        return pd.concat([df_approved, df_rejected], ignore_index=True)

2. Distribution Adaptation — DistributionAdaptation

Used when the training set and the OOT distribution differ.

Density Ratio Estimation

from Modeling_Tool import DistributionAdaptation

adapter = DistributionAdaptation(method="density_ratio")
sample_weights = adapter.estimate_density_ratio(
    X_train=train_df[features],
    X_oot=oot_df[features],
)

# Use sample_weight during training
import lightgbm as lgb
dtrain = lgb.Dataset(train_X, train_y, weight=sample_weights)

Covariate Shift Correction

adapter = DistributionAdaptation(method="covariate_shift")
weights = adapter.covariate_shift_weighting(train_X, oot_X)

The fit / get_weights interface

You can also use the unified fit(X_train, X_oot) + get_weights():

adapter = DistributionAdaptation(method="density_ratio")
adapter.fit(X_train=train_df[features], X_oot=oot_df[features])
sample_weights = adapter.get_weights()

3. Complete Pipeline

from Modeling_Tool import RejectInferenceFactory, DistributionAdaptation

# 1) Reject inference (expand the sample first)
inferrer = RejectInferenceFactory.create("parceling", target_col="bad_flag", score_col="prob")
df_combined = inferrer.infer(approved_df, rejected_df, score_col="prob")

# 2) Distribution adaptation (then weight)
adapter = DistributionAdaptation(method="density_ratio")
adapter.fit(X_train=df_combined[features], X_oot=oot_df[features])
sample_weights = adapter.get_weights()

# sample_weights can be used as sample weights for a downstream trainer (SMF currently has no built-in weighted-training interface for this).

4. Pipeline Layer: Automatic Sample-Weight Passing

Starting with v0.3.16, RejectInferencePipeline automatically passes the _weight column generated by each RI method to the downstream GradientBoostingModel.fit as sample_weight, with no explicit configuration needed:

RI method Generates _weight? Weighted during training?
simple_augment ❌ (each rejected sample is copied into 1 good row, with weight treated as 1.0) ❌
hard_cutoff ❌ ❌
fuzzy_augment ✅ (each rejected sample is copied into a good row and a bad row, with weights 1-p(bad) and p(bad) respectively) ✅
parceling ✅ (labels are assigned randomly by score band, with weights based on the within-band bad rate) ✅

Historical bug (v0.3.15 and earlier)

In v0.3.15 and earlier, RejectInferencePipeline passed _weight to GradientBoostingModel.fit as wgt=. Because the correct parameter name for fit is sample_weight, wgt was silently swallowed by **kwargs, and the sample weights never actually took effect. This was fixed in v0.3.16, and weighted training for fuzzy_augment / parceling now works as expected.

If you trained a post-RI model on v0.3.15 or earlier, retrain it once on v0.3.16 and compare the KS/AUC difference to confirm whether the weights significantly affect the model.

5. Pipeline Layer: Input & Config Validation (v0.3.17)

v0.3.17 tightened the fail-fast validation of RejectInferencePipeline, lifting two kinds of cases that previously either errored deep in the code or leaked data silently into the _validate_input / _validate_ri_approved_config stage at the start of run().

5.1 Implicit prescore needs target_col

Even with train_prescore=False, whenever the input DataFrame lacks score_col (default "prescore_prob"), the Pipeline still goes through one implicit prescore training (the internal _run logic: if cfg.train_prescore or cfg.score_col not in data.columns: _fit_prescore(...)).

In historical versions, _validate_input checked whether target_col existed only when train_prescore=True, so a missing target on that implicit path surfaced as an oddly placed KeyError inside _fit_prescore. Since v0.3.17, whenever the Pipeline determines that any prescore training path will be triggered, _validate_input explicitly raises a KeyError at the outermost level, with a message containing the actual state of train_prescore and score_col to make the cause easy to locate.

The valid combinations:

train_prescore Is score_col in the data? Is target_col required?
True Present or absent ✅ Required
False ✅ Provided ❌ Optional (the Pipeline uses the external score directly)
False ❌ Missing (implicit training) ✅ Required (explicitly demanded since v0.3.17)

5.2 oot_frac + ri_validation_frac must be < 1.0

After drawing the OOT by oot_frac, RejectInferencePipeline._sample_validation_ids draws the validation set from the remaining samples by ri_validation_frac. If the sum of the two is ≥ 1, the remaining pool is empty, and historical versions silently fell back to pool = approved.copy(), so the validation set overlapped with the OOT and OOT rows leaked into the validation metrics.

Since v0.3.17, _validate_ri_approved_config raises a ValueError directly at the start of run():

cfg = RejectInferencePipelineConfig(oot_frac=0.5, ri_validation_frac=0.5)
RejectInferencePipeline(cfg).run(df)
# ValueError: oot_frac (0.5) + ri_validation_frac (0.5) = 1.000 must be < 1.0
# to leave a non-empty training pool disjoint from OOT and validation

A separate boundary constraint was also added: oot_frac must be within [0, 1) (0 is allowed and means the OOT split is disabled).

6. Pipeline Layer: Pre-score Hardening (v0.3.18)

Starting with v0.3.18, RejectInferencePipeline adds two explicit checks on the pre-score training path, to surface early the silent errors that previously could only be found by inspecting the output.

6.1 score_col Overwrite Warning

When both of the following conditions hold:

  • cfg.train_prescore=True (the user explicitly asked to train a pre-score)
  • The column named by cfg.score_col already exists in the input DataFrame

the Pipeline raises a UserWarning before training actually starts:

train_prescore=True will overwrite existing column 'prescore_prob' in input data.
Set train_prescore=False to reuse the supplied scores, or rename cfg.score_col to
a fresh column name to avoid this overwrite.

Historical behavior (v0.3.17 and earlier): the Pipeline overwrote the user-supplied column with the newly trained model's scores, with no notice at all. Callers who wanted to keep the original scores usually discovered the swap only during downstream comparison.

Recommendations:

  • To reuse existing scores → train_prescore=False;
  • To train a new score while keeping the old column → change cfg.score_col to a new column name that does not exist (for example "prescore_prob_v2");
  • If you are sure you want to overwrite → just ignore the warning; behavior is the same as 0.3.17.

6.2 Fail Fast on an Empty Approved Sample

_fit_prescore first filters to rows where (approved_col==1) & target_col.notna(). If that subset is empty, v0.3.17 and earlier passed an empty DataFrame all the way to SampleSplitter, ending in a hard-to-locate stratify error inside sklearn. Since v0.3.18, this raises directly at the entry of _fit_prescore:

ValueError: Cannot train pre-score model: no rows satisfy
('approved'==1 & 'badflag' not null). Diagnostics: approved=0,
target-observed=0, total=1200. Check approved_col/target_col semantics or
supply a pre-computed score_col and set train_prescore=False.

The error message gives the approved count, the target-observed count, and the total row count, so you can quickly tell whether the approval flag is wrong, labels are missing, or the whole batch was rejected.

7. Pipeline Layer: Rows Without a Label

A model can only learn from rows that have a label. Two kinds of rows in the RI datasets have none:

  • approved rows whose target_col is missing, for example applications that have not performed yet;
  • hard_cutoff rejects whose score is missing, which HardCutoffInferrer deliberately leaves unlabelled.

Both stay in result.ri_datasets as they are, but the pre-score, every RI model, the no_ri_benchmark model, the validation sample and the random OOT use only rows with a label. The counts are reported, and a UserWarning names them:

Output Column Meaning
ri_summary N_approved_unlabelled Approved rows of the dataset without a target
ri_summary N_rejected_unlabelled Rejected rows without an inferred label (hard_cutoff with a missing score)
ri_model_perf train_unlabelled_n Rows of the model's training pool left out because their target is missing

ri_summary['prescore_AUC'] is computed on the approved rows with a target. Up to 0.9.0, the fit turned a missing target into 0, so these rows were trained as goods (see the changelog).

8. Pipeline Layer: One Score, and an OOT the Pre-score Never Saw

Three rules keep the pre-score consistent with the rest of the pipeline:

Rule Why
A pre-score trained by the Pipeline requires ri_score_direction="high_bad"; "high_good" raises ValueError The trained pre-score is P(bad). Read as a high-good score, it made hard_cutoff and fuzzy_augment label the riskiest rejects good (the hard-cutoff reject bad rate fell from 0.99 to 0.23 in the audit). Use "high_good" only with your own score and train_prescore=False
When the Pipeline trains the pre-score, an external ri_approved_data is scored with it, and an existing score_col there is replaced with a UserWarning The rules are fitted on the reference and applied to the rejects. A reference that kept its own score put the two on different scales (reject bad rate 0.09 for a true 0.41 in the audit)
The random OOT (oot_frac) is drawn before the pre-score is trained and is left out of its training data The inferred reject labels come from the pre-score. When it had learned the OOT labels, the RI models inherited them and their OOT AUC beat the benchmark on pure noise (0.54 against 0.51 for hard_cutoff)

The random OOT keeps the same rows as before; only the pre-score loses them. With ri_approved_scope="output_subset" the OOT is drawn from all labelled approved rows and only the drawn rows inside the subset are used, so its share of the subset is about oot_frac. The OOT from split_col or oot_data never entered the pre-score, so those runs are unchanged. The pre-score still trains on the validation rows of the RI models.

FAQ

Which reject inference method is the most accurate?

There is no academic consensus. Parceling performs steadily in most empirical studies and is worth trying first; in practice it is usually validated with an AB test.

Should I apply SMOTE to the inferred sample?

Usually not. Reject inference has already added minority-class samples, and applying SMOTE on top may amplify noise.

How do I evaluate the effect of weighting when the OOT sample is very small (< 1000)?
  1. Split the OOT into several time windows and evaluate each separately
  2. Use PSI rather than AUC as the primary metric