Split credit data by customer, not by row

The random forest scored 76.6% on held-out data, and that’s the headline on this project. Going back through the notebook with Claude Code for this write-up, it’s also the number I’d now read most carefully, because of how the held-out rows were chosen.

The problem: the test set had met its customers

The dataset is 12,500 customers with eight monthly rows each. Just before modelling, the notebook drops the customer ID and splits the rows at random: 70% to train, 30% to test. Nothing keeps a customer’s months together, so most customers end up with some months in training and others in the test set.

For a credit model that matters. The question it exists to answer is “which band is this new customer in?” A test row whose owner already sits in the training data asks an easier one: “have you seen this person before?”

Why it’s worse than it sounds here

Two things in this project make a returning customer easy to recognise:

• Cleaning made a customer’s months alike. Filling gaps from each customer’s other months, the right call for cleaning, copies the same values across their rows.
• The top feature is nearly an ID. Outstanding debt, the forest’s most important feature, barely changes from month to month: 12,203 distinct values across 100,000 rows from 12,500 customers.

So a forest can match a test row to the same customer’s training rows by debt alone, and lean on the band it learned for them. That’s recognition, not prediction.

The scaling step had a smaller version of the same leak. The min-max scaler was fitted on all rows before the split, so test rows helped set the 0 to 1 range. For tree models that barely matters. The principle does.

What I can and can’t say

The notebook never ran a customer-level split, so I can’t say how much of the 76.6% this explains. Maybe a little, maybe a lot. The second file that came with the data can’t settle it either: it has no credit-score column to check against.

What I can say is that 76.6% was scored mostly on customers the model had already met. For new applicants, the honest answer is “not yet tested”.

The fix

Split by the thing you’ll predict for. In scikit-learn that means a group-aware splitter, with the customer ID as the group:

split customers 70/30, not rows:
    each customer's months go entirely to train or entirely to test
fit scaling and feature selection on training customers only
score the test customers once, at the end

Everything learned from data, the scaling range, the feature shortlist and the forest’s top five, then comes from training customers only. The test customers are touched once.

Rules I keep from this

1. Ask what the model will be used on, then make the test set look like that.
2. Repeated entities need group splits: customers, patients, stores, devices.
3. Be suspicious of a near-unique top feature. It may be an ID in disguise.
4. Fit every preprocessing step after the split, on training data only.