Four feature selectors, three signals they all kept
After encoding, a credit model had 44 columns to choose from, most of them month and occupation flags. Four different selection methods each made a shortlist, and every one kept the same three signals.
The problem
44 candidate columns, most of them month and occupation flags.
The fix
Let four models each shortlist features, then trust what they agree on.
- 44 → 5
- features in the final forest
- 4 of 4
- methods kept debt, limit changes and credit mix
- 0
- month or occupation flags kept
Once the credit data was clean and encoded, there were 44 candidate columns: 16 numbers, a yes/no flag for each of 8 months, 15 occupations and 3 answers to “paid only the minimum?”, plus credit mix encoded two ways. Most of those flags looked unlikely to matter. The job was to find the few that did, without trusting one method’s opinion.
The method: four shortlists
For three of the four models I used recursive feature elimination (RFE). It fits the model, drops the feature the model leans on least, and repeats until the list is the size you asked for.
- Logistic regression: cut to 5.
- Linear SVM: cut to 10.
- Gradient boosting: cut to 10.
- Random forest: ranked all 44 by its own importance scores on the training rows, kept the top 5 and retrained on just those.
What they agreed on
Four methods, four different shortlists, and three signals on every one:
- Outstanding debt, what the customer still owes.
- Changed credit limit, the percentage change in their card limit.
- Credit mix, their mix of credit classed as Bad, Standard or Good.
Three more came close. Days late on payments made three shortlists, and logistic regression ranked it sixth, one place short of its five. Credit inquiries and the number of credit cards made three each, missing only the forest’s five. The forest’s final five were outstanding debt, interest rate, days late, changed credit limit and credit mix.
What nobody kept matters too: no month flag and no occupation flag made any shortlist. In this data, the band had little to do with someone’s job or the time of year, and a lot to do with how they handled debt and repayments.
Correlation disagreed once
A plain correlation check, which measures how well a straight line links each column to the band, mostly agreed. Credit mix had the strongest link (0.50), with credit inquiries (-0.44) and days late (-0.43) close behind, and every month and occupation flag scored under 0.02 either way. But it gave interest rate almost nothing (-0.007), while the forest ranked it second.
Part of that is the cleaning. The interest-rate limit was written but commented out, so rates in the thousands stayed in. A handful of extreme values can flatten a straight-line correlation. A tree only asks whether a value is above or below a cut-off, so extremes barely move it.
The catch: RFE doesn’t notice twins
Credit mix went into the table twice: once as an alphabetical label code, which puts Good between Bad and Standard, and once as an ordered Bad, Standard, Good scale. The SVM and gradient boosting both kept both copies, spending two of their ten slots on one signal. RFE ranks features by usefulness; it doesn’t check whether two of them say the same thing.
The fix isn’t a cleverer selector. It’s encoding each column once, deliberately, before selection starts.
Rules I keep from this
- Run more than one selector, and trust the overlap more than any single list.
- Read what got dropped, not just what got kept.
- Treat correlation as a first look. It only sees straight lines, and outliers drown it.
- Encode each category once, in an order that means something.