Four credit models compared on more than accuracy
A random forest won at 76.6%, but accuracy alone hid what mattered: guessing “Standard” for everyone scores 52.7%, and the costly mistakes are the ones that jump two bands.
The problem
Accuracy alone can’t rank models when half the customers share one band.
The fix
Compare on one split against a baseline, recall per band and two-band errors.
- 76.6%
- random forest accuracy
- 52.7%
- accuracy of always guessing Standard
- 0.8%
- forest predictions off by two bands
Four models went head to head on the same credit test set: logistic regression, a random forest, a linear support vector machine (SVM) and gradient boosting. The forest won on accuracy at 76.6%. On its own, that number says less than it seems.
The problem: one band is half the data
After cleaning, 53% of rows were Standard, 31% Poor and 16% Good. A “model” that answers Standard for everyone scores 52.7% on this test set without learning anything. So accuracy needs a floor next to it, and a view of which mistakes each model makes.
The setup: one split, four contenders
Every model used the same 70/30 split with the same random seed: 48,498 rows to train on and 20,785 held back. Each picked its own features first (that’s a separate post), then trained and predicted the test rows. For each I kept the classification report, with precision, recall and F1 for every band, and the confusion matrix, a 3×3 grid of actual band against predicted band.
The scoreboard
| Model | Accuracy | Macro F1 | Off by two bands |
|---|---|---|---|
| Random forest | 76.6% | 0.75 | 165 (0.8%) |
| Gradient boosting | 72.3% | 0.70 | 496 (2.4%) |
| Logistic regression | 63.4% | 0.60 | 490 (2.4%) |
| Linear SVM | 62.9% | 0.62 | 972 (4.7%) |
Macro F1 averages the F1 score of the three bands equally, so the small Good band counts as much as the big Standard one. Off by two bands means calling a Poor customer Good, or a Good one Poor: the mistakes a lender would least want.
Reading past the headline
- The forest rarely makes the bad mistake. Only 165 of 20,785 test rows jumped two bands. Its errors are mostly between neighbours: Standard and Poor, or Standard and Good.
- Good is the forest’s hardest band. Its recall, the share of each band it found, was 78% for Poor and Standard but 69% for Good. Good is also the band with the fewest examples.
- The SVM’s best number is a trap. It caught 83% of Good customers, more than any other model, but only half of its Good calls were right (precision 0.50). It got there by calling a lot of people Good, and it made the most two-band mistakes of the four.
- Straight lines finished last. Logistic regression and the SVM both draw straight-line boundaries between bands, and both landed near 63%, about ten points over the baseline. The two tree-based models, which can bend around the data, finished well ahead.
Gradient boosting was the closest challenger at 72.3%, with the same 69% recall on Good as the forest. Where it lost was the other end: it found only 66% of Poor customers to the forest’s 78%, and it made three times as many two-band mistakes.
Rules I keep from this
- Put the do-nothing baseline next to every accuracy.
- Use one split and one seed for every model, or the comparison isn’t fair.
- Read the confusion matrix for the costly cells, not just the diagonal.
- Look at recall per band, especially for the smallest one.
One caveat on the 76.6% itself: the split was by row, not by customer, which I come back to in another post.