Your first model reports 0.94 accuracy. You feel good for about an hour, until someone asks how it does on data from last month and the number collapses. That gap between a promising notebook and a model you would actually trust is where most first projects stall. The causes are rarely exotic. They are a short list of repeatable mistakes, and almost none of them are about the algorithm you chose.
1. Fitting Anything Before You Split
Data leakage is the most common and the most flattering mistake, because it makes your results look better than they are. It shows up in small, innocent steps: scaling features across the entire dataset before splitting, filling missing values with a column mean computed on all rows, choosing features by correlation with the target, or target-encoding a categorical column using every example.
The model never sees the future, but the statistics used to prepare its inputs do. That is enough to inflate validation scores.
The fix is a habit, not a trick:
- Split into train, validation, and test sets before you touch a single transformation.
- Wrap every preprocessing step in a pipeline so it refits on each training fold.
- For time-ordered data, split by time instead of randomly, and never shuffle.
- For each feature, ask: would I actually know this value at the moment I need a prediction?
That last question catches the subtle cases. A column recording "account closed" is not a feature if the label is "account closed."
2. Skipping the Baseline
A baseline is the dumbest reasonable answer, and it tells you whether your model learned anything. Predict the majority class. Predict the mean. Or fit a plain logistic regression on two or three features.
If your carefully tuned model reaches 0.91 and predicting the most common class already gets 0.90, you have not built a model. You have built a slightly noisier version of a constant. That result is useful information: the problem is probably the data, the features, or the metric, not the architecture.
Record the baseline number in your notes. Every later experiment gets compared against it, and you stop celebrating improvements that mean nothing.
3. Overfitting, and Peeking at the Test Set
Overfitting looks like a training score near perfection and a validation score that is far behind. The model memorized the training rows instead of learning a pattern. More data helps. So does a simpler model, stronger regularization, early stopping, or dropping features that only add noise.
There is a second version of this mistake that people commit without noticing: tuning against the test set. Each time you check test performance, adjust something, and check again, you leak information into your decisions. After twenty rounds, the test score reflects how well you fit that particular sample, not how the model will behave on new data.
Use validation data for decisions. Open the test set once, at the end, and report what you see.
If you would not accept the answer from a model you cannot re-run, you do not have a result yet.
4. Optimizing the Wrong Number
Accuracy is comfortable and often misleading. On a dataset where 2 percent of cases are positive, a model that predicts "no" every single time scores 98 percent accuracy while being useless.
Pick a metric that reflects what a wrong answer costs:
- Precision when false positives are expensive, such as flagging a customer account for review.
- Recall when missing a positive case is worse, such as screening for a rare failure.
- F1 or PR-AUC when both matter and the classes are lopsided.
- Mean absolute error or a similar regression metric when you are predicting a quantity, not a class.
Write down the reason you chose your metric. A false negative that means a missed defect and a false positive that means a wasted phone call are not equally bad, and your metric should say so.
5. Not Looking at Your Data or Your Errors
Before training, spend thirty minutes with the dataset. Plot distributions. Check ranges for impossible values. Count duplicates. Look for a column that is 99 percent one value. Read a sample of labels and ask whether you would have labeled them the same way. Label noise caps your ceiling no matter how good the model is.
After training, do the same thing with errors. Sort the misclassified examples by how confident the model was and read the top thirty. You will usually find a pattern within ten: a group of records with missing data, a category the model has never seen enough of, or a labeling rule that changed halfway through your data collection. Fixing that beats another round of hyperparameter search.
6. Reaching for the Biggest Model First
New practitioners tend to start with the most impressive option available. Larger models take longer to train, cost more to run, have more settings to tune, and fail in ways that are harder to diagnose. Start with something you can explain out loud.
Logistic regression, a decision tree, and a gradient boosting implementation with default settings are strong starting points on tabular data. If none of them beats the baseline, a larger model rarely saves the project. If one of them does beat it, you now have a reference point and a reason to justify anything more complex.
A Short Checklist Before You Call It Done
None of these steps is glamorous, and together they take an afternoon. They will also save you from the uncomfortable conversation where your results do not survive contact with real data.
- Split first; put every transformation in a pipeline.
- Record a baseline score and compare everything to it.
- Choose one metric, and write down why.
- Set random seeds and save your exact splits.
- Keep a plain text log of each experiment: what changed, what happened.
- Review misclassified examples, not just summary numbers.
- Touch the test set once, at the end.
Build the boring version first. Get it working end to end, from raw data to a saved prediction, and make sure you can reproduce it tomorrow. Then improve it, one measured change at a time. That order is what separates a notebook that impressed you for an afternoon from a model you can defend in a meeting.
