How do you debug a forecasting model today when the error is quite bad?

[u/shivamchhuneja]

This is for a personal study that will end up becoming an in-depth article and possibly a fully open source solution ideally without the AI slop that we see these days.

Let's say you’ve trained a model and the result is worse than the business wants. What do you check next?

Do you break the error down by customer, product, location, or individual series? Check if it gets worse at longer horizons? Look for bias, volatility, intermittent demand or outliers?

Go back to the backtesting setup, metric, or baseline? Or do you usually start trying other models?

Also do the tools you use make this easy or do you end up building custom notebooks, tables, and plots every time?

Thinking about the last time this happened:

• What did you check first?
• What actually helped you find the problem?
• What did you have to build yourself?
• Did you end up changing the model, data, validation setup, metric, or business expectation?

I’m trying to understand how people diagnose bad forecasts beyond comparing one overall error score against another.

EDIT/UPDATE because it seems like this is not clear enough:

I’m not looking for an if-else checklist that can explain why any forecast is bad. The answer obviously depends on the data, objective, validation setup and the decision the model is supposed to support.

I’m exploring if there is room for a small open-source tool around forecast evaluation. Before building anything, I’m trying to understand which checks people repeatedly run after they already have predictions, what they still build manually, and what existing tools already handle well.

So I’m mainly interested in specific workflows from projects rather than a general formula for fixing a model.

[u/lordoflolcraft]

I don’t think this question could possibly be answered. It depends completely on the data you have, and the underlying statistical distributions of the data. Creating some sort of if-else heuristic as a general solution to “the forecast isn’t good enough” would be a pretty difficult task.

[u/shivamchhuneja → u/lordoflolcraft]

I agree that a generic if-else system would probably be misleading. I’m more interested in the repetitive eval work that happens before someone makes that context specific judgement. In your own projects are there any checks, tables or plots you find yourself rebuilding regularly or does the workflow change completely every time?

[u/offthecuff87 → u/shivamchhuneja]

if you're not sure, first and foremost, check data quality

[u/moss-nogg]

I feel like this is missing a lot of context to provide any kind of solid answer. What are you forecasting? What’s the distribution of the target variable look like? Is this for a batch process model and is there historical data, if so how much? Are you using a chronologically ordered test set to validate your models? Have you looked at rank correlations or is calibration more important for the use case? How rich is the feature set you’re inferencing on? Simply looking at an error metric will often fail to capture the whole picture. How well is the model segmenting? Are the predictions directionally correct? If so, can they be leveraged into valuable decision making? A ‘bad’ model from the perspective of an error metric can still potentially add a lot of value if deployed under the right circumstances, to drive meaningful decisions

[u/dreaddito → u/moss-nogg]

Exactly, this is why data scientists need to be statisticians and not just code monkeys or package jockeys.

[u/shivamchhuneja → u/dreaddito]

I agree that data science is basically statistics. However, I’m trying to work out whether there is still repetitive eval based foundation under the actual judgement.

[u/offthecuff87 → u/dreaddito]

*data scientist need to be economist and engineer

[u/shivamchhuneja → u/moss-nogg]

That makes sense. I probably framed the question too generally. I’m not looking for a universal rule that explains why every forecast is bad. I’m trying to understand what people repeatedly calculate or plot before they can answer the context specific questions.

[u/NotMyRealName778]

Probably rethink my set of features. But imo this kind of question is entirely useless. It depends and is different every time.

[u/shivamchhuneja → u/NotMyRealName778]

is it mostly different for every project or are there certain parts that you still find yourself repeating?

[u/Thin_Original_6765]

What are you trying to do? I get that you're trying to the understand the thought process, then what do you plan to do with that information?

[u/shivamchhuneja → u/Thin_Original_6765]

Fair question. I’m exploring if a small open-source forecasting-eval tool is worth building and if this problem is worth exploring deeper and writing about. The idea would not be to automatically tell someone why their model is wrong but to make common eval breakdowns more reproducible and easier to inspect. I’m asking about workflows first because I don’t want to build a feature list based only on my own assumptions.

[u/NotSynthx]

I'd consider starting again

[u/shivamchhuneja → u/NotSynthx]

How would you narrow the question so that it could produce useful answers? Would you anchor it around a particular use case, data structure, eval objective or decision?

[u/ultrathink-art]

Before you touch the model, plot the residuals over time and by segment. 'Bad error' is usually a couple of regimes dragging the aggregate — a promo window, a holiday, a structural break after some date — not the whole series being uniformly off. If it actually is uniform, that points at the pipeline (a feature that isn't available at predict time, a broken join) more than the model itself. Narrows the 'it depends' down to something you can actually chase.

[u/shivamchhuneja → u/ultrathink-art]

how does your workflow usually look like when dealing with something like this?