Supply Chain · Digital

ML Forecasting Pilot

Start one machine-learning model on one product family, benchmark it against your current method for three months, then decide.

Time1 h
FormatSolo
StageDigital

ML Forecasting Pilot: what it is and why it works

An ML forecasting pilot tests whether a machine-learning model can forecast demand better than your current method on real data, under controlled conditions. It takes one product family with clean history, builds a straightforward ML baseline, such as a gradient-boosted tree model using sales history, calendar, price and promotion features, and runs it in parallel with the current method for about three months. Both are compared on accuracy, bias and stability, and the team then decides to scale, iterate or stop, with the reasoning written down.

The method works because it replaces opinions about AI with evidence from your own demand patterns. Machine learning tends to add value where many related series share patterns and drivers such as promotions, prices or weather are known in advance; it adds little on sparse, erratic demand with no explanatory data. Comparing against a simple benchmark, like a moving average or seasonal naive forecast, keeps the test honest. A pilot also surfaces data problems early, which is why it pairs naturally with Master Data Quality. The winning approach feeds the Demand Review, and results should be read alongside the broader Forecast Methods toolbox.

What you need

At least two to three years of clean demand history for one product family
Known demand drivers: promotions, prices, calendar events, customer orders
The current forecast method and its historical forecasts for comparison
Agreed accuracy, bias and stability metrics and the forecast horizon that matters for decisions
A data scientist or analyst and a planner who will use the results

What you get

Three months of parallel forecasts from both methods
A comparison on accuracy, bias and stability at the decision horizon
A documented decision to scale, iterate or stop, with reasons
A list of data issues found during the pilot

When to use it

When “AI will fix forecasting” is either feared or promised, but never tested.

How to do it, step by step

Pick one product family with clean history.
Benchmark: your current method versus a simple ML baseline.
Run three months of parallel forecasting.
Compare on accuracy AND bias AND stability.
Decide: scale, iterate, or stop — and write down why.

Worked example: Forecasting replacement filter demand for a distributor

Illustrative scenario — figures are realistic but not from a real company.

A distributor sells about 1,200 SKUs of industrial air and liquid filters. Planners use exponential smoothing with manual overrides. Weighted forecast error at a four-week lag averaged about 38%, and management received competing claims that AI would halve it.

The analyst chose one family of 180 SKUs with three years of clean history and added features: price changes, known distributor promotions, seasonality and installed-base trends for key filter housings.
She trained a gradient-boosted tree model and set up both methods to forecast weekly for the next 13 weeks, with the four-week lag as the scored horizon.
Scoring used weighted absolute percentage error for accuracy, mean signed error for bias, and week-to-week forecast change for stability.
Planners did not see the ML forecast during the pilot, so their overrides did not contaminate the comparison.

Result. The ML model reduced weighted error to about 31% and bias from a persistent 9% over-forecast to under 2%, but was less stable on low-volume SKUs. The team decided to scale it to high- and medium-volume SKUs only and keep smoothing for slow movers. The lesson: gains were real but smaller than promised, and uneven across the range.

Open it in the FieldKit Supply Chain app → Get the Supply Chain template pack — join the waitlist

Common pitfalls and how to avoid them

Comparing ML only against the current method's worst period.Run both methods in parallel on the same future weeks and the same horizon.
Scoring accuracy alone.Track bias and stability as well; a slightly more accurate but erratic forecast can hurt planning.
Letting planners see both forecasts during the test.Keep the pilot blind or record overrides separately so the comparison stays clean.
Choosing a product family with poor history.Pick a family with clean data for the pilot, and fix data issues before judging any model.

Frequently asked questions

Is machine learning better than traditional forecasting methods?

Sometimes. ML often helps where many related items share patterns and where external drivers such as price, promotions or weather are known ahead of time. For sparse, intermittent or short-history items, simple statistical methods can match or beat it. Only a parallel test on your own data answers the question.

How do you measure forecast accuracy?

Common measures include mean absolute percentage error, weighted absolute percentage error, which weights errors by volume, and mean absolute scaled error, which compares against a naive forecast. Always measure at the horizon that drives decisions, such as the supplier lead time, and pair accuracy with bias.

What is forecast bias?

Bias is the tendency of a forecast to be consistently too high or too low. It is usually measured as the average signed error or the ratio of total forecast to total actual demand. Bias matters because persistent over-forecasting builds excess stock and persistent under-forecasting causes shortages, even when absolute error looks acceptable.

Origin

ML forecasting — Amazon.com era practice; M5 competition lineage, 2020.

Used in these playbooks

Digital quick wins month 1 month

A month of pragmatic digitalization: clean masters, automate one document flow, stand up a lightweight tower, then pilot ML.

Related methods

More in “Digital”

Instrument the chain: data, integration and decision support.