When the best fit forecasts worst

The problem

To plan a process, you need a short-term forecast of what it will produce. The thorough way to get one is to build a physics-based simulation, which takes months and a budget to match. The capstone asked whether the operating data that gets measured routinely could give a usable forecast far faster.

The system has inputs an operator can control and outputs that get measured. When the operator changes a setting, the output responds, and the relationship is stable enough to learn from.

That input-to-output shape is the general case, so the method carries over to a production line, a utility or a pipeline, whatever industry the data came from.

Conceptual illustration. Not the project's data.

The data

Equinor's Volve open dataset: several producing units and two injectors, with rates, pressures, control settings and hours on stream. The analysis went deep on one unit.

Cleaning came before any model ran. Hours-on-stream values above the maximum possible were treated as outliers. The rates turned out to be log-normally distributed, not normal, and that changes what a residual plot is telling you.

How we approached it

Four families of model, compared on the same data:

1. Multiple linear regression, with predictors checked for significance and multicollinearity
2. A univariate ARIMA on the differenced series
3. A transfer-function model: each input pre-whitened, lags identified, then fitted
4. A multivariate VARMAX, with around fifty specifications tested and selected on an information criterion

Running all four side by side showed how differently models that describe the same history can behave once they have to predict.

How it is built

This was a study, with no production system behind it. Each model was prepared and fitted the same way, then asked to forecast a stretch of data it had not seen. The test that mattered was whether a forecast stayed inside what the process can physically do.

Methods and tools: SAS, JMP, Python and Tableau, which were the tools available at the time.

What we can show

The model with the best in-sample fit produced the least realistic forecast. The ARIMA fitted the history better than anything else and then predicted a future the process could not physically produce. Selecting on fit statistics alone would have chosen it.

The transfer-function model fitted slightly worse and forecast sensibly, because it encoded what drives the output: when a control setting is turned down, the forecast follows it down. The multivariate model was quick to build and captured how the outputs move together.

The result changes with the variable

For the main rate, the transfer-function forecast stayed close to the held-back validation data while the multivariate model drifted well above it. For the second variable the picture reverses later in the comparison: the multivariate model stays closer, and the transfer-function line climbs away from the data.

So this entry doesn't name a winning model. The useful lesson is to test on the variable you care about, because an average across outputs would have hidden the reversal completely.

What this page leaves out on purpose

You won't find an accuracy figure, an error percentage or a claim about how far ahead any of these models can be trusted. The original outputs were produced in software we no longer license, so nothing can be re-opened and checked against its source. Publishing a number we can't verify would be worse than publishing none, and the finding above doesn't depend on one.

The original presentation also estimated time and cost savings against building a simulation. Those were a student's projections, never measured outcomes, so they're left out too.

What it demonstrates

Fit statistics don't tell you whether a forecast is good enough to decide with. The test is whether it behaves correctly when someone changes a setting, and whether it holds up on the variable the decision depends on.

That's the test we apply to a forecast now. It's also why the forecasting package reports performance on data the model never saw, and not on the history it was fitted to.

What we would do differently today

We'd rebuild it in open Python, as a reproducible notebook anyone can run. Rolling-origin validation would report the error for each horizon separately, and a machine-learning baseline would sit alongside. That's the only route to numbers a reader could check. We've written it down as a project, which is not the same as a promise.