Reviews of Important Papers on Forecasting,
1985-1995 Reviews
Review of:

Robert Fildes (1989), "Evaluation of aggregate and individual forecast method selection rules," Management Science 35,  1056 – 1065.

This study examined 263 data series from a single organization. The series represent the number of telephone lines in use in each of 263 localities. The localities include only those with at least 1000 circuits in period one. The data cover the same time period and consist of 71 monthly observations. Most of the series were decreasing over this period. Two forecasting methods were considered: one was Holt’s exponential smoothing, and the other was a ‘robust trend estimate.’ The robust trend estimate uses the median of the first differences for the series and modifies this to adjust for outliers.

Here is the question. Is it better to find a model that works best overall, and to use this (referred to as an ‘aggregate selection’ strategy)? Or should a simulation be done for each series to determine which method is best (’individual selection’)? Or should you simply combine forecasts? Research to date suggests that combining is the best strategy, unless you know which method is best. And determining the best method is, of course, the purpose of simulation. In this study, models were calibrated on periods 1 to 24. Successive updating was then used to simulate over periods 25 to 48. The validation was conducted for periods 49 to 70.

Of the three strategies examined, ‘aggregate selection’ is the easiest to operate. ‘Ex ante aggregate selection’ has been used in previous evaluations of forecasting performance by basing the selection of the method on the performance of the contending methods over a number of data series with forecasts made at a single point of time. In contrast ‘ex post aggregate selection’ is based on the method that produced the best forecasts after successive updating has been used for all starting points.

‘Individual selection’ is more expensive than ‘aggregate selection.’ However, ex post analysis was shown to have the potential of leading to substantial improvements in accuracy. The key question addressed by the paper is whether the potential gains from individual selection are achievable in practice.

The results were that the three ex ante strategies performed almost identically for short range forecasts (from one to six periods in the future). For longer range forecasts (12 months ahead), the error from aggregate selection was about six percent higher than that for the individual and combining strategies. However, ‘ex post aggregate selection’ produced worthwhile improvements in accuracy for leads 6 and 12 over the three ex ante strategies. The paper concludes by arguing that the forecaster would do as well by choosing a single method for all series. It is desirable for the selection of this method to be based on a large number of data series as well as by successive updating over a substantial number of starting points.