Evaluating the Relationship Between Forecast Accuracy and Economic Value of Forecast: A Cost–Loss Based Evaluation of Influenza Forecasts

Abstract

Epidemiological forecast evaluation commonly focuses on statistical accuracy mea- sures such as Mean Absolute Error (MAE), Weighted Interval Score (WIS), and Log Score (LS). However, accurate forecasts do not necessarily translate into improved decision-making performance. Therefore, this thesis investigates the relationship between forecast accuracy and the value of a forecast within probabilistic influenza forecasting. The study uses data from the CDC FluSight forecasting challenge, where forecasting models provide probabilistic predictions of future Influenza-like Illness (ILI) preva- lence. Event-based forecasting scenarios are constructed through threshold defini- tions based on relative growth in ILI prevalence. Forecast probabilities are obtained from predictive distributions and used within a cost–loss decision framework to de- termine preventive actions. Ten forecasting models are evaluated across multiple event definitions, cost–loss ra- tios, and forecast horizons. Decision performance is quantified using value scores and integrated value measures. Furthermore, Spearman rank correlation analysis is used to investigate the relationship between forecast accuracy rankings and decision- based rankings. The results indicate that forecast accuracy and value score rankings are generally positively correlated across event definitions, cost–loss ratios, and forecast horizons, although the relationship is not perfectly aligned. While the ensemble model “target- type-based-weights” and the model “Reichlab-KCDE” perform strongly under both evaluation frameworks, differences between statistical and decision-based rankings arestillobserved. Thesefindingssuggestthattraditionalforecastaccuracymeasures and decision-based approaches provide complementary perspectives for evaluating the practical usefulness of probabilistic forecasts.