Guidance on the Choice of Threshold for Binary Forecast Modeling

This paper proposes useful guidance on the choice of threshold for binary forecasts. In weather forecast systems, the probabilistic forecast cannot be used directly when estimated too smoothly. In this case, the binary forecast, whether a meteorological event will occur or not, is preferable to the probabilistic forecast. A threshold is needed to generate a binary forecast, and the guidance in this paper encompasses the use of skill scores for the choice of threshold according to the forecast pattern. The forecast pattern consists of distribution modes of estimated probabilities, occurrence rates of observations, and variation modes. This study is performed via Monte-Carlo simulation, with 48 forecast patterns considered. Estimated probabilities are generated by random variate sampling from five distributions separately. Varying the threshold from 0 to 1, binary forecasts are generated by threshold. For the assessment of binary forecast models, a 2×2 contingency table is used and four skill scores (Heidke skill score, hit rate, true skill statistic, and threat score) are compared for each forecast pattern. As a result, guidance on the choice of skill score to find the optimal threshold is proposed.