Advanced Search
Article Contents

Evaluation of TIGGE Ensemble Forecasts of Precipitation in Distinct Climate Regions in Iran


doi: 10.1007/s00376-017-7082-6

  • The application of numerical weather prediction (NWP) products is increasing dramatically. Existing reports indicate that ensemble predictions have better skill than deterministic forecasts. In this study, numerical ensemble precipitation forecasts in the TIGGE database were evaluated using deterministic, dichotomous (yes/no), and probabilistic techniques over Iran for the period 2008-16. Thirteen rain gauges spread over eight homogeneous precipitation regimes were selected for evaluation. The Inverse Distance Weighting and Kriging methods were adopted for interpolation of the prediction values, downscaled to the stations at lead times of one to three days. To enhance the forecast quality, NWP values were post-processed via Bayesian Model Averaging. The results showed that ECMWF had better scores than other products. However, products of all centers underestimated precipitation in high precipitation regions while overestimating precipitation in other regions. This points to a systematic bias in forecasts and demands application of bias correction techniques. Based on dichotomous evaluation, NCEP did better at most stations, although all centers overpredicted the number of precipitation events. Compared to those of ECMWF and NCEP, UKMO yielded higher scores in mountainous regions, but performed poorly at other selected stations. Furthermore, the evaluations showed that all centers had better skill in wet than in dry seasons. The quality of post-processed predictions was better than those of the raw predictions. In conclusion, the accuracy of the NWP predictions made by the selected centers could be classified as medium over Iran, while post-processing of predictions is recommended to improve the quality.
    摘要: 目前数值天气预报产品的应用正与日俱增. 已有研究表明: 集合预报的技巧比单一确定性预报的更高. 本文利用确定性评估, 二分型评估, 概率评估等判别指标评估了TIGGE资料集合预报2008年至2016年间伊朗降水的预报质量, 评估选取了分布于伊朗8个典型降水区域的13个雨量计观测资料为参考. 首先将提前1-3天的数值预报结果通过距离反比加权, Kriging插值等方法插值到选取的站点, 并利用贝叶斯模式平均方法对数值预报产品预处理以进一步提高预报质量. 结果表明: 欧洲中期天气预报中心(ECMWF)的预报比其他产品得分更高. 所有预报中心的产品都低估了降水较多地区的降水, 同时高估了其他地区的降水. 这表明TIGGE集合预报中存在着系统性偏差, 需进一步应用偏差校订技术. 基于二分型评估, 美国环境预报中心(NCEP)的预测在大部分站点中表现较好, 尽管所有的中心均高估了降水事件的发生频次. 相较于ECMWF和NCEP的预报, 英国气象局(UKMO)的预报在山区表现更好, 但在其他地区表现较差. 此外, 所有中心的预报在湿季都比在干季表现更好. 经过预处理的预报比未经过处理的预报质量更好. 总的来说, 本研究选择的各中心数值预报对伊朗降水预报的精度可以评为中等, 同时对预报产品进行预处理可提高预报质量.
  • 加载中
  • Bao H.-J., L.-N. Zhao, Y. He, Z.-J. Li, F. Wetterhall, H. L. Cloke, F. Pappenberger, and D. Manful, 2011: Coupling ensemble weather predictions based on TIGGE database with Grid-Xinanjiang model for flood forecast.Advances in Geosciences, 29, 61-67, https://doi.org/10.5194/adgeo-29-61-2011
    Buizza R., P. L. Houtekamer, G. Pellerin, Z. Toth, Y. J. Zhu, and M. Z. Wei, 2005: A comparison of the ECMWF, MSC, and NCEP global ensemble prediction systems. Mon. Wea. Rev., 133, 1076-1097, https://doi.org/10.1175/MWR2905.1
    Fan K., H. J. Wang, and Y. J. Choi, 2008: A physically-based statistical forecast model for the middle-lower reaches of the Yangtze River Valley summer rainfall.Chinese Science Bulletin, 53, 602-609, https://doi.org/10.1007/s11434-008-0083-1
    Fan K., Y. Liu, and H. P. Chen, 2012: Improving the Prediction of the East Asian summer monsoon: New approaches.Wea. Forecasting, 27, 1017-1030, https://doi.org/10.1175/WAF-D-11-00092.1
    Gevorgyan A., 2013: Verification of daily precipitation amount forecasts in Armenia by ERA-Interim model.International Journal of Climatology, 33, 2706-2712, https://doi.org/10.1002/joc.3621
    Javanmard M., M. Delavar, and S. Morid, 2016: Evaluation and uncertainty analysis of the results of the global weather forecast models to apply in flood warning systems (case study: Karoon River basin, Iran). M.S. thesis, Tarbiat Modares University.
    Lee S.-M., J.-E. Nam, H.-W. Choi, J.-C. Ha, Y. H. Lee, Y.-H. Kim, H.-S. Kang, and C. Cho, 2016: A study on the predictability of the transition day from the dry to the rainy season over South Korea.Theor. Appl. Climatol., 125, 449-467, https://doi.org/10.1007/s00704-015-1504-0
    Liu J. G., Z. H. Xie, 2014: BMA probabilistic quantitative precipitation forecasting over the Huaihe Basin Using TIGGE multimodel ensemble forecasts.Mon. Wea. Rev., 142, 1542-1555, https://doi.org/10.1175/MWR-D-13-00031.1
    Liu Y., K. Fan, 2014: An application of hybrid downscaling model to forecast summer precipitation at stations in China.Atmos. Res., 143, 17-30, https://doi.org/10.1016/j.atmosres.2014.01.024
    Louvet S., B. Sultan, S. Janicot, P. H. Kamsu-Tamo, and O. Ndiaye, 2016: Evaluation of TIGGE precipitation forecasts over West Africa at intraseasonal timescale.Climate Dyn., 47, 31-47, https://doi.org/10.1007/s00382-015-2820-x
    Luitel B., G. Villarini, and G. A. Vecchi, 2016: Verification of the skill of numerical weather prediction models in forecasting rainfall from U.S.landfalling tropical cyclones. J. Hydrol., https://doi.org/10.1016/j.jhydrol.2016.09.019.
    Modarres R., 2006: Regional precipitation climates of Iran. Journal of Hydrology (New Zealand), 45, 13- 27.
    Mohammad D., Z. K. Suma, 2016: Evaluation of spatio-temporal accuracy of precipitation of European Center for medium-range weather forecasts (ECMWF) over Iran.Physical Geography Research Quarterly, 47, 651-675, https://doi.org/10.22059/jphgr.2015.56054
    Raftery A. E., T. Gneiting, F. Balabdaoui, and M. Polakowski, 2005: Using Bayesian model averaging to calibrate forecast ensembles.Mon. Wea. Rev., 133, 1155-1174, https://doi.org/10.1175/MWR2906.1
    Raziei T., F. Sotoudeh, 2017: Investigation of the accuracy of the European Center for Medium Range Weather Forecast (ECMWF) in forecasting observed precipitation in different climates of Iran.Journal of the Earth and Space Physics., 43, 133-147, https://doiorg/10.22059/jesphys.2017.57958
    Sene K., 2010: Hydrometeorology.Springer,345 pp.
    Shapiro M. A., A. J. Thorpe, 2004: THORPEX international science plan.WMO/TD No. 1246, WMO.
    Sodoudi S., A. Noorian, M. Geb, and E. Reimer, 2010: Daily precipitation forecast of ECMWF verified over Iran.Theor. Appl. Climatol., 99, 39-51, https://doi.org/10.1007/s00704-009-0118-9
    Su X., H. L. Yuan, Y. J. Zhu, Y. Luo, and Y. Wang, 2014: Evaluation of TIGGE ensemble predictions of Northern Hemisphere summer precipitation during 2008-2012.J. Geophys. Res., 119, 7292-7310, https://doi.org/10.1002/2014JD021733
    Swinbank, R., Coauthors, 2016: The TIGGE project and its achievements.Bull. Amer. Meteor. Soc., 97, 49-67, https://doi.org/10.1175/BAMS-D-13-00191.1
    Tao Y. M., Q. Y. Duan, A. Z. Ye, W. Gong, Z. H. Di, M. Xiao, and K. Hsu, 2014: An evaluation of post-processed TIGGE multimodel ensemble precipitation forecast in the Huai river basin.J. Hydrol., 519, 2890-2905, https://doi.org/10.1016/j.jhydrol.2014.04.040
    Zapata J. A. V., 2010: Evaluation of hydrological ensemble prediction systems for operational forecasting. PhD dissertation, UniversitéLaval.
    Zhao L.-N., F.-Y. Tian, H. Wu, D. Qi, J.-Y. Di, and Z. Wang, 2011: Verification and comparison of probabilistic precipitation forecasts using the TIGGE data in the upriver of Huaihe Basin.Advances in Geosciences, 29, 95-102, https://doi.org/10.5194/adgeo-29-95-2011
    Zhi X. F., L. Zhang, and Y. Q. Bai, 2011: Application of the multi-model ensemble forecast in the QPF.2011 International Conference on Information Science and Technology (ICIST), Nanjing, IEEE, 657- 660.
  • [1] ZHANG Jinqiang, XUAN Yuejian, YAN Xiaolu, LIU Mingyuan, TIAN Hongmin, XIA Xiang'ao, PANG Li, and ZHENG Xiangdong, 2014: Development and Preliminary Evaluation of a Double-cell Ozonesonde, ADVANCES IN ATMOSPHERIC SCIENCES, 31, 938-947.  doi: 10.1007/s00376-013-3104-1
    [2] LI Shan, RONG Xingyao, LIU Yun, LIU Zhengyu, Klaus FRAEDRICH, 2013: Dynamic Analogue Initialization for Ensemble Forecasting, ADVANCES IN ATMOSPHERIC SCIENCES, 30, 1406-1420.  doi: 10.1007/s00376-012-2244-z
    [3] Mengmeng SONG, Dazhi YANG, Sebastian LERCH, Xiang’ao XIA, Gokhan Mert YAGLI, Jamie M. BRIGHT, Yanbo SHEN, Bai LIU, Xingli LIU, Martin János MAYER, 2024: Non-crossing Quantile Regression Neural Network as a Calibration Tool for Ensemble Weather Forecasts, ADVANCES IN ATMOSPHERIC SCIENCES.  doi: 10.1007/s00376-023-3184-5
    [4] Lili LEI, Yangjinxi GE, Zhe-Min TAN, Yi ZHANG, Kekuan CHU, Xin QIU, Qifeng QIAN, 2022: Evaluation of a Regional Ensemble Data Assimilation System for Typhoon Prediction, ADVANCES IN ATMOSPHERIC SCIENCES, 39, 1816-1832.  doi: 10.1007/s00376-022-1444-4
    [5] ZHOU Baiquan, NIU Ruoyun, ZHAI Panmao, 2015: An Assessment of the Predictability of the East Asian Subtropical Westerly Jet Based on TIGGE Data, ADVANCES IN ATMOSPHERIC SCIENCES, 32, 401-412.  doi: 10.1007/s00376-014-4026-2
    [6] ZHU Jiang, LIN Caiyan, WANG Zifa, 2009: Dust Storm Ensemble Forecast Experiments in East Asia, ADVANCES IN ATMOSPHERIC SCIENCES, 26, 1053-1070.  doi: 10.1007/s00376-009-8218-0
    [7] T. N. Krishnamurti, Mukul Tewari, Ed Bensman, Wei Han, Zhan Zhang, William K. M. Lau, 1999: An Ensemble Forecast of the South China Sea Monsoon, ADVANCES IN ATMOSPHERIC SCIENCES, 16, 159-182.  doi: 10.1007/BF02973080
    [8] Yuejian ZHU, 2005: Ensemble Forecast: A New Approach to Uncertainty and Predictability, ADVANCES IN ATMOSPHERIC SCIENCES, 22, 781-788.  doi: 10.1007/BF02918678
    [9] SUN Yang, WANG Yuesi, XIU Tianyang, WANG Yinghong, XU Xin, 2007: Establishment and Evaluation of a Method for Analyzing Atmospheric Volatile Organic Compounds, ADVANCES IN ATMOSPHERIC SCIENCES, 24, 679-687.  doi: 10.1007/s00376-007-0679-4
    [10] Jingpeng ZHANG, Tianbao ZHAO, Zhi LI, Chunxiang LI, Zhen LI, Kairan YING, Chunxiang SHI, Lipeng JIANG, Wenyu ZHANG, 2021: Evaluation of Surface Relative Humidity in China from the CRA-40 and Current Reanalyses, ADVANCES IN ATMOSPHERIC SCIENCES, 38, 1958-1976.  doi: 10.1007/s00376-021-0333-6
    [11] Jeong-Gyun PARK, Dong-Kyou LEE, 2011: Evaluation of Heavy Rainfall Model Forecasts over the Korean Peninsula Using Different Physical Parameterization Schemes and Horizontal Resolution, ADVANCES IN ATMOSPHERIC SCIENCES, 28, 1233-1245.  doi: 10.1007/s00376-011-0058-z
    [12] JIANG Dabang, WANG Huijun, LANG Xianmei, 2005: Evaluation of East Asian Climatology as Simulated by Seven Coupled Models, ADVANCES IN ATMOSPHERIC SCIENCES, 22, 479-495.  doi: 10.1007/BF02918482
    [13] Zhenhua HUO, Wansuo DUAN, Feifan ZHOU, 2019: Ensemble Forecasts of Tropical Cyclone Track with Orthogonal Conditional Nonlinear Optimal Perturbations, ADVANCES IN ATMOSPHERIC SCIENCES, 36, 231-247.  doi: 10.1007/s00376-018-8001-1
    [14] Jihang LI, Zhiyan ZHANG, Lu LIU, Xubin ZHANG, Jingxuan QU, Qilin WAN, 2021: The Simulation of Five Tropical Cyclones by Sample Optimization of Ensemble Forecasting Based on the Observed Track and Intensity, ADVANCES IN ATMOSPHERIC SCIENCES, 38, 1763-1777.  doi: 10.1007/s00376-021-0353-2
    [15] Jiangshan ZHU, Fanyou KONG, Xiao-Ming HU, Yan GUO, Lingkun RAN, Hengchi LEI, 2018: Impact of Soil Moisture Uncertainty on Summertime Short-range Ensemble Forecasts, ADVANCES IN ATMOSPHERIC SCIENCES, 35, 839-852.  doi: 10.1007/s00376-017-7107-1
    [16] Kazutoshi SATO, Jun INOUE, Akira YAMAZAKI, Naohiko HIRASAWA, Konosuke SUGIURA, Kyohei YAMADA, 2020: Antarctic Radiosonde Observations Reduce Uncertainties and Errors in Reanalyses and Forecasts over the Southern Ocean: An Extreme Cyclone Case, ADVANCES IN ATMOSPHERIC SCIENCES, 37, 431-440.  doi: 10.1007/s00376-019-8231-x
    [17] Sun Shufen, Xue Yongkang, 2001: Implementing a New Snow Scheme in Simplified Simple Biosphere Model, ADVANCES IN ATMOSPHERIC SCIENCES, 18, 335-354.  doi: 10.1007/BF02919314
    [18] YE Xingnan, CHEN Tianyi, HU Dawei, YANG Xin, CHEN Jianmin, ZHANG Renyi, Alexei F. KHAKUZIV, WANG Lin, 2009: A Multifunctional HTDMA System with a Robust Temperature Control, ADVANCES IN ATMOSPHERIC SCIENCES, 26, 1235-1240.  doi: 10.1007/s00376-009-8134-3
    [19] Yuyang GUO, Yongqiang YU, Pengfei LIN, Hailong LIU, Bian HE, Qing BAO, Bo AN, Shuwen ZHAO, Lijuan HUA, 2020: Simulation and Improvements of Oceanic Circulation and Sea Ice by the Coupled Climate System Model FGOALS-f3-L, ADVANCES IN ATMOSPHERIC SCIENCES, 37, 1133-1148.  doi: 10.1007/s00376-020-0006-x
    [20] Jian RAO, Rongcai REN, Haishan CHEN, Xiangwen LIU, Yueyue YU, Yang YANG, 2019: Sub-seasonal to Seasonal Hindcasts of Stratospheric Sudden Warming by BCC_CSM1.1(m): A Comparison with ECMWF, ADVANCES IN ATMOSPHERIC SCIENCES, 36, 479-494.  doi: 10.1007/s00376-018-8165-8

Get Citation+

Export:  

Share Article

Manuscript History

Manuscript received: 14 April 2017
Manuscript revised: 13 July 2017
Manuscript accepted: 07 August 2017
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

Evaluation of TIGGE Ensemble Forecasts of Precipitation in Distinct Climate Regions in Iran

  • 1. Department of Technical and Engineering, Science and Research Branch, Islamic Azad University, Tehran 1477893855, Iran
  • 2. Department of Water Resources Engineering, Tarbiat Modares University, Tehran 14115-336, Iran

Abstract: The application of numerical weather prediction (NWP) products is increasing dramatically. Existing reports indicate that ensemble predictions have better skill than deterministic forecasts. In this study, numerical ensemble precipitation forecasts in the TIGGE database were evaluated using deterministic, dichotomous (yes/no), and probabilistic techniques over Iran for the period 2008-16. Thirteen rain gauges spread over eight homogeneous precipitation regimes were selected for evaluation. The Inverse Distance Weighting and Kriging methods were adopted for interpolation of the prediction values, downscaled to the stations at lead times of one to three days. To enhance the forecast quality, NWP values were post-processed via Bayesian Model Averaging. The results showed that ECMWF had better scores than other products. However, products of all centers underestimated precipitation in high precipitation regions while overestimating precipitation in other regions. This points to a systematic bias in forecasts and demands application of bias correction techniques. Based on dichotomous evaluation, NCEP did better at most stations, although all centers overpredicted the number of precipitation events. Compared to those of ECMWF and NCEP, UKMO yielded higher scores in mountainous regions, but performed poorly at other selected stations. Furthermore, the evaluations showed that all centers had better skill in wet than in dry seasons. The quality of post-processed predictions was better than those of the raw predictions. In conclusion, the accuracy of the NWP predictions made by the selected centers could be classified as medium over Iran, while post-processing of predictions is recommended to improve the quality.

摘要: 目前数值天气预报产品的应用正与日俱增. 已有研究表明: 集合预报的技巧比单一确定性预报的更高. 本文利用确定性评估, 二分型评估, 概率评估等判别指标评估了TIGGE资料集合预报2008年至2016年间伊朗降水的预报质量, 评估选取了分布于伊朗8个典型降水区域的13个雨量计观测资料为参考. 首先将提前1-3天的数值预报结果通过距离反比加权, Kriging插值等方法插值到选取的站点, 并利用贝叶斯模式平均方法对数值预报产品预处理以进一步提高预报质量. 结果表明: 欧洲中期天气预报中心(ECMWF)的预报比其他产品得分更高. 所有预报中心的产品都低估了降水较多地区的降水, 同时高估了其他地区的降水. 这表明TIGGE集合预报中存在着系统性偏差, 需进一步应用偏差校订技术. 基于二分型评估, 美国环境预报中心(NCEP)的预测在大部分站点中表现较好, 尽管所有的中心均高估了降水事件的发生频次. 相较于ECMWF和NCEP的预报, 英国气象局(UKMO)的预报在山区表现更好, 但在其他地区表现较差. 此外, 所有中心的预报在湿季都比在干季表现更好. 经过预处理的预报比未经过处理的预报质量更好. 总的来说, 本研究选择的各中心数值预报对伊朗降水预报的精度可以评为中等, 同时对预报产品进行预处理可提高预报质量.

1. Introduction
  • Nowadays, meteorological forecasts are produced using numerical models. Precipitation is one of the widely demanded meteorological factors. Improvement of Quantitative Precipitation Forecasts (QPFs) is the main objective of forecast centers and a major challenge for the meteorological research communities. Deterministic predictions have limitations in atmospheric conditions and change in initial conditions; thus, Ensemble Prediction Systems (EPSs) have been produced to enhance numerical and probabilistic prediction skill (Sene, 2010). EPSs involve different individual predictions produced by different physical parameterizations or different initial conditions. In the 1990s, EPSs were used practically in calculating the chaotic nature of climate processes, which could significantly reduce the uncertainties that had existed previously (Buizza et al., 2005). The first EPSs started in 1992 using data from ECMWF and NCEP (Zapata, 2010). The WMO organized THORPEX to further improve ensemble forecasts of severe meteorological events with one-day to two-week lead times (Shapiro and Thorpe, 2004). The THORPEX executive phase lasted from 2005 to 2014 (Swinbank et al., 2016) but later extended until 2019. TIGGE encompasses EPSs of 10 numerical weather prediction (NWP) centers whose data are made available by the China Meteorological Administration (CMA) and ECMWF centers. When various models that produce EPSs from different weather centers are aggregated, the probabilistic nature of the ensemble precipitation forecasts is better retained and accounted for (Bao et al., 2011).

    A number of researchers have evaluated the TIGGE data in different regions. For instance, (Zhao et al., 2011) showed that ECMWF was slightly better compared to NCEP and CMA in China region, whereas for lead times of over five days, none of the centers presented reliable predictions. Based on ensemble forecasting data of the CMA, UKMO, ECMWF, NCEP and JMA in the TIGGE datasets in the Northern Hemisphere, (Zhi et al., 2011) investigated the multi-model ensemble (MME) precipitation forecasting techniques and concluded that the bias-removed ensemble mean forecast was more skillful and more stable than each individual model. (Liu and Fan, 2014) post-processed TIGGE precipitation predictions using Bayesian Model Averaging (BMA) and showed that the post-processed prediction skill was better compared to that of the raw predictions. Moreover, UKMO and ECMWF yielded better predictions compared to those of NCEP and CMA. For the Northern Hemisphere, (Su et al., 2014) showed that the ECMWF product was better compared to those of other centers, while in central parts of the Northern Hemisphere better prediction skill was achieved compared to those of the equatorial regions.

    (Louvet et al., 2016) reported that ECMWF and UKMO provided better results in West Africa compared to those of other centers. (Luitel et al., 2016) evaluated the precipitation products, driven by North Atlantic tropical cyclone activities, of five prediction centers in the TIGGE database and concluded that the predictions were more suitable for lead times up to 48 h. In evaluating the prediction accuracies of TIGGE data over South Korea at six operational forecast centers, (Lee et al., 2016) showed that ECMWF and KMA (Korea Meteorological Administration) performed well, while CMC and CMA did poorly, in forecasts.

    Some researchers have used databases other than TIGGE in applications of MME forecasts. For instance, (Fan et al., 2012) evaluated the prediction ability of the three DEMETER models (CNRM, UKMO and ECMWF) as well as the MME in seasonal predictions of the East Asian summer monsoon. The interannual increment prediction approach was applied to improve the prediction ability of the models and it was concluded that the direct outputs of the models were better able to predict than its original form. (Liu and Fan, 2014) applied two statistical downscaling schemes based on three different DEMETER GCMs to predict station rainfall. The downscaling model based on any single predictor demonstrated lower prediction skill than the multi-predictor downscaling models.

    In the context of regional studies conducted in or around Iran, (Sodoudi et al., 2010) showed that ECMWF, to some extent, could better predict the location of precipitation bands in mountainous and high-elevation regions compared to those in desert plain. In addition, ECMWF provided better results in the Zagros Mountains, to the west of Iran, compared to the Alborz Mountains to the north of Iran. (Gevorgyan, 2013) concluded that changes in precipitation amounts throughout Armenia were not modeled properly by ECMWF precipitation data. (Mohammad and Suma, 2016) evaluated the 3-h precipitation product of the ECMWF's ERA-Interim over Iran and concluded that this product had adequate performance in precipitation prediction in the Zagros Mountains, southern shores of the Caspian Sea, and Northeast Iran. (Raziei and Sotoudeh, 2017) evaluated ERA-Interim data over Iran and concluded that, at most stations, sufficient accuracy was achieved. However, ECMWF underpredicted the precipitation at Caspian littoral stations, due to the inability of ERA-Interim to accurately predict heavy precipitation in the region. (Javanmard et al., 2016) evaluated TIGGE database predictions in the Karoon river basin, located in the southwest of Iran, over the period 2008-09. The results showed that ECMWF performed better compared to products of other centers. Moreover, after post-processing by the Bagging, Adaboost, and BMA methods, they concluded that post-processed predictions performed better compared to raw predictions.

    The accuracy of numerical ensemble precipitation predictions within the TIGGE database has not been evaluated over the whole country so far. This study aims to assess the TIGGE ensemble predictions of three meteorological centers——ECMWF, NCEP and UKMO——over the period 2008-16, with lead times of one to three days, covering 13 rain gauges from eight homogenous precipitation regimes as classified by (Modarres, 2006). The reason for selecting these three particular products out of ten available centers within the TIGGE database was due to their better abilities reported in previous studies. Evaluations were performed using (i) deterministic, (ii) dichotomous (yes/no) and (iii) probabilistic techniques. Finally, to assess the possible improvement in predictions, the ensemble predictions were post-processed using the BMA method, which constituted a grand ensemble prediction.

    Figure 1.  Classification map of Iran’s precipitation regimes according to (Modarres, 2006) overlaid on the topography (red circles are the selected stations in each region).

2. Data and methods
  • The 2008-16 50-km prediction products were extracted from the TIGGE database at the ECMWF with lead times of one, two and three days over Iran. Among the centers in the TIGGE database, three (ECMWF, NCEP and UKMO) were selected. The characteristics of the aforementioned centers are provided in Table 1. Observed data were extracted for 13 synoptic stations in Iran, spread over eight different regions as classified by (Modarres, 2006). Table 2 presents the characteristics of the stations. (Modarres, 2006) classified eight homogenous precipitation regimes over Iran, based on the application of Ward's technique to the annual and monthly precipitation of the selected rainfall stations. These eight regimes/clusters cover 90% of the precipitation variance within Iran. The first cluster (G1) is the largest and includes stations in arid and semi-arid regions in central Iran. The second cluster (G2) involves highland margins of G1, while G3 represents the northwestern cold region. The fourth cluster (G4) includes areas along the Persian Gulf coast south of Iran, while the sixth and the eighth clusters (G6, G8) involve areas located along the coast of the Caspian Sea. The major difference between the G6 and G8 regions in the north is the amount of precipitation decreasing from west to east. The fifth and seventh clusters (G5, G7) encompass regions in the Zagros Mountains, where precipitation in G5 is higher than in G7. The geographic distribution of the cluster regions is shown in Fig. 1. To make a direct comparison with precipitation spatial variation, Fig. 2 displays the average annual precipitation from 1984 to 2014. Interpolation of NWP predicted values at stations was implemented using the Inverse Distance Weighting (IDW) and Kriging methods.

    Figure 2.  Mean annual precipitation of Iran.

  • The evaluations were performed using deterministic, dichotomous (yes/no), and probabilistic approaches. For deterministic evaluation, four common criteria were adopted, including the Pearson correlation coefficient (Pearson's r), root-mean-square error (RMSE), mean absolute error (MAE), and the relative root-mean-square error (RRMSE). Furthermore, yes/no binary assessment criteria, including the probability of detection (POD), false alarm rate (FAR), bias score (BIAS), and equitable threat score (ETS), were used for the dichotomous evaluations. Finally, the Brier score (BS), Brier skill score (BSS), continuous ranked probability score (CRPS), and the area under the relative operating characteristic (ROC) (ROC.Area) were adopted for the probabilistic evaluation. All criteria formulations are given in Table 3 and contingency table are shown in Table 4.

  • BMA combines predictions from several statistical models with variable weighting coefficients. This method was used for ensemble predictions by (Raftery et al., 2005) to predict air temperature, surface and sea level pressure (Liu and Fan, 2014). The probability distribution function (PDF) of BMA is as follows (Raftery et al., 2005): \begin{equation} P(y|f_1,\cdots,f_K)=\sum_{k=1}^Kw_kg_k(y|f_k) , \nonumber \end{equation} where y is the prediction coefficient; gk(y|fk) is the conditional PDF of y based on fk, which is the best member of the ensemble prediction; wk is the posterior probability of forecast k which is non-negative with a summation equal to one and K is the number of models being combined. Since there were large numbers of zero precipitation events, the computational PDF in this paper was set to a Gamma distribution function, which was selected due to its high skewness. Detailed information regarding the calculation of gk(y|fk) and wk may be found in the literature (e.g. Raftery et al., 2005; Liu and Xie, 2014). This study took advantage of the ensemble BMA package in the R software.

3. Results and discussion
  • In what follows, the results of all evaluations associated with each of the eight regions are described. Due to a large number of results (prediction evaluation at 13 stations from 2008 to 2016), only the evaluation of 24-h precipitation at all stations is provided, and then the forecasts are evaluated for different lead times at the end of the section. Since in most parts of Iran precipitation is low in the dry seasons, the evaluations were carried out and reported for the wet seasons only. The wet seasons in Iran generally take place from November to April.

    As previously noted, the IDW and Kriging methods were used for spatial interpolation of precipitation forecasts. Nevertheless, the results of these two methods showed no significant difference. Hence, in what follows, only the IDW results are presented.

  • According to (Modarres, 2006), the G1 region is the dominant precipitation regime in Iran and has a high coefficient of variation with low precipitation in a predominantly arid and semi-arid climate condition. Due to the extent of this region, three stations (Esfahan, Semnan, and Zahedan) were selected. Figure 3 presents the total annual precipitation associated with this region. In most years, all centers overestimated the annual precipitation, while ECMWF offered better precipitation predictions at Semnan and Esfahan compared to that at Zahedan. On the contrary, NCEP performed better in predicting the annual precipitation at Zahedan but comparatively poorly at Esfahan and Semnan.

    Figure 3.  Total observed and predicted annual precipitation (mm yr-1) of three centers at rain gauge stations selected in precipitation regions: (a) Esfahan; (b) Mashhad; (c) Semnan; (d) Shahrekord; (e) Zahedan; (f) Tehran; (g) Ahvaz; (h) Bandarabbas; (i) Tabriz; (j) Sanandaj.

    In the G2 region, which essentially constitutes mountainous areas upstream of the G1 region, three stations were selected: Mashhad, Shahrekord, and Tehran. Similar to G1, all centers overestimated the annual precipitation for most years at Mashhad and Tehran. At Shahrekord, which receives higher precipitation than the other two stations, UKMO underestimated, whereas the other two generally overestimated, the precipitation.

    Some centers showed different performance in predicting precipitation in the wet seasons compared with those of the whole year. For example, UKMO, which performed better than the other two models at Shahrekord, was the weakest for the wet season. The total NCEP predicted precipitation over the study period was significantly different from the total observed precipitation at Tehran.

    In the G3 region, which encompasses cold regions in northwestern Iran, the station at Tabriz was studied. According to Fig. 3, NCEP predictions were the poorest in all years, except in 2010 and 2011, compared to those of the other centers, while better predictions were achieved by ECMWF compared to those of UKMO and NCEP.

    In the G4 region, the stations at Ahvaz and Bandar Abbas were selected. Based on Fig. 3, although all three centers overestimated the annual precipitation, UKMO did quite poorly. For Sanandaj station in the G5 region, similar to other regions, all centers overestimated the annual precipitation. Predictions made by UKMO were better compared to those of ECMWF. Moreover, poorer predictions were made by ECMWF in 2008 and 2009.

    In the rainy climate of the G6 region, the station at Babolsar was selected. Based on Fig. 4, the centers overestimated and underestimated precipitation in different years. At Ilam in G7, which generally receives more precipitation than G5, NCEP was the poorest of all the centers, whereas ECMWF's predictions were better than those of UKMO in most years. As shown in Fig. 4, in the G8 region, receiving higher precipitation than the G6 region, ECMWF offered better predictions compared to those of the other centers, while NCEP's was the poorest, underestimating the precipitation in all years.

    Overall, the products of all the centers underestimated the precipitation in the relatively wetter climate regions but overestimated the precipitation in dryer climate areas. This implies a systematic bias in forecasts and demands application of bias correction techniques, such as quantile mapping.

    Figure 4.  Total observed and predicted annual precipitation ( mm yr-1) of three centers at three rain gauge stations selected in precipitation regions: (a) Babolsar in the G6 region; (b) Ilam in the G7 region; (c) Rasht in the G8 region.

  • For the deterministic evaluation, this study adopted four criteria: the correlation coefficient (r), MAE, RMSE, and RRMSE, whose formulations are presented in Table 3. The results are shown in Fig. 5. Due to limitations in displaying all examined cases, the average performance of the stations in each cluster is presented. Moreover, the results of each station are presented in Table 5.

    At Esfahan and Semnan in the G1 region, ECMWF and NCEP yielded the best and poorest scores, respectively. In contrast, at Zahedan, ECMWF and NCEP were the poorest and the best predicting centers, respectively. All in all, in this region, ECMWF was the best and NCEP was the poorest.

    In the G2 region, and based on the correlation coefficient, ECMWF at all three selected stations produced the best scores, while NCEP was the poorest. At Shahrekord, UKMO performed well, but was poorest at Mashhad.

    In the cold climate of the G3 region, based on all three indicators, ECMWF was the best and NCEP was the poorest of all. In the hot and dry G4 region, NCEP yielded smaller prediction errors compared to those of the other centers, while UKMO performed comparatively poorly in terms of the deterministic evaluation scores.

    In the G5 region, of all three centers, UKMO resulted in smaller prediction error, whereas NCEP performed the poorest. In the G6 rainy region, ECMWF and NCEP had the best and poorest scores, respectively. However, in this region, due to higher precipitation relative to other areas in Iran, large prediction errors were produced by all three centers.

    At Ilam in the G7 region, ECMWF's predictions were slightly better than those of UKMO; NCEP was the poorest of all. In G8, based on the correlation coefficient and RMSE, ECMWF was the best and UKMO was the poorest.

    Figure 5.  Results of the deterministic evaluation of three centers for eight precipitation regions in Iran between observations and forecasts: (a) correlation coefficient; (b) mean absolute error (mm d-1); (c) root-mean-square error (mm d-1); (d) relative root-mean-square error.

    In general, based on deterministic evaluation, ECMWF in most regions of Iran, UKMO in mountainous regions, and NCEP in southern Iran, provided better results compared to other centers. In addition, TIGGE numerical precipitation predictions at Ilam within the G7 region performed best among all examined stations in terms of annual precipitation.

  • This study used four indicators (POD, FAR, ETS and BIAS) for dichotomous evaluation. The evaluation results are shown in Fig. 5. According to the BIAS criteria, which is the ratio of the number of predicted precipitation events to observed precipitation events, NCEP and UKMO respectively offered the best and poorest predictions of the number of precipitation days. ECMWF showed smaller BIAS in the G3 region compared to that of NCEP. All centers overestimated the number of precipitation days.

    Based on the ETS score, which measures the fraction of forecast events that were correctly predicted, NCEP achieved comparatively better scores at all stations, except in the G3 region. In addition, the prediction quality of UKMO was poor. However, the very low scores of ETS at most stations represents an inappropriate prediction accuracy of the number of precipitation events.

    According to Fig. 5d, POD values are high, which is due to a high BIAS score at most stations. Of all centers, UKMO, due to the higher values of BIAS compared to those of other centers, yielded better POD, while NCEP had the lowest scores. Based on FAR, which represents the number of false alarms in precipitation events, UKMO was the poorest and NCEP, in most regions, was better than other centers. The number of false identifications was quite high in the G1 and G4 regions, most likely due to the rarity of precipitation events in these regions. In conclusion, the number of precipitation events predicted by all three centers was higher than observed, while NCEP had better scores in most regions.

    Figure 6.  Dichotomous (yes/no) evaluation of three centers for eight precipitation regions in Iran between observations and forecasts: (a) bias score (frequency bias); (b) equitable threat score (Gilbert skill score); (c) false alarm ratio; (d) probability of detection (hit rate).

    Figure 7.  Results of the probabilistic evaluation of three centers for eight precipitation regions in Iran between observation and forecasts: (a) Brier score; (b) Brier skill score; (c) continuous ranked probability score; (d) area under the relative operating characteristic (ROC) curve.

  • In this section, the gamma PDF was used to represent the QPF distribution. Four common methods (ROC.Area, CRPS, BS and BSS) were used for the probabilistic evaluation and the results are presented in Fig. 6. BS, which is a function of resolution, uncertainty and reliability, measures the mean squared probability error. BSS, which expresses the BS skill score relative to the reference BS, is usually determined by climatology predictions. CRPS evaluates the accuracy of the probabilistic forecast distribution. The ROC curve is a measure of the prediction's isolation skill in occurrence/non-occurrence of precipitation. The area under the curve is also an evaluation criterion. The values closer to 1.0 represent higher confidence in predictions.

    Figure 7 shows the average probabilistic evaluations over the eight study years. Based on BS, precipitation at stations in the G4 region was better predicted than that at other selected stations. However, based on BSS, predictions were poor due to, as previously mentioned, the rarity of precipitation events. In all regions, based on BSS, NCEP showed better prediction capability compared to ECMWF, except in G1 and G3, whereas UKMO was the poorest based on both BS and BSS. Moreover, based on CRPS, UKMO and ECMWF had higher scores in some regions while NCEP did poorly compared to other models. Based on ROC.Area, ECMWF and NCEP yielded the highest and lowest scores, respectively.

    As a whole, according to the probabilistic evaluations in Table 5, precipitation at Semnan and Zahedan in the G1 region, as well as Bandar Abbas in G4, were poorly predicted. Mashhad, Zahedan, Ilam had better scores than those of other stations. ECMWF and NCEP performed almost the same, while UKMO performed poorer in the probability of precipitation occurrence/non-occurrence criteria.

    Summary results are presented in Table 5, showing ECMWF performed better in all regions. UKMO had slightly better performance compared to NCEP in precipitation prediction. However, according to the dichotomous evaluation, NCEP performed better in almost all regions and could predict precipitation occurrence/non-occurrence better than other centers. Figure 8 presents the evaluation results for lead times of between one and three days. The results clearly illustrate that the precipitation prediction skill decreases with an increase in lead time. This reduction is quite obvious based on CRPS. According to Fig. 8, region G7 had the best scores, while the poorest performance in precipitation prediction was achieved in G1 and G4.

    Also, Fig. 9 compares the performance of the models in the dry and wet seasons. Only the results of the rainy regions of G6 and G8 are presented because other regions receive very little precipitation in the dry season. Based on Fig. 9, all models performed better in the wet than in the dry season, whereas UKMO failed in the G8 region for the dry season.

    Overall, the results indicate that better numerical prediction performance is expected in regions with high precipitation.

    Figure 8.  Results of the three prediction centers' assessments for eight precipitation regions with different lead times between observation and forecasts: (a) correlation coefficient; (b) bias score; (c) Brier score; (d) continuous ranked probability score.

4. Grand ensemble prediction
  • A grand ensemble that includes EPS forecasts from several forecasting centers may improve the accuracy of numerical weather forecasts by taking uncertainties in the initial conditions, lateral boundary conditions, and model physics into account. The ensemble is potentially able to provide a better representation of the probable distribution of true predictions (Liu and Fan, 2014). For this purpose, ensemble predictions of the three centers were combined to constitute a grand ensemble prediction using two techniques. First, the products of the three centers were combined with the same weighting coefficients and without post-processing (raw) to constitute a grand ensemble prediction. In the second scenario, the predictions of each of the three centers were post-processed with variable weighting coefficients via BMA. Using the ensemble BMA package in the R software package, observed precipitation and ensemble predictions associated with the three centers from 2008 to 2016 were selected and, by a training period of 30 days based on the cube root of the ensemble mean, were post-processed.

    Figure 9.  Comparison of dry season and wet season ensemble predictions for rainy climates: (a) correlation coefficient; (b) relative root-mean-square error; (c) bias score.

    Figure 10.  Comparison of grand raw and post-processed ensemble predictions: (a) continuous ranked probability score; (b) Brier skill score; (c) mean absolute error (mm d-1).

    To evaluate the performance of the BMA prediction model in both deterministic and probabilistic forecasts, MAE was used to measure the former skill, while CRPS and BSS were selected to measure the latter skill. Figure 10 presents the results, showing the prediction capability improved after post-processing. This implies that raw TIGGE ensemble predictions must be post-processed to be used in hydrological applications. Furthermore, the grand ensemble prediction showed better performance compared to individual model predictions.

5. Summary and conclusions
  • In this paper, TIGGE numerical ensemble precipitation predictions of the UKMO, NCEP and ECMWF centers for the Iran region were extracted with lead times of one, two and three days over the period 2008-16. To spatially break down the evaluation process, eight precipitation regions, as classified by (Modarres, 2006), were adopted. Deterministic, dichotomous (yes/no), and probabilistic evaluations were carried out for 13 selected stations in eight homogenous regions. The major findings were as follows:

    (1) Comparison of the observed annual precipitation in the wet season at each station with the predicted values indicates that, in rainy regions, such as G6 and G8, the predicted precipitation overestimated the observed in most years. Conversely, annual precipitation was underestimated in other regions subject to a drier climate. ECMWF's predictions were closer to the observations in most regions, while UKMO predicted the annual precipitation quite well, mainly in mountainous regions, such as at the stations of Shahrekord and Sanandaj. Interestingly, UKMO underestimated precipitation in regions of high precipitation but overestimated observations in low-precipitation regions. NCEP predicted annual precipitation better than UKMO and ECMWF over the rim of the Persian Gulf in the G4 region.

    (2) Based on deterministic evaluation, ECMWF performed best at most stations, while UKMO had better scores in mountainous regions, such as at Shahrekord and Sanandaj. Additionally, NCEP performed best in the G4 region.

    (3) According to dichotomous (yes/no) evaluations and the BIAS indicator, all centers over-predicted the number of precipitation events, being much higher at some stations, such as in Bandar Abbas, with rare precipitation. In general, UKMO performed very poorly in precipitation occurrence/non-occurrence, compared to those of the other two centers. Moreover, NCEP performed better compared to UKMO and ECMWF.

    (4) According to probabilistic evaluations, which represent the occurrence probability, reliability, and prediction quality, ECMWF had better scores, with NCEP coming close and UKMO rated last. Based on BSS, all centers were weaker than their climatology.

    (5) Two- and three-day lead time predictions were also evaluated and, as expected, these predictions showed poorer skill compared to those of the one-day predictions.

    (6) Comparing the wet and dry seasons in rainy regions, the evaluations showed that all three centers had better skill in the wet than in the dry season. This may indicate that it is easier to predict rainfall occurrence during the wet season.

    (7) Ensemble predictions of the three centers were post-processed using BMA, constituting a grand ensemble prediction. The evaluation results showed that the quality of predictions improved considerably over the raw ensemble prediction.

    All in all, it can be stated that, over Iran, ECMWF performs better compared to UKMO and NCEP. Overall, however, the evaluation scores returned a "medium" result, suggesting that precipitation predictions must be post-processed before application in operational forecasts.

Reference

Catalog

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return