Advanced Search
Article Contents

An Assessment of the Predictability of the East Asian Subtropical Westerly Jet Based on TIGGE Data


doi: 10.1007/s00376-014-4026-2

  • The predictability of the position, spatial coverage and intensity of the East Asian subtropical westerly jet (EASWJ) in the summers of 2010 to 2012 was examined for ensemble prediction systems (EPSs) from four representative TIGGE centers, including the ECMWF, the NCEP, the CMA, and the JMA. Results showed that each EPS predicted all EASWJ properties well, while the levels of skill of all EPSs declined as the lead time extended. Overall, improvements from the control to the ensemble mean forecasts for predicting the EASWJ were apparent. For the deterministic forecasts of all EPSs, the prediction of the average axis was better than the prediction of the spatial coverage and intensity of the EASWJ. ECMWF performed best, with a lead of approximately 0.5-1 day in predictability over the second-best EPS for all EASWJ properties throughout the forecast range. For probabilistic forecasts, differences in skills among the different EPSs were more evident in the earlier part of the forecast for the EASWJ axis and spatial coverage, while they departed obviously throughout the forecast range for the intensity. ECMWF led JMA by about 0.5-1 day for the EASWJ axis, and by about 1-2 days for the spatial coverage and intensity at almost all lead times. The largest lead of ECMWF over the relatively worse EPSs, such as NCEP and CMA, was approximately 3-4 days for all EASWJ properties. In summary, ECMWF showed the highest level of skill for predicting the EASWJ, followed by JMA.
  • 加载中
  • Chen Y., P. M. Zhai, 2013a: Two types of typical circulation pattern for persistent extreme precipitation in Central-Eastern China. Quart. J. Roy. Meteor. Soc., doi: 10.1002/qj.2231.
    Chen Y., P. M. Zhai, 2013b: Persistent extreme precipitation events in China during 1951-2010. Climate Res., 57, 143- 155.
    Du Y., Y. C. Zhang, Z. Q. Xie, 2008: Impacts of longitude location changes of East Asian westerly jet core on the precipitation distribution during Meiyu period in middle-lower reaches of Yangtze River valley. Acta Meteorologica Sinica, 66, 566- 576. (in Chinese)
    Du Y., Y. C. Zhang, Z. Q. Xie, 2009: Location variation of the East Asia subtropical westerly jet and its effect on the summer precipitation anomaly over eastern China. Chinese J. Atmos. Sci., 33, 581- 592. (in Chinese)
    Froude, L. S. R., 2009: Regional differences in the prediction of extratropical cyclones by the ECMWF Ensemble Prediction System. Mon. Wea. Rev., 137, 893- 911.
    Froude, L. S. R., 2011: TIGGE: Comparison of the prediction of southern hemisphere extratropical cyclones by different ensemble prediction systems. Wea.Forecasting, 26, 388- 398.
    Hagedorn R., R. Buizza, T. M. Hamill, M. Leutbecher, T. N. Palmer, 2012: Comparing TIGGE multimodel forecasts with reforecast-calibrated ECMWF ensemble forecasts. Quart. J. Roy. Meteor. Soc., 138, 1814- 1827.
    Hamill T. M., 2012: Verification of TIGGE multimodel and ECMWF reforecast-calibrated probabilistic precipitation forecasts over the contiguous United States. Mon. Wea. Rev., 140, 2232- 2252.
    He.Y.l Coauthors, 2010: Ensemble forecasting using TIGGE for the July-September 2008 floods in the Upper Huai catchment: A case study. Atmos. Sci. Lett., 11, 132- 138.
    Hersbach H., 2000: Decomposition of the continuous ranked probability score for ensemble prediction systems. Wea. Forecasting, 15, 559- 570.
    Ji Y. M., J. Chen, M. Y. Jiao, C. P. Chen, 2011: The preliminary experiment of GRAPES-MESO ensemble prediction based on TIGGE data. Meteor. Mon., 37, 392- 402. (in Chinese)
    Jin R. H., W. J. Li, B. Zhang, C. X. Yan, 2012: A study of the relationship between East Asia subtropical westerly jet and abnormal Meiyu in the middle-lower reaches of the Yangtze River. Chinese J. Atmos. Sci., 36, 722- 732. (in Chinese)
    Koch P., 2004: Novel Perspectives of Jet-Stream Climatologies and Events of Heavy Precipitation on the Alpine Southside. Logos Verlag, 217 pp.
    Krishnamurti T. N., C. M. Kishtawal, D. W. Shin, C. Eric Williford, 2000: Improving tropical precipitation forecasts from a multianalysis superensemble. J.Climate, 13, 4217- 4227.
    Kuang X. Y., Y. C. Zhang, 2006: Impact of position abnormalities of the East Asian subtropical westerly jet on summer precipitation in Middle-Lower reaches of Yangtze River. Plateau Meteor., 25, 382- 389. (in Chinese)
    Lorenz E. N., 1963: Deterministic nonperiodic flow. J. Atmos. Sci., 20, 130- 141.
    Mason S. J., 2004: On using "Climatology" as a reference strategy in the brier and ranked probability skill scores. Mon. Wea. Rev., 132, 1891- 1895.
    Niu R. Y., P. M. Zhai, 2013: Synoptic verification of medium-extended-range forecasts of the Northwest Pacific subtropical high and South Asian high based on multi-center TIGGE data. Acta Meteorologica Sinica, 27, 725- 741.
    Niu R. Y., Z. G. Zhang, R. H. Jin, 2012: The atmospheric circulation features of two persistent heavy rainfalls over southern China in the summer of 2010. J. Appl. Meteor. Sci., 23, 385- 394. (in Chinese)
    Park Y. Y., B. Roberto, M. Leutbecher, 2008: TIGGE: Preliminary results on comparing and combining ensembles. Quart. J. Roy. Meteor. Soc., 134, 2029- 2050.
    Sun F. H., Y. C. Zhang, L. L. Guo, 2009: Relationship between the East Asia subtropical westerly jet anomaly and summer precipitation over eastern China. Plateau Meteor., 28, 1308- 1315. (in Chinese)
    Tao S. Y., Y. J. Zhao, X. M. Chen, 1958: The relationship between plum rains in far east and the behavior of circulation over Asia. Acta Meteorologica Sinica, 29, 119- 134. (in Chinese)
    Taylor K. E., 2001: Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res., 106, 7183- 7192.
    Toth Z., E. Kalnay, 1993: Ensemble forecasting at NMC: The generation of perturbations. Bull. Amer. Meteor. Soc., 174, 2317- 2330.
    Toth Z., O. Talagrand , G. Cand ille, Y. Zhu, 2003: Probability and ensemble forecasts. Forecast Verification: A Practitioner's Guide in Atmospheric Science, I. T. Jolliffe and D. B. Stephenson, Eds., Wiley, 137- 163.
    Wedam G. B., L. A. McMurdie, C. F. Mass, 2009: Comparison of model forecast skill of sea level pressure along the east and west coasts of the United States. Wea. Forecasting, 24, 843- 854.
    Ye D. Z., S. Y. Tao, M. C. Li, 1958: The catastrophe of general circulation in June and October. Acta Meteorologica Sinica, 29, 250- 263.
    Zhang Y. C., L. L. Guo, 2005: Relationship between the simulated East Asian westerly jet biases and seasonal evolution of rain belt over eastern China. Chinese Science Bulletin, 50, 1503- 1508. (in Chinese)
    Zhao L. N., H. Wu, F. Y. Tian, D. Qi, J. Y. Di, Q. Y. Duan, Z. Wang, 2010: Assessment of probabilistic precipitation forecasts for the Huaihe Basin using TIGGE data. Meteorological Monthly, 36, 133- 142. (in Chinese)
    Zheng F., J. Zhu, H. Wang, 2007: The verifications for ENSO ensemble prediction system. Climatic Environ. Res., 12, 587- 594. (in Chinese)
    Zhi X. F., H. X. Qi, Y. Q. Bai, C. Z. Lin, 2012: A comparison of three kinds of multimodel ensemble forecast techniques based on the TIGGE data. Acta Meteorologica Sinica, 26, 41- 51.
    Zhu Y. J., Z. Toth, 2008: Ensemble based probabilistic forecast verification. 19th Conf. on Probability and Statistics, New Orleans, LA, Amer. Meteor. Soc., 20- 24.
  • [1] ZHU Jiangshan, Fanyou KONG, LEI Hengchi, 2012: A Regional Ensemble Forecast System for Stratiform Precipitation Events in Northern China. Part I: A Case Study, ADVANCES IN ATMOSPHERIC SCIENCES, 29, 201-216.  doi: 10.1007/s00376-011-0137-1
    [2] ZHU Jiangshan, KONG Fanyou, LEI Hengchi, 2013: A Regional Ensemble Forecast System for Stratiform Precipitation Events in the Northern China Region. Part II: Seasonal Evaluation for Summer 2010, ADVANCES IN ATMOSPHERIC SCIENCES, 30, 15-28.  doi: 10.1007/s00376-012-1043-x
    [3] Saleh AMINYAVARI, Bahram SAGHAFIAN, Majid DELAVAR, 2018: Evaluation of TIGGE Ensemble Forecasts of Precipitation in Distinct Climate Regions in Iran, ADVANCES IN ATMOSPHERIC SCIENCES, 35, 457-468.  doi: 10.1007/s00376-017-7082-6
    [4] GUO Lanli, ZHANG Yaocun, WANG Bin, LI Lijuan, ZHOU Tianjun, BAO Qing, 2008: Simulations of the East Asian Subtropical Westerly Jet by LASG/IAP AGCMs, ADVANCES IN ATMOSPHERIC SCIENCES, 25, 447-457.  doi: 10.1007/s00376-008-0447-0
    [5] KUANG Xueyuan, ZHANG Yaocun, 2005: Seasonal Variation of the East Asian Subtropical Westerly Jet and Its Association with the Heating Field over East Asia, ADVANCES IN ATMOSPHERIC SCIENCES, 22, 831-840.  doi: 10.1007/BF02918683
    [6] Kazutoshi SATO, Jun INOUE, Akira YAMAZAKI, Naohiko HIRASAWA, Konosuke SUGIURA, Kyohei YAMADA, 2020: Antarctic Radiosonde Observations Reduce Uncertainties and Errors in Reanalyses and Forecasts over the Southern Ocean: An Extreme Cyclone Case, ADVANCES IN ATMOSPHERIC SCIENCES, 37, 431-440.  doi: 10.1007/s00376-019-8231-x
    [7] MA Juhui, Yuejian ZHU, Richard WOBUS, Panxing WANG, 2012: An Effective Configuration of Ensemble Size and Horizontal Resolution for the NCEP GEFS, ADVANCES IN ATMOSPHERIC SCIENCES, 29, 782-794.  doi: 10.1007/s00376-012-1249-y
    [8] Zhaorong ZHUANG, Nusrat YUSSOUF, Jidong GAO, 2016: Analyses and Forecasts of a Tornadic Supercell Outbreak Using a 3DVAR System Ensemble, ADVANCES IN ATMOSPHERIC SCIENCES, 33, 544-558.  doi: 10.1007/s00376-015-5072-0
    [9] Jie FENG, Jianping LI, Jing ZHANG, Deqiang LIU, Ruiqiang DING, 2019: The Relationship between Deterministic and Ensemble Mean Forecast Errors Revealed by Global and Local Attractor Radii, ADVANCES IN ATMOSPHERIC SCIENCES, 36, 271-278.  doi: 10.1007/s00376-018-8123-5
    [10] Swapan MALLICK, Devajyoti DUTTA, Ki-Hong MIN, 2017: Quality Assessment and Forecast Sensitivity of Global Remote Sensing Observations, ADVANCES IN ATMOSPHERIC SCIENCES, 34, 371-382.  doi: 10.1007/s00376-016-6109-8
    [11] ZHU Jiang, LIN Caiyan, WANG Zifa, 2009: Dust Storm Ensemble Forecast Experiments in East Asia, ADVANCES IN ATMOSPHERIC SCIENCES, 26, 1053-1070.  doi: 10.1007/s00376-009-8218-0
    [12] TAN Jiqing, XIE Zhenghui, JI Liren, 2003: A New Way to Predict Forecast Skill, ADVANCES IN ATMOSPHERIC SCIENCES, 20, 837-841.  doi: 10.1007/BF02915409
    [13] XIAO Ziniu, LIU Hua, ZHANG De, 2012: Progress in Climate Prediction and Weather Forecast Operations in China, ADVANCES IN ATMOSPHERIC SCIENCES, 29, 943-957.  doi: 10.1007/s00376-012-1194-9
    [14] Guokun DAI, Mu MU, Zhina JIANG, 2019: Evaluation of the Forecast Performance for North Atlantic Oscillation Onset, ADVANCES IN ATMOSPHERIC SCIENCES, , 753-765.  doi: 10.1007/s00376-019-8277-9
    [15] Isidora JANKOV, Scott GREGORY, Sai RAVELA, Zoltan TOTH, Malaquías PEÑA, 2021: Partition of Forecast Error into Positional and Structural Components, ADVANCES IN ATMOSPHERIC SCIENCES, 38, 1012-1019.  doi: 10.1007/s00376-021-0251-7
    [16] Yuejian ZHU, 2005: Ensemble Forecast: A New Approach to Uncertainty and Predictability, ADVANCES IN ATMOSPHERIC SCIENCES, 22, 781-788.  doi: 10.1007/BF02918678
    [17] Keon Tae SOHN, Sun Min PARK, 2008: Guidance on the Choice of Threshold for Binary Forecast Modeling, ADVANCES IN ATMOSPHERIC SCIENCES, 25, 83-88.  doi: 10.1007/s00376-008-0083-8
    [18] Keon Tae SOHN, Jeong Hyeong LEE, Young Seuk CHO, 2009: Ternary Forecast of Heavy Snowfall in the Honam Area, Korea, ADVANCES IN ATMOSPHERIC SCIENCES, 26, 327-332.  doi: 10.1007/s00376-009-0327-2
    [19] T. N. Krishnamurti, Mukul Tewari, Ed Bensman, Wei Han, Zhan Zhang, William K. M. Lau, 1999: An Ensemble Forecast of the South China Sea Monsoon, ADVANCES IN ATMOSPHERIC SCIENCES, 16, 159-182.  doi: 10.1007/BF02973080
    [20] Song YANG, WEN Min, Rongqian YANG, Wayne HIGGINS, ZHANG Renhe, 2011: Impacts of Land Process on the Onset and Evolution of Asian Summer Monsoon in the NCEP Climate Forecast System, ADVANCES IN ATMOSPHERIC SCIENCES, 28, 1301-1317.  doi: 10.1007/s00376-011-0167-8

Get Citation+

Export:  

Share Article

Manuscript History

Manuscript received: 10 February 2014
Manuscript revised: 16 June 2014
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

An Assessment of the Predictability of the East Asian Subtropical Westerly Jet Based on TIGGE Data

  • 1. State Key Laboratory of Severe Weather, Chinese Academy of Meteorological Sciences, China Meteorological Administration, Beijing 100081
  • 2. National Meteorological Center, China Meteorological Administration, Beijing 100081

Abstract: The predictability of the position, spatial coverage and intensity of the East Asian subtropical westerly jet (EASWJ) in the summers of 2010 to 2012 was examined for ensemble prediction systems (EPSs) from four representative TIGGE centers, including the ECMWF, the NCEP, the CMA, and the JMA. Results showed that each EPS predicted all EASWJ properties well, while the levels of skill of all EPSs declined as the lead time extended. Overall, improvements from the control to the ensemble mean forecasts for predicting the EASWJ were apparent. For the deterministic forecasts of all EPSs, the prediction of the average axis was better than the prediction of the spatial coverage and intensity of the EASWJ. ECMWF performed best, with a lead of approximately 0.5-1 day in predictability over the second-best EPS for all EASWJ properties throughout the forecast range. For probabilistic forecasts, differences in skills among the different EPSs were more evident in the earlier part of the forecast for the EASWJ axis and spatial coverage, while they departed obviously throughout the forecast range for the intensity. ECMWF led JMA by about 0.5-1 day for the EASWJ axis, and by about 1-2 days for the spatial coverage and intensity at almost all lead times. The largest lead of ECMWF over the relatively worse EPSs, such as NCEP and CMA, was approximately 3-4 days for all EASWJ properties. In summary, ECMWF showed the highest level of skill for predicting the EASWJ, followed by JMA.

1. Introduction
  • A discrete atmospheric model is an approximation of the real atmosphere in mathematical and physical terms. There exists considerable uncertainty in such numerical weather prediction (NWP) systems, and this uncertainty tends to increase with an extension in lead time, resulting in a decline in forecast accuracy (Lorenz, 1963; Toth and Kalnay, 1993). TIGGE (THORPEX Interactive Grand Global Ensemble), constructed by THORPEX (The Observing System Research and Predictability Experiment), collects ensemble prediction data from the leading global NWP centers on a standardized platform, providing a convenient way to verify and evaluate multimodel ensemble prediction systems (EPSs), and thus contributes to improvements in the accuracy of 1-day to 2-week high impact weather forecasts. The verification, evaluation and comparison of different EPSs has become a globally important research topic. Many verification methods have been introduced (Toth et al., 2003; Zhu and Toth, 2008) and preliminary results that compare and combine ensembles for predictions of 500 hPa geopotential height and 850 hPa temperature have been achieved by (Park et al., 2008). (Zhi et al., 2012) studied multimodel ensemble forecasts of temperature over the Northern Hemisphere. (He et al., 2010) revealed that TIGGE ensemble forecasts exhibit satisfactory flood forecasting skill, with clear signals of floods up to 10 days in advance. (Hagedorn et al., 2012) used a reduced multimodel system to improve the performance of the best single model in forecasting 850 hPa and 2 m temperatures. Various other analyses and comparisons of EPSs for the forecasting of precipitation, cyclones, and sea surface pressure have also been performed (Krishnamurti et al., 2000; Wedam et al., 2009; Zhao et al., 2010; Froude, 2011; Ji et al., 2011, Hamill, 2012). (Niu and Zhai, 2013) evaluated the performances of four EPSs for predicting the Northwest Pacific subtropical high and the South Asian high. However, relatively few studies have focused on the East Asian subtropical westerly jet (EASWJ).

    The subtropical westerly jet is a strong and narrow high-speed airflow band existing in the upper troposphere of the subtropical region in the Northern and Southern Hemispheres. The jet stream flow is quasi-horizontal in three-dimensional space and exhibits strong vertical and horizontal lateral wind shear. The EASWJ is an essential part of the East Asian summer monsoon and generally corresponds with the upper-level frontal zone, meaning the EASWJ has important impacts not only on general circulation in East Asia but also on local severe weather (Ye et al., 1958; Koch, 2004). The onset, cessation, location and amount of precipitation in East Asia are each affected by the position and intensity of the EASWJ (Tao et al., 1958; Zhang and Guo, 2005; Kuang and Zhang, 2006; Du et al., 2008; Sun et al., 2009; Du et al., 2009). The establishment and shifting of the upper-level jet also plays an important role in the onset and cessation of persistent extreme precipitation (Niu et al., 2012; Chen and Zhai, 2013a). Studies suggest that the EASWJ is a key circulation system for persistent precipitation in eastern China, and analyzing the medium-range characteristics of the variation and evolution of its northward jump, intensity, and location is of great benefit for forecasting persistent precipitation at the same scale (Jin et al., 2012). Therefore, investigating the forecast abilities of EPSs with respect to the EASWJ is important for forecasters and has critical scientific relevance for improving medium-range forecasts of persistent precipitation in East Asia.

    In this study, we verified and compared the predictability of the position, spatial coverage, and intensity of the EASWJ according to different EPSs from four representative operational centers for the summers of 2010 to 2012. The strengths and weaknesses of each EPS are presented, and it is hoped that the results will be a useful reference for future improvements in the accuracy of 1-day to 2-week high-impact weather forecasts. Furthermore, we report results from a verification case study of the EASWJ to investigate the differences between the performances of each EPS during persistent and non-persistent extreme precipitation periods, and verify that the EASWJ is a favorable predictor of persistent extreme precipitation.

    This paper continues with a description of the data and methodology in section 2, followed by a verification of the overall activity of the EASWJ in section 3. The performance of deterministic and probabilistic forecasts for the EASWJ are verified and compared in sections 4 and 5. In section 6, the forecast performances in a period of persistent extreme precipitation are presented, and the paper finishes with a discussion and conclusions in section 7.

2. Data and methodology
  • We analyzed EPS data for the time period of 1 June to 31 August during 2010 to 2012 for four different operational weather centers: the European Center for Medium-Range Weather Forecasts (ECMWF); the National Centers for Environmental Prediction (NCEP); the China Meteorological Administration (CMA); and the Japan Meteorological Agency (JMA). Table 1 lists the key characteristics of each EPS. Besides, ERA-Interim data for the summers of 1980 to 2012 were also used, which are the latest global atmospheric reanalysis data produced by the ECMWF. The core of its simulation system is the use of 12-hour 4DVAR simulations in the upper atmosphere, and the horizontal resolution is T255 (∼79 km). The TIGGE data and ERA-Interim data were interpolated to a common grid spacing of 1°. Daily rainfall data for the summer of 2010 were obtained from the database of the National Meteorological Center in China.

  • 2.2.1 Calculation of indices for all EASWJ properties

    The average axis index of the EASWJ (Kuang and Zhang, 2006) was calculated as the average of the latitude with maximum westerly wind above 30 m s-1 at each longitude of the area (20°-60°N, 100°-125°E) at 200 hPa. If there was no grid reaching the threshold value of 30 m s-1 in the above-mentioned area, we reduced the threshold value to 20 m s-1 to calculate the average axis index of the EASWJ. The area index of the EASWJ was the total number of grid points with westerly wind speed above 30 m s-1 in the area (25°-55°N, 70°-125°E) at 200 hPa. The Intensity index of the EASWJ was computed as the average speed of zonal wind at all grid points with westerly wind speed above 30 m s-1 in the area (25°-55°N, 70°-125°E) at 200 hPa.

    2.2.2.Verification methods

    In this study, the control forecast started from a "central" analysis, which is usually an unperturbed analysis generated by a data assimilation procedure, and the ensemble mean forecasts were then derived by equal weighting of all ensemble members of each TIGGE center (Park et al., 2008). The performance of each EPS was assessed against its corresponding analysis (defined as the control forecast at step zero), i.e. the analysis of each EPS was taken as the corresponding observation data. The EASWJ was considered as active at a grid point when the westerly wind speed was no less than 30 m s-1 at 200 hPa. Whether or not an active EASWJ could be observed constituted a dichotomous event, so we used threat scores to verify the overall activity of the EASWJ:

    The threat score (TS) was used to assess the ability of each forecast system to predict categorical events. H,A and M are the number of hits, false alarms and misses, respectively. If an active EASWJ occurred in both the EPS forecast and its analysis, this was regarded as a "hit" event. False alarms and misses were deduced in the same way: \begin{equation} \label{eq1} { TS}=\dfrac{H}{H+A+M} . (1)\end{equation}

    We chose mean error and Taylor diagrams to assess the performance of the deterministic forecasts for the position, spatial coverage and intensity of the EASWJ for all EPSs. The mean error (ME)——the range of which is \(-\infty\) to \(\infty\) and a perfect score is 0——was used to reflect the overall forecast error tendencies. Fi and Oi are the ith values in the forecast and observation sequences, while n is the sequence length: \begin{equation} { ME}=\dfrac{1}{n}\sum_{i=1}^n(F_i-O_i) . (2)\label{eq2} \end{equation}

    Taylor diagrams (Taylor, 2001) can display correlation, standard deviation, and centered root-mean-square difference on a single diagram to (in our case) graphically summarize how closely the forecasts of the EPSs matched observations. The standard deviation of the forecast relative to observations, which is proportional to the radial distance from the origin, is defined as \begin{eqnarray} \label{eq3} \sigma&=&\sigma_{ f}/\sigma_{ o} ,(3)\\ \label{eq4} \sigma_{ f}&=&\sqrt{\dfrac{1}{n}\sum_{i=1}^n(F_i-\overline {F})^2}, \sigma_{ o}=\sqrt{\dfrac{1}{n}\sum_{i=1}^n(O_i-\overline {O})^2} , (4)\end{eqnarray} where σ f is the standard deviation of the forecast and σ o is the standard deviation of observations. \(\overline F\) and \(\overline O\) represent the mean values of the forecast and observation sequences respectively.

    The correlation coefficient between the forecast field and observation field was computed by the cosine of the azimuthal angle: \begin{equation} \label{eq5} R=\dfrac{\sum\limits_{i=1}^n(F_i-\overline {F})(O_i-\overline {O})}{\sqrt{\sum\limits_{i=1}^n(F_i-\overline {F})^2}\sqrt{\sum\limits_{i=1}^n(O_i-\overline {O})^2}} . (5)\end{equation} According to the correlation coefficient (R) and the normalized standard deviation, we used the Law of Cosines to obtain the centered root-mean-square (CRMS), given by the distance from the forecast field to the observation field: \begin{equation} \label{eq6} { CRMS}=\dfrac{\sqrt{\frac{1}{n}\sum\limits_{i=1}^n[(F_i-\overline {F})-(O_i-\overline {O})]^2}}{\sigma_{ o}} . (6)\end{equation}

    In the Taylor diagrams, a forecast point that is near to the observation point indicates close standard deviation, small CRMS, and large R between the two fields. Thus, for a given EPS, the closer its forecast point was to the observation point, the better the performance of the EPS.

    Figure 1.  TSs of EASWJ activity in the area with >10 days in summer (2010-12) when the EASWJ was active. The red solid lines indicate the TS of the ensemble mean forecast; colored shading denotes the differences between the ensemble mean forecast and the control forecast.

    For probabilistic forecasts, which consist of forecasts of all members of each EPS, the verification statistics we used included the continuous ranked probability skill score and the score of the relative operating characteristics curve. The continuous ranked probability score (CRPS) is a diagnostic measure focusing on the entire possible range of the examined variable. Its skill score is the continuous ranked probability skill score (CRPSS) (Hersbach, 2000; Toth et al., 2003). The CRPS evolved from the Brier score. It integrates the squared difference between the cumulative probability distribution of forecasts and observations for all possible thresholds: \begin{eqnarray} \label{eq7} { CRPS}&=&\int_{-\infty}^{+\infty}[F(x)-O(x)]^2dx ,(7)\\ \label{eq8} F(x)&=&\int_{-\infty}^x\rho(y)dy ,(8)\\ \label{eq9} O(x)&=&\left\{ \begin{array}{ll} 0,& x<x_{ a} \\[2mm] 1,& x>=x_{ a} \end{array} \right.. (9)\end{eqnarray} F(x) and O(x) are the cumulative probability distributions of forecasts and observations respectively; ρ(x) is the probability density distribution of forecasts; and x a is the observational data. Forecasts of all members of each EPS comprised a set of values of the variable we examined, and the probability density distribution was calculated using these values. The CRPS denotes the error of the cumulative probability distribution instead of the forecast ability of the EPS, so we chose a reference forecast, which is usually the climatology (Mason, 2004) or persistence, to carry out a comparison. A three-year sample mean was used in our work as the reference forecast (see section 5.1 for details). Accordingly, we were able to obtain the CRPSS, which ranges between \(-\infty\) and 1, and where a perfect score is 1. A CRPSS above 0 indicates the EPS possesses forecast ability relative to the reference forecast: \begin{equation} \label{eq10} { CRPSS}=1-\dfrac{\overline {{ CRPS}_{ f}}}{\overline {{ CRPS}_{ r}}} . (10)\end{equation} Relative operating characteristics (ROC) can examine categorical events (Zheng et al., 2007). Using observations to verify forecasts, the result would be one of the following situations: hit; false alarm; miss; correct negative. The hit rate (HR) and false alarm rate (FAR), which are also referred to as the probability of detection (POD) and the probability of false detection (POFD) respectively, are calculated as \begin{equation} \label{eq11} { POD}=\dfrac{H}{H+M} , { POFD}=\dfrac{A}{C_{ n}+A} , (11)\end{equation} where H,A,M and C n are the number of hits, false alarms, misses, and correct negatives respectively. For the axis, spatial coverage and intensity of the EASWJ, we took departures between these indices and a 30-yr climatology less than certain standards as an event (see section 5.2 for further details). By investigating whether or not the event occurred in the forecast and observation, we obtained H,A,M and C n. For forecasts of all members in an EPS, we obtained a probability of the event occurring. By changing the probability threshold (0%-100%) of probabilistic forecasts to accept the occurrence of an event, one can achieve a range of HRs and FARs. The ROC curve is the result of plotting HRs against FARs, and the area under the ROC curve is frequently used as its score. When the ROC curve is exactly diagonal with a score of 0.5, this means there is no decision-making skill. If the ROC curve falls below the diagonal, with its score being less than 0.5, this indicates that the forecast skill is lower than random probabilistic forecasts. The more the ROC curve bends to the left of the diagonal, the larger than 0.5 its score is, and the better the performance of the forecast.

3. Forecast verification of the overall activity of the EASWJ
  • As shown in Fig. 1, for the four TIGGE EPSs, the distribution of averaged active days of the EASWJ (the number of days with westerly wind speed ≥30 m s-1 at 200 hPa at each grid point) in summer over the period 2010-12 was quite similar. The favorable active range of EASWJ concentrated mainly in the latitude band of 35°-45°N, in which area the highest frequency of the EASWJ activity was up to about 50 days. All of the four TIGGE EPSs showed appreciable forecast skill for the EASWJ in summer throughout the forecast range, with the TS reaching up to 0.8 at a lead time of 3 days in the relatively more active area of the EASWJ. The TS gradually decreased for all EPSs with increasing lead time, indicating that the level of skill in predicting the activity of the EASWJ declines as the lead time extends. The area with higher TSs usually corresponded to the more active EASWJ area, suggesting that all four EPSs provide more accurate predictions in the area with a high frequency of activity, as compared to peripheral areas. The colored shading in Fig. 1 shows that there was some improvement in the prediction of EASWJ activity from the control to the ensemble mean forecasts within a lead time of 9 days in most of the area with averaged active days above 10. However, for ECMWF and NCEP, the area where the ensemble mean had an advantage over the control forecasts greatly reduced at the lead time of 15 days. These results based on a comparison of TSs show that ECMWF and JMA perform better in forecasting EASWJ activity.

4. Performance of deterministic forecasts
  • We analyzed the overall forecast error tendencies of the position, spatial coverage and intensity of the EASWJ (represented by the average axis index, area index and intensity index) using mean errors (Fig. 2). For the position of the EASWJ, the mean forecast errors of the ensemble mean and control forecasts were close for all four EPSs. The ECMWF forecast was to the north of its analysis, while the NCEP and JMA forecasts generally diverged to the south. Mean errors of the CMA forecast oscillated around zero with an increase in lead time. As for the spatial coverage and intensity of the EASWJ, the ECMWF and JMA ensemble mean forecasts underestimated these for all lead times, except the JMA forecast for spatial coverage on day 1 of the lead times. The CMA and NCEP ensemble mean forecasts exhibited an overestimation of these features in the early forecast range, and both turned to an underestimation around the lead time of 3 days. The negative errors of all ensemble mean forecasts showed a dramatic increase. In general, the negative errors of the ECMWF ensemble mean forecasts were larger than those of other ensemble mean forecasts for spatial coverage and intensity. Mean forecast errors of spatial coverage and intensity for all control forecasts showed no obvious change. The spatial coverage and intensity predicted by these control forecasts all remained close to their respective analyses. Among them, the CMA and NCEP control forecasts were slightly larger and stronger than their respective analyses, while the JMA control forecast was slightly smaller and weaker than its analysis.

  • The Taylor diagrams of all EASWJ properties for all EPSs are shown in Fig. 3. The Taylor diagram provides a statistical summary of how well patterns match each other, which is especially useful in evaluating multiple aspects of complex models or for gauging the relative skill of many different models. In the Taylor diagram, the closer the forecast field of an EPS is to the reference field (represented by its own analysis), the better its performance. Whether for the ensemble mean or the control forecasts of all EPSs, the distance between the forecasts fields of all EASWJ properties and their reference fields basically showed an increase as lead time extended. Thus, the corresponding forecast performance of all EPSs becomes poorer with increased lead time. For the position of the EASWJ (Figs. 3a1 and a2), the variations of its shift in the ensemble mean forecast and the respective analysis for all EPSs were consistent to a high degree, with all showing a correlation coefficient above 0.7. If we measure the number of skill days as the longest forecast lead days with R≥0.6, then the number of skill days of all ensemble mean forecasts for all the EPSs examined could reach their own longest forecast lead time. The variability of the ensemble mean forecast of the position of the EASWJ for each EPS was small relative to their analyses, and became increasingly smaller with an extension of the lead time. This reveals that the level of skill in predicting variations of the position of the EASWJ declines simultaneously with the lead time extends. The ensemble mean forecasts of the different EPSs displayed little advantage over their control forecasts within a lead time of 5 days, while the performances of the ensemble mean forecasts increasingly improved relative to their control forecasts beyond the lead time of day 5 for the position of the EASWJ. The control forecasts predicted variations in the position of the EASWJ slightly better than the ensemble mean forecasts for all EPSs. For the control forecast, ECMWF showed an overestimation of the variations, while the other EPSs exhibited an underestimation. By comparing EPSs' ensemble mean forecasts which showed superiority over their control forecasts, it was clear that ECMWF possessed the highest level of skill in terms of the position of the EASWJ, because of its smaller CRMS and larger R. JMA was next best, while the performances of NCEP and CMA were relatively inferior. Overall, for the position of the EASWJ, the results showed ECMWF to have the equivalent of about a 0.5-1-day lead in predictability when compared with the second-best, JMA, for all lead times. It was also noted that the level of skill of JMA declined faster than the other EPSs between day 1 and 2 of the forecast time, both for the ensemble mean and control forecasts. Additionally, the largest lead of ECMWF over NCEP and CMA in predictability reached as long as 3-4 days (i.e. the CRMS and R of the ECMWF ensemble mean forecasts at a lead time of 13 days were comparable to those of NCEP and CMA at a lead time of 9 days).

    Figure 2.  Mean forecast errors of all EPSs during the summers of 2010-12 for the (a) position, (b) spatial coverage, and (c) intensity of the EASWJ (pf, ensemble mean forecasts; cf, control forecast).

    Figure 3.  Taylor diagrams of all EPSs during the summers of 2010-12 for the (a) position, (b) spatial coverage, and (c) intensity of the EASWJ. The horizontal and vertical axes both represent standard deviation normalized by the reference field, and the radial lines are labeled by the cosine of the angle made with the horizontal axis. The analysis of each EPS is used as the reference field, plotted as `ref' in the diagrams. The distance from the forecast field to the coordinate axis origin means the standard deviation compared to the ref; the correlation coefficient between the forecast field and the ref is shown by the cosine of the azimuthal angle; and its centered root-mean-square difference is given by the distance from the forecast field to the ref. Panels (a1, b1, c1) are the ensemble forecasts and panels (a2, b2, c2) are the control forecasts. The number above each forecast point denotes the lead time.

    The Taylor diagram of the area index of the EASWJ (Figs. 3b1 and b2) showed that the forecast performance in terms of the spatial coverage of the EASWJ for all EPSs was not as good as that of the position of the EASWJ at longer lead times. For the spatial coverage of the EASWJ, the skill days of the JMA and CMA ensemble mean forecasts can reach their own longest forecast lead time. However, for ECMWF and NCEP, the number of ensemble mean forecast skill days could only reach 13 and 11 days respectively, which were both shorter than in the prediction of the position of the EASWJ. For the spatial coverage of the EASWJ, the ensemble mean forecasts of all EPSs overpredicted the variations, and the variations predicted by the control forecasts were closer to their analyses. However, the performance of the control forecasts was slightly worse than the ensemble mean forecasts for spatial coverage when considering R and CRMS. Within the forecast range of 1-9 days, ECMWF performed best in the forecasts of spatial coverage. JMA was next best, followed by CMA and NCEP. ECMWF showed higher skill than NCEP in the longer forecast range. For spatial coverage, ECMWF led JMA in predictability by about 0.5-1 day, and the lead increased to around 1-2 days when comparing ECMWF with CMA and NCEP throughout the forecast range.

    For the intensity of the EASWJ (Figs. 3c1 and c2), the number of skill days of JMA reached its longest forecast lead time. As for the ECMWF, NCEP and CMA ensemble mean forecasts, the numbers of skill days were 10, 11 and 8 days respectively. Variations in the intensity predicted by the ensemble mean forecasts of all EPSs were close to their analyses for the lead time of 1-6 days, but beyond that time the variations predicted by the ensemble mean forecasts turned increasingly smaller, while the variations in intensity predicted by the ECMWF and NCEP control forecasts simultaneously became increasingly larger than their respective analyses. On the whole, the ensemble mean was also found to be superior to the control forecasts for the intensity of the EASWJ. ECMWF provided the most accurate predictions of the intensity with a lead time up to 9 days, and in this period ECMWF had a lead in predictability of about 0.5-1 days over JMA and NCEP, and about 2-2.5 days over CMA. In the later part of the forecast range, NCEP was more skillful than ECMWF for the intensity forecast.

5. Probabilistic forecast skill
  • In this section, we report the results from computing the CRPS of the indices of the axis at 120°E, the area, and the intensity of the EASWJ. CRPS measures the sum of squared differences in cumulative probability space for a probabilistic forecast, and the CRPSS denotes the skill of predicting the cumulative probability distribution. For investigating the forecast abilities of each EPS, ERA-Interim data averaged from 2010 to 2012 were selected as the reference forecast. We computed the CRPS of ERA-Interim data in the same way as for the forecast of each EPS, and achieved the CRPSS by comparing the CRPSs of the EPS forecast data and ERA-Interim data. Apart from the forecasts of spatial coverage by NCEP at the lead time of days 14 and 15, the errors of the probabilistic forecasts of all EPSs for all EASWJ properties were smaller than those of the reference forecast, with a CRPSS greater than 0. This suggested that all the EPSs possess skill relative to the reference forecast (Fig. 4). Differences between the skills of the different EPSs for predicting the EASWJ axis were more evident in the forecast range of 2-6 days, while for spatial coverage the differences were distinct in the first two lead days, and for intensity the differences departed obviously throughout the forecast range. ECMWF showed the highest level of skill for either the axis or the spatial coverage and intensity of the EASWJ at almost every forecast time. For the EASWJ axis, ECMWF led the second-best JMA in predictability by about 0.5-1 day, and led CMA by 1-2 days for the lead time of 1-7 days. The lead of ECMWF over NCEP extended as long as 3 days. As for the spatial coverage and intensity of the EASWJ, ECMWF produced a lead in predictability of about 1-2 days over JMA at almost every forecast lead time. Overall, the prediction of CMA was found to be slightly inferior to JMA with respect to spatial coverage, except for the first two lead days, and at day 9 of the forecast, CMA even outperformed JMA. For the intensity of the EASWJ, NCEP generally fell behind ECMWF in predictability by 1-3 days throughout the forecast range, while the skill of CMA trailed by 3-4 days when compared with ECMWF. Considering the skill of each EPS in predicting all EASWJ properties, ECMWF provided the best probabilistic forecasts, followed by JMA and then NCEP and CMA.

  • According to the mean absolute errors of the different EPSs for all EASWJ properties (not shown), we chose the climatological mean of ERA-Interim data over the period 1980-2009 as the reference, and identified differences between the forecast or analysis and the climatological mean value of the jet axis of less than 1° as the norm. For the area and intensity indices, the difference between the forecast or analysis and the climatological mean value less than 20 and 1.5 m s-1,

    Figure 4.  CRPSSs of all EPSs during the summers of 2010-12 for the (a) axis, (b) spatial coverage, and (c) intensity of the EASWJ.

    respectively, were taken as the norm. We considered the occurrence of normal conditions in both the forecast and analysis as a "hit" event. Moreover, we selected 10%,20%…90% as probability thresholds to accept the occurrence of normal conditions of the forecast. By plotting a range of HRs against FARs, we obtained the ROC curve. The ROC score is the area under the ROC curve.

    The ROC score measures the ability of the forecast to discriminate between two alternative outcomes, but provides no information on reliability. Figure 5 presents the ROC scores of the indices of the axis at 120°E, the area and intensity of the EASWJ. Apart from the forecasts of spatial coverage by ECMWF, NCEP and CMA at the final day of the lead times, all EPSs possessed decision-making skill in terms of predicting all EASWJ properties, with ROC scores above 0.5, but the forecast performances of different EPSs showed large differences. For the EASWJ axis, ECMWF was generally the most skillful, with the highest ROC score, followed by JMA. An oscillation with large amplitude of the ROC scores for CMA and NCEP appeared at day 7 and 5 of the forecast time respectively. The performance of NCEP was comparable to ECMWF in the late forecast range. As for spatial coverage and the intensity of the EASWJ, the differences among the four EPSs were larger than that for the EASWJ axis at the first two days of the lead times, which was also reflected by the verification of the CRPSS. Overall, ECMWF took the lead, and JMA performed second best, followed by NCEP and CMA. CMA was comparable to NCEP for the forecasting of the spatial coverage within the forecast range of 1-7 days. JMA outperformed ECMWF at the lead time of 1-2 days, and turned to be comparable with NCEP beyond the lead time of 2 days for intensity. A slight periodic oscillation appeared for the ROC scores of CMA and NCEP for the spatial coverage and intensity. Based on the assessment and comparison of the four EPSs for all EASWJ properties, the ECMWF probabilistic forecast was found to provide the most accurate predictions of the EASWJ, while JMA performed second best. The performance differences revealed by comparing ROC scores were consistent with those established by verifying the CRPSSs.

6. Verification case study in a period of persistent extreme precipitation
  • The analysis and forecast results of the ECMWF ensemble mean forecast for the average EASWJ during a period of persistent extreme precipitation (Chen and Zhai, 2013b) over the Yangze-Huai river basin from 17 to 25 June 2010 are shown in Fig. 6. (Chen and Zhai, 2013b) revealed that the jet axis always shifts further southward when a persistent extreme precipitation event occurs above the region of 80°-130°E. The Yangze-Huai river basin is located right along the entrance region of the jet, which is characterized by strong divergence. The divergence contributes to conditions that are conducive to the initiation and maintenance of persistent extreme precipitation. Within lead times of 6 days, ECMWF basically provided accurate predictions of the position and spatial coverage of the EASWJ, but the intensity was slightly weak relative to the analysis. Deviations of the forecast for all EASWJ properties grew with increased lead time. Forecasts at day 9 of the lead time provided a rough position and an obviously weak intensity of the EASWJ. Even worse, the jet band parted in the middle. The performance of the forecast at a lead time of 15 days was poorer. Hence, forecasts of the EASWJ still have room to be improved in periods of persistent extreme precipitation. Taking ECMWF as an example, Fig. 7 shows how the average axis of the EASWJ over 100°-125°E changed with time in June when the persistent extreme precipitation event happened. Due to the fact that the upper-level jet can establish and shift ahead of the onset and cessation of persistent heavy precipitation, we focused on the performance of the EASWJ in the period ahead of the persistent heavy precipitation (between the black dashed lines in Fig. 7). The results showed that the position of the EASWJ remained stable ahead of the persistent heavy precipitation, and within a lead time of 9 days predictions of the average axis of the EASWJ were more accurate than in the remaining periods of June. The average spatial coverage predicted in the period between the black dashed lines was also closer to the analysis than during other periods. As a result, the ECMWF predictions of the EASWJ in the period when the EASWJ remained stable, ahead of the persistent extreme precipitation, were superior to those in other periods, confirming that the EASWJ is a useful predictor of persistent extreme precipitation. Therefore, further improving the forecasts of the EASWJ will bring great benefits to the accuracy of predicting persistent extreme precipitation in East Asia.

    Figure 5.  ROC scores of all EPSs during the summers of 2010-2012 for the (a) axis, (b) spatial coverage, and (c) intensity of the EASWJ.

    Figure 6.  Average EASWJ predicted by the ECMWF ensemble mean forecast during the period of a persistent heavy precipitation event from 17 to 25 June 2010. Colored shading shows the average EASWJ of the analysis, and the black dotted line is its axis. Purple, green, blue and orange contour lines are the forecasts at lead days 3, 6, 9 and 15 respectively.

    Figure 7.  Change in the average position of the EASWJ axis over 100°-125°E with time in June 2010. The blue line is the analysis and the red, green, black and orange lines are the forecasts of the average position of the EASWJ axis at lead days 3, 6, 9 and 15 respectively. Blue and red shading represent the average spatial coverage over 100°-125°E of the analysis and forecasts at day 3 of the forecast respectively. The green histogram indicates mean observed daily rainfall in the area (26°-27°N, 117°-118°E).

7. Discussion and conclusion
  • In this study, we performed a verification and comparison of four different EPSs (ECMWF, NCEP, JMA and CMA) for predicting the position, spatial coverage and intensity of the EASWJ. The conclusions can be summarized as follows.

    All four EPSs provided more accurate predictions of EASWJ activities in the area with a high frequency activity than in its peripheral areas. For both deterministic and probabilistic forecasts, all EPSs predicted the activity and all properties of the EASWJ well, while the levels of skill of all the EPSs declined as the lead time extended. Overall, the ensemble mean forecasts better predicted EASWJ activity and properties compared to the control forecasts.

    In terms of deterministic forecasts, deviations existed among all the EPSs' predictions of the EASWJ. For the position of the EASWJ predicted by the ensemble mean and control forecasts, ECMWF's was to the north of its analysis, while that of NCEP and JMA generally diverged to the south. Mean errors of CMA oscillated around zero with an increase in lead time. Generally, for the spatial coverage and intensity, the ensemble mean forecasts of all the EPSs produced an underestimation, while the control forecasts were all closer to their respective analyses, with the CMA and NCEP forecasts being slightly larger and stronger, and the JMA forecasts being slightly smaller and weaker than their respective analyses. The performances of all the EPSs differed greatly with the different properties of the EASWJ. The number of skill days of all ensemble mean forecasts for predicting the position of the EASWJ could reach their own longest forecast lead time, but for the spatial coverage the number of skill days of the ECMWF and NCEP ensemble mean forecasts could only reach 13 and 11 days respectively. When predicting the intensity, the number of skill days of the ECMWF, NCEP and CMA ensemble mean forecasts were 10, 11, and 8 days respectively (Table 2). As a comparison, the number of skill days of all control forecasts for all EASWJ properties are also listed in Table 2. Hence, the performances of predicting the position of the EASWJ were found to be superior to the performances of predicting the spatial coverage and intensity, for all EPSs. The performances of predicting all the EASWJ properties also showed large differences among the different EPSs. Apart from the forecasts of intensity in the later part of the forecast range, ECMWF possessed the highest level of skill in predicting all properties. Among the other EPSs, JMA outperformed CMA and NCEP in its predictions of the position and spatial coverage of the EASWJ, while NCEP was comparable to JMA in predicting the intensity. The second best EPS for the prediction of all properties had around 0.5-1 day less skill than ECMWF throughout the forecast range. The difference between the skills of the best and the worst EPSs were as long as 3-4 days in terms of the predictability of the position of the EASWJ, and about 1-2.5 days for the predictability of spatial coverage and intensity.

    For the probabilistic forecasts, those of all four EPSs showed good skill when compared with reference forecasts and random probabilistic forecasts. In addition, large differences among the skills of the different EPSs for all EASWJ properties were apparent. Differences in terms of the EASWJ axis were more evident in the forecast range of 2-6 days, while for spatial coverage the differences were distinct at the first two lead days, and for intensity the differences departed obviously throughout the forecast range. ECMWF still provided the most accurate predictions for all EASWJ properties. For forecasts of the spatial coverage and intensity of the EASWJ, JMA led NCEP and CMA at almost every lead time. However, the performance of JMA in predicting the axis was inferior to the performance of CMA in the late forecast range, which could be found in the verification of both the CRPSS and ROC scores. Periodic oscillations were prone to appear in the ROC scores of the CMA and NCEP forecasts, but this phenomenon was even more evident in the NCEP forecast. Such an oscillating pattern was also noted by Froude (2009, 2011) and (Niu and Zhai, 2013) when verifying the mean error. Froude (2009, 2011) attributed the phenomenon to the data assimilation process, and associated it with the frequency with which the EPS suites are run. The reasons may be complicated, and need a more in-depth investigation in future work. In terms of predictability, for EASWJ axis of the ECMWF forecast had a lead of 0.5-1 day over JMA for all lead times, with the largest leads of ECMWF over CMA and NCEP being 2 and 3 days respectively. For the spatial coverage and intensity, the difference between ECMWF and JMA, which was 1-2 days of predictability, was obviously larger than that for the EASWJ axis. NCEP and CMA had 1-3 days less skill than ECMWF, apart from their worse performances in the early forecast range.

    To summarize these verifications of deterministic and probabilistic forecasts, ECMWF showed the highest level of skill for predicting the EASWJ. Meanwhile, JMA was second best because of its advantage over NCEP and CMA for all EASWJ properties at most lead times. In terms of extreme weather, forecasts of the EASWJ still have room for improvement in periods of persistent extreme precipitation, and further enhancements to the forecasts of the EASWJ, which evolves ahead of persistent extreme precipitation, will bring great benefits to the accuracy of predicting persistent extreme precipitation. Finally, it is important to note that these results are only for the selected time period, and the performance characteristics are likely to differ in the future as systems evolve, since all EPSs are subject to continuous development.

Reference

Catalog

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return