Collating Historic Weather Observations for the East Asian Region: Challenges, Solutions, and Reanalyses

Historic instrumental weather observations, made on land or at sea from as early as the 17th century (e.g., Camuffo et al., 2010), are integral to extending our understanding of the decadal and centennial variations of Earth's climate and for comparison with paleo-proxy data. The potential of such data is shown to best effect when used in dynamical 4D global reanalyses to reconstruct climate patterns and fluctuations over more than 250 years, improving climate projections and contributing to climate change detection and attribution studies. This longer temporal dimension permits the resolution of more realizations of decadal to multi-decadal climate variations (Bengtsson et al., 2007). The reduction of errors in reanalyses depends heavily on the homogeneity and geographic and temporal coverage of the data assimilated into them, particularly when downscaling climate change simulations. For some regions of the world, a paucity of observational data requires a global, multi-disciplinary effort to source and recover previously unknown repositories of instrumental weather observations, and to preserve them in digital formats suitable for modern-day use. This is the premise behind ACRE (Atmospheric Circulation Reconstructions over the Earth) China——a dedicated effort within the wider CSSP (Climate Science for Service Partnership) China project.

The region covered by ACRE China covers a wide area including the mainland of China, Hong Kong China, Macau, China, the China seas, and locations bordering the mainland of China and the China seas within the Asian monsoon region. It also benefits from close ties with a similar effort covering India and Southeast Asia under the wider ACRE Southeast Asia focus. The recovery of instrumental observations for both regions entails different stages, with sourcing, imaging and digitization, enabled by cooperation amongst cross-disciplinary investigators from around the globe. Sources of data include weather observations taken on board "stationary ships" in Hong Kong harbor and vessels patrolling Chinese seas; those made on ships during voyages of exploration or naval surveys in Chinese waters or in Southeast Asia; those taken on terrestrial explorations——for example, the Clark Expedition in northern China between 1908 to 1909; as well as observations made at meteorological registering stations, by observatories, government bodies, and port authorities at sub-daily scales. Usually pressure, temperature, wind and precipitation information are available.

Some of the earliest marine observations for this region were made during the late 18th century, while from terrestrial stations they have so far been obtained since the 1820s. Early historic observers often focused on unusual and extreme events. Later observations were more standardised, in the form of daily or sub-daily reports. The mainland of China has one of the earliest known typhoon records——for example, dating to AD 816 (Louie and Liu, 2003)——and also noted are floods, droughts and storms (Ho, 2003). As the 19th century progressed, this emphasis changed as widespread observations were increasingly considered key to understanding the climate (Anderson, 2005), and technological innovation in observational equipment kept apace (Friedman, 1993). In China, government officials made observations that were published in gazettes in the Song and Qing dynasties (Ho, 2003). In the 19th century, port customs officials and harbor staff took on a major role in registering observations (Bickers, 2016). From the 1840s, harbor authorities and government hospitals made observations in Hong Kong (MacKeown, 2011). Their role was centralized after 1883 with the establishment of the Hong Kong Observatory (HKO; Ho, 2003; Lee, 2016).

After locating the original records, the next stage in the recovery process and, arguably, the most expensive and time-consuming, is to scan the original documents in order to aid the preservation of the source material and to enable the data to be extracted by digitization. The digital technologies now available for storing and making available such data have revolutionized the potential of old weather observations (Veale et al., 2017); and yet, much of the work to get to this stage still necessitates labor intensive data entry.

Three meetings have been held so far under, or linked to, the above program: an inaugural workshop organized by ACRE China, the China Meteorological Administration (CMA), Beijing Climate Centre (BCC) and China University of Geosciences (Wuhan), held at the CMA offices in Beijing, 23-24 August 2016 (Williamson et al., 2017); a CMOC (Centre for Marine-Meteorological and Oceanographic Climate Data) meeting at Tianjin, 29 August to 1 September 2016 (WMO/IOC/JCOMM, 2016); and and ACRE China 2, hosted by HKO, in combination with an event at the Hong Kong Maritime Museum during 1-3 March 2017 (Fig. 1).

Figure 1.  Workshop participants outside the historic HKO, 2 March 2017.

The remit of these workshops and the meeting was to share knowledge on extant sources of historical instrumental weather observations from terrestrial and marine sources; to discuss the current state of international data recovery (DARE) for China-region sources; and to discuss how gaps in the historic record might be filled. This is especially critical for the 1939-45 period where, due to the disruption in normal peacetime meteorological services, data were lost or scattered.

As result of the CSSP China partnership, a great deal of data rescue work has been undertaken by the CMA and BCC, who are recovering observations from the mainland of China. The datasets produced as a result of this project will help improve regional data density and add value when integrated into global baseline data productions after quality control and homogenization (e.g., Yang et al., 2016; Xu et al., 2017). Digitization of daily air pressure, temperature and precipitation has been completed so far for 19 stations in China [Beijing, Shanghai, Qingdao, Guangzhou (Shantou), Harbin, Wuhan, Shenyang, Dalian, Tianjin, Jinan, Nanjing, Fuzhou, Chongqing, Kunming, Taiyuan, Hohhot (Jining), Lanzhou, Wulumuqi, Nanning]. Hourly surface air pressure data have also been digitized for Beijing (1841-1953), Shanghai (1877-1953), Qingdao (1902-53), Guangzhou (1915-53), Harbin (1900-53), and Wuhan (1880-1953). For Northeast China, pre-1950s daily temperature and precipitation records have been digitized for 68 stations, along with a daily climate dataset for Shenyang Station (1905-2014) developed by combining the digitized records with the post-1951 data. For Shanxi Province, yearly data from 105 county stations and monthly data from eight climate observatories (1921-27) were recovered. More work is ongoing, including the digitization of observations made by Father Joseph Marie Amiot, a French Jesuit missionary stationed in Beijing between 1757-62. Hourly surface air pressure data recovered from Beijing, Shanghai, Qingdao, Guangzhou, Harbin, and Wuhan have been sent to the International Surface Pressure Databank (ISPD).

ACRE China also collaborates with other regional partners——relationships that work in parallel with, and complimentary to, the efforts under CSSP China. The Hong Kong Observatory (HKO), for example, has recently become a collaborator, sharing information on its wealth of historic observations and past/extant recovery projects. In Hong Kong, before the establishment of the HKO in 1883, regular meteorological observations were conducted at the Seamen's Hospital, Government Civil Hospital, Government Lock Hospital, Victoria Peak and the Harbour Master's Office. Historical records of these meteorological measurements (such as pressure, temperature, rainfall and weather remarks) have been recovered back to the 1850s.

In 1883, Dr. W. DOBERCK, the first director of the HKO (called the Government Astronomer at the time), set out instructions for making meteorological observations in various treaty ports, customs stations and lighthouses in China (Doberck, 1883). Thus, starting in 1884, HKO regularly conducted meteorological observations (such as dry and wet bulb temperatures, rainfall, pressure, wind speed/direction, cloud amount, duration of bright sunshine, state of weather, etc.) at its headquarters in Tsimshatsui, apart from a break during 1940 to 1946 owing to World War II (Lee, 2016). It is one of the World Meteorological Organization's recognized Centennial Observing Stations (WMO station code: 45005) with over a century of continuous observations of essential surface meteorological observations. Information on the meteorological observations were regularly included in the Observatory's annual data publications (e.g., "Meteorological results part I——surface observations" from 1884 to 1939 and from 1947 to 1986). As part of the HKO's wide range of climate services, it has made available more than 130 years of climate data together with a whole range of other climate information and statistics on its Climatological Information Services website (http://www.hko.gov.hk/cis/climat_e.htm), for easy access by the public and users in different sectors.

Since the 1970s, the HKO has digitized most of the available hourly observations at its headquarters from 1884 to 1970——apart from the hourly relative humidity (RH) data, of which records in "Meteorological results part I" were only available since 1961. With a view to reconstructing the hourly RH data at the HKO back to 1884, a project has recently been carried out to digitize all RH records from the HKO weather log books between 1947 and 1960 and the wet bulb temperatures in "Meteorological results part I" from 1884 to 1939. For the archiving of pre-WWII historical weather charts, the HKO has commenced a project to scan all available historical weather charts before 1939 into digital image formats.

Besides meteorological observations at the HKO, efforts have been made to recover the daily rainfall data at the Botanic Gardens between 1892 and 1926 from the annual report of the Botanical and Afforestation Department and the weather observations from the weather log books of Gap Rock Lighthouse (a small island about 43 km to the southwest of Hong Kong) from 1919 to 1938. With a view to narrowing the gap of missing data between 1940 and 1946, some of the daily and monthly temperature and rainfall records in Hong Kong were recovered from historical government reports and newspapers in 1940, 1941 and 1946. Wartime meteorological records observed and prepared by HKO officers and other volunteers in the internment camp at Stanley in 1942-45 have also been recovered (Ho, 2003; PRO). Furthermore, the HKO has been liaising with overseas researchers in searching and recovering meteorological records measured by the Japanese military stationed in Hong Kong during the war.

Scholars working under the umbrella of ACRE Japan (http://www.met-acre.net/chapters.htm) have been active in contributing to the historical data for China. A Japanese weather station was constructed in the Taiwan region in November 1896, and operations expanded to the mainland of China in 1904. By the early 1940s, the Japanese held more than 100 weather stations in the region (Yamamoto, 2014). Observational data were recorded in the Meteorological Monthly Bulletin of the Central Meteorological Observatory of Japan, Northern China and Northeast China, and the rainfall, surface pressure and temperature data were recently digitized (Kubota et al., 2016). Twice-daily observations published in the China Coast Meteorological Register (CCMR) from 1873 to 1932 have also been collected and digitized in the ACRE China DARE effort. This includes previously "missing" data from the CCMR for 1882, 1884, 1886, and 1888-91, all of which were located in the Japan Meteorological Agency library and digitized. Additional observations from the Shanghai Observatory were found by ACRE from 1906 to 1941, with the 1933 to 1941 portion digitized to extend the CCMR coverage, and records from the WWII years are being sorted by ACRE Japan. These digitized data were provided to the ISPD. ACRE Japan has also recovered Chinese data from other sources——most notably, 19th century temperature data from Beijing, 1840s-1880s (Zaiki et al., 2008), and observational data from Canton, Hong Kong and Shanghai, found in English newspapers from the 1830s to 1860s (Tsukahara, 2013). Daily rainfall data at several stations, for the period 1901 to 1939, and all the daily perimeters noted at Zi-Ka-Wei Observatory (Shanghai), 1945-8, were digitized. From this information, some preliminary research on the seasonal changes of the rainy season has been conducted (Hirano and Matsumoto, 2010; Matsumoto et al., 2010).

Within the CSSP China effort, sub-daily meteorological data for Indochina were digitized for the first 40 years of the 20th century. These data were taken from the Indian Daily Weather Report (IDWR; http://browse.ceda.ac.uk/browse/badc/corral/images/metobs/indian_ocean/India/IndianDailyWeatherReports). The number of stations per reading varies between approximately 180 and 220 (about 9 307 500 observations). Sub-daily station information includes pressure, wet and dry bulb temperature, minimum and maximum temperature, wind, humidity, cloud, rainfall and general weather conditions. The data were quality controlled and different checks for homogeneity applied, and offer the possibility for different statistical analyses and the study of extreme monsoon precipitation events. The data also provide the basis for studies of monsoon variability and trends, monsoon onset and retreat on high spatial and temporal time scales for the first part of the last century, understanding key mechanisms controlling the diversity of the Indian and southeastern monsoon relationship, and the role of ENSO and the boreal summer teleconnections on daily to intraseasonal timescales. The dataset is the basis for validations of reanalyses, data/model comparisons, and the study of early 20th century typhoon activity.

ACRE China contractors and ACRE Southeast Asia activities have digitized and made available the following to the ISPD in the 2016-17 period: China and surrounding countries (9133 observations); Papua New Guinea, Port Daru, 1894-95 (1314 observations) and Port Moresby, 1892-95 (11 340 observations); Sarawak, Kuching, December 1906 to May 1907 (1820 observations); Medan, Sumatra, Indonesia, 1910-25 (591 300 observations), ongoing.

Team member Clive WILKINSON has been working on documenting and recovering instrumental observations for the China seas and coasts from records extant in British archives. Most of these observations are held in ship logbooks from British commercial and naval vessels that journeyed to China or across the China seas in the 19th and early 20th centuries. Wilkinson has compiled an inventory for the vessels——so far encompassing more than 6500 merchant and 2000 naval logbooks and meteorological forms——and has begun the process of photographing some of these. From the UK National Archives, London, for instance, 40 logbooks of moored ("stationary") harbor vessels in Hong Kong (1843-1901), with eight additional logs of vessels in Chinese waters, have been rescued and digitized. These ships made observations whilst at sea and in port. At the National Meteorological Archive in Exeter, more than 380 meteorological logbooks from British Royal Navy vessels over the period 1853-1950 have been photographed, with a scattering of earlier logs. This in fact represents the entire collection of British Royal Navy logbooks in Chinese waters from this one archive up until 1950. This photography yielded a total of 32 600 pages of meteorological data, including temperature, pressure and winds. Many of the logbooks also enclose typhoon reports and, from the 1920s, there are frequently synoptic charts, upper air observations from pilot balloons, radio reports and typhoon warnings. The 32 600 images represent data from just 420 logbooks——a tiny fraction of the 8500 identified——with a further 3000 ships/voyages whose logbooks are yet to be sourced.

In Hong Kong, under the Voluntary Observing Ships' Scheme of the then International Meteorological Organization (the predecessor of the World Meteorological Organization), the HKO started to recruit a fleet of locally based voluntary weather observing ships in 1949. Archives of their observations are available since then. Prior to 1949, limited ship observations have been recovered by the HKO (details as follows):

(1) Extracts of ship observations for the years of 1892, 1894 and 1895 were included among the descriptions for tropical cyclones in the South China Sea and western North Pacific region in the Report of the Director of the Observatory for the respective years.

(2) Hand-written ship observations were also recovered from the reverse side of the pages of weather maps published by the HKO for the period 1 January 1934 -to 1 July 1937, and 1 January 1938 -to 1 July 1939.

(3) Some more ship observations for 1931 have also been recovered from the publication on "Weather Observations from Ships" (Jeffries, 1932).

In 1887, based on meteorological data received from the telegraph companies, the HKO published the CCMR booklet (also addressed by ACRE China, as noted previously) to report on the daily weather conditions of the South China Sea (Ho, 2003). By 1890, this booklet provided information on 16 cities or regions including Vladivostok, Tokyo, Nagasaki, Shanghai, Fuzhou, Xiamen, Kaiping, Shantou, Victoria Peak, Guangzhou, Macau, Haikou, Kaifeng, Borneo, Manila and Cape Saint James. Archives of the CCMR for the period 1894-1912 have also been recovered. The number of stations included in the Register have increased to 43 towards the end of 1912, including the reports for Hong Kong, Victoria Peak and Gap Rock. This was superseded by the Monthly Meteorological Bulletin from 1913 to 1932. From 1933 to 1940, these bulletins were published in the form of Daily Weather Reports. Scanning of the documents mentioned above have been conducted, but digitization of any observations not done by ACRE China (such as of the CCMR) has not yet been carried out. In the United States, Kevin WOOD and Mark MOLLAN identified more than 200 CSSP China-relevant naval vessel logbook series in the holdings of the U.S. National Archives (NARA), representing many thousands of pages of weather data and operational remarks. These logbooks span the period from 1830 to 1946. All these ships were stationed in China and elsewhere in the western Pacific for years——some for their entire commission. Of these, the logbooks of 15 ships have been photographed to archival standards (full color; resolution ≥300 dpi) and posted on the National Archives Catalogue website (https://www.archives.gov/research). This set includes 471 volumes totaling 235 000 pages. The logbooks of an additional nine ships that spent a significant period of time in the region were imaged incidentally as part of an earlier project. This set includes 454 volumes totaling 145 000 pages, and spans 1844-1946. Of these, 1 300 000 hourly weather records have been extracted by the citizen-scientists under Old Weather (https://www.oldweather.org) and are in the process of integration into the International Comprehensive Ocean Atmosphere Data Set (ICOADS) and the ISPD. A further 80 volumes (63 000 pages) of the IDWR from the holdings of the NOAA Central Library were also photographed and provided to the UK Met Office in order to reconstruct a complete set of volumes of this resource in digital format; the weather data in them are slowly being digitized.

Using high-resolution digital photography to facilitate DARE from logbooks and other primary sources can have several important benefits. In the US case, the objectives of project partners align well with the output of the joint imaging program of NARA, satisfying equal but distinct objectives consistent with free and open global access to data and other digital products, in addition to weather DARE. This opens the door for all kinds of research otherwise hindered by the need to find and access unique documents that exist only in one place, including options to use citizen-science or other distributed approaches for the second essential step in DARE: manual transcription. However, the effort required to carry out a comprehensive digital imaging campaign is not insignificant considering the massive scale of the potential data resources. For example, NARA holds 98 979 Navy and Coast Guard logbooks, of which roughly 25 000 cover the particularly interesting period of 1801-1940. These contain many millions of unutilized weather records, and often these are the only existing source of instrumental data for an entire region. Hence, it is necessary to focus strategically on periods and regions where the impact of new observations is likely to be greatest, and the scope is consistent with imaging and transcription capacity, and the means to carry out required quality-control and database integration tasks.

ACRE China contractors have digitized and made available the following to ICOADS in the 2016/17 period: Hong Kong seven "stationary" and moving ships 1857-74 (53 916 observations); voyages and surveys HMS Providence 1796/97 (340 observations), L'Erigone 1840-44 (78 000 observations), Doudart de Lagrèe 1866-68 (4182 observations); Thetis and Esperance 1824-26 (10 008 observations); HMS Samarang 1843-47 (19 692 observations).

The new and improved marine observations have been packaged in IMMA (the International Maritime Meteorological Archive) format and distributed as a candidate update to ICOADS, designated ICOADS3+ (https://github.com/oldweather/ICOADS3.plus), though not yet formally released as part of ICOADS. These marine, plus the terrestrial observations, have been sent to the US NCEI (National Center for Environmental Information) and the NOAA/University of Colorado CIRES (Cooperative Institute for Research in Environmental Sciences) for inclusion in ISPD, version 5. One of the main uses of these data will be their assimilation into various dynamical 4D global reanalyses, especially the 20th Century Reanalysis Project (20CR; Compo et al., 2011). As outlined by Reanalysis.org (http://reanalyses.org/), "… observations and a numerical model that simulates one or more aspects of the Earth system are combined objectively to generate a synthesized estimate of the state of the system. A reanalysis typically extends over several decades or longer, and covers the entire globe from the Earth's surface to well above the stratosphere".

Improving and extending reanalyses through the inclusion of more observations will enhance these tools and allow for their use in better understanding past climate variability and change. Further enhancement can be provided by "downscaling" these reanalyses, and this is being done under CSSP China by the Met Office PRECIS (Providing Regional Climates for Impacts Studies) group (https://www.metoffice.gov.uk/research/applied/international-development/precis). The aim is to "downscale" 20CR over China from 1831, in order to provide high-resolution (25 km) information on meteorological parameters for use in climate service applications such as wind energy analysis, flood planning and urban health services.

The authors acknowledge the ongoing support of CSSP China under the BEIS UK-China Research & Innovation Partnership Fund through the Met Office Climate Science for Service Partnership (CSSP) China as part of the Newton Fund. Prof. Rob ALLAN is also supported by funding from the EU Copernicus Climate Change Service (C3S) and acknowledges the University of Southern Queensland, Toowoomba, Australia, and the Centre for Maritime Historical Studies, University of Exeter, UK, where he is an Adjunct and Honorary Professor, respectively. The authors also wish to thank Mr. Chi-ming SHUN, Director of the HKO, for his valuable comments on the manuscript.