This site uses cookies, tags, and tracking settings to store information that help give you the very best browsing experience. Dismiss this warning

Enhancing the Data Coverage in the Integrated Global Radiosonde Archive

Imke Durre NOAA/National Centers for Environmental Information, Asheville, North Carolina

Search for other papers by Imke Durre in
Current site
Google Scholar
PubMed
Close
,
Xungang Yin ERT, Inc., Asheville, North Carolina

Search for other papers by Xungang Yin in
Current site
Google Scholar
PubMed
Close
,
Russell S. Vose NOAA/National Centers for Environmental Information, Asheville, North Carolina

Search for other papers by Russell S. Vose in
Current site
Google Scholar
PubMed
Close
,
Scott Applequist NOAA/National Centers for Environmental Information, Asheville, North Carolina

Search for other papers by Scott Applequist in
Current site
Google Scholar
PubMed
Close
, and
Jeff Arnfield NOAA/National Centers for Environmental Information, Asheville, North Carolina

Search for other papers by Jeff Arnfield in
Current site
Google Scholar
PubMed
Close
Full access

Abstract

The Integrated Global Radiosonde Archive (IGRA) is a collection of historical and near-real-time radiosonde and pilot balloon observations from around the globe. Consisting of a foundational dataset of individual soundings, a set of sounding-derived parameters, and monthly means, the collection is maintained and distributed by the National Oceanic and Atmospheric Administration’s National Centers for Environmental Information (NCEI). It has been used in a variety of applications, including reanalysis projects, assessments of tropospheric and stratospheric temperature and moisture trends, a wide range of studies of atmospheric processes and structures, and as validation of observations from other observing platforms. In 2016, NCEI released version 2 of the dataset, IGRA 2, which incorporates data from a considerably greater number of data sources, thus increasing the data volume by 30%, extending the data back in time to as early as 1905, and improving the spatial coverage. To create IGRA 2, 40 data sources were converted into a common data format and merged into one coherent dataset using a newly designed suite of algorithms. Then, an overhauled version of the IGRA 1 quality-assurance system was applied to the integrated data. Last, monthly means and sounding-by-sounding moisture and stability parameters were derived from the new dataset. All of these components are updated on a regular basis and made available for download free of charge on the NCEI website.

© 2018 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Imke Durre, imke.durre@noaa.gov

Abstract

The Integrated Global Radiosonde Archive (IGRA) is a collection of historical and near-real-time radiosonde and pilot balloon observations from around the globe. Consisting of a foundational dataset of individual soundings, a set of sounding-derived parameters, and monthly means, the collection is maintained and distributed by the National Oceanic and Atmospheric Administration’s National Centers for Environmental Information (NCEI). It has been used in a variety of applications, including reanalysis projects, assessments of tropospheric and stratospheric temperature and moisture trends, a wide range of studies of atmospheric processes and structures, and as validation of observations from other observing platforms. In 2016, NCEI released version 2 of the dataset, IGRA 2, which incorporates data from a considerably greater number of data sources, thus increasing the data volume by 30%, extending the data back in time to as early as 1905, and improving the spatial coverage. To create IGRA 2, 40 data sources were converted into a common data format and merged into one coherent dataset using a newly designed suite of algorithms. Then, an overhauled version of the IGRA 1 quality-assurance system was applied to the integrated data. Last, monthly means and sounding-by-sounding moisture and stability parameters were derived from the new dataset. All of these components are updated on a regular basis and made available for download free of charge on the NCEI website.

© 2018 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Imke Durre, imke.durre@noaa.gov

1. Introduction

The National Centers for Environmental Information (NCEI) of the National Oceanic and Atmospheric Administration (NOAA) maintains a collection of historical radiosonde and pilot balloon (pibal) observations known as the Integrated Global Radiosonde Archive (IGRA). The central component of this collection is a dataset consisting of quality-assured individual soundings from many areas of the globe. Ancillary products include monthly means computed from the individual observations and a set of additional parameters describing the moisture and stability of each profile (Durre et al. 2006; Durre and Yin 2008).

Since its release in 2004 by NCEI’s predecessor, the National Climatic Data Center (NCDC), version 1 of IGRA (IGRA 1) has proven to be useful for both research and practical applications. Studies on a variety of atmospheric processes and structures (Añel et al. 2007; Seidel et al. 2010, 2012; Rapp et al. 2011; Sorokina and Esau 2011), analyses of trends in tropospheric and stratospheric temperature and water vapor content (Thompson and Solomon 2005; Durre et al. 2009; Serreze et al. 2012), and comparisons of water vapor measurements from various platforms (Ho et al. 2010; Huang et al. 2013; Moradi et al. 2013; Van Malderen et al. 2014; Schröder et al. 2016) have all relied on observations taken from IGRA 1. IGRA data also serve as input to the NOAA Products Validation System (Reale et al. 2012), products of homogeneity-adjusted radiosonde observations (Thorne et al. 2005; McCarthy et al. 2009; Dai et al. 2011; Haimberger et al. 2012; Ramella Pralungo and Haimberger 2014; Sherwood and Nishant 2015), and the reanalysis data in the JRA-55 (Kobayashi et al. 2015). In particular, IGRA contributes post-1997 data to NOAA’s Radiosonde Atmospheric Temperature Products for Assessing Climate (RATPAC; Free et al. 2005), a collection of homogeneity-adjusted large-area time series of monthly temperature anomalies for various pressure levels and layers between the surface and lower stratosphere. RATPAC is updated once a month at NCEI and has been referenced in the American Meteorological Society’s annual State of the Climate report (e.g., Blunden and Arndt 2016) as well as in assessments of the Intergovernmental Panel on Climate Change (IPCC 2007, 2013).

In the years following the release of IGRA 1, additional digital collections of upper-air data became available that covered areas or times with relatively sparse data in IGRA 1, including Africa, South America, and China during the 1970s and 1980s, and much of the world prior to the 1960s. Their inclusion in IGRA 2 required a redesign of the process for creating merged data records from multiple sources to take into account the greater diversity of data and metadata characteristics compared to those that were represented in IGRA 1. For example, while IGRA 1 was designed to contain only radiosonde soundings from stations with World Meteorological Organization (WMO) station identification numbers (ID), newly acquired data from before the 1950s were largely characterized by wind-only pibal observations from stations that were not indexed by WMO station identifiers. In addition, IGRA 1 did not include relative humidity (RH), the only humidity variable available in soundings from before the 1970s.

The culmination of NCEI’s effort to incorporate the additional data is version 2 of IGRA (IGRA 2; Durre et al. 2016a,b). Like IGRA 1, IGRA 2 is intended to be a comprehensive dataset of radiosonde and pibal observations that can serve as a foundation for reanalysis, for value-added products such as bias-adjusted datasets, and for many other applications. Consequently, it is constructed from multiple data sources to maximize temporal and spatial coverage and is subjected to quality-assurance (QA) procedures to remove grossly erroneous values; homogeneity adjustments are not applied.

The purpose of this paper is to describe this new dataset as well as the procedures used to create it. A brief description of the data sources used and how they were processed into a standard format is provided in section 2, and the methods for generating the integrated, quality-assured IGRA 2 sounding data from these sources are documented in section 3. In section 4, a comparison between the data coverage in the two versions of IGRA is presented. Section 5 contains information about how the data and derived products can be accessed, and a summary is provided in section 6. Figure 1 gives the sequence of steps to process the source data and create the sounding data, as described in more detail in sections 2 and 3.

Fig. 1.
Fig. 1.

Data-flow diagram showing the progress of source data from their original format to the final step of creating the IGRA dataset.

Citation: Journal of Atmospheric and Oceanic Technology 35, 9; 10.1175/JTECH-D-17-0223.1

2. Source data

The primary aim of creating version 2 of IGRA was to enhance spatial and temporal coverage, particularly prior to the 1970s. Toward that end, a comprehensive search for suitable datasets was conducted by checking NCEI’s archive for the presence of any upper-air datasets that were not already included in IGRA 1, searching the peer-reviewed literature, and reviewing the data holdings at the National Center for Atmospheric Research (NCAR). The types of data considered included not only land-based radiosonde observations but also observations from ships and moving platforms as well as profiles in which height, either above the surface or above sea level, was the only available vertical coordinate. Other types of upper-air observations, such as those from dropsondes, aircraft, and wind profilers, were not considered because of their considerably different processing requirements. The following sections cover the steps involved in selecting and preparing suitable data sources for subsequent integration into IGRA 2.

a. Selection of data sources

The more than 90 potential data sources identified during the comprehensive search were first checked for relevance to IGRA, availability, and adequacy of documentation and station location information. Approximately one-third of the sources were eliminated from consideration because the data turned out to not be available for distribution, because the documentation was inadequate, because they turned out to contain estimated or modeled values rather than the actual measurements, or because they had been superseded by a newer version of the same data. Another four sources were set aside when extensive data problems of one kind or another were encountered during the processing. Thirteen of the remaining 53 data sources contained either field campaign observations or fewer than two years of data and were set aside because of both their limited contribution to the dataset as a whole and the often considerable effort required for reformatting any one data source.

There remained a total of 40 data sources, which were identified in IGRA 2 by 33 distinct data-source identifiers. These datasets, listed in Table 1, include 10 of the 11 data sources that were contained in IGRA 1 (Durre et al. 2006). One of the IGRA 1 sources, Global Telecommunication System (GTS) messages originating from the National Centers for Environmental Prediction (NCEP) between January 1973 and October 1999, was replaced by a comparable dataset obtained from the U.S. Air Force (USAF) 14th Weather Squadron. In comparison with the NCEP dataset, the USAF version of the data is more complete during the 1970s and 1980s, particularly in China, and exhibits very similar spatial completeness thereafter. Furthermore, the USAF source correctly represents the dates of observations reported for 0000 coordinated universal time (UTC), which are incorrectly reported according to the date of the time of balloon release rather than according to the date of the nominal hour in the NCEP data. The pre-1970 station density was increased with two hemispheric- to global-scale datasets from NCAR (UCAR 1971; MIT and U. Missouri 1980), with data digitized for the European Reanalysis of Global Climate Observations (ERA-CLIM) project (Stickler et al. 2014), and with the Comprehensive Historical Upper Air Network (CHUAN), version 1.01 (V1.01), data (Stickler et al. 2010; Brönnimann and Stickler 2013). Coverage over Africa between the 1930s and 2000s was further enhanced by 12 collections of pibal observations and one set of radiosonde measurements digitized under two independent data rescue efforts, one directed by the U.S. Climate Database Modernization Program (CDMP; Dupigny-Giroux et al. 2007) and the other under the auspices of Météo-France.

Table 1.

Basic provenance information for IGRA data sources. MIT = Massachusetts Institute of Technology, NSIDC = National Snow and Ice Data Center, U. Missouri = University of Missouri–Columbia, and USAF14WS = U.S. Air Force 14th Weather Squadron.

Table 1.

b. Processing of data sources

To simplify the subsequent integration process, the chosen datasets and accompanying station metadata were reformatted into a common data format, cleansed of invalid entries, freed of duplicate soundings and data levels, and placed into a logical order. Invalid entries included records whose format or contents did not conform with the accompanying documentation, soundings with nonexistent dates (e.g., 31 April) or without a valid observation time, and data values identified as estimated, edited, or interpolated.

To achieve a logical ordering of the data, soundings were then grouped by source station identifier (e.g., WMO station number) and sorted chronologically according to date, nominal hour, and launch time. Analogously, levels within a sounding were arranged in order of descending pressure and ascending height. In addition, two steps were taken to standardize observation times. First, nominal hours of 24 and release times of 2400 were changed to the corresponding 0000 UTC time for the next day. The other step involved the correction of certain observation hours assigned to soundings received from the NWS. When the release time is between 31 and 60 min before a synoptic hour (0000, 0600, 1200, or 1800 UTC), the observation hour is frequently listed as 1 h earlier (2300, 0500, 1100, 1700) than it should be in NCEI’s archive of NWS soundings. These observation hours were corrected on the basis of the release time that was provided along with the data. For example, for a release time of 1105 UTC, the nominal hour should be 1200, not 1100.

The most complex component of the reformatting process was the elimination of duplicates of soundings and data levels. Cases in which multiple soundings were present for the same station, date, and time can be the result of data entry or processing errors, of a deliberate retaking of the same observation by the observer, or of the division of one sounding’s data into several records. Since no information typically exists on the reason for any one particular case of duplication, the duplicate removal algorithm (Durre et al. 2016a) relied on data comparison and completeness criteria to select and combine soundings. It included provisions for handling three types of duplication: identical sounding records, complementary soundings (e.g., one containing pressure levels and one containing nonpressure levels), and soundings whose data values at the same pressures or heights differed. Cases of differing soundings were resolved by choosing the sounding with the largest number of data levels, or, in the case of a tie, the sounding with the largest number of standard pressure levels. When sets of duplicates contained a mix of identical, complementary, and differing sounding records, records that differed from the greatest number of soundings were removed iteratively before combining any remaining identical and complementary soundings into one sounding.

The repetition of data levels within a sounding (e.g., two sets of data values at 500 hPa) was addressed in a similar manner. Identical copies of the same data level were removed, differences between multiple sets of data values at the same pressure or height were resolved through a logical decision-making algorithm, and copies containing complementary sets of data values were combined.

Some data sources required some additional special handling. For example, for some of the U.S. pibal data digitized by CDMP (source cdmp-us2 in Table 1), the only available vertical coordinate was elapsed time since balloon launch. These elapsed times were converted to height above sea level using the NWS standard conversion table as well as the station elevation and appropriate balloon weight (30 or 100 g) that were supplied with the data.

Information on station name, location, and elevation was collected for each station in a data source and also written in a common format for use in later processing steps. A station’s WMO ID and International Civil Aviation Organization (ICAO) call letters were also collected whenever possible. Stations without valid coordinates or station names were eliminated. Station metadata either originated from the data provider or were taken from independent station catalogs (e.g., Air Force Weather Agency/U.S. Air Force/U.S. Department of Defense 1980; WMO 2016a,b,c).

The individual lists of source stations, totaling 19 473 stations for data ending in 2016, were then combined into one consolidated list. From these, approximately 40% were removed from further consideration. Half of the removed stations had insufficient data (no year with 28 or more soundings), and the other half exhibited various data or metadata problems that either could not be resolved or were absent from other sources containing the same data.

After all exclusions and corrections had been applied, the source stations that were considered for inclusion in IGRA 2 all had a valid location and name or, in the case of a mobile station, only a valid name, and had a minimum of 28 soundings in at least one year.

Among the 11 561 stations that remained, some station names were edited to further standardize them in preparation for station name matching, and some additional station-specific corrections to coordinates, elevation, station name, call letters, or WMO numbers were made on the basis of specially designed input files in which these corrections were recorded. In general, the need for these corrections was identified during the effort to match source stations for the same observing site on the basis of data and metadata (see section 3a). Corrections were gleaned from metadata found in the station lists of other data sources, Internet searches, and NCEI’s archive of station history information.

3. Creation of the sounding dataset

Once the data sources had been standardized, the IGRA 2 sounding dataset was created in four steps. First, the source stations were arranged into groups, each of which essentially represented the data for one observing site. Second, within each of these groups, the data from the source stations composing the group were integrated into a single time series containing one sounding per observation time. Next, a comprehensive suite of QA procedures was applied to the integrated data. Last remaining stations that lay in close proximity to each other and whose periods of record did not overlap were composited into longer data records. Each of these steps is described in one of the following sections, and results are presented in section 3e.

a. Source station matching

If stations were consistently identified according to the same station identification system and with the same coordinates and names across different data sources, then the task of matching duplicate source station records for the same location would be straightforward. Indeed, when creating IGRA 1, only data sources in which stations were identified with five-digit WMO station numbers were considered, and source station records were matched on the basis of those station IDs. This approach, however, occasionally resulted in mismatches as a result of errors in coordinates, data, or station ID assignments that had to be detected and corrected via an iterative semiautomatic process that was time consuming, was difficult to reproduce, and could not be extended to include data from the first half of the twentieth century to which no WMO number had been assigned (Durre et al. 2006).

For IGRA 2, a mostly automated, multistep algorithm was therefore designed that matched the source station records on the basis of the similarity of their data and metadata. The output from this algorithm was a set of “mingle groups,” each of which provided the ingredients of the data record for one IGRA 2 station in terms of one or more source station records.

As a first step, each possible pair of source stations was classified according to the degree to which the data and metadata of the two member stations matched with each other. Four criteria were used: data similarity, equality of station IDs, distance, and station name similarity. If none of the four criteria indicated a match, the stations were considered to be separate from each other. Stations were assigned to the same group if they lay within 40 km of each other and one of the following two conditions was true: 1) 50% or more of the soundings found in both station records matched or 2) the station IDs or names of the two source stations were identical and their data did not overlap. Station pairs with conflicting matching criteria (e.g., data identical, but distance greater than 40 km) were specially labeled for resolution in subsequent processing steps.

Second, stations for which multiple pairings showed conflicting matching criteria were examined to determine whether their metadata required any correction, whether the station should be removed from the dataset, or whether the elimination of the conflict could be left to the algorithm. In this way, it was discovered, for example, that the latitude or longitude of some stations had the incorrect sign. If reliable location information could be obtained, either from other data sources or from the Internet, then such errors were corrected manually, and the entire pairing process was repeated with the corrected metadata. Stations whose obviously incorrect metadata could not be corrected were removed from the dataset.

Next, an automated decision-making algorithm was applied to the station pair classification results to identify groups of source stations that should be combined into IGRA 2 stations. An initial set of these mingle groups was constructed by finding sets of stations such that the members of each set were interconnected by way of a match based on at least one of the matching criteria. For example, the mingle group for the city of Karonga, Malawi, consisted of station records from three sources (cdmp-adp, cdmp-awc, and usaf-ds3), all of which matched with each other because their WMO station IDs (67587), coordinates (9.95°S, 33.883°E), and names (KARONGA) were identical, and their data did not overlap.

Each of these groups was then checked for transitivity violations, namely, cases in which station A matched station B, station B matched station C, but A and C did not match. In an iterative process, a station involved in such a violation was eliminated if it met one of the following conditions: 1) it had the largest number of conflicts among all of the group’s stations or 2) it had the same number of conflicts as another member, and a smaller number of soundings than the other member. Once all violations had been removed, it was verified that all of the group’s remaining members were still interconnected. If not, the group was split into two or more groups accordingly. For example, suppose if a group contained five members after the removal of transitivity violations, and two of them matched with each other but not with any of the other group members, then those two stations were split off into a separate new two-member mingle group.

Source stations that were identified as being clearly distinct from all other stations on the basis of either distance or data comparisons were added to IGRA 2 without being merged with any other data records. All other source stations were either matched with one or more source stations or, when their matching results revealed problems, removed from consideration.

Each mingle group was assigned an 11-character IGRA 2 station ID. The first two characters of the station ID represented the Federal Information Processing Standards (FIPS) country code, which is consistent with the geographical location of the station. When available, the country code was derived from metadata provided along with the station’s data sources; otherwise, the Google or geonames.org Internet services were used to retrieve the country code corresponding to the station’s latest available coordinates. Whenever possible, the remainder of the IGRA station ID was based on one of the standard station IDs—ship call sign, WMO number, Weather Bureau Army Navy (WBAN) number, or ICAO call sign, in that order of preference. If none of the four standard station IDs was associated with the IGRA 2 station record, then a custom station ID was constructed from the source station ID. For example, the most current WMO number for Key West, Florida, is 72201. Therefore, the corresponding IGRA 2 station ID became USM00072201. On the other hand, chuan101 station 3671 was not associated with any of the four standard types of station identifiers; therefore, its IGRA 2 station ID was based on the chuan101 ID and took the form USXUAC03671.

b. Data integration

Once the mingle groups had been created, data from all source stations in each such group were merged into one integrated IGRA 2 station data record. Since the source records often overlapped in time, simply concatenating and sorting them would have resulted in the presence of multiple soundings for the same observation time and location. Consequently, an algorithm was designed with the aim of creating data records containing only one sounding for each distinct station and time.

This task was complicated by three factors: differences in how time was reported in the various data sources, periods of high-frequency observations with more than four soundings per day, and 1-day errors in date stamps. The most common intersource discrepancy occurred when a sounding launched within an hour prior to the synoptic hour (e.g., 1200 UTC) was identified with that synoptic hour in one source and with the observation hour closest to the release time (e.g., 1100 UTC) in another. Only in the case of NCEI’s archive of NWS soundings was it possible to correct this issue prior to the integration stage (see section 2b) because both the release times and the convention used for assigning the observation hour from that release time were known. In most data sources, however, time was identified only by an observation hour, precluding any corrections. As a result, in addition to true time differences between distinct observations, there existed apparent time differences between soundings from different data sources that actually represented the same observation.

The integration algorithm handled these issues by way of a three-step process consisting of the removal of duplicated soundings on consecutive days, a cleanup of clusters of high-frequency observations, and the selection of one sounding from groups of soundings with similar observation times. For each IGRA 2 station, the algorithm was applied to the soundings from the sources specified in the associated mingle group.

First, each sounding was compared to all soundings that preceded it by between 21 and 24 h to identify cases in which the same sounding was attributed to different dates in different data sources as well as cases in which identical data were repeated on consecutive days. For each sounding pair, a data similarity percentage was computed separately for pressure levels and nonpressure levels. If the calculated percentage or percentages equaled or exceeded 90%, the soundings were considered to match.

If such matches were found, an iterative process was applied to remove those soundings that were most likely to be in error based on the comparison results. That process worked as follows. Moving through the IGRA 2 station’s entire record, a count of the number of matches found was tracked for each sounding. If there were soundings with more than one match, they were removed, and all remaining soundings were compared again. If the maximum number of matches throughout the entire record was 1, all soundings with at least one match were removed. The process ended for that station when there were no more matches throughout the entire record or no more soundings remained.

For example, if a 0000 UTC sounding on 2 October 2015 were identical to the 0000 UTC 3 October 2015 soundings from two other sources, then the 2 October sounding would be removed because it matched with two soundings, while the 3 October soundings each matched with only one. If the 2 October sounding matched only with one 3 October sounding, then both of these soundings would be removed.

A total of 71 412 source soundings (0.1%) were eliminated with this step. The approach proved to be particularly useful for identifying cases in which the data of one source were systematically shifted by one sounding or by 24 h relative to the data in another source, as was the case between the NCAR DS353.4 and U.S. Air Force DS3 datasets, eventually leading to the exclusion of the entire DS353.4 dataset from IGRA 2.

Second, periods of frequent observations were identified, and only soundings from one data source were retained within such clusters. This step was necessary because, within frequent-observation periods, it was difficult to distinguish truly unique observations from those whose observation times differed because of the usual intersource reporting differences. A high-frequency cluster was defined as a sequence of soundings within which all pairs of consecutive observations were at least 60 and less than 300 min apart and that was separated from adjacent soundings by at least 24 h. There were approximately 300 000 such clusters containing 2.2 million (3%) of all source soundings.

If all soundings in a high-frequency cluster originated from the same source, they were assumed to represent true high-frequency observations and were retained. If more than one data source was represented, the data source with the largest number of soundings in the cluster was chosen. In the event of a tie, each sounding was assigned a “utility score” that is based on the hierarchy of sounding characteristics shown in Tables 2 and 3, and the source that included the sounding with the highest such score was retained. For example, from the cluster shown in Table 4, soundings with source ncdc6323 were retained because one of its soundings had the highest variable score among them (Table 2), and the sources involved resided at the same level in the data-source hierarchy (Table 3).

Table 2.

Criteria used when choosing one source or sounding from a cluster of soundings during data integration (section 3b). Criteria are listed in order of decreasing priority; e.g., variable availability is used only when two soundings have the same priority ranking in terms of source hierarchy. The numbers in parentheses refer to the scores given to each of the three types of variables.

Table 2.
Table 3.

Data-source hierarchy used during data integration. Data-source codes are defined in Table 1.

Table 3.
Table 4.

Example of a high-frequency cluster containing three soundings from each of two data sources (3 Sep 1967 on Macquarie Island, Australia). Both sources rank at the second-lowest level in the source hierarchy (Table 3). The third sounding in the group contains more variables than the others and therefore is the highest-ranking sounding (Table 2). Thus, the three soundings from source ncdc6323 are retained in IGRA 2. Obs hr = observation hour (UTC), NA = not available, RT = release time (hh:mm), hum = humidity, and temp = temperature.

Table 4.

Last, in an effort to retain only one sounding for each station, date, and time, soundings were grouped into four types of clusters, listed in Table 5, according to what appeared to be the same observation time, and one sounding from each group was chosen for retention based on the five criteria listed in Table 2. This process was applied in an iterative fashion. First, a station’s chronologically sorted collection of source soundings was scanned for clusters of types 1a and 1b (Table 5), and appropriate selections were made. The same step was repeated for cluster types 2 and 3, in turn. If any soundings were removed from the collection as a result of these three passes, the entire process was repeated on the modified collection of source soundings. The iterations continued until no more modifications were made in the previous iteration.

Table 5.

Types of sounding clusters from which one sounding was chosen during data integration (section 3b). Examples show pairs of soundings, but clusters can have more than two soundings. Observation times are identified by the observation hour and release time in UTC followed by the date; NA = missing. The time relative to the “reference time” [0000 UTC on the first day of the earliest years of source data (1 Jan 1899)] was calculated from the sounding’s observation hour or, when the observation hour is unavailable, from its release time.

Table 5.

c. Quality assurance

The IGRA 2 QA system consists of a sequence of targeted algorithms, each of which is designed to look for a typical type of gross error. Gross errors most commonly occur as a result of equipment failure, data transmission problems, and data processing mistakes. Typical characteristics of such errors include exceedances of physically plausible limits; internal, vertical, and temporal inconsistencies; unrealistic repetitions of values across data levels or from sounding to sounding; excessive departures from climatological values; and unrealistic sequences of station elevations and ship positions. After each check has been applied, the values “flagged” as erroneous by that check are removed from the dataset, such that subsequent checks are no longer impacted by their presence.

A complete list of all IGRA 2 QA procedures is provided in Table 6. The overall structure of the system and many of the procedures are the same as in IGRA 1 and are described in Durre et al. (2006). A more detailed explanation of the tests for climatological outliers and vertical inconsistencies in temperature are found in Durre et al. (2008b). For IGRA 2, a number of enhancements were implemented to accommodate data characteristics that were not present in IGRA 1 and to improve the quality of the final wind and humidity data. The most significant of these modifications are described in the following sections and are summarized in Tables 7 and 8.

Table 6.

IGRA QA procedures and their impact. Abbreviations refer to data elements as follows: DD = dewpoint depression, ET = elapsed time since launch, LT = level-type indicator, p = pressure, RH = relative humidity, T = temperature, WD = wind direction, WS = wind speed, and z = geopotential height.

Table 6.
Table 7.

Semiautomatic QA checks on the positions of mobile stations. A procedure’s “flag rate” indicates the percentage of tested values that were removed from IGRA by this procedure.

Table 7.
Table 8.

New QA procedures in the automated QA system. A procedure’s flag rate indicates the percentage of tested values that were removed from IGRA by this procedure. False-positive rates were estimated on the basis of manual inspection of samples of flag values as described in Durre et al. (2008a).

Table 8.

1) Modified handling of surface levels

As discussed in Durre et al. (2006), a sounding typically contains a set of surface observations at or near the site where the balloon was launched. These observations, when provided, are identified in one way or another as the so-called surface level. Since IGRA 2 contains both thermodynamic and pibal soundings, the elevation of a surface level may be identified by pressure, height, both pressure and height, or neither. Since the height of the surface level is important for the calculation of stability parameters such as convective available potential energy (CAPE), it was deemed important that a reasonably accurate surface height be provided whenever surface observations are available as part of a sounding. Although, in theory, temporal changes in surface height and pressure reflect changes in station location, true changes in station elevation are often difficult to distinguish from spurious spikes and shifts that are caused by incorrect metadata, by the incorrect identification of surface levels, or by the integration of multiple sources reporting different elevations for the same station and time (Durre et al. 2006). Consequently, procedures were developed for removing gross errors from time series of surface heights and augmenting surface levels without a height with an estimated station elevation.

For IGRA 1, these procedures were implemented in a semiautomatic fashion that required extensive manual inspection and therefore were not reproducible. For IGRA 2, many aspects of the IGRA 1 approach were integrated into a new algorithm that can be applied fully automatically.

The algorithm calculated a monthly median elevation for each station/year/month in which a surface height was available in at least five soundings. Each time series was then checked for three types of problems: elevations that never persisted for more than three consecutive months, elevations that were grossly inconsistent with either the most recently available monthly median elevation or the elevation provided in the IGRA 2 station list, and uncharacteristically large and short spikes in elevation. These checks flagged 1.2% of all monthly median elevations as erroneous, a flag rate that was comparable to the 1% flag rate of the analogous IGRA 1 checks (Durre et al. 2006).

Every unflagged median elevation was inserted into the corresponding station/year/month’s surface levels. In 0.4% of all soundings, the newly inserted monthly median elevation differed from the originally provided surface height by more than 10 m. When a median elevation was missing or flagged, each surface height available during that month was checked against the last unflagged median elevation from a previous month or, at the beginning of the period of record, against the first unflagged median elevation in the record. If the surface height differed from that proxy elevation, it was removed from the sounding. This approach for handling months without a median elevation represented a change from the IGRA 1 system, which removed all surface heights in such months. The modified IGRA 2 approach led to the retention of 50% more surface heights in these situations compared to IGRA 1.

The algorithm also checked both the latest monthly median elevation and the station list elevation against gridded elevation model values obtained using Google and geonames.org Internet services as well as against the station’s median surface pressure, when available. Gross differences with these reference elevations were inspected manually, as were stations where all median elevations were flagged, stations for which no median elevations were produced, and stations with a significant inconsistency between the median surface pressure and the gridded elevation model. In a few cases, these inspections led to the removal of the station from consideration or to the correction of either the station coordinates or station list elevation. In addition, 14 stations for which no monthly median elevation could be calculated, and which contained a very small number of soundings, were excluded from IGRA 2.

2) Handling of mobile stations

The locations of mobile observation platforms present a special challenge in quality control. Their data are not suitable for any check that is based on a station’s climatological range or time series and therefore were excluded from such tests. However, new procedures were implemented that identified unrealistic ship movements by searching the record for anomalous behavior in the velocity of the ship. These checks were designed to be performed in a semiautomatic fashion, and the resulting list of soundings whose positions were considered to be erroneous was then provided to the automated QA process for removal from the dataset. Table 7 and Fig. 2 illustrate the types of position errors identified. All in all, 0.3% of all soundings from mobile stations were removed as a result of these checks.

Fig. 2.
Fig. 2.

Ship tracks exemplifying unrealistic ship movement. (left) Northeast course of Cargo Ship (AK) Merkur Portugal across the Atlantic Ocean in March 1992. The coordinates for the observation at 1100 UTC 11 Mar 1992 suggest the ship traveled to the south-southeast at over 100 kt (51 m s−1) for 12 h, then to the north-northeast at over 100 kt, before resuming its path to the northeast. (right) R/V Kiefu Maru in late January 1991. The ship is nearly stationary in the northeastern region of the Sea of Japan, except for one observation due west and 150 km inland in northeast China at 1200 UTC 27 Jan 1991.

Citation: Journal of Atmospheric and Oceanic Technology 35, 9; 10.1175/JTECH-D-17-0223.1

3) Other new QA procedures

Six additional checks were incorporated into the automated QA system (Table 8). Following Durre et al. (2008b), these checks were designed to detect specific types of gross errors in such a way that at least 90% of the values flagged were truly errors; that is, their false-positive rates were less than 10%. As in IGRA 1, all flagged values were removed from the dataset.

Two new checks applied to humidity. The first tested for excessive vapor pressure, as computed from either RH or dewpoint depression, whichever was available. When the vapor pressure exceeded 10% of the atmospheric pressure at the same level, both the humidity variable and temperature from which the vapor pressure was derived were removed from the data. Upon inspection of samples of the identified cases, all of them were related either to excessively high temperatures or to excessive levels of humidity. In the second test, the consistency between RH and dewpoint depression was checked. When the difference between the reported RH and the RH computed from dewpoint depression exceeded 20%, both humidity values and the temperature at the same level were removed. This check was applied to levels at which both the usually observed RH and the dewpoint depression derived therefrom by the data provider were available. Judging from an inspection of the results, the 0.0004% of all temperature–humidity pairs removed by this procedure represented inexplicable gross inconsistencies between the two variables. The 20% threshold was chosen because many of the smaller discrepancies could be explained either by differences in the computational methods employed when converting between the humidity variables (Elliott and Gaffen 1993) or by the former NWS practice of setting the dewpoint depression to 30°C when RH was less than 20% (Elliott and Gaffen 1991; Elliott et al. 1998). These discrepancies therefore represent biases, rather than gross errors, in the humidity record (McCarthy et al. 2009; Dai et al. 2011) and are retained in IGRA 2.

Given that excessive wind speeds had been found to be prevalent in a few specific months in IGRA 1, several techniques were used to determine whether similar systematic issues were present in IGRA 2. The first involved the application of the standard normalized homogeneity test (SNHT; Alexandersson 1986) to time series of monthly mean maximum wind speeds at the expected altitude of the jet stream (500–100 hPa; 6–18 km). In IGRA 1, the SNHT detected the known periods of excessive wind speeds. In IGRA 2, however, no similarly widespread or systematic issues were found. Consequently, no automated check for systematic shifts in wind speed magnitude was implemented.

However, a new procedure was added that identified individual profiles with consistently excessively low or excessively high wind speeds. Pressure-level and non-pressure-level profiles were checked separately, and a minimum of two wind speeds was required in the profile. The entire wind profile was removed when all its wind speeds were equal to zero or when all speeds equaled or exceeded 60 m s−1. Approximately one-third of the cases with zero wind speeds were in India, and approximately half were from years between 1998 and 2006, suggesting that these cases generally did not represent true observations but likely reflected processing errors such as the use of a zero in place of the missing value code. On the other end of the spectrum, the threshold of 60 m s−1 was chosen after an inspection of random samples of profiles for various categories of minimum wind speed (Durre et al. 2008a) showed that most of these profiles were clearly erroneous or extremely incomplete. Nearly 75% of the profiles thus removed originated from limited time periods in two countries, 59% from China during 1990–97 and 13% from the United States during 1973–86, suggesting that data problems (e.g., incorrect units) affected wind speed observations from those countries much more frequently during those years than at other times. The remaining profiles removed for excessive minimum wind speed were spread over several hundred stations across the world and generally contained fewer than five levels with wind speed.

For the newly added variable of elapsed time, three basic checks were implemented. First, elapsed times outside of the range between 0 and 240 min were set to missing. The upper threshold of 240 min was chosen, since soundings can last for more than 3 h (W. Blackmore 2013, personal communication), yet the frequency distribution of elapsed times longer than 240 min looks unreasonable. A second procedure looked for cases in which the same elapsed time appeared at two levels whose geopotential heights differed by 10 m or less. If one of the two levels was a mandatory pressure level, surface level, tropopause, or freezing level, then the other level was removed. Last, the logic of the height sequence check (Durre et al. 2006) was applied to elapsed time, thereby removing levels whose elapsed time was out of order with respect to pressure at pressure levels or with respect to height at nonpressure levels. All three checks combined removed approximately 0.05% of all elapsed times.

d. Compositing

While the integration scheme merged observations from different record sets with matching data, station identifiers, or station names, it did not manage to merge collocated, yet nonoverlapping, source stations when their station identifiers and names differed. To provide the longest records representative of a location in IGRA2, stations within 5 km of each other that did not have observations at the same time were composited. Additionally, Canadian stations whose WMO block number changed from 72 or 74 to 71 were combined. For example, CAM00072877 and CAM00071877 were combined to form a single record for Calgary, Alberta, Canada. This compositing step was performed on the post-QA data.

e. Results

When the station matching and integration processes were applied to the collection of data ending in the year 2016, 247 of the 11 561 source stations were eliminated from consideration because of conflicts among their matching criteria. The remainder were combined into 3113 integrated station records. During the subsequent QA process, 0.6% of the data values, including 195 entire station records containing 100 or fewer soundings, were removed from the integrated dataset. Another 131 were composited with other stations, yielding a final total of 2787 IGRA 2 stations. One-third of these had only one source, while 9% had 10 or more source stations. Station RSM00027612 in Moscow, Russia, had the largest number of source stations, with a total of 27.

4. Dataset description

IGRA 2 consists of 2787 stations worldwide, one-third of which were operating in 2016. The dataset contains roughly 46 million soundings, of which 52% include temperature measurements, with the remainder having only pibal (i.e., wind) data. The U.S. Air Force dataset contributes the largest number of observations (44%), followed by the NCEI archive (29%) and the real-time GTS collection (17%). The most common observation times are 0000 and 1200 UTC, particularly after 1958, with each hour accounting for about 30% of all soundings. Twice-daily soundings are available on approximately half of the 21 million station days in the dataset, with the other half approximately equally divided between once-per-day and more-than-twice-per-day observations.

Spatial coverage generally improves through time until the early 1990s and then declines modestly thereafter, with IGRA 2 having greater density than its predecessor for most of the record (Fig. 3). For instance, IGRA 2 has several hundred stations as early as 1940, concentrated in North America, western Europe, West Africa, and southern Asia; IGRA 1, in contrast, has only one station in Tasmania in that year. Coverage improved dramatically a quarter century later, with IGRA 2 having over 1000 stations across most global land areas, whereas IGRA 1 has large gaps in South America and Africa. Station density is greatest around 1990 in both datasets, with IGRA 2 having notably more stations in several regions (China as the most prominent example). Coverage is roughly comparable as of 2016, with each dataset having reasonable coverage in most areas except the eastern half of Africa.

Fig. 3.
Fig. 3.

Locations of IGRA 1 and IGRA 2 stations during the years of (a) 2015, (b) 1990, (c) 1965, and (d) 1940.

Citation: Journal of Atmospheric and Oceanic Technology 35, 9; 10.1175/JTECH-D-17-0223.1

Observations from fixed and mobile ships account for 0.6% of all soundings in IGRA 2. Their geographical distribution is determined by the locations and density of ship tracks (Fig. 4). Coverage is densest over the North Atlantic Ocean, the ocean with the most ship traffic, followed by the western Pacific Ocean and South Atlantic.

Fig. 4.
Fig. 4.

Locations of all observations from mobile stations and fixed ships in IGRA 2. Colors distinguish different ship tracks from each other.

Citation: Journal of Atmospheric and Oceanic Technology 35, 9; 10.1175/JTECH-D-17-0223.1

Because pibals were invented before radiosondes, there are differences in the time evolution of the network by climatic element (Fig. 5). For wind, there is a gradual increase in the number of stations from the mid-1920s until the early 1970s, then an abrupt increase in 1973 stemming from the initiation of GTS reporting, followed by a general decline starting around 1980 and continuing until present. For temperature and humidity, the number of stations increases rapidly from the mid-1940s to the mid-1950s, then more gradually until about 1990, and levels off at about 850 stations through present day. (Note that RH was generally reported until GTS was implemented in 1973; dewpoint depression has been the preferred humidity metric ever since.)

Fig. 5.
Fig. 5.

Time series of the number of IGRA 2 stations per year reporting temperature, RH, dewpoint depression, and wind.

Citation: Journal of Atmospheric and Oceanic Technology 35, 9; 10.1175/JTECH-D-17-0223.1

Owing to gradual improvements in the quality of both the balloons and the equipment used for tracking them, there is a general increase in the vertical extent of soundings through time (Fig. 6). This is particularly apparent for soundings reaching 30 hPa, which increased in number almost monotonically from 1950 to the present. The increase is also evident for soundings reaching 500 and 100 hPa, except for an abrupt jump in 1973 coinciding with the beginning of the U.S. Air Force data (Table 1) and a similar drop in 1990, around the time at which many of the country-specific data sources end.

Fig. 6.
Fig. 6.

Number of IGRA 2 soundings reaching the 500-, 100-, and 30-hPa level in each year.

Citation: Journal of Atmospheric and Oceanic Technology 35, 9; 10.1175/JTECH-D-17-0223.1

There are a total of 1.4 billion levels across all soundings. Approximately two-thirds of these are pressure levels, while the remaining levels have only geopotential height as a vertical coordinate. Pressure levels include both mandatory pressure levels and fixed pressures between 1000 and 1 hPa, and additional pressure levels whose location varies from sounding to sounding.

5. Ancillary products, versioning, and updates

Like IGRA 1, IGRA 2 also includes two derived products, sounding-derived parameters, and monthly means, as well as a collection of station metadata.

The sounding-derived parameters include a variety of moisture and stability metrics intended to facilitate studies of the vertical structure of the troposphere and lower stratosphere (Durre and Yin 2008, 2011). They are available at the subset of approximately 1500 stations at which at least 100 soundings contain temperature, pressure, and height at the surface as well as temperature at one or more nonsurface pressure levels. Relative to the IGRA 1–derived parameters, this represents an increase of 34%. As a beneficial result of the inclusion of RH in IGRA 2, parameters relying on humidity measurements are now provided in many more soundings. For example, the number of soundings containing precipitable water (PW) increased from 9 million in IGRA 1 to 23 million in IGRA 2.

Monthly means of temperature, geopotential height, vapor pressure, and zonal and meridional wind components at the surface and standard pressure levels for the nominal times of 0000 and 1200 UTC are provided whenever at least 10 values are available for a particular station/month/level/variable (Durre et al. 2006). The monthly mean of vapor pressure, which is new in IGRA 2, is calculated by first converting the individual dewpoint depression or RH values to vapor pressure, following Durre and Yin (2008), and then averaging the resulting vapor pressure values. The number of stations with monthly means varies by variable and nominal time, ranging from around 1400 for pressure and temperature to around 1900 for wind.

Two types of station history information are also provided to assist users in the interpretation of the data: documented station metadata and instrument/equipment codes received during automated transmissions of some soundings. The documented station metadata, which were already included in IGRA 1, were enhanced to include documented changes in instrumentation, observing and reporting practices, processing conventions, and station locations for stations that are new in IGRA 2. The information was originally compiled by Gaffen (1993) and was later augmented at NCDC for IGRA 1, through personal communication with representatives from WMO member countries and from a variety of WMO reports and other publications. Because this material has not been updated since 2010 and information for the 2000s is less complete than that for earlier decades, radiosonde and equipment codes extracted from GTS reports dating back to 2000 were made available separately to supplement the documented information. No attempt was made to check the supplied codes for accuracy or for consistency with other station history information, except insofar as invalid nonnumeric codes were removed. The codes identify the type of radiosonde and tracking equipment used at any given time. However, because the WMO routinely reassigns radiosonde codes as old instruments are retired and new ones are placed into operation, users are advised to carefully examine the relevant WMO code tables when determining the meaning of any such code.

Beginning with IGRA 2.0.0, a single-version identifier is assigned to the entire suite of IGRA 2 sounding data, sounding-derived parameters, monthly means, and station metadata. The version number is of the form “X.Y.Z,” where X, Y, and Z are defined as follows: X is incremented when a major change to the source data or processing procedures is implemented, a change in Y represents the modification or addition of a few percent of the sounding data or one of its derived products, and a change in Z reflects less extensive changes.

The sounding data and sounding-derived parameters are updated once a day such that soundings are first published in IGRA 2 some 1–2 days after they are observed. On the sixth day of every month, the monthly means are also recalculated to include data from the previous month. Every year or two, the entire IGRA 2 dataset and associated products will be recreated to incorporate newly opened stations and to subject the most recent complete year’s data to the full set of QA procedures, including checks for temporal runs and outliers that cannot be applied during daily updates (Durre et al. 2006, 2016a). Updated versions of data sources and processing procedures may also be included at that time. Any fully reprocessed dataset will be identified by a new version number whose first, second, or third digit is incremented consistent with the extent of changes made.

6. Summary

Version 2 of IGRA was designed to serve as a foundational dataset of radiosonde and pibal observations that were drawn from multiple different data sources and from which gross errors were removed. As such, it is suitable for a variety of modeling applications, atmospheric process studies, and many other purposes. Analyses of long-term trends, however, require the consideration of inhomogeneities resulting from instrument biases and station moves. Adjustments for such inhomogeneities have intentionally not been applied to IGRA 2, thus allowing the dataset to serve as the basis for the application of different bias-adjustment approaches.

The increased coverage in IGRA 2 relative to IGRA 1 was achieved primarily by including approximately 30 additional data sources, allowing RH as a humidity variable in addition to dewpoint depression, implementing the design procedures for merging multiple data sources to form unique IGRA 2 station records, and enhancing the QA system. With this paper we have attempted to provide an overview of these enhancements. Readers interested in greater detail about the processing methods and data sources are referred to the dataset description document that is distributed along with the IGRA 2 data (Durre et al. 2016a).

Acknowledgments

The authors thank Anne Markel for her assistance in editing this manuscript, James Kossin and the three anonymous reviewers for their constructive comments, and individuals at the agencies listed in Table 1 for providing the source data for IGRA. We also are indebted to the many individuals from around the world who, since the beginning of the twentieth century, have been collecting and recording the observations that make a collection such as IGRA possible.

REFERENCES

Save
  • Fig. 1.

    Data-flow diagram showing the progress of source data from their original format to the final step of creating the IGRA dataset.

  • Fig. 2.

    Ship tracks exemplifying unrealistic ship movement. (left) Northeast course of Cargo Ship (AK) Merkur Portugal across the Atlantic Ocean in March 1992. The coordinates for the observation at 1100 UTC 11 Mar 1992 suggest the ship traveled to the south-southeast at over 100 kt (51 m s−1) for 12 h, then to the north-northeast at over 100 kt, before resuming its path to the northeast. (right) R/V Kiefu Maru in late January 1991. The ship is nearly stationary in the northeastern region of the Sea of Japan, except for one observation due west and 150 km inland in northeast China at 1200 UTC 27 Jan 1991.

  • Fig. 3.

    Locations of IGRA 1 and IGRA 2 stations during the years of (a) 2015, (b) 1990, (c) 1965, and (d) 1940.

  • Fig. 4.

    Locations of all observations from mobile stations and fixed ships in IGRA 2. Colors distinguish different ship tracks from each other.

  • Fig. 5.

    Time series of the number of IGRA 2 stations per year reporting temperature, RH, dewpoint depression, and wind.

  • Fig. 6.

    Number of IGRA 2 soundings reaching the 500-, 100-, and 30-hPa level in each year.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 2727 1055 154
PDF Downloads 2025 554 45
  翻译: