NPOESS Advisory Committee for Observing Systems Simulation Experiments

Report No. 3

September 1999

Introduction

The NPOESS Observing System Simulation Experiments (OSSE) Advisory Committee provides technical oversight and scientific guidance to the investigators involved in the NPOESS OSSE project. These investigators have requested input from a committee representing the potential users of the polar-orbiting satellite data forthcoming from the future NPOESS satellites' sensor suites. The Advisory Committee has been asked to convene as necessary to review progress on development and implementation of the OSSE system, and report on the progress to the NPOESS Integrated Program Office. This offering constitutes the third in a series of such reports.

Progress Summary

The third committee meeting was held at the National Centers for Environmental Prediction, Environmental Modeling Center's office in the World Weather Building, Camp Springs, MD on September 28, 1999. In attendance at that meeting were the following committee members: Akira Kasahara, T. N. Krishnamurti, Greg Mandt, Jan Paegle, Edwin Eloranta, and chairperson Donald Norquist. Prior to the meeting, the committee received a set of documents from the NPOESS OSSE project investigators (hereafter referred to as the OSSE Team) that described progress on the project since the last committee meeting in May 1998. At the meeting, the committee heard presentations from several of the OSSE Team members, and from a representative of the NPOESS Integrated Program Office (IPO). The following discussion summarizes the collective information provided in the document set and at the meeting.

Steve Lord of NCEP/EMC, the OSSE Team leader, presented an overview of the progress of the OSSE Team since the last advisory committee meeting. He also distributed a document entitled "Observing System Simulation Experiments for NPOESS" which was his presentation in written form. Steve listed the participants on the OSSE Team, representing NOAA/NWS/NCEP/EMC, NASA/GSFC, Simpson Weather Associates, and NOAA/NESDIS. He reminded the committee that the two objectives of the NPOESS OSSEs are: to test the NWP impact of advanced instruments using simulated data and make recommendations for configuration based on quantitative NWP impact of NPOESS instruments, and to prepare for use of advanced instrument suite as soon as possible after space deployment. Candidate instruments being simulated in the OSSEs are Doppler Wind Lidar (DWL), Cross track infrared counter (CrIS), Conically scanning Microwave Imager/Sounder (CMIS), and Advanced Technology Microwave Sounder (ATMS).

Steve reported that the following new work had been completed since the last committee meeting:

report on Nature Run cloud evaluation and adjustment (with input from ECMWF),

OSSE data base upgraded to accommodate DWL line-of-sight (LOS) winds,

NCEP global data assimilation model (SSI) upgraded to use LOS winds,

conventional and DWL observations simulated, TOVS data simulation underway,

real data sensitivity test for existing instruments repeated using current SSI and TOVS 1B radiance data,

OSSE data assimilation environment setup continuing, including enabling of software for application on massively parallel computer architecture,

evaluated simulated DWL winds scenario sensitivity results,

evaluated simulated conventional observations.

Steve next discussed the present status of computing resources available to NCEP for the NPOESS OSSEs. First, the NCEP operational massively parallel computer, the IBM SP, is not available to the project by NWS mandate. This and other "reimbursable" projects (that is, not NWS core funded) will not receive support of any kind from NWS operations. Next, computer resources that have been used will become unavailable soon. An example is the NCEP Cray 5 (J916) computer, which will be taken out of service in Spring 2000. Steve stated that a lack of computer resources has greatly hindered progress on the OSSE project, and the situation will only worsen now that the OSSE experiments themselves are imminent. To alleviate these shortfalls in computing resources, the OSSE project has teamed with other NCEP-based reimbursable projects to purchase an SGI Origin 2000 computer with 32 processors. The OSSE Team is currently involved in porting all OSSE-related software (configured for massively-parallel architectures) from the IBM SP. The NCEP-based reimbursable projects will have to share the Origin 2000, and are expected to overtax the system at times. This will likely present the need for the purchase of another "projects" computer if funds from the projects are available.

In conclusion, Steve presented the following planned tasks for the OSSE Team in fiscal year 2000:

complete sensitivity tests for real data (existing instruments): forecasts and analysis of results,

complete sensitivity tests for simulated data (existing instruments) "baseline" OSSE, including completion of simulated TOVS, calibration with real data experiments, and any necessary adjustment of simulated data error levels,

continue analysis sensitivity tests for simulated DWL data, including DWL technology scenarios,

consider non-NPOESS data that may be available in the NPOESS-operational era,

perform OSSEs with simulated DWL data,

develop and test adaptive observation strategies for DWL data,

develop and execute simulation strategies for other potential NPOESS instruments: CrIS, CMIS, ATMS.

Michiko Masutani of NCEP/EMC next presented results from the real data sensitivity tests for existing instruments, repeated using current SSI and TOVS 1B radiance data. The resolution of the global forecast model (the NCEP MRF) used in the data assimilation was triangular 62 wave truncation with 28 vertical levels. The NCEP/NCAR Reanalysis data set was used for initial conditions and verification. The tests were conducted over the period 1 Jan 28 Feb 1993. Three types of conventional data withheld in the experiments were TOVS 1B (NOTOVS), rawinsonde winds (NOWINS), and radiosonde temperature (NOTEMS). Evaluation of the impacts of withholding combinations of the data types showed that: in the Northern Hemisphere, NOWINS has more impact than NOTEMS, NOTOVS has slightly less impact than NOTEMS; in the Southern Hemisphere, NOTOVS has much greater impact than NOTEMS or NOWINS; in the tropics, NOTEMS had greater impact than NOWINS or NOTOVS. In general, the results showed that wind observations are more important in the wind analysis than are temperature observations, wind observations are at least as important as temperature observations in the temperature/height analysis, TOVS radiances are most important in the Southern Hemisphere, temperature is most important in the tropical western Pacific. Initial adjustment (to the withholding of data) takes about 20 days to complete. This may have implications for the minimum duration of the nature run in the OSSEs. Forecasts from the tests' analyses are yet to be executed and evaluated.

Michiko Masutani then presented results and conclusions of the evaluation of the nature run (NR) cloud cover to be used in the OSSEs. A written report was distributed to committee members before the meeting. NR clouds were compared with cloud analyses from the Air Force (RTNEPH), International Satellite Cloud Climatology Project (ISCCP), NOAA/NESDIS (CLAVR, currently just total cloud cover), and Warren (surface observations only). Some of the major findings from the study were as follows:

NR total cloud cover is in general agreement with the cloud analyses,

large differences between the cloud analyses preclude definitive conclusions about NR cloud quality,

NR appears to have too much high cloud, but uncertainties in reference analyses do not justify adjustments,

uncertainties in mid-level cloud do not justify adjustments in NR clouds,

NR had less low cloud than ISCCP and RTNEPH to an extent that adjustments seemed necessary: excessive clouds over snow were reduced, insufficient marine stratocumulus were enhanced.

Subsequent discussion centered on the implications of the reference analysis uncertainties, particularly in the high cloud estimate. Steve Lord suggested that since there is so much uncertainty in the specification of the real cloud cover state, it may be necessary to run a range of OSSEs in which the cloud cover are set to a range of values bracketed by possible bounds.

Sid Wood of Simpson Weather Associates (SWA) presented an overview of their lidar simulation model ( ""

Joe Terry of NASA/GLA gave a report on the simulation of the conventional observations from the NR data set. BUFR-formatted real observations (that were used in the NCEP/NCAR Reanalysis) and the NR data are combined in a process to extract simulated "observations" from the NR data set. The next step is to introduce random error in the simulated observations, using published observing error standard deviations. This represents an attempt to account for instrument error and spatial representativeness error. Then the rawinsonde drift simulator algorithm is used to apply modifications to the rawinsonde simulations to account for spatial and temporal displacement of the sonde during ascent. To date, the following types of conventional data have been simulated for January and February 1993: rawinsonde and pilot balloons, aircraft reports, buoy, PAOB, surface observations, and cloud motion winds.

Discussion following the report centered on two topics. First, how to account for the increase in types of observing systems that are available globally since 1993? How do we account for this in the OSSEs, so that we can realistically predict the impact of the NPOESS instrument suite when it is fielded? Second, random error is being added to the extracted observations, but not systematic error. We know there are biases in the observations (notably, relative humidity). Is it important to account for this in simulating the conventional observations, and if so, how?

Jack Woollen of NCEP/EMC-GSC gave a presentation on the Doppler Wind Lidar database for use with the NCEP global data assimilation model (SSI). In addition to time and location of the DWL LOS wind profile, the following parameters are included for each vertical level in the profile: altitude, elevation angle, azimuth, number of laser shots in product, backscatter source index, atmospheric depth of product layer, horizontal LOS component, standard deviation of LOS estimates, DWL quality mark. The simulated DWL LOS winds assumes a conical scan strategy, having two LOS wind estimates in each 200 km grid cell. These could be used to compute the (u,v) component wind profile, but the SSI can assimilate the LOS estimates directly.

Tom Kleespies of NESDIS gave a brief update of the progress on the simulation of the satellite sounding data. The two primary goals of the effort are to make realistic simulations of TOVS radiances from the NR data to test the OSSE sounding infrastructure, and then to extend the simulation technology to the advanced sounders planned for NPOESS. The necessary ingredients in the simulation process are: radiative transfer model, specification of geotemporal locations for the soundings, the nature run, specification of realistic instrument noise, and specification of scene noise. The phenomena that are necessary for inclusion in the radiative transfer model are transmittance, emissivity, cloud, cloud liquid water, and precipitation. For the OSSE, the radiative transfer model (also known as the forward model) to apply to the NR data to simulate radiances should be similar to but sufficiently different from that used in the data assimilation process. The radiative transfer model known as OPTRAN has been selected for use in the assimilation process, and the model known as RTTOV5 will be used to simulate the radiance data. Tom presented tables comparing the features of these two forward models. Specification of geotemporal locations should be realistic, and ideally taken from orbit geometry and sampling strategies planned for the NPOESS sensor suite. For simulated TOVS, these can be obtained from the real TOVS 1B data corresponding to the OSSE period. For the NPOESS advanced instruments, this will require a knowledge of the scan patterns and orbital parameters. Parameters used from the NR data for satellite sounding simulations include profiles of temperature, moisture, cloud liquid water, and cloud fraction, as well as surface temperature and soil moisture. Ozone profiles for the period will be obtained from the Climate Prediction Center. Realistic instrument noise can best be specified from the TOVS existing HIRS and MSU instrument estimates by monitoring space and black body calibration targets. Instrument noise for advanced sounders will have to be based on engineering "best guesses." Scene noise specification should include inhomogeneous features in field of view, unresolved clouds, precipitation, and other similar effects. The current status of the satellite sounding simulation component of the OSSE project is: (1) the infrastructure for simulating the cloud cleared, limb adjusted radiances was almost completed when the contract expired, (2) NCEP data assimilation model is now assimilating TOVS 1B radiance data, which are not cloud-cleared or limb adjusted, so (3) the modified simulation infrastructure will have to account for the presence of clouds in the simulated 1B radiances. One-time-only funds have been identified to accomplish the simulation of the TOVS 1B radiances, and the work is to start October 1. Continued funding from the NPOESS IPO is required to accomplish the simulation of the advanced sounders.

Michiko Masutani presented an evaluation of simulated DWL winds scenario sensitivity results. Some preliminary OSSEs were conducted using the February 6 and 7 real and simulated observational data for the following experiments: real conventional data (identical to NCEP analysis), simulated conventional data, simulated DWL data and real conventional data, and simulated DWL data and simulated conventional data. The latter experiment was conducted in three DWL technology scenarios: best case, moderate technology, and worst case (see paragraph above on Sid Wood's presentation). Starting from the NCEP analysis without TOVS radiances, each OSSE involved the assimilation of the designated observational data over the two-day period. Because it has no data simulated from (extracted from) the NR, the comparison of the real conventional data analysis and the NR should yield the largest differences. Then each OSSE involving simulated data should bring the resulting analysis closer to the NR (truth). In the DWL OSSEs the simulated DWL winds were assimilated as LOS estimates directly, which required the transformation of the analysis background fields of (u,v) wind components to LOS estimates to compute the observation increment. A six-hour window (± 3 hours) about the analysis time was used in the 3DVar assimilation. Results showed that the best case DWL simulated data reduced differences considerably from the real data-NR differences, while the worst case DWL simulated data had little effect on the differences.

Dave Emmitt of Simpson Weather Associates made a presentation on progress made in Doppler Wind Lidar (DWL) wind measurements. His talk focussed on developments occurring across the DWL research and development community. Dave gave a brief description of some initiatives in DWL measurement systems being undertaken by agencies of the U.S. government. Also, at least two European agencies are conducting research and development activities in DWL measurements and impact studies. In summary, much activity is ongoing in agencies sponsoring DWL research that is of relevance to NPOESS DWL project.

Dave next focused his remarks on the current NPOESS DWL simulation study. He commented on the bracketing scenarios being considered for the OSSEs. These are described on the SWA web site given above. He stated that what is not being accounted for in the proposed bracketing assumptions includes realistic hardware, realistic backscatter distribution, attenuation factors, and situation dependent cases. He feels that not accounting for these factors puts them in a position to do trade studies that is, how much gain can be gotten from how much extra cost of the DWL system.

Dave described the consensus agreements made at a meeting on metrics for measuring impacts of the OSSEs, conducted in July 1999 and involving OSSE Team members. The metrics that were agreed upon at that meeting were:

number of cyclones, positions, and intensities,

number of jets, their positions and core speeds,

occurrence and maximum and minimum temperature locations (including number of extreme events)

anomaly correlation by spectral binning,

data rejection statistics,

flight path winds,

shear layers, location and strength.

Subsequent discussion centered on the need for a water vapor metric. Advisory committee members expressed the need for the OSSE Team to consider such a metric, for example water vapor anomaly. The Team is somewhat dubious at this time as to how much information can be gained from the inclusion of a water vapor metric. One reason is that the impacts on water vapor distribution can be very much dependent on the characteristics of the prediction model used in the OSSEs.

Joe Terry briefly presented work being done at NASA/GLA on cyclone tracking methods. The characteristics of individual cyclones are of interest in the OSSE evaluations, and there is a need to associate cyclones simulated in the OSSEs with corresponding cyclones in the NR. Joe described a method under study at NASA/GLA that identifies individual cyclones from the global weather parameter grids using the sea level pressure field to find the centroid of the innermost closed contour. The cyclone tracking method he described estimates the forecast position of the cyclone center by extrapolation or 500 hPa steering. A search is conducted in the resulting area of estimate to determine the whether the cyclone's state is one of continuation, cyclolosis, or cyclogenesis.

Steve Lord next informed the advisory committee of highlights from a workshop on the evaluation of space-based lidar technology, hosted by the Office of Oceanographic and Atmospheric Research on September 23, 1999. NESDIS is seeking guidance from OAR on a possible future data buy. More information on this is contained in the following paragraph. In addition, regional and global lidar wind impact OSSEs may be conducted under the auspices of OAR. This represents a separate OSSE study for which funding may be limited to one year.

Jim Ellickson of NOAA/NESDIS next gave a brief discussion of the proposed NOAA/NESDIS commercial global data buy. This idea began in fiscal year 1998 when NESDIS began consideration of the feasibility of commercializing satellite weather data collection. Jim explained that in a data buy program, the government enters into a contract with a private company. The contract is taken to a lending institution to borrow capital needed to build, launch, and operate the observing platform and system. The contract would obligate the government to purchase the data if contractual agreements on data quality, timeliness, and coverage were met. Public meetings and requests for information are used by the government to collect information from potential contractors necessary to formulate the request for proposals. OSSEs and other NPOESS-funded projects would supply information to the contract winners, as this information is all public domain. The data buy approach is a possible alternative to a government build and field approach. A cost study is being conducted at NASA-Stennis to estimate the cost to the government of the latter approach.

Steve Mango of the NPOESS IPO presented some remarks on the NPOESS program in general and the OSSE project in particular. The IPO wants to continue to fund the OSSEs. The payloads depend on it, as they need the information in their design and fabrication. Payload proposals have been evaluated and winners have been announced. Now the instruments are in the engineering management and development phase. By the year 2008, a constellation of three coordinated polar orbiters will be in operation two from NPOESS, and one from Europe (the METOP satellite). Steve showed a list of notional payloads necessary to satisfy IORD-1, the existing operational requirements document. There are instruments planned for the NPOESS sensor suite for which no OSSEs are planned. Steve said that the timelines of fielding these instruments dictates that their development has to be done in parallel with the OSSEs. The OSSE team has to make the hard choices to select which instruments to simulate in their OSSEs the instruments likely to have the most impact on global weather analysis and prediction. NPOESS currently faces an $80M reduction in FY2000 out of a total $160M budget. The combined DOC and DoD contribution is $35M less than the FY1999 appropriation. In spite of these projected budget shortfalls, the NPOESS IPO remains committed to fund the OSSE effort.

Issues and Recommendations

Issue: Definition of "cloud" in the ECMWF Nature Run data set. In particular, the need for a standard "threshold" measure of cloud on which cloud coverage is based, such as a threshold level of optical depth, emissivity, or ice water content. It is accepted that most conventional satellite analyses of high level clouds report too little high cloud. Since the greatest impact of clouds on the conduct of the OSSE is likely to be in the simulation of DWL wind observations, it may be necessary to use a supplemental measure of cloud density, like threshold optical depth or ice/liquid water content.

Recommendation: The OSSE Team should supplement their study of the nature run cloud cover with a evaluation of the consistency between NR cloud cover and cloud water content. If the two cloud indicators are consistent in time and space, then threshold levels of cloud water content as dictated by the characteristics of the bracketing lidar technologies should be used to indicate the essential presence or absence of cloud insofar as the lidar beam interception or non-interception is concerned. Cloud cover may be used as before to estimate the probability of interception by cloud that is considered opaque.

Issue: "Bracketing experiments" have been proposed to test the sensitivity of weather analyses and forecasts to a range of lidar power and sampling options. Do these experiments give us confidence that a particular technology will be realistically simulated for use in the OSSEs? Are these options relevant enough to feasible lidar technologies?

Recommendation: The NPOESS OSSE Team should comprehensively survey all proposed or operating lidar technologies and, with space deployment in mind, use these to check the bracketing assumptions proposed for the OSSEs. Conservatively, all assumptions should lie within the range of viable proposed or operating lidar technologies. The survey should include both domestic and international lidar researchers. The Oregon lidar meeting held earlier this year should provide substantial information for such a survey.

Issue: The number of global observations routinely available is (1999) already greater than in 1993 (date of nature run). How do we account for this in the OSSEs? Perhaps more importantly, how can we project the observational coverage and its impact for when the NPOESS sensor package is flown? How do we reconcile the need to determine likely future impact with the need to simulate exactly the same observations present in 1993 for OSSE calibrations? Does this suggest the need for a series of sequentially more future OSSEs, in which incrementally more and newer observing systems are simulated, culminating in the NPOESS sensor suite? Is there a possibility that NPOESS instruments' impacts will be overtaken by progress in other sensor systems?

Recommendation: The advisory committee does not expect the OSSE Team to foretell the future of availability and accuracy of observing system in the next ten years. However, we feel that it is crucial to estimate the impact of NPOESS sensors on global weather analysis and prediction with the future and not the present constellation of non-NPOESS observing systems in mind. It is possible that, with the advances in observing systems and atmospheric modeling systems seen recently, it may be very hard to show appreciable impact from the NPOESS sensors, particularly in the Northern Hemisphere. One aspect of the projection of assimilation of data in global weather analysis systems that must be factored in is the capacity and efficiency of computer systems available in the future. One OSSE Team task planned for FY2000 is entitled "Consider the 2008 data world." We recommend that a series of non-NPOESS OSSEs be conducted that attempt to account for the evolution of observing systems and prediction systems between now and 2008. Another suggestion is to rerun critical OSSE experiments between now and 2008 whenever major changes occur in the assimilation system or the globally available observations. Finally, the OSSEs might profit by their expansion to the wider research community. This would require making a community model, perhaps using the NCEP assimilation system as a template. Then researchers (especially instrument scientists) could run their own OSSEs and estimate their sensor's impacts on the community system. This allows for competition and intercomparison of results, and distributes the labor and computational burden. It does require significant documentation, management, and servicing.

Issue: Random error is being added to the simulated conventional observations, but not systematic (mean) error. We know that there are biases in the observations (notably relative humidity). Is it important to account for this in the simulation of the observations? If so, how can we do this? Is there any such information that can be used? Or can the (known) systematic error simply be removed from the observations before they are assimilated?

Recommendations: The OSSE Team could review other recent OSSEs reported in the literature to determine if systematic error of sensor systems was accounted for in the simulations. Then the sensitivity of the OSSEs to imposed systematic error could be tested by repeating a preliminary OSSE experiment in which the magnitude of systematic error imposed on certain simulated conventional observations is varied between experiments. Is the response linear with the imposed error level? If the error is imposed on a moisture observation, what effect does it have on a mass or motion field analysis or prediction, or vice versa? The impacts from these experiments may suggest the benefit of removing systematic errors (if known) from observations before they are assimilated.

Issue: Metrics for evaluating the impact of future observing systems were presented at the meeting. The advisory panel suggested that a water vapor metric be included, such as water vapor anomaly. Another was a wind impact, more spatially widespread than the list presented at the meeting. That list was:

number of cyclones, positions, intensities

number of jets, positions, core speeds

number occurrence and locations of max. and min. temperatures (plus # of extremes)

anomaly correlation by spectral binning

data rejection statistics

flight path winds

shear layers, location, and strength.

Recommendation: The OSSE Team should look at recent observing system experiments conducted at NCEP and elsewhere to assess the metrics used in the assessment of impact by candidate observing systems. With the increase emphasis on cloud and precipitation prediction in the meteorological community, it seems likely that suitable moisture impact metrics may be identified. On the other hand, it may be found through such a study that attempts to measure impacts from or on the atmospheric moisture distribution are frustrated by the sensitivity of the individual NWP model formulations. Wind impact is expected to be high from high powered lidar technologies. Translating the physical impact into an economic impact would be a highly beneficial metric.

Issue: There seems to be more instruments planned for NPOESS than there are OSSEs planned for impact assessment. Unfortunately, the timelines of fielding these instruments dictate that their development has to be done in parallel with the OSSEs. The NPOESS IPO is asking the OSSE team to make the hard choices to select which instruments to simulate in their experiments, and how to simulate them. Does this mean to choose to include the instruments hypothesized to have the more marginal impact, to guide the NPOESS IPO in their funding decisions as to which instruments actually get deployed? Or does this mean to choose to include the instruments likely to have the greatest impact, so that assimilation software systems can be fully developed, tested, and in place to use these major-contributing sensors as soon as they begin sending real data?

Recommendation: In cooperation with the NPOESS IPO, the OSSE Team should re-examine the subset of sensors they have chosen for inclusion in the simulation impact studies. The advisory panel has yet to see a rational justification for the choice of sensors to be simulated in the OSSEs. What do these sensors potentially have to offer that makes them a better choice for representation in the OSSEs than the sensors not chosen? Do we run the risk of not being prepared to assimilate the data from NPOESS sensors that may have significant but undiscovered potential? The OSSE Team should provide an interim report on this topic to the advisory committee, with a reasoned and comprehensive survey of each notional payload planned for IORD-I, and why the payload is or is not represented in the planned OSSEs, and an explanation of the simulation approach for the selected payloads.

Issue: A lack of affordable and accessible computing resources looms as a bottleneck in the execution of the many OSSEs planned in the next phase of the project. This shortfall could cause significant delays in the processing of the experiments, which in turn could delay the information getting to the NPOESS mission planners. Creative solutions to this problem must be found this fiscal year (FY2000). What are the options, to avoid the costly purchase of additional computer hardware? It was mentioned that special project money (like NPOESS) has been pooled from several such projects to purchase some limited computer resources. Much more are needed because these have to be shared among the projects. Are there accessible government computer systems available? How do we find out, and can they be used for the OSSEs with little or no cost to the project?

Recommendation: The OSSE Team should determine the availability, accessibility, and feasibility of use of DoD computing resources, so that not all of the burden rests on computers local to NCEP. The DoD High Performance Computing Modernization Office http://www.hpcmo.hpc.mil/ would be a great place to start. While it may be felt that the disadvantages of shipping files back and forth from remote computational centers would outweigh their usefulness, it should be kept in mind that the OSSE project does not require real-time turnaround. It may be that the government already has adequate computational resources to handle the OSSE computations in a timely manner, obviating the need for the purchase of more. This avenue should be seriously explored before committing additional scarce resources to the purchase of new computational hardware. Other possible solutions include: international cooperation (for example, cooperation with Japanese scientists who will have enormous supercomputing capabilities in two or

three years in connection with the development of the earth simulator at the Frontier Research System for Global Change, Tokyo, Japan); asking the other sponsoring organizations, NOAA and NASA, to help with computer time support (operational data assimilation centers, like NOAA and NASA, will benefit from the development of assimilation methodologies for the OSSEs).