NPOESS Advisory Committee for Observing Systems Simulation Experiments

 

Report No. 4

 

May 2001

 

Introduction

 

The NPOESS Observing System Simulation Experiments (OSSE) Advisory Committee provides technical oversight and scientific guidance to the investigators involved in the NPOESS OSSE project.  These investigators have requested input from a committee representing the potential users of the polar-orbiting satellite data forthcoming from the future NPOESS satellites’ sensor suites.  The Advisory Committee has been asked to convene as necessary to review progress on development and implementation of the OSSE system, and report on the progress to the NPOESS Integrated Program Office.  This offering constitutes the fourth in a series of such reports.

 

The fourth committee meeting was held at the National Centers for Environmental Prediction, Environmental Modeling Center’s office in the World Weather Building, Camp Springs, MD on May 22-23, 2001.  In attendance at that meeting were the following committee members: Akira Kasahara, T. N. Krishnamurti, Jeff McQueen, Jan Paegle, Edwin Eloranta, and chairperson Donald Norquist. At the meeting, the committee heard presentations from the NPOESS OSSE project investigators (hereafter referred to as the OSSE Team). This meeting added an additional half-day to allow the committee to discuss the information presented and prepare a feedback briefing. The briefing was presented to the OSSE Team at the conclusion of the meeting.  The following discussion summarizes the information provided by the OSSE Team presentations and the feedback given by the committee. Issues considered important by the committee and corresponding recommendations are contained within the feedback sections of the discussion.

 

Overview – Steve Lord, NCEP/EMC

 

Steve first presented the intended purposes for this NPOESS OSSE Advisory Committee Meeting. The first purpose was to update the committee on OSSE Team activities since the last committee meeting (September 1999). The second purpose was to have the committee consider the following issues: the level of readiness of the OSSE calibration experiments, and the appropriateness of the strategy for the Doppler wind lidar (DWL) OSSEs. He then reviewed the committee’s recommendations from the previous committee meeting. They were:

1.      Consistency between Nature Run cloud cover and cloud water content

2.      Non-NPOESS OSSEs to assess potential changes in actual observing system between 1993 (Nature Run date) and present, and between present and 2007 (NPOESS flight)

3.      Inclusion of systematic error in simulated observations

4.      Cloud and precipitation metrics

5.      Survey of NPOESS sensors and rational choices of candidates for OSSEs

6.      Sufficient computing resources

Information on items 1, 2, 3 and 6 were presented at the current committee meeting. These will be discussed in the relevant sections of this report.

 

Steve also introduced the committee to the newly established NASA/NOAA Joint Center for Satellite Data Assimilation (JCSDA). Under the current arrangement, the NPOESS OSSE project is receiving funding from three separate sources. Under the proposal, the project would be a funded activity of the JCSDA. Among other advantages, this arrangement opens the possibility of OSSE and related observing system experiment projects continuing after the operation of the NPOESS satellites begin. It is hoped that more funding stability and increased computing resources might also be forthcoming under the JCSDA.

 

Committee Feedback:

 

The committee pointed out a need for a written plan that describes in detail the organization, inter-relationship of responsibilities, and accomplishments of the various task principals on the OSSE Team. The committee felt that such a document, which can and should be dynamic (subject to continual updating) in nature, would be helpful both to the team and onlookers (such as the committee). Lucid communication and understanding of roles among team members is always a benefit to a team effort, especially when several institutions are involved. In a related recommendation, the committee feels that development of a timeline for when each OSSE accomplishment will be implemented in NCEP operations would be of value to both NPOESS/IPO decision-makers (hereafter referred to as “management”) as well as onlooking groups that depend on NCEP global data assimilation system (GDAS) products. The committee described an “accomplishment” as the completion of algorithm or technique development, stand-alone testing, and testing within the OSSE framework. Such a timeline allows tracking of progress by committee and management, calls attention to need to routinely provide proposed sensor impact to IPO for decision-making, and gives NCEP guidance on lead-times available to be prepared to assimilate new observing systems. The committee stressed that the proposed timeline should not force decisions related to the execution of the OSSE project. That is, serious compromises should not be made in the OSSE project just to meet the timelines. Instead, the timeline should be like the recommended organization plan in that it is primarily intended for information and general guidance. Both can serve as reference for when quick answers are required regarding organization and schedule – if all team members are subscribers to these “living” documents, then all will answer the questions with the same answers. Finally, the committee emphasized the need for the OSSE Team to get the information from the OSSEs to the IPO as soon as possible. Instead of waiting until calendar-dictated report times, a mechanism should be established to effectively communicate OSSE sensor impact findings (and a thorough explanation of their implications and caveats) to the OSSE Team’s immediate contacts at the IPO. The primary reason for the committee’s concern is the perception that the IPO instrument selection is ahead of OSSEs. Immediate communication of OSSE findings (of any type) will likely prove useful to decision-makers in authorizing and selecting instrument design strategies. In addition, it will improve the perceived value of the OSSE effort in the view of the IPO.

 

NCEP/EMC Global Data Assimilation System Activities – John Derber, NCEP/EMC

 

John first described a number of changes made to the NCEP/EMC GDAS in the period since the last OSSE committee meeting. One of the changes made was the inclusion of a capability to assimilate precipitation information from the Defense Meteorological Satellite Organization’s SSM/I satellite sensor. A related fundamental change to the NCEP global numerical weather prediction model (an integral part of the GDAS) is the implementation of a prognostic cloud water formulation. Other satellite-related GDAS developments include assimilation of NOAA-16 Advanced Microwave Sounding Unit (AMSU) – A and AMSU – B and NOAA-15 AMSU – B sounding radiances, changes to AMSU – A quality control and cloud liquid water bias correction, less thinning of AMSU – A measurements, enhancements to the data handling code within the GDAS to allow easier inclusion of new satellite or other observing systems, and inclusion of GOES-8 and GOES-10 sounding radiances over the ocean. Other general enhancements to the GDAS recently implemented include accounting for actual position (vice launch position) of rawinsonde, dropsonde and pilot balloon sensors (considered increasingly important with decreasing model grid spacing), improvements in ozone assimilation, improved parallelization of computer code to increase throughput on massively parallel computing platforms, and a corresponding overall code documentation improvement for the GDAS.

 

The next part of John’s presentation focused on preparations for the assimilation of radiances from the satellite-borne Advanced Infrared Sounder (AIRS) sensor. A vast spatial coverage of data being informally obtained from NESDIS is being thinned to ¼ density. In addition, from the approximately 2000 sounder channels available, some 280 channels are being assessed for actual use. Because this is currently considered an experimental (not routinely operational) satellite sensor, the formal NASA to NESDIS to NCEP data path is not in effect. A temporary setup has been arranged to allow the flow of some of the data to NCEP for the purposes of GDAS preparation. In terms of actual GDAS enhancements to accommodate the AIRS radiances, the relevant radiative transfer code has been recoded, made more portable, and has included new scientific aspects. The scientific enhancements include development of quality control procedures (related to cloud effect considerations), re-evaluation of channel selection and improved surface emissivity over land and ice.

 

John next presented near-term plans for the modification of the GDAS techniques. NCEP/EMC will develop a capability for computing and using situation-dependent background error covariances for both the global and regional assimilation systems. A recursive filter tool, similar to that being used in the regional system, will be employed in the GDAS to define error covariances. Another modification planned is the introduction of bias correction of the background (short-term forecast) fields. This is considered important, since assimilation theory assumes that both background fields and observations are unbiased. Other plans include the improvement of the moisture field background error covariances, cross-correlations between moisture/ozone and mass/wind fields, inclusion of surface skin temperature and soil moisture analysis, and beginning the development of a cloud analysis system. Also during the next 1-2 years, NCEP/EMC plans to add the following types of observations to those being assimilated by the GDAS: QuickScat sea surface winds, satellite winds, GPS radio-occultation, GIFTS satellite high resolution interferometer, DMSP SSM/IS, imager radiances from MODIS, GOES, and AVHRR sensors, and NPP/NPOESS experimental sensors as they become available.

 

Committee Feedback:

 

The committee stated that a higher priority should be placed on preparation for operational ingest of “research” satellite sensor observations (e.g., AIRS) into GDAS. Use of these sensors allows NCEP to learn how to use the information effectively, potentially results in beneficial impacts to operational forecasts, and provides information on the utility of the observations to the sensor’s principal investigators. NCEP and NESDIS operations should take early initiative to formalize the data flow from satellite to operations rather than waiting until the sensor is flying to construct an informal arrangement. Interagency cooperation needs to be carried out at management levels to arrange for effective satellite to operations data flow. A selling point to convince management of the benefit of such an undertaking is the opportunity for NCEP to advance in operational capability with respect to the international community. Thus there is a need for the interagency development of a time schedule for the “research” satellites and when the NCEP GDAS will incorporate their observations.

 

Another panel recommendation was the use of the information obtained from observing system experiment impacts to select the important channels from experimental satellite sensors from the massive number available. The framework that the OSSE effort has built in carrying out its “real data experiments” could serve as a parallel test capability to run such experiments. This is an example of a satellite sensor coming on line before OSSEs could be conducted to determine the best way to use the sensor data. Alternatively, stand-alone techniques such as principle component analysis could be employed to “down select” useful channels from the thousands available from the newest satellite sensors.

 

Finally, the committee commented that the planned bias correction of the background fields in the GDAS should not be developed using the OSSE Nature Runs as a reference. Instead, an automated bias correction formulation should be designed that can adjust automatically to any changes in the data assimilation system. Again, the OSSE project software infrastructure can serve as a useful parallel testing bench for such a modification to the GDAS.

 

Simulation of Conventional and GOES Wind Observations – Joe Terry and Bob Atlas, NASA/DAO

 

Joe Terry began the presentation with a discussion of the global distribution of conventional and satellite observations and how they have changed from 1993 to the present. While the number of operational rawinsonde soundings have decreased slightly worldwide, the number of aircraft, ship and buoy observations have seen a significant increase. Expected beyond 2001 is an expansion of ACARS aircraft observations, additional buoy observations, water vapor winds, and an increased number of satellite-obtained ocean surface winds. The implications for the OSSE project is that all of these near-term additions to the global observation suite will have to be simulated so that meaningful estimates of the additional impact of NPOESS sensors can be obtained.

 

Joe next discussed the issue of deriving cloud drift wind CDW information from GOES satellite imagery. Up to now, the CDW algorithm was applied to real satellite imagery from the period of the Nature Run. A new effort is being undertaken to simulate cloud drift winds with respect to the Nature Run clouds. Joe showed comparisons between the satellite observed cloud fraction and the real cloud drift wind locations versus the locations of clouds as depicted by the corresponding Nature Run scenes. In simulating other observing systems great pains are taken to locate the observations where they were actually taken. In cloud drift winds, this is not always possible because the Nature Run cloud locations may not match the locations of the original imagery clouds. Because clouds are often associated with important weather features, cloud drift winds can provide crucial information to the initial state of a forecast that may not be available from other routines observing systems. Differences in assigned locations of the cloud drift winds can lead to differences in the initial state circulations and thus differences in ensuing forecasts. It is considered especially important to run comprehensive tests of simulated cloud drift winds in data assimilation calibration experiments, to make sure their impact is similar to real cloud drift wind experiments.

 

Joe next covered the simulation of satellite ocean surface winds, such as obtained from ASCAT and QuickScat sensor systems. A simulation strategy being considered is to obtain representative current samples of the data and use them to generate perfect simulated observations from the Nature Run  10 meter winds. Then appropriate errors could be added based on information obtained from validation error statistics.

 

Bob Atlas briefly commented on the simulation of lidar winds. One of the concerns in the present OSSE design is that a lot of small-scale motion information is unresolved by the current Nature Run data set. Given the finer spatial scales that the proposed Doppler wind lidar (DWL) will be likely to detect, this poses a possible limitation of the usefulness of the Nature Run to simulate DWL winds. With this in mind, NASA/DAO has generated a 3.5 month-long nature run data set at 0.5 degree latitude-longitude grid spacing using their Finite Volume Community Climate Model (FVCCM). Bob showed animations of the simulated wind vectors that demonstrated the fine-scale circulations generated by the FVCCM. Another possible benefit of the FVCCM nature run is the greater number of cyclones it generated during the NPOESS OSSE Nature Run period compared to the Nature Run currently being used – closer to the actual number depicted in the “reanalysis” data set of the real conditions.

 

Committee Feedback

 

The NASA/DAO presentations raised the following question in the minds of committee members: What is the OSSE team role of the NASA/DAO effort? This was not clear to the panel. There seemed to be duplication of effort vis-à-vis simulation of conventional observations (DAO vs. NCEP), efforts on simulation of DWL (DAO vs SWA), role in defining nature run to be used in OSSE (ECMWF vs FVCCM. Subsequent discussion at the meeting cleared up this issue to some degree. For example, the committee was told that DAO is responsible for the simulation of the observations from existing observing systems for the OSSEs including the addition of random errors, while NCEP is investigating methods of suitably adding systematic error (biases) to the simulated observations (see following discussion of Jack Woollen’s presentation). It was explained to the committee that simulation of proposed DWL winds was indeed the role of Simpson Weather Associates, but that NASA/DAO was anticipating the need for proper nature run conditions to do so realistically. Additionally, the FVCCM may be used to generate nature run data sets for other seasons, and would be subjected to the same level of pre-experimentation scrutiny as was applied to the ECMWF Nature Run.

 

Still, to avoid the appearance of duplication among the various activities of the OSSE Team members, the committee recommended that a detailed restatement of the current and proposed future assignments of the OSSE Team members be developed as a part of an overall plan for organization and accomplishments of principal investigators (see General Feedback section below). A related question asked by the committee was: What is the role of the JCSDA in coordinating the activities of the various OSSE team members? Subsequent communication with the OSSE team suggests less of a coordinating role and more of a source of consistent and sufficient funding for the JCSDA. This needs to be spelled out and communicated. Perhaps the transfer of authority for the NPOESS OSSE Project to the JCSDA is the right time to redefine roles and responsibilities of the members of the OSSE team.

 

Simulation of Satellite Radiances – Tom Kleespies, NOAA/NESDIS

 

Tom first reviewed the status of this part of the NPOESS OSSE project as of the previous committee meeting. At that time, the infrastructure for simulating the cloud cleared and limb adjusted TOVS radiances was almost completed. NCEP/EMC had just been modified to assimilate the raw unadjusted (called level 1-B) radiances, and it was clear that the simulation infrastructure would have to be adapted accordingly. One-time-only funding was identified to complete the work for the TOVS radiances, but future funding would have to be forthcoming to simulate the proposed NPOESS sounder radiances. The level 1-B TOVS radiances were simulated as follows. First, the actual 1-B radiance soundings for the period of the Nature Run were established as the reference for the simulations. The Nature Run temperature, mixing ratio, pressure profile, cloud amount, surface wind speed, and skin temperature were interpolated in time and space to the actual level 1-B TOVS profiles locations and times within three hours of each Nature Run time. The calibration coefficients used with the real level 1-B radiances were used in the radiative transfer models used to compute the simulated 1-B radiances. Tom then showed the various radiative transfer models used to perform the various necessary functions involved in the simulations. In virtually all cases, the radiative transfer model chosen for simulation was different from the model used in the NCEP GDAS. Tom then discussed the assumptions made in assigning cloud geometries for the simulations and the assignment of instrumental noise to the simulated radiances. At this point in time, the TOVS level 1-B radiances have been simulated for the HIRS and MSU sensors on board the NOAA 11 and 12 satellites for the Nature Run period. The advanced microwave sounding unit (AMSU-A) level 1-B radiances, while not available in 1993, have been simulated as if they were measured by the same two satellites in the Nature Run period. Tom then showed example map plots depicting the brightness temperatures from certain channels of both the real and simulated level 1-B radiances. While the spatial patterns were generally similar, some degree of difference was apparent in the magnitudes between the depictions.

 

Tom then gave a brief summary of the current status of the NPOESS OSSE Project satellite radiance simulation task. Once again, funding for the contractor doing the work ran out in September 2000. Another one-time-only allotment was obtained to carry the contractor through until December 2000. Because the Department of Commerce budget was not signed until mid-December, the contractor did not continue past December. A new contract was signed in April 2001, and work will resume pending the assignment of new contract personnel. Future work on this task includes the simulation of the AIRS, IASI, and CrIS which are infrared sensors proposed for NPOESS, as well as the NPOESS-proposed SSMIS, ATMS and CMIS microwave sounders. Some work on the AIRS simulation infrastructure has begun.

 

Panel Feedback:

 

One point of concern raised by the committee was the fact that no discrimination of ice versus liquid cloud information is contained in the current Nature Run. This means that there can be no accounting for this phase discrimination in the simulation of the IR radiances. Yet these phases have vastly different effects on the radiative transfer equation solutions. Another concern voiced was that a blackbody assumption is being used for the clouds in the simulations, which can result in colder brightness temperatures where high clouds exist in the nature run than from actual radiances which partially see through the transmissive high clouds. No committee recommendations were given to correct these deficiencies.  The committee recognizes the importance to the OSSEs of properly simulating existing and GDAS-implemented satellite sensor radiances. The committee acknowledges the difficulty of the problem, suggesting that higher funding and manpower support should be given, as meaningful OSSEs await the successful simulations of the current satellite sensors. Regarding currently deployed or near-term experimental (non-operational) satellite sensors - what is their priority in simulation for the OSSEs, how can we be sure their radiative transfer algorithms work well? This revisits the recommendation given in the GDAS section above regarding a schedule for the operational use of data from the research satellites. For the future proposed NPOESS sensors, the committee is concerned that satellite radiance simulation is lagging behind other NPOESS OSSE efforts, even their implementation into GDAS. For example, work is well underway to include AIRS radiances in the GDAS, but their simulation has barely begun. The committee recommends that significant resources be committed to development of radiative transfer algorithms for proposed NPOESS sensors in a timely manner for use by sensor designers and NPOESS decision-makers in the Integrated Program Office (IPO). Finally, much information is now known about future sensor design as a result of progress in the sensor design and selection process. The OSSE Team should use this information to prioritize and determine the order of the simulation of these sensors for the OSSEs.

 

Simulation of Doppler Wind Lidar Data – Dave Emmitt, Simpson Weather Associates

 

A Doppler wind lidar is the only active remote sensor proposed for the NPOESS suite. Dave Emmitt is responsible for the simulation of the line of sight (LOS) winds that should be detected by a typical but unspecified lidar technology. The fact that the task is to remain “technology neutral” naturally leads to the employment of “bracketing OSSEs.” A range of achievable specifications are simulated in separate experiments in terms of vertical extent, coverage, and degree of penetration through cloud by the sensor “shots.” The scenarios are constrained by the assumption of a single satellite in sun-synchronous polar orbit. Each bracketing experiment specifies the coverage (horizontal and vertical) and accuracy considering both random and correlated errors. Dave described some general DWL concepts, including the difference between the major two lidar technologies (direct detection and coherent detection) and scanning versus non-scanning schemes.

 

The Nature Run winds are being used to simulate the DWL winds for the OSSEs. The coverage being assumed in the simulations is based on a 2000 km swath width, which is the maximum attainable by a conical scanning system (assumed), leaving a gap of approximately 500 km between scans at the Equator. Each simulated line of sight (LOS) wind observation represents a 200 X 200 km target area, allowing for approximately 30 individual shots to be used to form a single LOS estimate. The NCEP GDAS is actually assimilating simulated LOS winds from two different angle “looks” of the same target area, from two (displaced in time) different satellite positions (a forward look and and an aft look). In the vertical, a 1 km spacing of the LOS observations is assumed above 2 km altitude, and 500 m spacing below 2 km. Accuracy is accounted for in the simulated winds first by assignment of random error taking into consideration signal-to-noise ration, pointing jitter, and other instrument sources of mis-measurement. A typical range for random error assigned is 0.5 – 10 m/s. The error assumption also accounts for the number of shots that actually fall incident within a target area. Because partial cloud cover can block some shots from reaching the lowest cloud-free altitude of their penetration (thus leaving fewer shots to measure wind at this destination altitude), it is assumed that the error increases as the number of incident shots decreases. Factors that go into assigning systematic error include horizontal and vertical error correlations, and the instrument and platform types as well as the actual atmospheric conditions. Dave presented a list of factors that contribute to systematic error that are instrument unique, platform induced, atmosphere independent and atmosphere/earth surface condition dependent. Cloud effects that are being considered in the simulations are the degree of attenuation (ranging from weak in thin cirrus to strong in optically thick clouds), strong signals from the clouds themselves, and solar bias (only in the case of direct detection).

 

Dave discussed the four OSSE experiments for which simulations have been prepared to date. They include the differences between clustered and distributed shot design, single and dual (forward and aft), aerosol, molecular and cloud returns, and the number of shots used to produce the reported LOS observation. The actual experiment names and descriptions will be given in the “Calibration Experiments” section below. Dave closed his talk by giving a perspective on the number of simulated LOS wind observations that were generated for the four experiments. Something like 72,000 wind profiles were produced from the Nature Run for each 24-hour period using the cloud effect assumptions. The number of actual wind reports ranged from 58,000 assuming cluster shots and only cloud returns, to 1.6 million assuming distributed shots and aerosol and molecular returns.

 

Committee Feedback:

 

The committee had formerly pointed out a need to account for consistency between cloud cover and cloud water content in the Nature Run when simulating shot penetration. There is now a recognition that the information content of the Nature Run does not provide sufficient detail to account for both cloud aspects. Therefore, the committee supports the current approach to dealing with clouds – the Nature Run clouds (lack of fidelity) cannot support attempts to account for lidar propagation through clouds. The distributed vs. clustered simulation approach is a concern: it is not clear the Nature Run will correctly predict the number of shots that get through the reported cloud coverage given its fidelity. Thus there is a need to be aware that the vertical distribution of lidar returns from Nature Run data is not necessarily a true reflection of reality. This means that this approach does not give enough support for selecting one sensing strategy over another. This reinforces the earlier contention that the OSSE impacts from DWL can be very sensitive to the distribution of cloudiness in the database from which the DWL LOS winds are simulated.

 

Adding Systematic Errors to Simulated Conventional Observations – Jack Woollen, NCEP/EMC

 

Observations are simulated for assimilation in OSSEs by combining a true value extracted from a nature run with an estimate of noise, also known as observation error. The random component of the error consists of instantaneous mis-measurement by the sensor (instrument error) and information on spatial/temporal scales that cannot be resolved by the assimilating model (representativeness error). The systematic component of the error is a bias resulting from some innate time-independent inability of the sensor to measure the absolute value of the actual environmental conditions. Jack’s presentation discussed two methods of assigning error to simulated observations. Method 1 involves adding randomly generated noise to the true value, using error levels consistent with objective estimates of real instrument and representative error typical for each sensor type and parameter measured. Method 2 adds an empirically derived estimate of observation error to each true value in the global, sensor ensemble over the duration of the nature run. The observation error estimates are derived from real data assimilations. The focus of Jack’s report was an investigation of Method 2 estimates for a single parameter - zonal wind component - resulting from real data assimilation experiments conducted twice daily over a week-long segment of the Nature Run period (February 1993).

 

The basic assumption used in estimating the Method 2 errors is that the observation – analysis difference resulting from assimilation of a particular sensor type and parameter represent the portion of the observation information not influencing the resulting analysis at the observation site. This is considered “noise” or error to the assimilation system. The difference would consist of both random and systematic error at any place and time, but if averaged over time and space the random component would vanish leaving an estimate of just the systematic component. Jack showed a number of time-height cross sections of the rms and mean of the observation – analysis difference of zonal wind component for the North American rawinsondes for the seven-day period of the investigation. The results showed that the mean difference in space and time is not insignificant. Jack contended that the point by point observation – analysis difference from real experiments may be a reasonable estimate of the actual observation errors. Because they match the locations and times of the simulated true values from the nature run, they could be added to the nature run true values to approximate real observation errors on a point by point (in time and space) basis. However, due to the nature of the extraction techniques to produce the true values from the current Nature Run, the number of real and true observations diverge increasingly below (lower in altitude than) 700 hPa. Because TOVS retrievals were used in the real data assimilation experiments while TOVS radiances were used in the OSSEs, the no-TOVS assimilation had to be used as the basis for the point by point empirical error experiment. The point by point observation – analysis differences were added to true values as the Method 2 observation error to the OSSE assimilation. Forecasts using both Method 1 and Method 2 error assignments were conducted and verified against the true observations. He compared the results to the same forecast errors from the control (including TOVS radiances) real data assimilation. His conclusions from the results were that the forecasts using only the Method 1 (random error) were unrealistically good, while those from Method 2 (full empirical error) were more realistic. The observation error rejection rate by the data assimilation is another measure of realism. Jack found that the number of rejected observations in the Method 1 experiments was unrealistically small, while the number of rejected observations from Method 2 was much more in agreement with that of the control experiment. Empirical errors contain gross error as well as bias, and are applied at the proper times and locations.  His results suggested that the empirical error prescription method has merit and should be investigated further.

 

Committee Feedback:

 

The committee stated a need to clarify the role of this project with respect to the simulation of observing errors by NASA/DAO. It seems that the results shown suggest that the approach being used by DAO to assign errors to conventional observations (like Method 1) could be improved upon. The OSSE Team needs to decide if the NCEP/EMC investigation will likely produce definitive results in time to affect the current set of simulated conventional observations, or whether a latter round of OSSEs might possibly be conducted if the current investigation shows the merit of the empirical error assignment method. The panel recognizes the need for including realistic high-resolution “noise” (non-resolvable scales of motion) in the simulation of observation error (the nature run may be too smooth in comparison with current operational GDAS resolutions). It is possible that the point by point empirical method could take care of this if both real and OSSE experiments are conducted at the same grid spacing (so that noise level in the OSSEs is imposed at the same level as the real experiments).

 

Calibration Experiments – Michiko Masutani, NCEP/EMC

 

In conducting OSSEs, it is important to ensure that the analysis and forecast impacts of the simulated observations of existing observing systems are similar to their counterparts in real data experiments. Only then can the additional impact of new proposed observing systems be realistically assessed from the OSSEs, because they are an attempt to “look ahead” to the time when the new observations would be added to the observational suite in future real data assimilation. Michiko described her task of conducting experiments to determine the level of similarity between counterpart synthetic and real data assimilations, often called calibration experiments. The basic methodology is to conduct the data assimilation in both synthetic and real environments over some period of time beginning with the “no data” experiment as a worst case reference. In successive data assimilation experiments, the experiment is repeated over the time period, but now using different combinations of the real and simulated existing observing system observations in the respective environment. The degree of impact that each additional observing system observation type has on the analysis or short-term forecasts is compared between the real and synthetic environments. If there are significant differences between the two for any given combination of included observation types, this suggests that there may be an error in the way the synthetic observations have been simulated or are being used in the data assimilation.

 

The present set of calibration experiments were conducted using the March 1999 version of the NCEP/EMC GDAS, an earlier version of the spectral statistical interpolation still in current use. The assimilating model used a triangular 62 spectral truncation and 28 vertical levels. Both real and synthetic experiments used the distribution of actual observations existing in February 1993, including RAOBs and other conventional data sources, ACARS aircraft observations, HIRS and MSU level 1-B radiances from NOAA-11 and NOAA-12 satellites, and satellite cloud track winds. The ECMWF Nature Run was used to simulate the conventional and ACARS data and cloud track winds (NASA/DAO) and the level 1-B radiances (NESDIS) using the methods described in current and past presentations by these groups. The Nature Run has a triangular 213 truncation and 31 vertical levels, and covers the period 06 UTC 5 February 1993 to 00 UTC 7 March 1993. In order to attempt to simulate the level of difference between the real analyses and the real world in the synthetic environment, Michiko ran a five day forecast from the Nature Run initial conditions to 10 February, then ran full observing system data assimilations in the synthetic environment to 13 February. Thus at 13 February she had prepared a “starting point” atmospheric state in the synthetic environment that should be similar in observation sensitivity to that of the real analysis environment. At 00 UTC 13 February she began the synthetic and real data assimilation experiments using or withholding the various combinations of the following four groups of observation types: TOVS 1-B radiances, RAOB temperature, RAOB winds, and all other conventional data (ACARS, cloud track wind, surface observations). The control experiment used all four groups, and Experiments 1-6 used varying combinations of the groups. Experiment 7 used only the “all other conventional data” group and represented the sparsest data experiment shown. The experiment forecast – control forecast root-mean-square-errors from six-hour forecasts four times daily resulting from the respective real and synthetic data assimilations were computed as a measure of observing system impact. Michiko showed a number of charts of the RMSEs for zonal wind and temperature in differing latitude zones and at several isobaric levels over a four-day period. The patterns of growth of the error between the real and synthetic experiments compared favorably in most cases. One exception was the 850 hPa temperature in the 20S to 20N latitude zone, in which some of the synthetic experiments showed higher RMSEs than their real counterparts. Michiko suggested that this may be due to the use of a fixed sea surface temperature in the Nature Run (from which the synthetic observations are simulated) compared to a variable sea surface temperature in the real analyses. She intends to study this case further to assure that the simulations are correct. Michiko also showed charts of the RMSE of 500 hPa wind for both analysis and 72-hour forecasts averaged for the period 16-18 February at 12-hourly intervals. Later she showed RMSE for height, temperature and zonal wind at 500 hPa for 0-120 hour forecasts. Also shown was anomaly correlation of 500 hPa height for 12-hour intervals to 120 hours of forecast time for various wave number bands. In all validations the respective control analyses were used as reference for forecasts from both the real and synthetic analyses.

 

In the next part of her presentation, Michiko went on to the first of the OSSEs conducted in this project. She described four experiments involving simulated Doppler wind lidar line-of-sight (LOS) observations as provided by Simpson Weather Associates. They were: (Exp 1) full tropospheric LOS soundings, clouds permitting, (Exp 2) only wind observations from clouds and (where clear) the planetary boundary layer, (Exp 3) mid and upper tropospheric wind observations down to the levels of significant cloud coverage, (Exp 4) wind observations from a non-scanning instrument providing full tropospheric LOS soundings, clouds permitting, along a single line that parallels the ground track. All of the DWL simulation assumptions are discussed in the SWA presentation described earlier in this report. Two additional experimental factors were considered for each experiment: clustered versus distributed, and greater observation weight versus lesser observation weight (no measurement error was assigned). In the clustered assumption, the data product is based on averaging lidar shots clustered within a very small area compared to the target area. In the distributed assumption, the data product is based on averaging the lidar shots distributed throughout the target area as would result from continuous conical scanning.

 

Michiko showed a number of charts of RMSE differences for Exp 1 and Exp 4, both with both assumptions of clustered-distributed and lesser-greater observation weights (eight experiments in all). RMSE differences was calculated from the RMSE of experiment – nature run minus RMSE of control – nature run for each of the eight OSSEs. For both experiment and control the 6h forecast was used, and the verifying time nature run field was used as reference. Thus all OSSEs with a beneficial impact should have negative values of the RMSE differences, the larger the negative value the greater the impact. Results were shown for zonal wind at 200 and 850 hPa in three latitude zones, and temperature at 850 hPa in three latitude zones. In virtually all results shown, the Exp 1 experiments showed greater beneficial impact that the Exp 4 experiments. Among the Exp1 results, the two distributed assumption experiments showed more impact than the two clustered, and for both the greater observation weighting assumption experiments showed more impact than the lesser. In all experiments, greatest impact in the zonal wind was seen in the tropics, with a lesser but significant impact in the Southern (summer) hemisphere and and least amount in the Northern (winter) hemisphere extratropics. The only significant temperature impact was seen in the Southern hemisphere extratropics.

 

Committee Feedback:

 

In the presentation of calibration experiments and the observing system simulation experiments, the committee recommends that the goals and objectives should be clearly stated up front. It is important that any audience, especially the uninitiated, should be able to understand what is being done and why. For example, it could be confusing to a lay person to see calibration experiments in which existing sensor observations are withheld when he/she thinks the goal is to show the benefit of using new sensor observations. Also, the level of detail that should be presented will depend on the audience to which it is directed. While the kind of information shown (including goals and objectives up front) might be suitable for a group of peer researchers, it would be confusing and unnecessary detail to higher level management. The committee felt that a simple explanation of the what and why of the calibration experiments and a word summary of the findings would suffice for project management. Management wants to be convinced that the OSSE Team is satisfied that the OSSE experimental design is sound and that all observing systems have been simulated correctly and are used in a realistic manner in the assimilations. Even for technical audiences, there is a need to synthesize and summarize the calibration study results.

 

There is a need to include a reference magnitude for observing system impacts in both the calibration experiments and the OSSEs. For example, how do these impacts compare with the difference between two different data assimilation systems (e.g., NCEP vs DAO)? Such a reference should be provided so that level of significance can be identified.

 

Other issues that arose from this presentation included the implications of unknowns and uncertainties of the Nature Run, and the need for spectral analyses of motion scales in the experiments compared with those that can be measured by proposed observing systems. Because these issues have implications for all aspects of the OSSE Team activities, they will be discussed in further detail in the General Committee Feedback section below.

 

General Committee Feedback

 

The committee identified a number of strong points in the OSSE Team presentations. They included:

·        Careful attention to observational impacts – significant amount of work on assessing the OSSE impacts

·        Good use of metrics in the impact experiments (e.g., anomaly correlation by wave number groups)

·        Color handouts were useful – panel found them helpful

·        We found the coordination of efforts to be good

connection between analysis of obs-anal differences and the observation impacts DWL simulations used in observation impacts (DWL OSSEs).

 

The committee had a number of general concerns and suggestions for the OSSE team. First, the committee considers it vital to distill the OSSE Team’s products to be more consumable by decision makers. Much information has been given, but it is not organized in a manner that can effectively inform project management and funding decision makers. So the next step is to pull together an efficient telling of the OSSE Team’s story. Included in this presentation should the next step in OSSE assessment - the potential forecast improvement needs to be shown in order to emphasize the utility of the OSSE procedure. Second, the committee is concerned with OSSE Team concerns about the perception of funding agencies regarding rate of progress. A lot of the behind the scenes work that goes into ensuring an accurate assessment of impact (simulation of observations, assignment of errors, calibration experiments) takes the bulk of the time and effort, yet is never seen by management. The actual OSSE impacts of the proposed sensors is the tip of the iceberg in the overall OSSE project. One way to allay management’s and funding agency concern about rate of progress is to report on an ongoing basis. This reiterates the suggestion given in the “Overview” section above to present pertinent information on progress to the immediate contact persons at the IPO and other funding agencies. Perhaps this issue will be alleviated to some extent in the future by reporting to just one authority – the JCSDA. An important selling point to management is the perceived importance of gaining ground in international competition in forecast improvement from NPOESS sensors. The U.S. is on the leading edge of sensor technology, and it should be more argued that this advanced technology can be more quickly and effectively exploited in weather prediction if financial resources are applied now to preparing assimilation systems to use and demonstrating the impact of new sensor technologies. On a related issue, the OSSE Team expressed concerns about prospective large amounts of data, especially from near-term experimental satellites, and how they can be handled. The committee advises that the OSSE Team communicate NCEP GDAS plans to decision-makers for preparation for and use of these data. This reiterates the earlier suggestion of a timeline for the implementation of research satellite data in the assimilation system.

 

The committee raised concerns about the level of complexity of the OSSE design, especially the detailed focus on simulating observations and the details presented in the calibration experiments. This makes the findings of the project difficult for decision-makers to understand. One possible simplification is to emphasize impacts on forecasts, but do so in a way the viewer can understand and appreciate. There is concern by some committee members about level of technical detail being considered to date. A suggestion for simplification of the calibration experiments is adding incrementally increasing numbers of randomly located observations, without regard to simulating current observing systems. This can be used to demonstrate the principle of forecast impact which is the main purpose for the OSSEs. So much focus on carefulness in simulating the various observing technologies is only worthwhile after impacts are shown in this type of simple experiment.

 

Another issue mentioned was the complications to impact experiments from unknowns and uncertainties of the Nature Run. The lack of a variable sea surface temperature specification in the Nature Run and the lack of a consequent ocean surface temperature anomaly in the tropics was cited as an example. Considerable attention has been given to certifying the realism of the Nature Run, especially in the clouds. Experiments may uncover other Nature Run deficiencies that will have to be investigated on a case by case basis. In the case of the sea surface temperature issue, it was suggested that the real experiments be conducted with the same fixed sea surface temperature field used in the Nature Run.

 

Still another committee suggestion was to perform spectral analysis on the scales of motion of simulated observations vs. first guess fields, and control experiment (all conventional observations and existing satellite radiances) vs. control plus DWL. Then compare the spectral analyses from the synthetic environment with that of the corresponding components of the real experiments. This is another way to assess the realism of the OSSEs: is the degree of high frequency information contained in the simulated observations (and thus, the Nature Run itself) similar to that of the real atmosphere – is a realistic level noise being used? At T62, too much of the higher frequency information may be filtered out in the assimilating model integration – the spectral analyses is one means of quantifying this. The committee encourages the use of alternative NWP model investigation of high-resolution motion scales (e.g., MM5, FVCCM, ECMWF) and the simulated observation impacts on these scales. In fact, on the basis of the aforementioned motion scales investigation, it may be found that the current Nature Run contains too little energy in the higher-frequency scales of motion to properly capture the noise associated with real observations. Another committee suggestion was to perform diagnostic studies, including spectral analysis study, on various geographic regions of the full-resolution Nature Run, then compare the motion scales from the T62 impact experiments to see what scales the latter are missing. Use this information to design higher-resolution impact studies.

 

The committee recommends that emphasis should be placed on demonstrating the OSSE infrastructure utility at higher spatial resolution. The committee recognizes the fact that computational resources have limited the spatial resolution at which the calibration experiments and OSSEs to date have been conducted. Priority should be placed on acquiring or accessing computing assets to accomplish the more realistic, higher resolution experiments reflecting current and future operational data assimilation systems. This should be briefed as a high priority requirement to the JCSDA.

 

It is likely that not all of the suggestions given above are feasible for implementation. Time, manpower and fiscal constraints may preclude the OSSE Team from acting on all of the suggestions. This is understandable and the committee in no way implies an obligation to try them all. The OSSE Team should consider each suggestion and determine which should be done, and which have the most payoff for the least amount of investment of time and energy.

 

Summary

 

The committee was satisfied with the level of progress achieved since the last meeting. The OSSE Team has made favorable advances in all areas of the project, in spite of funding, manpower and computational limitations. The committee feels that the calibration experiments, besides their likely limitation of coarse resolution (to be further investigated in the suggested spectral studies), demonstrate that the “control experiment” that serves as a baseline for new sensor impact is ready to go. This positions the team well to focus their efforts on the OSSEs themselves. It is advantageous for the team to do so, to provide information on new sensor impact even as the proposed sensor design is being considered and selected. If the OSSE team could gear up to perform “on demand” OSSEs for each proposed sensor in a timely manner by establishing a strong dialogue with and connection to with the NPOESS instrument specialists through the IPO, they can perform a valuable service to the NPOESS project. The OSSE Team is nearly in a position to offer their services in this way, and should do so by putting together their story in a concise and effective manner and selling it to the IPO and instrument specialists. The team needs to emphasize a “Look what we can do for you” sales pitch to the NPOESS sensor teams and get their support for the OSSE Team’s active role in the sensor design and selection process. This is the best guarantee for continued future funding.