NPOESS Advisory Committee for Observing Systems Simulation Experiments

Report No. 2

May 1998

Introduction

The NPOESS Observing System Simulation Experiments (OSSE) Advisory Committee provides technical oversight and scientific guidance to the investigators involved in the NPOESS OSSE project. These investigators have requested input from a committee representing the potential users of the polar-orbiting satellite data forthcoming from the future NPOESS satellites' sensor suites. The Advisory Committee has been asked to convene as necessary to review progress on development and implementation of the OSSE system, and report on the progress to the NPOESS Integrated Program Office. This offering constitutes the second in a series of such reports.

Progress Summary

The second committee meeting was held at the National Centers for Environmental Prediction, Environmental Modeling Center's office in the World Weather Building, Camp Springs, MD on May 19, 1998. In attendance at that meeting were the following committee members: Akira Kasahara, T. N. Krishnamurti, Greg Mandt, Jan Paegle, Paul Hays, and chairperson Donald Norquist. Prior to the meeting, the committee received a copy of the fiscal year 1997 Final Report from the NPOESS OSSE project investigators (hereafter referred to as the OSSE Team). At the meeting, the committee heard presentations from several of the OSSE Team members, and from a representative of the NPOESS Integrated Program Office (IPO). The following discussion summarizes the collective information provided in the report and at the meeting.

Steve Lord of NCEP/EMC, the OSSE Team leader, reviewed the agenda and gave an introduction. He stated that the issue of cost savings was a major factor for the motivation of the OSSEs in support of the NPOESS program. Spending a modest amount of money now in carrying out studies evaluating the impact of atmospheric measurements from various satellite sensor designs could provide valuable information in determining the allocation of large amounts of money later for sensor fabrication. He cited a program being conducted by NOAA called the North American Atmospheric Observing System (NAOS), which involves a series of observing system experiments, as being a potential source of useful information for the current NPOESS OSSE project. Steve then reviewed the basic OSSE design, and listed a number of issues that a well-designed OSSE study must address to have validity. Among these concerns were: present numerical weather prediction (NWP) models and assimilation systems are different from what they will be when the proposed sensor data is available; importance of fairness and objectivity in proposed instrument specification, to avoid a vested interest;

methodological concerns, such as how close a proxy is the "nature run" (the global NWP model simulation used in the OSSE to represent a simulation of the real atmosphere) to the real atmosphere; the realism of simulated observations; calibration of the OSSEs against real data observing system experiments. Steve then briefly summarized the current status of the present OSSE project:

ECMWF nature run data was received, distributed to investigators, and evaluated;

NCEP pre-OSSE forecast system (T62/L28) was completed;

real observation sensitivity experiments (control, NOTOVS, NORAOBWINDS) were completed for February 1993 (same period as nature run);

TOVS soundings were simulated from the nature run and evaluated against observed TOVS;

full data assimilation and archiving system for quality control, analysis, and forecast model has been constructed.

Steve stated that though two letters have been sent to the IPO requesting guidance on recommended sensor specifications to be used in the OSSE study, no such information has been received. Finally, he said that NCEP computer resources could pose a limitation to the extent of the OSSE experiments that can be conducted.

Michiko Masutani of NCEP/EMC next presented an introduction to the evaluation of the nature run. The nature run is a 30-day forecast from the T213/L31 version of the ECMWF global NWP model covering the period 5 February - 7 March 1993. Besides the forecast spectral coefficients, simulated observations from most conventional observing systems were provided in the nature run data set acquired from ECMWF. Michiko showed that no El Nino was occurring during this time, so that the month is representative a typical Northern Hemisphere winter/Southern Hemisphere summer month. She showed evidence that suggests that the major features of outgoing longwave radiation pattern are captured by the nature run. The nature run cloud cover, as generated by the prognostic cloud scheme in the ECMWF model, revealed some possible problems in its realism. First, the zonal mean high cloud cover decreases about 15 % during the first several days of the nature run, then levels out for the duration. Second, in comparison with RTNEPH clouds for the same period, the nature run has a greater total cloud cover worldwide. The largest differences are in polar regions where uncertainties are greatest. Perhaps the most disturbing finding is that the nature run has greater high cloud cover than low cloud cover, while RTNEPH has greater low cloud cover than high cloud cover. This discrepancy is most pronounced in the tropics. This finding, if confirmed by subsequent study, could have profound implications for satellite sensor sampling of the nature run.

Sid Wood of Simpson Weather Associates (SWA), an OSSE Team member, gave a short presentation of some statistical results of their evaluation of the nature run. Their study focused on an examination of those aspects of the nature run atmosphere that would affect sampling distribution and simulation design for the Doppler Wind Lidar (DWL). For example, SWA determined the spatial and temporal extent of the nature run atmosphere that can be sampled given cloud attenuation affects. Sid showed the percentage of grid points that were cloud free as a function of latitude, which showed that the tropics were particularly cloudy in the nature run. He stressed that the distribution of clouds can have an effect on which DWL instrument design and sampling strategy is deemed to give the best weather prediction impact. Thus, an unrealistic representation of the cloud distribution could lead to faulty recommendations for design and use of the DWL. Sid pointed out the difficulty in using the nature run to give guidance in representing the subgrid scale wind variance in the DWL wind simulations. SWA had been using a variance of 20% of the mean wind speed. Results have shown that such specification may underestimate the jet level variance and overestimate the wind variance in the boundary layer.

Bob Atlas of NASA/GLA, an OSSE Team member, presented major conclusions from a climatological and synoptic evaluation of the nature run. Most of his evaluation of the nature run was conducted by comparing it with the real ECMWF global analysis for the same 30-day period. His overall conclusion is that the nature run is a generally realistic simulation of the large-scale and synoptic-scale systems and processes seen in the real atmosphere. His findings included both strengths and weaknesses in the realism of the nature run. Among the strengths, Bob showed evidence for the following conclusions: mean fields generally portray a reasonable climatology; patterns of cyclones and other synoptic-scale phenomena are realistic; distribution and evolution of fronts agree with synoptic frontal models; simulated cloud patterns show characteristic features consistent with those of real cyclones and fronts. Key deficiencies Bob saw in his evaluation of the nature run were: wind speeds in the cores of upper tropospheric jet streams were underestimated; number of cyclogenesis and cyclolysis cases and number of cyclones present at any one time was slightly fewer in the nature run than in the real atmosphere analysis; frontal boundaries, as indicated by pressure gradients and wind changes, are not as sharp in the nature run as they are in the analysis - this along with other similar evidence indicated that subsynoptic features were not resolved as well as should be expected by a T213 forecast model; cloud patterns associated with weather systems often appear too broad in the nature run. Bob also presented results of his evaluation of the simulated observational data that was included in the nature run data set. His key findings were: there were fewer simulated rawinsondes and surface buoys simulated than were available in reality; there were more aircraft, cloud track winds, and surface land and ship observations than were available in reality; cloud track wind simulations don't match up with the nature run cloud locations - they appear to be done independently of the nature run clouds; mean errors, both RMS and bias, used in the simulated rawinsonde observations are larger in magnitude than what has been typically used for simulating these observations. Bob stated that an important criteria for reliable OSSE results is that the differences between the nature run NWP model and the OSSE NWP model be sufficient to represent the departure of a typical NWP model's forecast from a real analysis. His evaluation showed that the NCEP model will indeed be a suitable "fraternal twin" to the ECMWF model, in that the NCEP model's forecasts depart from the nature run in a realistic way.

John Derber of NCEP/EMC gave a presentation of the recent and near-future upgrades to the NCEP global analysis system. This work is an ongoing project at NCEP/EMC that is independent of the NPOESS OSSE project, but the final OSSE data assimilation system will include many of the upgrades. NCEP/EMC has completely revamped their analysis system, and the new system is due to be implemented operationally on June 1, 1998. Currently, NCEP/EMC is using the revamped analysis system in testing a T140/L42 version of the NCEP global NWP model. The improvements to the analysis system that John mentioned were: a reformulated background error covariance matrix, that has resulted in improved forecasts of tropical winds; an improvement in the operator that transforms analysis quantities to the actually observed quantities; improvement in the interpolation to the time of observation; improved transformation operator for the satellite sounding data - the adaptation of the OPTRAN radiative transfer model; inclusion of new data sources, such as TOVS Level 1b radiances and GOES-8/9 radiances (completed), and GPS occultation observations and precipitation observations (in progress). Overall, many significant changes have been made to the NCEP global analysis system. The best result that has been observed to date is a lessening of the forecast model's tendency to reduce wind speeds with increasing forecast duration. Unfortunately, with so many changes being made to the model it is impossible to determine which change(s) might have contributed to this improvement in the forecasts. On the other hand, there is still a large amount of observational data available that is not being used in the global analysis system. With appropriately designed transformation operators, many of these observation types could be used in the model. An example of this is the line-of-sight winds that are available from active remote sensors. A preliminary transformation operator for these observations has been developed and is in the check-out phase now. John mentioned the uncertainties in specifying the observational error covariance matrix in the analysis system's cost function equation, stating that in OSSEs it may be necessary to account for the error in this specification. Though the details of the data assimilation system's formulation are extremely important, the system is not sufficiently documented to inform others of these details. Thus, though the emerging new NCEP global analysis system is state of the art, there are many improvements that could be included.

Steve Lord gave some very brief remarks on the simulated conventional data included with the nature run data set. He referred to the study of these data mentioned earlier in Bob Atlas' talk. The inconsistencies between the number of the various kinds of observations simulated and the number of real observations available during the 30-day period may require remedial action to be taken. It is important that in the pre-OSSE experiments conducted for calibration of the final OSSEs, the same types, amounts, and locations of observations are available to both the real data and simulated data experiments. The other issue is the magnitude of the mean errors used in simulating the observations. The identified discrepancies may require reconfiguration, or possibly even total re-simulation, of the simulated conventional observations.

Tom Kleespies of NESDIS gave a brief update of the progress on the simulation of the satellite sounding data. The existing simulated TOVS radiance soundings acquired with the nature run data set were compared with actual TOVS radiances from the same time period as the nature run. The statistics from the comparison showed that the simulated TOVS were in good agreement with the actual TOVS soundings. The actual TOVS radiances were used in the pre-OSSE real observation sensitivity experiments that will contribute to the calibration of the actual OSSE experiments conducted later. With respect to the simulation of advanced sounder radiances for the OSSEs, the OSSE team has proposed two initial candidate instrument types to the IPO for simulation: a Grating spectrometer and a Michelson Interferometer. These were submitted to the IPO in an attempt to narrow the possible instrument types to a manageable set when considering that the number of OSSEs that can be executed will be limited. They have asked the IPO for their reaction to these initial candidates, and are awaiting their reply. The work on developing the simulated observations for any advanced satellite sounder has been stopped due to fiscal year 1998 funding shortfalls. The NESDIS contractor working on this project has been reassigned to another project as a result. It is hoped that the work can resume when additional funding is provided by the IPO.

Dave Emmitt of Simpson Weather Associates made a presentation of issues involving the simulation of a Doppler Wind Lidar (DWL) instrument typical of that which may be a part of the NPOESS sensor suite. The first set of issues he discussed were those that pertain to the design of the instrument and sampling strategies. They were: baseline designs for coherent and direct detection DWLs; bracketing designs, which would set limits to narrow the possibilities to the best two or three designs for the OSSEs; dealing with the anticipated massive data volumes that would be obtained using currently envisioned DWL systems; shot combinations by averaging or accumulation; forward models for the assimilation of the DWL line-of-sight winds into the NCEP analysis system; coherent DWL signal processing for the simulated data. The second set of issues presented deal with the actual simulation of the DWL line-of-sight winds from the nature run data. They were: deriving the subgrid scale variance of the winds, which involves specification of the illumination of volume scale, the shot spacing scale, and the shot cluster scale; the cloud effects on the lidar returns, including the return signal strength - it was noted that while very accurate cloud top winds can be obtained from DWL, this is a secondary product of the instrument, and that the unique DWL contribution are the wind measurements from aerosol returns; the porosity of clouds with respect to the DWL - it was found from the nature run data the 50% of the time that you have a cloud return from the lidar signal, you also have a surface return; cloud distribution as depicted in the nature run, with lidar implications from both fractional cloud coverage and cirrus thickness; backscatter and attenuation of signal; and organization of lidar data voids - attenuation of the signal due to backscatter by coarser interceptors.

Steve Lord next briefly presented current status of and plans for the pre-OSSE calibration experiments. The pre-OSSE real observation sensitivity experiments have been completed for the entire nature run period. These involved the use of the NCEP analysis system assimilating all observations that were actually available during the 5 February - 7 March 1993 period, and a T62/L28 version of the NCEP global NWP model. Three real observation data assimilation experiments were conducted: control (all observations), NOTOVS (TOVS radiance soundings withheld), and NORAOBWINDS (rawinsonde winds withheld). The impact of withholding observations in the latter two experiments will be compared with the corresponding impacts in the analogous simulated observation sensitivity experiments to determine the degree of realism of the latter. The ratio of the impacts from the real and simulated pre-OSSE experiments is the calibration term that can be applied to the final OSSEs to correct the magnitude of the impacts obtained in those experiments. It will be necessary to make sure that the same observations available in the real pre-OSSE experiments will be available in simulated form in the simulated pre-OSSE experiments in order to insure that the calibration term reflects just the realism of the OSSE system and not the number of available observations. Steve showed anomaly correlation statistics from the three real observation data assimilation experiments. The statistics showed that withholding TOVS radiances had no noticeable effect on the anomaly correlations in the Northern Hemisphere, and moderate negative impact in the Southern Hemisphere. When rawinsonde winds were withheld, there was a large decrease in the anomaly correlation in the Northern Hemisphere, and a moderate decrease in the Southern Hemisphere.

Steve Mango of the IPO presented some perspectives of the IPO that are pertinent to each of the efforts being funded in the NPOESS program, including the OSSE project. Constant change is a reality in most aspects of the program. There have been two major satellite payload changes in the last 18 months. The governing requirements document is constantly changing. From the perspective of the IPO, their view of the importance of the OSSE project is how well it can help the future user to know how to use the future measurements, and how well it can determine the usefulness of the measurements. In order to have an effect on the initial payload design, the OSSEs will have to provide definitive information on sensor measurement impact on NWP by the end of FY2000. The primary hindrance to meeting this goal is funding. FY1998 saw a 60% actual allocation of funds compared to the NPOESS budget request, which will result in a six month delay in the program. Steve stressed, however, that the IPO cannot fund all of the work necessary to conduct the projects that are funded by NPOESS. Government labs must be able to leverage the basic and applied research that they would otherwise conduct to benefit their NPOESS-related efforts. Steve stated that he needs to have a schedule from the OSSE Team as to when they will have results and the type of results to expect to use as a justification for funding support from NPOESS.

Steve Lord briefly presented the OSSE Team near-term plans for the future. They are:

continue simulated observation studies, including pre-OSSE simulated observation sensitivity experiments (control, NOTOVS, NORAOBWINDS);

decide on advanced sounder and DWL specifications;

create simulated conventional observations data sets for the final OSSEs;

conduct preliminary evaluation of analysis requirements for line-of-sight winds;

determine experimental setup for "mini-OSSEs" (few days of data assimilation)

examine the effect of horizontal and vertical sampling

examine the effect of observational wind errors.

Steve stated that the OSSE Team hopes to have some one or more of the final OSSE experiments conducted by May 1999.

Issues and Recommendations

Issue: The single most important issue in the NPOESS OSSE project is the value of the effort to the NPOESS/IPO. The IPO must be convinced that the funding of this project will provide substantial guidance toward the selection, design, and deployment of the eventual sensor suite. In order to assure the responsiveness of the OSSE research team to what the IPO needs from them, the NPOESS OSSE Advisory Committee recommended in our previous report that the IPO state the specific questions that they need to have answered and objectives that need to be met through this project. The committee felt that defining the essential questions and objectives could serve to keep the OSSE research team focused on what the IPO sees as important and allow us as a committee to judge progress toward these objectives. To our knowledge, this information has not been provided formally to the OSSE Team. Steve Mango stated in the meeting two such objectives for the project in the meeting: to help the eventual user to know how to use the measurements and to know if the measurements will be useful. In order for the OSSE Team to provide information that will be of value to the IPO, the committee feels that the IPO needs to put these and any other questions and objectives in writing with any necessary elaboration. In response, the OSSE research team should state the format of how the questions will be answered and objectives met, with a proposed schedule for the answers and fulfilled objectives. The advisory committee would be willing to play a role in this exchange if the IPO and OSSE research team deem such participation to be helpful.

The OSSE Team has requested information from the IPO on instrument specifications for the advanced sounder and the Doppler Wind Lidar. In their most recent letter, they identified initial candidates for the advanced sounder specifications, and described a study to identify specifications for the DWL. The committee recommends that the IPO gives timely responses to these "in good faith" requests for information, especially considering the fact that the team did the ground work of identifying workable options. Steve Mango did verbally suggest an order for the advanced sounder instrument systems (first AIRS, then Michelson) but the committee recommends that the IPO guidance should be given in written form in sufficient detail. For example, the IPO needs to give guidance on the number of channels of information likely to be available from the sounders, so that they can be accounted for in the radiative transfer model. Such models require a significant amount of time to integrate into the NCEP analysis system (OPTRAN took two years) such the timely delivery of such guidance from the IPO to the OSSE Team is crucial.

The suitability of the ECMWF nature run data set for use in the proposed OSSEs is not yet a closed issue. Evaluation study results presented in the meeting raised concerns particularly in the realism of the clouds produced by the prognostic cloud scheme used in the nature run. Of significant concern is the apparent deficiency of low clouds and surplus of high cloudiness. All agree that this problem poses significant implications for the realism of the satellite sensor sampling distribution. The committee recommends that further study be conducted to confirm the presented cloud distribution results. This may require contacting the ECMWF to see if their analyses of the nature run clouds yielded the same results. If they are confirmed, the committee suggests that serious consideration be given to reconstructing the clouds from the other nature run data using a diagnostic approach, insuring consistency between the cloudiness and meteorological conditions from which they were derived.

The synoptic study of the nature run reported at the meeting identified some deficiencies in the realism in the nature run as well. Particularly noteworthy are the understated jet core wind speeds, the magnitude of the errors assigned to and used in the simulation of the conventional observations, the number of observations simulated for each conventional observation type, and the apparent lack of subsynoptic scale resolution in the nature run fields. The committee recommends that the understatement of the winds in the jet core be kept in mind in both the pre-OSSE calibration results, and of course in the interpretation of the final OSSE impacts. There may be a tendency to obtain results in the data assimilation that are too optimistic when both the forecast model (background field) and the truth (nature run) have understated jet core winds. The committee recommends that the magnitude of the conventional observations be studied further, along with the discrepancy in the number of simulated conventional observations compared with actual observations available. If these issues cannot be suitably resolved using the provided conventional observations, it may be necessary (although costly in time and manpower) to re-simulate the conventional observations. This course of action should only be taken if it is judged that the quality of the OSSEs will be significantly compromised by using the provided simulated observations. Finally, the committee recommends that at spectral analysis of the nature run fields be conducted to insure that all spectral wave numbers are being represented adequately in the fields. The OSSE Team should check to see if unrealistically low spectral energy is present in the highest wave numbers.

Representation of possible lidar technologies to represent in the lidar wind simulations should be balanced and even-handed. Concern was raised at the meeting that the coherent lidar technology was being over emphasized at the expense of the other technologies, such as direct detection. The focus for the DWL simulations should not be on what instrument takes the data, but rather on how to sample the data, represent the variance, and represent the cloud effects in such a way that would be applicable to a generic lidar design. This approach is preferred by the committee over an attempt to explicitly represent all forms of lidar technologies in separate simulation of the DWL winds.

Finally, the seemingly ever-present issue of funding shortfalls was raised at the meeting. Discussion revolved around a way to make the funding decision makers aware of the importance of the OSSE project to the overall NPOESS program. It was suggested that the combination of a letter of support from the Advisory Committee combined with a presentation by Steve Lord, both directed to the Satellite User's Advisory Group (SUAG), be used as appropriate and effective advocacy for the continued funding of the OSSE project. The committee recommends that the OSSE team give further consideration and study to the most effective avenues of advocacy. The committee agrees to play any role in this process that might be considered helpful, but would need further guidance on what forms of advocacy might be most effective.

Next Meeting

After the previous meeting, the committee felt that, rather than arbitrarily planning to meet at regular time intervals, we would meet at the suggestion and request of the OSSE Team. This approach is in keeping with the primary role envisioned for the committee, as stated in the introduction of this report. As no date was specified for the next meeting at the end of the current meeting, the date for the next meeting remains open for the suggestion and request of the OSSE Team.

T