1. INTRODUCTION

Global atmospheric observing systems provide the basic data for Numerical Weather Prediction (NWP) forecasts and the means to monitor and assess climate. Any contribution of a particular instrument to this observing system must be taken in the context of other instruments that provide similar or complementary information. The forecast impact of current instruments can be assessed by Observing System Experiments, in which already existing observations are denied or added to observations from a standard data base, but future instruments must be assessed with experiments using simulated observations. These experiments, known as Observing System Simulation Experiments (OSSEs), have been performed in the past (e.g., Atlas et al. 1985a, b) for space-based sensors. At the present time, a new series of OSSEs are needed to take account of the major recent advances in data assimilation methodology. These OSSEs will aid a variety of future decisions, such as the instrument configuration for the National Polar-Orbiting Operational Environmental Satellite System (NPOESS). This paper describes, in general terms, issues surrounding the value and credibility of OSSEs, and a proposed configuration and methodology that addresses weaknesses in past OSSEs.

2. THE VALUE AND CREDIBILITY OF THE OSSE APPROACH

A considerable number of OSSE's have been conducted to evaluate the potential for future observing systems to improve NWP and to plan for the Global Weather Experiment and more recently for NASA's Earth Observing System (Atlas et al., 1985b; Arnold and Dey, 1986; Hoffman et al., 1990). In addition, OSSE's have been run to evaluate trade-offs in the design of observing systems and observing networks (Atlas and Emmitt,1991; Rohaly and Krishnamurti,1993), and to test new methodology for data assimilation (Atlas and Bloom, 1989).

These experiments have contributed to the design of observing systems, and to the use of new data types in the analysis and forecasting of the atmosphere. Nevertheless, there has also been criticism of OSSEs, particularly from the instrument community. The most extreme form of this criticism asserts that modeling, data assimilation, and OSSEs have no place in the design and selection of instruments. This argument can be easily countered as follows: (1) The data assimilation and modelling community is one of the more important customers for instrument observations. This community uses OSSEs as one of its main tools in assessing the utility of proposed instruments. Instrument selection and design without the input of a major customer is not a very rational process and has often led to major under-utilization of expensive instruments in the past. (2) If the data assimilation community is involved in instrument design and selection from the outset, it is more likely to be able to extract the maximum amount of information from new sensors soon after launch, rather than taking over 15 years as in the case of current polar-orbiting temperature sounders. (3) If models and data assimilation systems are really so bad that they shouldn't be used for assessing new instruments, then they are unlikely to be able to make use of the new observations upon launch.

While this extreme form of OSSE criticism is easily discredited, there remain milder, more rational concerns about the OSSE process that have considerable validity. These concerns may be divided into three classes - intrinsic, ethical and methodological.

(1) Intrinsic Concerns

OSSEs are usually carried out with respect to sensors or systems that may not be in place for another 5 or 10 years. The models and assimilation systems of 10 years hence will be different (and hopefully much better) than the systems in use at the time of the OSSE. Therefore, the impact of a particular sensor may be somewhat different at the time of launch than at the time of the OSSE. While there is obviously no ideal solution to this problem, use of the most advanced data assimilation systems currently available, with the proviso that these systems be affordable and well-tested, is a requirement for OSSEs. Experiments should be re-run as time proceeds and more advanced systems become available.

(2) Ethical Concerns

In the past, a number of OSSEs have originated from a very close relationship between certain instrument engineers, retrieval experts and OSSE experimenters. While these OSSEs were performed in an objective manner, it may have been perceived by proponents of rival prospective instruments that the purpose of such experiments was primarily to demonstrate the (positive) impact of a particular sensor. OSSE experimenters should remain completely neutral with respect to different instruments. OSSEs should be concerned with evaluation, not advocacy, so that decisions based (in part) on the results of OSSEs can be as objective as possible.

(3) Methodological Concerns

An OSSE experiment requires a nature run from a numerical model to provide the "truth" against which the assimilated analyses can be compared. Instrument forward models (including an assessment of the relevant error statistics) are required in order to generate simulated observations that would have the same characteristics as real observations. These instrument models (and statistics) are also used in the assimilation procedure in the same way as they would be if real observations were being assimilated. The potential difficulties in this procedure are as follows:

(a) The nature run is meant to be a proxy for the real atmosphere. It is produced by a numerical model that can be sampled in any way desired. The question is "How close a proxy is the nature run to the real atmosphere"? Numerical models are improving in realism and provide much better proxies than the models of 10 years ago. There do, however, remain serious errors in all numerical models, and some errors are highly inter-model correlated, e.g. climate drift. Thus, it may well be that there are some similarities between the biases of the assimilating model and the model that produced the nature run even though these models might have come from different institutions. Clearly, such biases can contaminate the results of OSSE experiments and must be handled carefully.

Benchmark experiments can be done with the system using real observations to assess the actual impact of current operational systems by adding or removing them from the real observational data stream (Fig.1). The OSSE system can then be run under the same experimental conditions to see if the OSSE impacts agree with those obtained from the real data experiments. If the OSSEs and the real data experiments produce the same conclusions, then the OSSE system can be used with more confidence to assess the impact of proposed sensors. The innovation (difference between model guess and observations) and other forecast error statistics (e.g., anomaly correlations) can be compiled both for the OSSEs and for real data experiments. These statistics can then be compared for the real and simulated data and also with the error statistics that are specified in the data assimilation algorithms. If these sets of statistics are substantially in agreement, then more confidence can be placed in the OSSE conclusions. This procedure has been employed by GSFC in their OSSEs since 1985.

(b) Early OSSEs did not use instrument forward models and, therefore, the assumed observational error characteristics were crude (e.g. white-noise). This led, inevitably, to overly favorable OSSE impact assessments. More recent OSSEs have used appropriate instrument models (including error characteristics) both in creating the simulated observations and in the assimilation process itself. Knowledge of the instrument characteristics is increasingly important in data assimilation, and it can be expected that a considerable amount of effort will be expended in developing and applying such instrument models. This must be done by working closely with all candidate instrument teams.

If the instrument characteristics are properly specified and utilized in the assimilation process, then all instruments should have either a distinguishable positive impact or an impact that is so close to zero as to be below the noise level. If any instrument has a distinguishable negative impact, this indicates a problem in the OSSE system. The data assimilation system must be improved and the OSSE experiment rerun.

3. A PROPOSED FRAMEWORK FOR AN OSSE SYSTEM

Our proposed OSSE system (Fig.1) will include:

a) A "nature" run, which is a long numerical integration of a high resolution, state-of-the-science atmospheric model. The nature run is assumed to be a true atmospheric evolution from which simulated observations can be extracted.

b) Forward models for each instrument to generate all the simulated observations that will be assimilated from the nature run. These simulated observations will have realistic observational error properties that will then be specified in the assimilation algorithms. The simulated observations will range from conventional surface and radiosonde observations to multi-channel radiances that would be measured by satellite-borne radiometers.

c) A state-of-the-science data assimilation system, which includes a forecast model and a data analysis module, to ingest the simulated data, make the forecasts and provide representative measures of forecast performance. The forecast performance measures provide guidance on questions related to instrument configurations and their potential impacts on numerical weather forecasts and climate analyses.

d) A series of validation procedures to ensure that the results of the simulations are realistic (Fig.1). For example, a critical test is to compare the impact of simulated data created from algorithms that emulate already existing instrument packages, such as TOVS, with the impact obtained using real TOVS observations. It is also critical to compare the statistics of simulated versus observed innovations.

Each of these components needs to be state-of-the-science, since the impact obtained with real or simulated data will depend on the system used. An inferior forecast model and data assimilation system can result in positive impacts that are not realized with an advanced system, because the accuracy requirements in the better system are more stringent. For example, satellite temperature retrievals will produce large positive impacts in the Northern Hemisphere (NH) if one uses a system with persistence or climatology as a first guess, but not in systems with 4-Dimensional Data Assimilation (4-DDA). On the other hand, the very small impact of satellite retrievals obtained previously in the NH (Mo et al, 1995), has been transformed into a large positive impact (Derber and Wu, 1996) through improved use of the data in a 3-D VAR scheme.

4. CONCLUDING REMARKS

The advanced OSSE system proposed here can provide the quantitative basis for a rational design of observing systems that will be used primarily for numerical weather prediction and climate monitoring. As such, it has the potential for significant programmatic cost savings. In particular, for the NPOESS Program, OSSEs will provide answers to fundamental questions such as: What is the relative impact on forecast accuracy of the different proposed advanced atmospheric sounders? What is the relative importance of lidar wind soundings compared to atmospheric temperature and humidity soundings? What is the impact of coverage from multiple platforms rather than a single one? Since the answers will depend on the assimilation system to some extent, using multiple, state-of-the-science systems within the same OSSE framework will add credibility to those results that corroborate each other and will help to pinpoint weaknesses in those areas where differences occur. In addition, our proposed OSSE system should contribute to accelerated operational use of the real data, once the tested systems come on line, since much of the needed data assimilation development has taken place beforehand.

5. ACKNOWLEDGMENTS

This work has been supported by the National Polar-Orbiting Operational Environmental Satellite System Integrated Program Office.

6. REFERENCES

Arnold,C. P.,Jr., and C. H. Dey, 1986: Observing-system simulation experiments: Past, present, and future. Bull. Amer. Meteor. Soc., 67, 687-695.

Atlas, R., E. Kalnay and M. Halem, 1985a: The impact of satellite temperature sounding and wind data on numerical weather prediction. Optical Engineering., 24, 341-346.

Atlas, R., E. Kalnay, J. Susskind, W. E. Baker and M. Halem, 1985b: Simulation studies of the impact of future observing systems on weather prediction. Proc. Seventh Conf. On NWP. 145-151.

Atlas, R. and G. D. Emmitt, 1991: Implications of several orbit inclinations for the impact of LAWS on global climate studies. Second Symposium on Global Change Studies., New Orleans, 28-32.

Atlas, R. And S. C. Bloom, 1989: Global surface wind vectors resulting from the assimilation of satellite wind speed data in atmospheric general circulation models. Oceans 89 Proceedings, IEEE Publication 89CH2780-5, 260-265.

Derber, J. C. and W. S. Wu, 1996: The use of cloud-cleared radiances in the NCEP's SSI analysis system. 11th Conference on Numerical Weather Prediction, 19-23 August, 1996, Norfolk, VA.

Hoffman, R. N., C. Grassotti, R. Isaacs, J. Louis, T. Nehrkorn, and D. Norquist, 1990: Assessment of the impact of simulated satellite lidar wind and retrieved 183 GHz water vapor observations on a global data assimilation system. Mon. Wea. Rev., 118, 2513-2542.

Mo, K. C., X. L. Wang, R. Kistler, M. Kanamitsu, and E. Kalnay, 1995: Impact of satellite data on the CDAS-reanalysis system. Mon. Wea. Rev., 123, 124-139.

Rohaly, G.D. and T. N. Krishnamurti, 1993: An observing system simulation experiment for the Laser Atmospheric Wind Sounder (LAWS). J. Applied Meteor., 32, 1453-1471.