NPOESS Advisory Committee for Observing Systems Simulation Experiments
Report No. 2
May 1998
Introduction
The NPOESS Observing System Simulation Experiments (OSSE) Advisory Committee
provides technical oversight and scientific guidance to the investigators involved in the
NPOESS OSSE project. These investigators have requested input from a committee
representing the potential users of the polar-orbiting satellite data forthcoming from the
future NPOESS satellites' sensor suites. The Advisory Committee has been asked to
convene as necessary to review progress on development and implementation of the
OSSE system, and report on the progress to the NPOESS Integrated Program Office.
This offering constitutes the second in a series of such reports.
Progress Summary
The second committee meeting was held at the National Centers for Environmental
Prediction, Environmental Modeling Center's office in the World Weather Building,
Camp Springs, MD on May 19, 1998. In attendance at that meeting were the
following committee members: Akira Kasahara, T. N. Krishnamurti, Greg Mandt, Jan
Paegle, Paul Hays, and chairperson Donald Norquist. Prior to the meeting, the
committee received a copy of the fiscal year 1997 Final Report from the NPOESS
OSSE project investigators (hereafter referred to as the OSSE Team). At the meeting,
the committee heard presentations from several of the OSSE Team members, and from
a representative of the NPOESS Integrated Program Office (IPO). The following
discussion summarizes the collective information provided in the report and at the
meeting.
Steve Lord of NCEP/EMC, the OSSE Team leader, reviewed the agenda and gave an
introduction. He stated that the issue of cost savings was a major factor for the
motivation of the OSSEs in support of the NPOESS program. Spending a modest
amount of money now in carrying out studies evaluating the impact of atmospheric
measurements from various satellite sensor designs could provide valuable information
in determining the allocation of large amounts of money later for sensor fabrication.
He cited a program being conducted by NOAA called the North American Atmospheric
Observing System (NAOS), which involves a series of observing system experiments,
as being a potential source of useful information for the current NPOESS OSSE
project. Steve then reviewed the basic OSSE design, and listed a number of issues that
a well-designed OSSE study must address to have validity. Among these concerns
were: present numerical weather prediction (NWP) models and assimilation systems
are different from what they will be when the proposed sensor data is available;
importance of fairness and objectivity in proposed instrument specification, to avoid a
vested interest;
methodological concerns, such as how close a proxy is the "nature run" (the global
NWP model simulation used in the OSSE to represent a simulation of the real
atmosphere) to the real atmosphere; the realism of simulated observations; calibration
of the OSSEs against real data observing system experiments. Steve then briefly
summarized the current status of the present OSSE project:
ECMWF nature run data was received, distributed to investigators, and
evaluated;
NCEP pre-OSSE forecast system (T62/L28) was completed;
real observation sensitivity experiments (control, NOTOVS, NORAOBWINDS)
were completed for February 1993 (same period as nature run);
TOVS soundings were simulated from the nature run and evaluated against
observed TOVS;
full data assimilation and archiving system for quality control, analysis, and
forecast model has been constructed.
Steve stated that though two letters have been sent to the IPO requesting guidance on
recommended sensor specifications to be used in the OSSE study, no such information
has been received. Finally, he said that NCEP computer resources could pose a
limitation to the extent of the OSSE experiments that can be conducted.
Michiko Masutani of NCEP/EMC next presented an introduction to the evaluation of
the nature run. The nature run is a 30-day forecast from the T213/L31 version of the
ECMWF global NWP model covering the period 5 February - 7 March 1993. Besides
the forecast spectral coefficients, simulated observations from most conventional
observing systems were provided in the nature run data set acquired from ECMWF.
Michiko showed that no El Nino was occurring during this time, so that the month is
representative a typical Northern Hemisphere winter/Southern Hemisphere summer
month. She showed evidence that suggests that the major features of outgoing
longwave radiation pattern are captured by the nature run. The nature run cloud cover,
as generated by the prognostic cloud scheme in the ECMWF model, revealed some
possible problems in its realism. First, the zonal mean high cloud cover decreases
about 15 % during the first several days of the nature run, then levels out for the
duration. Second, in comparison with RTNEPH clouds for the same period, the nature
run has a greater total cloud cover worldwide. The largest differences are in polar
regions where uncertainties are greatest. Perhaps the most disturbing finding is that the
nature run has greater high cloud cover than low cloud cover, while RTNEPH has
greater low cloud cover than high cloud cover. This discrepancy is most pronounced in
the tropics. This finding, if confirmed by subsequent study, could have profound
implications for satellite sensor sampling of the nature run.
Sid Wood of Simpson Weather Associates (SWA), an OSSE Team member, gave a
short presentation of some statistical results of their evaluation of the nature run. Their
study focused on an examination of those aspects of the nature run atmosphere that
would affect sampling distribution and simulation design for the Doppler Wind Lidar
(DWL). For example, SWA determined the spatial and temporal extent of the nature
run atmosphere that can be sampled given cloud attenuation affects. Sid showed the
percentage of grid points that were cloud free as a function of latitude, which showed
that the tropics were particularly cloudy in the nature run. He stressed that the
distribution of clouds can have an effect on which DWL instrument design and
sampling strategy is deemed to give the best weather prediction impact. Thus, an
unrealistic representation of the cloud distribution could lead to faulty recommendations
for design and use of the DWL. Sid pointed out the difficulty in using the nature run
to give guidance in representing the subgrid scale wind variance in the DWL wind
simulations. SWA had been using a variance of 20% of the mean wind speed. Results
have shown that such specification may underestimate the jet level variance and
overestimate the wind variance in the boundary layer.
Bob Atlas of NASA/GLA, an OSSE Team member, presented major conclusions from
a climatological and synoptic evaluation of the nature run. Most of his evaluation of
the nature run was conducted by comparing it with the real ECMWF global analysis for
the same 30-day period. His overall conclusion is that the nature run is a generally
realistic simulation of the large-scale and synoptic-scale systems and processes seen in
the real atmosphere. His findings included both strengths and weaknesses in the
realism of the nature run. Among the strengths, Bob showed evidence for the
following conclusions: mean fields generally portray a reasonable climatology;
patterns of cyclones and other synoptic-scale phenomena are realistic; distribution and
evolution of fronts agree with synoptic frontal models; simulated cloud patterns show
characteristic features consistent with those of real cyclones and fronts. Key
deficiencies Bob saw in his evaluation of the nature run were: wind speeds in the cores
of upper tropospheric jet streams were underestimated; number of cyclogenesis and
cyclolysis cases and number of cyclones present at any one time was slightly fewer in
the nature run than in the real atmosphere analysis; frontal boundaries, as indicated by
pressure gradients and wind changes, are not as sharp in the nature run as they are in
the analysis - this along with other similar evidence indicated that subsynoptic features
were not resolved as well as should be expected by a T213 forecast model; cloud
patterns associated with weather systems often appear too broad in the nature run. Bob
also presented results of his evaluation of the simulated observational data that was
included in the nature run data set. His key findings were: there were fewer simulated
rawinsondes and surface buoys simulated than were available in reality; there were
more aircraft, cloud track winds, and surface land and ship observations than were
available in reality; cloud track wind simulations don't match up with the nature run
cloud locations - they appear to be done independently of the nature run clouds; mean
errors, both RMS and bias, used in the simulated rawinsonde observations are larger in
magnitude than what has been typically used for simulating these observations. Bob
stated that an important criteria for reliable OSSE results is that the differences between
the nature run NWP model and the OSSE NWP model be sufficient to represent the
departure of a typical NWP model's forecast from a real analysis. His evaluation
showed that the NCEP model will indeed be a suitable "fraternal twin" to the ECMWF
model, in that the NCEP model's forecasts depart from the nature run in a realistic
way.
John Derber of NCEP/EMC gave a presentation of the recent and near-future upgrades
to the NCEP global analysis system. This work is an ongoing project at NCEP/EMC
that is independent of the NPOESS OSSE project, but the final OSSE data assimilation
system will include many of the upgrades. NCEP/EMC has completely revamped their
analysis system, and the new system is due to be implemented operationally on June 1,
1998. Currently, NCEP/EMC is using the revamped analysis system in testing a
T140/L42 version of the NCEP global NWP model. The improvements to the analysis
system that John mentioned were: a reformulated background error covariance matrix,
that has resulted in improved forecasts of tropical winds; an improvement in the
operator that transforms analysis quantities to the actually observed quantities;
improvement in the interpolation to the time of observation; improved transformation
operator for the satellite sounding data - the adaptation of the OPTRAN radiative
transfer model; inclusion of new data sources, such as TOVS Level 1b radiances and
GOES-8/9 radiances (completed), and GPS occultation observations and precipitation
observations (in progress). Overall, many significant changes have been made to the
NCEP global analysis system. The best result that has been observed to date is a
lessening of the forecast model's tendency to reduce wind speeds with increasing
forecast duration. Unfortunately, with so many changes being made to the model it is
impossible to determine which change(s) might have contributed to this improvement in
the forecasts. On the other hand, there is still a large amount of observational data
available that is not being used in the global analysis system. With appropriately
designed transformation operators, many of these observation types could be used in
the model. An example of this is the line-of-sight winds that are available from active
remote sensors. A preliminary transformation operator for these observations has been
developed and is in the check-out phase now. John mentioned the uncertainties in
specifying the observational error covariance matrix in the analysis system's cost
function equation, stating that in OSSEs it may be necessary to account for the error in
this specification. Though the details of the data assimilation system's formulation are
extremely important, the system is not sufficiently documented to inform others of
these details. Thus, though the emerging new NCEP global analysis system is state of
the art, there are many improvements that could be included.
Steve Lord gave some very brief remarks on the simulated conventional data included
with the nature run data set. He referred to the study of these data mentioned earlier in
Bob Atlas' talk. The inconsistencies between the number of the various kinds of
observations simulated and the number of real observations available during the 30-day
period may require remedial action to be taken. It is important that in the pre-OSSE
experiments conducted for calibration of the final OSSEs, the same types, amounts, and
locations of observations are available to both the real data and simulated data
experiments. The other issue is the magnitude of the mean errors used in simulating
the observations. The identified discrepancies may require reconfiguration, or possibly
even total re-simulation, of the simulated conventional observations.
Tom Kleespies of NESDIS gave a brief update of the progress on the simulation of the
satellite sounding data. The existing simulated TOVS radiance soundings acquired with
the nature run data set were compared with actual TOVS radiances from the same time
period as the nature run. The statistics from the comparison showed that the simulated
TOVS were in good agreement with the actual TOVS soundings. The actual TOVS
radiances were used in the pre-OSSE real observation sensitivity experiments that will
contribute to the calibration of the actual OSSE experiments conducted later. With
respect to the simulation of advanced sounder radiances for the OSSEs, the OSSE team
has proposed two initial candidate instrument types to the IPO for simulation: a
Grating spectrometer and a Michelson Interferometer. These were submitted to the
IPO in an attempt to narrow the possible instrument types to a manageable set when
considering that the number of OSSEs that can be executed will be limited. They have
asked the IPO for their reaction to these initial candidates, and are awaiting their reply.
The work on developing the simulated observations for any advanced satellite sounder
has been stopped due to fiscal year 1998 funding shortfalls. The NESDIS contractor
working on this project has been reassigned to another project as a result. It is hoped
that the work can resume when additional funding is provided by the IPO.
Dave Emmitt of Simpson Weather Associates made a presentation of issues involving
the simulation of a Doppler Wind Lidar (DWL) instrument typical of that which may
be a part of the NPOESS sensor suite. The first set of issues he discussed were those
that pertain to the design of the instrument and sampling strategies. They were:
baseline designs for coherent and direct detection DWLs; bracketing designs, which
would set limits to narrow the possibilities to the best two or three designs for the
OSSEs; dealing with the anticipated massive data volumes that would be obtained
using currently envisioned DWL systems; shot combinations by averaging or
accumulation; forward models for the assimilation of the DWL line-of-sight winds into
the NCEP analysis system; coherent DWL signal processing for the simulated data.
The second set of issues presented deal with the actual simulation of the DWL
line-of-sight winds from the nature run data. They were: deriving the subgrid scale
variance of the winds, which involves specification of the illumination of volume scale,
the shot spacing scale, and the shot cluster scale; the cloud effects on the lidar returns,
including the return signal strength - it was noted that while very accurate cloud top
winds can be obtained from DWL, this is a secondary product of the instrument, and
that the unique DWL contribution are the wind measurements from aerosol returns; the
porosity of clouds with respect to the DWL - it was found from the nature run data the
50% of the time that you have a cloud return from the lidar signal, you also have a
surface return; cloud distribution as depicted in the nature run, with lidar implications
from both fractional cloud coverage and cirrus thickness; backscatter and attenuation
of signal; and organization of lidar data voids - attenuation of the signal due to
backscatter by coarser interceptors.
Steve Lord next briefly presented current status of and plans for the pre-OSSE
calibration experiments. The pre-OSSE real observation sensitivity experiments have
been completed for the entire nature run period. These involved the use of the NCEP
analysis system assimilating all observations that were actually available during the 5
February - 7 March 1993 period, and a T62/L28 version of the NCEP global NWP
model. Three real observation data assimilation experiments were conducted: control
(all observations), NOTOVS (TOVS radiance soundings withheld), and
NORAOBWINDS (rawinsonde winds withheld). The impact of withholding
observations in the latter two experiments will be compared with the corresponding
impacts in the analogous simulated observation sensitivity experiments to determine the
degree of realism of the latter. The ratio of the impacts from the real and simulated
pre-OSSE experiments is the calibration term that can be applied to the final OSSEs to
correct the magnitude of the impacts obtained in those experiments. It will be
necessary to make sure that the same observations available in the real pre-OSSE
experiments will be available in simulated form in the simulated pre-OSSE experiments
in order to insure that the calibration term reflects just the realism of the OSSE system
and not the number of available observations. Steve showed anomaly correlation
statistics from the three real observation data assimilation experiments. The statistics
showed that withholding TOVS radiances had no noticeable effect on the anomaly
correlations in the Northern Hemisphere, and moderate negative impact in the Southern
Hemisphere. When rawinsonde winds were withheld, there was a large decrease in the
anomaly correlation in the Northern Hemisphere, and a moderate decrease in the
Southern Hemisphere.
Steve Mango of the IPO presented some perspectives of the IPO that are pertinent to
each of the efforts being funded in the NPOESS program, including the OSSE project.
Constant change is a reality in most aspects of the program. There have been two
major satellite payload changes in the last 18 months. The governing requirements
document is constantly changing. From the perspective of the IPO, their view of the
importance of the OSSE project is how well it can help the future user to know how to
use the future measurements, and how well it can determine the usefulness of the
measurements. In order to have an effect on the initial payload design, the OSSEs will
have to provide definitive information on sensor measurement impact on NWP by the
end of FY2000. The primary hindrance to meeting this goal is funding. FY1998 saw
a 60% actual allocation of funds compared to the NPOESS budget request, which will
result in a six month delay in the program. Steve stressed, however, that the IPO
cannot fund all of the work necessary to conduct the projects that are funded by
NPOESS. Government labs must be able to leverage the basic and applied research
that they would otherwise conduct to benefit their NPOESS-related efforts. Steve
stated that he needs to have a schedule from the OSSE Team as to when they will have
results and the type of results to expect to use as a justification for funding support
from NPOESS.
Steve Lord briefly presented the OSSE Team near-term plans for the future. They are:
continue simulated observation studies, including pre-OSSE simulated
observation sensitivity experiments (control, NOTOVS, NORAOBWINDS);
decide on advanced sounder and DWL specifications;
create simulated conventional observations data sets for the final OSSEs;
conduct preliminary evaluation of analysis requirements for line-of-sight winds;
determine experimental setup for "mini-OSSEs" (few days of data
assimilation)
examine the effect of horizontal and vertical sampling
examine the effect of observational wind errors.
Steve stated that the OSSE Team hopes to have some one or more of the final OSSE
experiments conducted by May 1999.
Issues and Recommendations
Issue: The single most important issue in the NPOESS OSSE project is the value of
the effort to the NPOESS/IPO. The IPO must be convinced that the funding of this
project will provide substantial guidance toward the selection, design, and deployment
of the eventual sensor suite. In order to assure the responsiveness of the OSSE
research team to what the IPO needs from them, the NPOESS OSSE Advisory
Committee recommended in our previous report that the IPO state the specific
questions that they need to have answered and objectives that need to be met through
this project. The committee felt that defining the essential questions and objectives
could serve to keep the OSSE research team focused on what the IPO sees as important
and allow us as a committee to judge progress toward these objectives. To our
knowledge, this information has not been provided formally to the OSSE Team. Steve
Mango stated in the meeting two such objectives for the project in the meeting: to help
the eventual user to know how to use the measurements and to know if the
measurements will be useful. In order for the OSSE Team to provide information that
will be of value to the IPO, the committee feels that the IPO needs to put these and any
other questions and objectives in writing with any necessary elaboration. In response,
the OSSE research team should state the format of how the questions will be answered
and objectives met, with a proposed schedule for the answers and fulfilled objectives.
The advisory committee would be willing to play a role in this exchange if the IPO and
OSSE research team deem such participation to be helpful.
The OSSE Team has requested information from the IPO on instrument specifications
for the advanced sounder and the Doppler Wind Lidar. In their most recent letter, they
identified initial candidates for the advanced sounder specifications, and described a
study to identify specifications for the DWL. The committee recommends that the IPO
gives timely responses to these "in good faith" requests for information, especially
considering the fact that the team did the ground work of identifying workable options.
Steve Mango did verbally suggest an order for the advanced sounder instrument
systems (first AIRS, then Michelson) but the committee recommends that the IPO
guidance should be given in written form in sufficient detail. For example, the IPO
needs to give guidance on the number of channels of information likely to be available
from the sounders, so that they can be accounted for in the radiative transfer model.
Such models require a significant amount of time to integrate into the NCEP analysis
system (OPTRAN took two years) such the timely delivery of such guidance from the
IPO to the OSSE Team is crucial.
The suitability of the ECMWF nature run data set for use in the proposed OSSEs is not
yet a closed issue. Evaluation study results presented in the meeting raised concerns
particularly in the realism of the clouds produced by the prognostic cloud scheme used
in the nature run. Of significant concern is the apparent deficiency of low clouds and
surplus of high cloudiness. All agree that this problem poses significant implications
for the realism of the satellite sensor sampling distribution. The committee
recommends that further study be conducted to confirm the presented cloud distribution
results. This may require contacting the ECMWF to see if their analyses of the nature
run clouds yielded the same results. If they are confirmed, the committee suggests that
serious consideration be given to reconstructing the clouds from the other nature run
data using a diagnostic approach, insuring consistency between the cloudiness and
meteorological conditions from which they were derived.
The synoptic study of the nature run reported at the meeting identified some
deficiencies in the realism in the nature run as well. Particularly noteworthy are the
understated jet core wind speeds, the magnitude of the errors assigned to and used in
the simulation of the conventional observations, the number of observations simulated
for each conventional observation type, and the apparent lack of subsynoptic scale
resolution in the nature run fields. The committee recommends that the understatement
of the winds in the jet core be kept in mind in both the pre-OSSE calibration results,
and of course in the interpretation of the final OSSE impacts. There may be a tendency
to obtain results in the data assimilation that are too optimistic when both the forecast
model (background field) and the truth (nature run) have understated jet core winds.
The committee recommends that the magnitude of the conventional observations be
studied further, along with the discrepancy in the number of simulated conventional
observations compared with actual observations available. If these issues cannot be
suitably resolved using the provided conventional observations, it may be necessary
(although costly in time and manpower) to re-simulate the conventional observations.
This course of action should only be taken if it is judged that the quality of the OSSEs
will be significantly compromised by using the provided simulated observations.
Finally, the committee recommends that at spectral analysis of the nature run fields be
conducted to insure that all spectral wave numbers are being represented adequately in
the fields. The OSSE Team should check to see if unrealistically low spectral energy is
present in the highest wave numbers.
Representation of possible lidar technologies to represent in the lidar wind simulations
should be balanced and even-handed. Concern was raised at the meeting that the
coherent lidar technology was being over emphasized at the expense of the other
technologies, such as direct detection. The focus for the DWL simulations should not
be on what instrument takes the data, but rather on how to sample the data, represent
the variance, and represent the cloud effects in such a way that would be applicable to a
generic lidar design. This approach is preferred by the committee over an attempt to
explicitly represent all forms of lidar technologies in separate simulation of the DWL
winds.
Finally, the seemingly ever-present issue of funding shortfalls was raised at the
meeting. Discussion revolved around a way to make the funding decision makers
aware of the importance of the OSSE project to the overall NPOESS program. It was
suggested that the combination of a letter of support from the Advisory Committee
combined with a presentation by Steve Lord, both directed to the Satellite User's
Advisory Group (SUAG), be used as appropriate and effective advocacy for the
continued funding of the OSSE project. The committee recommends that the OSSE
team give further consideration and study to the most effective avenues of advocacy.
The committee agrees to play any role in this process that might be considered helpful,
but would need further guidance on what forms of advocacy might be most effective.
Next Meeting
After the previous meeting, the committee felt that, rather than arbitrarily planning to
meet at regular time intervals, we would meet at the suggestion and request of the
OSSE Team. This approach is in keeping with the primary role envisioned for the
committee, as stated in the introduction of this report. As no date was specified for the
next meeting at the end of the current meeting, the date for the next meeting remains
open for the suggestion and request of the OSSE Team.
T