NPOESS Advisory Committee for Observing Systems Simulation Experiments
Report No. 4
May 2001
Introduction
The NPOESS Observing System Simulation Experiments (OSSE) Advisory Committee provides technical oversight and scientific guidance to the investigators involved in the NPOESS OSSE project. These investigators have requested input from a committee representing the potential users of the polar-orbiting satellite data forthcoming from the future NPOESS satellites’ sensor suites. The Advisory Committee has been asked to convene as necessary to review progress on development and implementation of the OSSE system, and report on the progress to the NPOESS Integrated Program Office. This offering constitutes the fourth in a series of such reports.
The fourth committee meeting was held at the National Centers for Environmental Prediction, Environmental Modeling Center’s office in the World Weather Building, Camp Springs, MD on May 22-23, 2001. In attendance at that meeting were the following committee members: Akira Kasahara, T. N. Krishnamurti, Jeff McQueen, Jan Paegle, Edwin Eloranta, and chairperson Donald Norquist. At the meeting, the committee heard presentations from the NPOESS OSSE project investigators (hereafter referred to as the OSSE Team). This meeting added an additional half-day to allow the committee to discuss the information presented and prepare a feedback briefing. The briefing was presented to the OSSE Team at the conclusion of the meeting. The following discussion summarizes the information provided by the OSSE Team presentations and the feedback given by the committee. Issues considered important by the committee and corresponding recommendations are contained within the feedback sections of the discussion.
Overview – Steve Lord, NCEP/EMC
Steve first presented the intended purposes for this NPOESS OSSE Advisory Committee Meeting. The first purpose was to update the committee on OSSE Team activities since the last committee meeting (September 1999). The second purpose was to have the committee consider the following issues: the level of readiness of the OSSE calibration experiments, and the appropriateness of the strategy for the Doppler wind lidar (DWL) OSSEs. He then reviewed the committee’s recommendations from the previous committee meeting. They were:
1. Consistency between Nature Run cloud cover and cloud water content
2. Non-NPOESS OSSEs to assess potential changes in actual observing system between 1993 (Nature Run date) and present, and between present and 2007 (NPOESS flight)
3. Inclusion of systematic error in simulated observations
4. Cloud and precipitation metrics
5. Survey of NPOESS sensors and rational choices of candidates for OSSEs
6. Sufficient computing resources
Information on items 1, 2, 3 and 6 were presented at the current committee meeting. These will be discussed in the relevant sections of this report.
Steve also introduced the committee to the newly established NASA/NOAA Joint Center for Satellite Data Assimilation (JCSDA). Under the current arrangement, the NPOESS OSSE project is receiving funding from three separate sources. Under the proposal, the project would be a funded activity of the JCSDA. Among other advantages, this arrangement opens the possibility of OSSE and related observing system experiment projects continuing after the operation of the NPOESS satellites begin. It is hoped that more funding stability and increased computing resources might also be forthcoming under the JCSDA.
Committee Feedback:
The committee pointed out a need for a written plan that describes in detail the organization, inter-relationship of responsibilities, and accomplishments of the various task principals on the OSSE Team. The committee felt that such a document, which can and should be dynamic (subject to continual updating) in nature, would be helpful both to the team and onlookers (such as the committee). Lucid communication and understanding of roles among team members is always a benefit to a team effort, especially when several institutions are involved. In a related recommendation, the committee feels that development of a timeline for when each OSSE accomplishment will be implemented in NCEP operations would be of value to both NPOESS/IPO decision-makers (hereafter referred to as “management”) as well as onlooking groups that depend on NCEP global data assimilation system (GDAS) products. The committee described an “accomplishment” as the completion of algorithm or technique development, stand-alone testing, and testing within the OSSE framework. Such a timeline allows tracking of progress by committee and management, calls attention to need to routinely provide proposed sensor impact to IPO for decision-making, and gives NCEP guidance on lead-times available to be prepared to assimilate new observing systems. The committee stressed that the proposed timeline should not force decisions related to the execution of the OSSE project. That is, serious compromises should not be made in the OSSE project just to meet the timelines. Instead, the timeline should be like the recommended organization plan in that it is primarily intended for information and general guidance. Both can serve as reference for when quick answers are required regarding organization and schedule – if all team members are subscribers to these “living” documents, then all will answer the questions with the same answers. Finally, the committee emphasized the need for the OSSE Team to get the information from the OSSEs to the IPO as soon as possible. Instead of waiting until calendar-dictated report times, a mechanism should be established to effectively communicate OSSE sensor impact findings (and a thorough explanation of their implications and caveats) to the OSSE Team’s immediate contacts at the IPO. The primary reason for the committee’s concern is the perception that the IPO instrument selection is ahead of OSSEs. Immediate communication of OSSE findings (of any type) will likely prove useful to decision-makers in authorizing and selecting instrument design strategies. In addition, it will improve the perceived value of the OSSE effort in the view of the IPO.
NCEP/EMC Global Data Assimilation System Activities – John Derber, NCEP/EMC
John first described a number of changes made to the NCEP/EMC GDAS in the period since the last OSSE committee meeting. One of the changes made was the inclusion of a capability to assimilate precipitation information from the Defense Meteorological Satellite Organization’s SSM/I satellite sensor. A related fundamental change to the NCEP global numerical weather prediction model (an integral part of the GDAS) is the implementation of a prognostic cloud water formulation. Other satellite-related GDAS developments include assimilation of NOAA-16 Advanced Microwave Sounding Unit (AMSU) – A and AMSU – B and NOAA-15 AMSU – B sounding radiances, changes to AMSU – A quality control and cloud liquid water bias correction, less thinning of AMSU – A measurements, enhancements to the data handling code within the GDAS to allow easier inclusion of new satellite or other observing systems, and inclusion of GOES-8 and GOES-10 sounding radiances over the ocean. Other general enhancements to the GDAS recently implemented include accounting for actual position (vice launch position) of rawinsonde, dropsonde and pilot balloon sensors (considered increasingly important with decreasing model grid spacing), improvements in ozone assimilation, improved parallelization of computer code to increase throughput on massively parallel computing platforms, and a corresponding overall code documentation improvement for the GDAS.
The next part of John’s presentation focused on preparations for the assimilation of radiances from the satellite-borne Advanced Infrared Sounder (AIRS) sensor. A vast spatial coverage of data being informally obtained from NESDIS is being thinned to ¼ density. In addition, from the approximately 2000 sounder channels available, some 280 channels are being assessed for actual use. Because this is currently considered an experimental (not routinely operational) satellite sensor, the formal NASA to NESDIS to NCEP data path is not in effect. A temporary setup has been arranged to allow the flow of some of the data to NCEP for the purposes of GDAS preparation. In terms of actual GDAS enhancements to accommodate the AIRS radiances, the relevant radiative transfer code has been recoded, made more portable, and has included new scientific aspects. The scientific enhancements include development of quality control procedures (related to cloud effect considerations), re-evaluation of channel selection and improved surface emissivity over land and ice.
John next presented near-term plans for the modification of the GDAS techniques. NCEP/EMC will develop a capability for computing and using situation-dependent background error covariances for both the global and regional assimilation systems. A recursive filter tool, similar to that being used in the regional system, will be employed in the GDAS to define error covariances. Another modification planned is the introduction of bias correction of the background (short-term forecast) fields. This is considered important, since assimilation theory assumes that both background fields and observations are unbiased. Other plans include the improvement of the moisture field background error covariances, cross-correlations between moisture/ozone and mass/wind fields, inclusion of surface skin temperature and soil moisture analysis, and beginning the development of a cloud analysis system. Also during the next 1-2 years, NCEP/EMC plans to add the following types of observations to those being assimilated by the GDAS: QuickScat sea surface winds, satellite winds, GPS radio-occultation, GIFTS satellite high resolution interferometer, DMSP SSM/IS, imager radiances from MODIS, GOES, and AVHRR sensors, and NPP/NPOESS experimental sensors as they become available.
Committee Feedback:
The committee stated that a higher priority should be placed on preparation for
operational ingest of “research” satellite sensor observations (e.g., AIRS)
into GDAS. Use of these sensors allows NCEP to learn how to use the information
effectively, potentially results in beneficial impacts to operational
forecasts, and provides information on the utility of the observations to the
sensor’s principal investigators. NCEP and NESDIS operations should take early
initiative to formalize the data flow from satellite to operations rather than
waiting until the sensor is flying to construct an informal arrangement.
Interagency cooperation needs to be carried out at management levels to arrange
for effective satellite to operations data flow. A selling point to convince
management of the benefit of such an undertaking is the opportunity for NCEP to
advance in operational capability with respect to the international community.
Thus there is a need for the interagency development of a time schedule for the
“research” satellites and when the NCEP GDAS will incorporate their
observations.
Another
panel recommendation was the use of the information obtained from observing
system experiment impacts to select the important channels from experimental
satellite sensors from the massive number available. The framework that the
OSSE effort has built in carrying out its “real data experiments” could serve
as a parallel test capability to run such experiments. This is an example of a
satellite sensor coming on line before OSSEs could be conducted to determine
the best way to use the sensor data. Alternatively, stand-alone techniques such
as principle component analysis could be employed to “down select” useful
channels from the thousands available from the newest satellite sensors.
Finally, the
committee commented that the planned bias correction of the background fields
in the GDAS should not be developed using the OSSE Nature Runs as a reference.
Instead, an automated bias correction formulation should be designed that can
adjust automatically to any changes in the data assimilation system. Again, the
OSSE project software infrastructure can serve as a useful parallel testing
bench for such a modification to the GDAS.
Simulation of
Conventional and GOES Wind Observations – Joe Terry and Bob Atlas, NASA/DAO
Joe Terry
began the presentation with a discussion of the global distribution of
conventional and satellite observations and how they have changed from 1993 to
the present. While the number of operational rawinsonde soundings have
decreased slightly worldwide, the number of aircraft, ship and buoy
observations have seen a significant increase. Expected beyond 2001 is an
expansion of ACARS aircraft observations, additional buoy observations, water
vapor winds, and an increased number of satellite-obtained ocean surface winds.
The implications for the OSSE project is that all of these near-term additions
to the global observation suite will have to be simulated so that meaningful estimates
of the additional impact of NPOESS sensors can be obtained.
Joe next
discussed the issue of deriving cloud drift wind CDW information from GOES
satellite imagery. Up to now, the CDW algorithm was applied to real satellite
imagery from the period of the Nature Run. A new effort is being undertaken to
simulate cloud drift winds with respect to the Nature Run clouds. Joe showed
comparisons between the satellite observed cloud fraction and the real cloud
drift wind locations versus the locations of clouds as depicted by the
corresponding Nature Run scenes. In simulating other observing systems great
pains are taken to locate the observations where they were actually taken. In
cloud drift winds, this is not always possible because the Nature Run cloud locations
may not match the locations of the original imagery clouds. Because clouds are
often associated with important weather features, cloud drift winds can provide
crucial information to the initial state of a forecast that may not be
available from other routines observing systems. Differences in assigned
locations of the cloud drift winds can lead to differences in the initial state
circulations and thus differences in ensuing forecasts. It is considered
especially important to run comprehensive tests of simulated cloud drift winds
in data assimilation calibration experiments, to make sure their impact is
similar to real cloud drift wind experiments.
Joe next
covered the simulation of satellite ocean surface winds, such as obtained from
ASCAT and QuickScat sensor systems. A simulation strategy being considered is
to obtain representative current samples of the data and use them to generate
perfect simulated observations from the Nature Run 10 meter winds. Then appropriate errors could be added based on
information obtained from validation error statistics.
Bob Atlas
briefly commented on the simulation of lidar winds. One of the concerns in the
present OSSE design is that a lot of small-scale motion information is
unresolved by the current Nature Run data set. Given the finer spatial scales
that the proposed Doppler wind lidar (DWL) will be likely to detect, this poses
a possible limitation of the usefulness of the Nature Run to simulate DWL
winds. With this in mind, NASA/DAO has generated a 3.5 month-long nature run
data set at 0.5 degree latitude-longitude grid spacing using their Finite
Volume Community Climate Model (FVCCM). Bob showed animations of the simulated
wind vectors that demonstrated the fine-scale circulations generated by the
FVCCM. Another possible benefit of the FVCCM nature run is the greater number
of cyclones it generated during the NPOESS OSSE Nature Run period compared to
the Nature Run currently being used – closer to the actual number depicted in
the “reanalysis” data set of the real conditions.
Committee
Feedback
The NASA/DAO
presentations raised the following question in the minds of committee members:
What is the OSSE team role of the NASA/DAO effort? This was not clear to the
panel. There seemed to be duplication of effort vis-à-vis simulation of
conventional observations (DAO vs. NCEP), efforts on simulation of DWL (DAO vs
SWA), role in defining nature run to be used in OSSE (ECMWF vs FVCCM.
Subsequent discussion at the meeting cleared up this issue to some degree. For
example, the committee was told that DAO is responsible for the simulation of
the observations from existing observing systems for the OSSEs including the
addition of random errors, while NCEP is investigating methods of suitably
adding systematic error (biases) to the simulated observations (see following
discussion of Jack Woollen’s presentation). It was explained to the committee
that simulation of proposed DWL winds was indeed the role of Simpson Weather
Associates, but that NASA/DAO was anticipating the need for proper nature run
conditions to do so realistically. Additionally, the FVCCM may be used to
generate nature run data sets for other seasons, and would be subjected to the
same level of pre-experimentation scrutiny as was applied to the ECMWF Nature Run.
Still, to avoid the
appearance of duplication among the various activities of the OSSE Team
members, the committee recommended that a detailed restatement of the current
and proposed future assignments of the OSSE Team members be developed as a part
of an overall plan for organization and
accomplishments of principal investigators (see General Feedback section
below). A related question asked by the committee was: What is the role of the
JCSDA in coordinating the activities of the various OSSE team members?
Subsequent communication with the OSSE team suggests less of a coordinating
role and more of a source of consistent and sufficient funding for the JCSDA.
This needs to be spelled out and communicated. Perhaps the transfer of
authority for the NPOESS OSSE Project to the JCSDA is the right time to
redefine roles and responsibilities of the members of the OSSE team.
Simulation
of Satellite Radiances – Tom Kleespies, NOAA/NESDIS
Tom first
reviewed the status of this part of the NPOESS OSSE project as of the previous
committee meeting. At that time, the infrastructure for simulating the cloud
cleared and limb adjusted TOVS radiances was almost completed. NCEP/EMC had
just been modified to assimilate the raw unadjusted (called level 1-B)
radiances, and it was clear that the simulation infrastructure would have to be
adapted accordingly. One-time-only funding was identified to complete the work
for the TOVS radiances, but future funding would have to be forthcoming to
simulate the proposed NPOESS sounder radiances. The level 1-B TOVS radiances
were simulated as follows. First, the actual 1-B radiance soundings for the
period of the Nature Run were established as the reference for the simulations.
The Nature Run temperature, mixing ratio, pressure profile, cloud amount,
surface wind speed, and skin temperature were interpolated in time and space to
the actual level 1-B TOVS profiles locations and times within three hours of
each Nature Run time. The calibration coefficients used with the real level 1-B
radiances were used in the radiative transfer models used to compute the
simulated 1-B radiances. Tom then showed the various radiative transfer models
used to perform the various necessary functions involved in the simulations. In
virtually all cases, the radiative transfer model chosen for simulation was
different from the model used in the NCEP GDAS. Tom then discussed the
assumptions made in assigning cloud geometries for the simulations and the
assignment of instrumental noise to the simulated radiances. At this point in
time, the TOVS level 1-B radiances have been simulated for the HIRS and MSU
sensors on board the NOAA 11 and 12 satellites for the Nature Run period. The
advanced microwave sounding unit (AMSU-A) level 1-B radiances, while not
available in 1993, have been simulated as if they were measured by the same two
satellites in the Nature Run period. Tom then showed example map plots
depicting the brightness temperatures from certain channels of both the real
and simulated level 1-B radiances. While the spatial patterns were generally
similar, some degree of difference was apparent in the magnitudes between the
depictions.
Tom then
gave a brief summary of the current status of the NPOESS OSSE Project satellite
radiance simulation task. Once again, funding for the contractor doing the work
ran out in September 2000. Another one-time-only allotment was obtained to
carry the contractor through until December 2000. Because the Department of
Commerce budget was not signed until mid-December, the contractor did not
continue past December. A new contract was signed in April 2001, and work will
resume pending the assignment of new contract personnel. Future work on this
task includes the simulation of the AIRS, IASI, and CrIS which are infrared
sensors proposed for NPOESS, as well as the NPOESS-proposed SSMIS, ATMS and
CMIS microwave sounders. Some work on the AIRS simulation infrastructure has
begun.
Panel
Feedback:
One point of concern
raised by the committee was the fact that no discrimination of ice versus liquid
cloud information is contained in the current Nature Run. This means that there
can be no accounting for this phase discrimination in the simulation of the IR
radiances. Yet these phases have vastly different effects on the radiative
transfer equation solutions. Another concern voiced was that a blackbody
assumption is being used for the clouds in the simulations, which can result in
colder brightness temperatures where high clouds exist in the nature run than
from actual radiances which partially see through the transmissive high clouds.
No committee recommendations were given to correct these deficiencies. The committee recognizes the importance to
the OSSEs of properly simulating existing and GDAS-implemented satellite sensor
radiances. The committee acknowledges the difficulty of the problem, suggesting
that higher funding and manpower support should be given, as meaningful OSSEs
await the successful simulations of the current satellite sensors. Regarding
currently deployed or near-term experimental (non-operational) satellite
sensors - what is their priority in simulation for the OSSEs, how can we be
sure their radiative transfer algorithms work well? This revisits the
recommendation given in the GDAS section above regarding a schedule for the
operational use of data from the research satellites. For the future proposed
NPOESS sensors, the committee
is concerned that satellite radiance simulation is lagging behind other NPOESS
OSSE efforts, even their implementation into GDAS. For example, work is well
underway to include AIRS radiances in the GDAS, but their simulation has barely
begun. The committee recommends that significant resources be committed to
development of radiative transfer algorithms for proposed NPOESS sensors in a
timely manner for use by sensor designers and NPOESS decision-makers in the
Integrated Program Office (IPO). Finally, much information is now known about
future sensor design as a result of progress in the sensor design and selection
process. The OSSE Team should use this information to prioritize and determine
the order of the simulation of these sensors for the OSSEs.
Simulation
of Doppler Wind Lidar Data – Dave Emmitt, Simpson Weather Associates
A Doppler
wind lidar is the only active remote sensor proposed for the NPOESS suite. Dave
Emmitt is responsible for the simulation of the line of sight (LOS) winds that
should be detected by a typical but unspecified lidar technology. The fact that
the task is to remain “technology neutral” naturally leads to the employment of
“bracketing OSSEs.” A range of achievable specifications are simulated in
separate experiments in terms of vertical extent, coverage, and degree of
penetration through cloud by the sensor “shots.” The scenarios are constrained
by the assumption of a single satellite in sun-synchronous polar orbit. Each
bracketing experiment specifies the coverage (horizontal and vertical) and
accuracy considering both random and correlated errors. Dave described some
general DWL concepts, including the difference between the major two lidar
technologies (direct detection and coherent detection) and scanning versus
non-scanning schemes.
The Nature
Run winds are being used to simulate the DWL winds for the OSSEs. The coverage
being assumed in the simulations is based on a 2000 km swath width, which is
the maximum attainable by a conical scanning system (assumed), leaving a gap of
approximately 500 km between scans at the Equator. Each simulated line of sight
(LOS) wind observation represents a 200 X 200 km target area, allowing for
approximately 30 individual shots to be used to form a single LOS estimate. The
NCEP GDAS is actually assimilating simulated LOS winds from two different angle
“looks” of the same target area, from two (displaced in time) different
satellite positions (a forward look and and an aft look). In the vertical, a 1
km spacing of the LOS observations is assumed above 2 km altitude, and 500 m
spacing below 2 km. Accuracy is accounted for in the simulated winds first by
assignment of random error taking into consideration signal-to-noise ration,
pointing jitter, and other instrument sources of mis-measurement. A typical
range for random error assigned is 0.5 – 10 m/s. The error assumption also
accounts for the number of shots that actually fall incident within a target
area. Because partial cloud cover can block some shots from reaching the lowest
cloud-free altitude of their penetration (thus leaving fewer shots to measure
wind at this destination altitude), it is assumed that the error increases as
the number of incident shots decreases. Factors that go into assigning
systematic error include horizontal and vertical error correlations, and the
instrument and platform types as well as the actual atmospheric conditions.
Dave presented a list of factors that contribute to systematic error that are
instrument unique, platform induced, atmosphere independent and
atmosphere/earth surface condition dependent. Cloud effects that are being
considered in the simulations are the degree of attenuation (ranging from weak
in thin cirrus to strong in optically thick clouds), strong signals from the
clouds themselves, and solar bias (only in the case of direct detection).
Dave
discussed the four OSSE experiments for which simulations have been prepared to
date. They include the differences between clustered and distributed shot
design, single and dual (forward and aft), aerosol, molecular and cloud
returns, and the number of shots used to produce the reported LOS observation.
The actual experiment names and descriptions will be given in the “Calibration
Experiments” section below. Dave closed his talk by giving a perspective on the
number of simulated LOS wind observations that were generated for the four
experiments. Something like 72,000 wind profiles were produced from the Nature
Run for each 24-hour period using the cloud effect assumptions. The number of
actual wind reports ranged from 58,000 assuming cluster shots and only cloud
returns, to 1.6 million assuming distributed shots and aerosol and molecular
returns.
Committee
Feedback:
The committee had
formerly pointed out a need to account for consistency between cloud cover and
cloud water content in the Nature Run when simulating shot penetration. There
is now a recognition that the information content of the Nature Run does not
provide sufficient detail to account for both cloud aspects. Therefore, the
committee supports the current approach to dealing with clouds – the Nature Run
clouds (lack of fidelity) cannot support attempts to account for lidar
propagation through clouds. The distributed vs. clustered simulation
approach is a concern: it is not clear the Nature Run will correctly predict
the number of shots that get through the reported cloud coverage given its
fidelity. Thus there is a need to be aware that the vertical distribution of
lidar returns from Nature Run data is not necessarily a true reflection of
reality. This means that this approach does not give enough support for
selecting one sensing strategy over another. This reinforces the earlier
contention that the OSSE impacts from DWL can be very sensitive to the
distribution of cloudiness in the database from which the DWL LOS winds are
simulated.
Adding Systematic Errors
to Simulated Conventional Observations – Jack Woollen, NCEP/EMC
Observations are
simulated for assimilation in OSSEs by combining a true value extracted from a
nature run with an estimate of noise, also known as observation error. The
random component of the error consists of instantaneous mis-measurement by the
sensor (instrument error) and information on spatial/temporal scales that
cannot be resolved by the assimilating model (representativeness error). The
systematic component of the error is a bias resulting from some innate
time-independent inability of the sensor to measure the absolute value of the
actual environmental conditions. Jack’s presentation discussed two methods of
assigning error to simulated observations. Method 1 involves adding randomly
generated noise to the true value, using error levels consistent with objective
estimates of real instrument and representative error typical for each sensor
type and parameter measured. Method 2 adds an empirically derived estimate of
observation error to each true value in the global, sensor ensemble over the
duration of the nature run. The observation error estimates are derived from
real data assimilations. The focus of Jack’s report was an investigation of
Method 2 estimates for a single parameter - zonal wind component - resulting
from real data assimilation experiments conducted twice daily over a week-long
segment of the Nature Run period (February 1993).
The basic assumption
used in estimating the Method 2 errors is that the observation – analysis
difference resulting from assimilation of a particular sensor type and
parameter represent the portion of the observation information not influencing
the resulting analysis at the observation site. This is considered “noise” or
error to the assimilation system. The difference would consist of both random
and systematic error at any place and time, but if averaged over time and space
the random component would vanish leaving an estimate of just the systematic
component. Jack showed a number of time-height cross sections of the rms and
mean of the observation – analysis difference of zonal wind component for the
North American rawinsondes for the seven-day period of the investigation. The
results showed that the mean difference in space and time is not insignificant.
Jack contended that the point by point observation – analysis difference from real
experiments may be a reasonable estimate of the actual observation errors.
Because they match the locations and times of the simulated true values from
the nature run, they could be added to the nature run true values to
approximate real observation errors on a point by point (in time and space)
basis. However, due to the nature of the extraction techniques to produce the
true values from the current Nature Run, the number of real and true
observations diverge increasingly below (lower in altitude than) 700 hPa.
Because TOVS retrievals were used in the real data assimilation experiments
while TOVS radiances were used in the OSSEs, the no-TOVS assimilation had to be
used as the basis for the point by point empirical error experiment. The point
by point observation – analysis differences were added to true values as the
Method 2 observation error to the OSSE assimilation. Forecasts using both
Method 1 and Method 2 error assignments were conducted and verified against the
true observations. He compared the results to the same forecast errors from the
control (including TOVS radiances) real data assimilation. His conclusions from
the results were that the forecasts using only the Method 1 (random error) were
unrealistically good, while those from Method 2 (full empirical error) were
more realistic. The observation error rejection rate by the data assimilation
is another measure of realism. Jack found that the number of rejected
observations in the Method 1 experiments was unrealistically small, while the
number of rejected observations from Method 2 was much more in agreement with
that of the control experiment. Empirical errors contain gross error as well as
bias, and are applied at the proper times and locations. His results suggested that the empirical
error prescription method has merit and should be investigated further.
Committee Feedback:
The committee stated a
need to clarify the role of this project with respect
to the simulation of observing errors by NASA/DAO. It seems that the results
shown suggest that the approach being used by DAO to assign errors to
conventional observations (like Method 1) could be improved upon. The OSSE Team
needs to decide if the NCEP/EMC investigation will likely produce definitive
results in time to affect the current set of simulated conventional
observations, or whether a latter round of OSSEs might possibly be conducted if
the current investigation shows the merit of the empirical error assignment
method. The panel recognizes the need for including realistic high-resolution
“noise” (non-resolvable scales of motion) in the simulation of observation
error (the nature run may be too smooth in comparison with current operational
GDAS resolutions). It is possible that the point by point empirical method
could take care of this if both real and OSSE experiments are conducted at the
same grid spacing (so that noise level in the OSSEs is imposed at the same
level as the real experiments).
Calibration
Experiments – Michiko Masutani, NCEP/EMC
In
conducting OSSEs, it is important to ensure that the analysis and forecast
impacts of the simulated observations of existing observing systems are similar
to their counterparts in real data experiments. Only then can the additional
impact of new proposed observing systems be realistically assessed from the
OSSEs, because they are an attempt to “look ahead” to the time when the new
observations would be added to the observational suite in future real data
assimilation. Michiko described her task of conducting experiments to determine
the level of similarity between counterpart synthetic and real data
assimilations, often called calibration experiments. The basic methodology is
to conduct the data assimilation in both synthetic and real environments over
some period of time beginning with the “no data” experiment as a worst case
reference. In successive data assimilation experiments, the experiment is
repeated over the time period, but now using different combinations of the real
and simulated existing observing system observations in the respective
environment. The degree of impact that each additional observing system
observation type has on the analysis or short-term forecasts is compared
between the real and synthetic environments. If there are significant
differences between the two for any given combination of included observation
types, this suggests that there may be an error in the way the synthetic
observations have been simulated or are being used in the data assimilation.
The present
set of calibration experiments were conducted using the March 1999 version of
the NCEP/EMC GDAS, an earlier version of the spectral statistical interpolation
still in current use. The assimilating model used a triangular 62 spectral
truncation and 28 vertical levels. Both real and synthetic experiments used the
distribution of actual observations existing in February 1993, including RAOBs
and other conventional data sources, ACARS aircraft observations, HIRS and MSU
level 1-B radiances from NOAA-11 and NOAA-12 satellites, and satellite cloud
track winds. The ECMWF Nature Run was used to simulate the conventional and
ACARS data and cloud track winds (NASA/DAO) and the level 1-B radiances
(NESDIS) using the methods described in current and past presentations by these
groups. The Nature Run has a triangular 213 truncation and 31 vertical levels,
and covers the period 06 UTC 5 February 1993 to 00 UTC 7 March 1993. In order
to attempt to simulate the level of difference between the real analyses and
the real world in the synthetic environment, Michiko ran a five day forecast
from the Nature Run initial conditions to 10 February, then ran full observing
system data assimilations in the synthetic environment to 13 February. Thus at
13 February she had prepared a “starting point” atmospheric state in the
synthetic environment that should be similar in observation sensitivity to that
of the real analysis environment. At 00 UTC 13 February she began the synthetic
and real data assimilation experiments using or withholding the various
combinations of the following four groups of observation types: TOVS 1-B
radiances, RAOB temperature, RAOB winds, and all other conventional data
(ACARS, cloud track wind, surface observations). The control experiment used
all four groups, and Experiments 1-6 used varying combinations of the groups.
Experiment 7 used only the “all other conventional data” group and represented
the sparsest data experiment shown. The experiment forecast – control forecast
root-mean-square-errors from six-hour forecasts four times daily resulting from
the respective real and synthetic data assimilations were computed as a measure
of observing system impact. Michiko showed a number of charts of the RMSEs for
zonal wind and temperature in differing latitude zones and at several isobaric
levels over a four-day period. The patterns of growth of the error between the
real and synthetic experiments compared favorably in most cases. One exception
was the 850 hPa temperature in the 20S to 20N latitude zone, in which some of
the synthetic experiments showed higher RMSEs than their real counterparts.
Michiko suggested that this may be due to the use of a fixed sea surface
temperature in the Nature Run (from which the synthetic observations are
simulated) compared to a variable sea surface temperature in the real analyses.
She intends to study this case further to assure that the simulations are
correct. Michiko also showed charts of the RMSE of 500 hPa wind for both
analysis and 72-hour forecasts averaged for the period 16-18 February at
12-hourly intervals. Later she showed RMSE for height, temperature and zonal
wind at 500 hPa for 0-120 hour forecasts. Also shown was anomaly correlation of
500 hPa height for 12-hour intervals to 120 hours of forecast time for various
wave number bands. In all validations the respective control analyses were used
as reference for forecasts from both the real and synthetic analyses.
In the next
part of her presentation, Michiko went on to the first of the OSSEs conducted
in this project. She described four experiments involving simulated Doppler
wind lidar line-of-sight (LOS) observations as provided by Simpson Weather
Associates. They were: (Exp 1) full tropospheric LOS soundings, clouds
permitting, (Exp 2) only wind observations from clouds and (where clear) the
planetary boundary layer, (Exp 3) mid and upper tropospheric wind observations
down to the levels of significant cloud coverage, (Exp 4) wind observations
from a non-scanning instrument providing full tropospheric LOS soundings,
clouds permitting, along a single line that parallels the ground track. All of
the DWL simulation assumptions are discussed in the SWA presentation described
earlier in this report. Two additional experimental factors were considered for
each experiment: clustered versus distributed, and greater observation weight
versus lesser observation weight (no measurement error was assigned). In the
clustered assumption, the data product is based on averaging lidar shots
clustered within a very small area compared to the target area. In the
distributed assumption, the data product is based on averaging the lidar shots
distributed throughout the target area as would result from continuous conical
scanning.
Michiko
showed a number of charts of RMSE differences for Exp 1 and Exp 4, both with
both assumptions of clustered-distributed and lesser-greater observation
weights (eight experiments in all). RMSE differences was calculated from the
RMSE of experiment – nature run minus RMSE of control – nature run for each of
the eight OSSEs. For both experiment and control the 6h forecast was used, and
the verifying time nature run field was used as reference. Thus all OSSEs with
a beneficial impact should have negative values of the RMSE differences, the
larger the negative value the greater the impact. Results were shown for zonal
wind at 200 and 850 hPa in three latitude zones, and temperature at 850 hPa in
three latitude zones. In virtually all results shown, the Exp 1 experiments
showed greater beneficial impact that the Exp 4 experiments. Among the Exp1
results, the two distributed assumption experiments showed more impact than the
two clustered, and for both the greater observation weighting assumption
experiments showed more impact than the lesser. In all experiments, greatest
impact in the zonal wind was seen in the tropics, with a lesser but significant
impact in the Southern (summer) hemisphere and and least amount in the Northern
(winter) hemisphere extratropics. The only significant temperature impact was
seen in the Southern hemisphere extratropics.
Committee
Feedback:
In the presentation of calibration experiments and the observing system simulation experiments, the committee recommends that the goals and objectives should be clearly stated up front. It is important that any audience, especially the uninitiated, should be able to understand what is being done and why. For example, it could be confusing to a lay person to see calibration experiments in which existing sensor observations are withheld when he/she thinks the goal is to show the benefit of using new sensor observations. Also, the level of detail that should be presented will depend on the audience to which it is directed. While the kind of information shown (including goals and objectives up front) might be suitable for a group of peer researchers, it would be confusing and unnecessary detail to higher level management. The committee felt that a simple explanation of the what and why of the calibration experiments and a word summary of the findings would suffice for project management. Management wants to be convinced that the OSSE Team is satisfied that the OSSE experimental design is sound and that all observing systems have been simulated correctly and are used in a realistic manner in the assimilations. Even for technical audiences, there is a need to synthesize and summarize the calibration study results.
There is a
need to include a reference magnitude for observing system impacts in both the
calibration experiments and the OSSEs. For example, how do these impacts
compare with the difference between two different data assimilation systems
(e.g., NCEP vs DAO)? Such a reference should be provided so that level of
significance can be identified.
Other issues
that arose from this presentation included the implications of unknowns and
uncertainties of the Nature Run, and the need for spectral analyses of motion
scales in the experiments compared with those that can be measured by proposed
observing systems. Because these issues have implications for all aspects of
the OSSE Team activities, they will be discussed in further detail in the
General Committee Feedback section below.
General
Committee Feedback
The
committee identified a number of strong points in the OSSE Team presentations.
They included:
·
Careful
attention to observational impacts – significant amount of work on assessing
the OSSE impacts
·
Good
use of metrics in the impact experiments (e.g., anomaly correlation by wave
number groups)
·
Color
handouts were useful – panel found them helpful
·
We
found the coordination of efforts to be good
connection between analysis of obs-anal differences and the observation impacts DWL simulations used in observation impacts (DWL OSSEs).
The committee had a number of general concerns and
suggestions for the OSSE team. First, the committee considers it vital to distill the OSSE Team’s products
to be more consumable by decision makers. Much information has been given, but
it is not organized in a manner that can effectively inform project management
and funding decision makers. So the next step is to pull together an efficient telling
of the OSSE Team’s story. Included in this presentation should the next step in
OSSE assessment - the potential forecast improvement needs to be shown in order
to emphasize the utility of the OSSE procedure. Second, the committee is
concerned with OSSE Team concerns about the perception of funding agencies
regarding rate of progress. A lot of the behind the scenes work that goes into
ensuring an accurate assessment of impact (simulation of observations,
assignment of errors, calibration experiments) takes the bulk of the time and
effort, yet is never seen by management. The actual OSSE impacts of the
proposed sensors is the tip of the iceberg in the overall OSSE project. One way
to allay management’s and funding agency concern about rate of progress is to
report on an ongoing basis. This reiterates the suggestion given in the
“Overview” section above to present pertinent information on progress to the
immediate contact persons at the IPO and other funding agencies. Perhaps this
issue will be alleviated to some extent in the future by reporting to just one
authority – the JCSDA. An important selling point to management is the
perceived importance of gaining ground in international competition in forecast
improvement from NPOESS sensors. The U.S. is on the leading edge of sensor
technology, and it should be more argued that this advanced technology can be
more quickly and effectively exploited in weather prediction if financial
resources are applied now to preparing assimilation systems to use and demonstrating
the impact of new sensor technologies. On a related issue, the OSSE Team
expressed concerns about prospective large amounts of data, especially from
near-term experimental satellites, and how they can be handled. The committee
advises that the OSSE Team communicate NCEP GDAS plans to decision-makers for
preparation for and use of these data. This reiterates the earlier suggestion
of a timeline for the implementation of research satellite data in the
assimilation system.
The committee raised
concerns about the level of complexity of the OSSE design, especially
the detailed focus on simulating observations and the details presented in the
calibration experiments. This makes the
findings of the project difficult for decision-makers to understand. One
possible simplification is to emphasize
impacts on forecasts, but do so in a way the viewer can understand and
appreciate. There is concern by
some committee members about level of
technical detail being considered to date. A suggestion for simplification of the calibration
experiments is adding incrementally increasing numbers of randomly located
observations, without regard to simulating current observing systems. This can
be used to demonstrate the principle of forecast impact which is the main
purpose for the OSSEs. So much focus on carefulness in simulating the various
observing technologies is only worthwhile after impacts are shown in this type
of simple experiment.
Another
issue mentioned was the complications to impact experiments from unknowns and uncertainties
of the Nature Run. The lack of a variable sea surface temperature specification
in the Nature Run and the lack of a consequent ocean surface temperature
anomaly in the tropics was cited as an example. Considerable attention
has been given to certifying the realism of the Nature Run, especially in the
clouds. Experiments may uncover other Nature Run deficiencies that will have to
be investigated on a case by case basis. In the case of the sea surface
temperature issue, it was suggested that the real experiments be conducted with
the same fixed sea surface temperature field used in the Nature Run.
Still
another committee suggestion was to perform spectral analysis on the scales of
motion of simulated observations vs. first guess fields, and control experiment
(all conventional observations and existing satellite radiances) vs. control
plus DWL. Then compare the spectral analyses from the synthetic environment
with that of the corresponding components of the real experiments. This is
another way to assess the realism of the OSSEs: is the degree of high frequency
information contained in the simulated observations (and thus, the Nature Run
itself) similar to that of the real atmosphere – is a realistic level noise
being used? At T62, too much of the higher frequency information may be
filtered out in the assimilating model integration – the spectral analyses is
one means of quantifying this. The committee encourages the use of alternative
NWP model investigation of high-resolution motion scales (e.g., MM5, FVCCM,
ECMWF) and the simulated observation impacts on these scales. In fact, on the
basis of the aforementioned motion scales investigation, it may be found that
the current Nature Run contains too little energy in the higher-frequency
scales of motion to properly capture the noise associated with real
observations. Another committee suggestion was to perform diagnostic studies,
including spectral analysis study, on various geographic regions of the
full-resolution Nature Run, then compare the motion scales from the T62 impact
experiments to see what scales the latter are missing. Use this
information to design higher-resolution impact studies.
The committee recommends that emphasis should be placed on demonstrating the OSSE infrastructure utility at higher spatial resolution. The committee recognizes the fact that computational resources have limited the spatial resolution at which the calibration experiments and OSSEs to date have been conducted. Priority should be placed on acquiring or accessing computing assets to accomplish the more realistic, higher resolution experiments reflecting current and future operational data assimilation systems. This should be briefed as a high priority requirement to the JCSDA.
It is likely
that not all of the suggestions given above are feasible for implementation.
Time, manpower and fiscal constraints may preclude the OSSE Team from acting on
all of the suggestions. This is understandable and the committee in no way
implies an obligation to try them all. The OSSE Team should consider each
suggestion and determine which should be done, and which have the most payoff
for the least amount of investment of time and energy.
Summary
The
committee was satisfied with the level of progress achieved since the last
meeting. The OSSE Team has made favorable advances in all areas of the project,
in spite of funding, manpower and computational limitations. The committee
feels that the calibration experiments, besides their likely limitation of
coarse resolution (to be further investigated in the suggested spectral
studies), demonstrate that the “control experiment” that serves as a baseline
for new sensor impact is ready to go. This positions the team well to focus
their efforts on the OSSEs themselves. It is advantageous for the team to do
so, to provide information on new sensor impact even as the proposed sensor
design is being considered and selected. If the OSSE team could gear up to
perform “on demand” OSSEs for each proposed sensor in a timely manner by
establishing a strong dialogue with and connection to with the NPOESS
instrument specialists through the IPO, they can perform a valuable service to
the NPOESS project. The OSSE Team is nearly in a position to offer their
services in this way, and should do so by putting together their story in a
concise and effective manner and selling it to the IPO and instrument
specialists. The team needs to emphasize a “Look what we can do for you” sales
pitch to the NPOESS sensor teams and get their support for the OSSE Team’s
active role in the sensor design and selection process. This is the best
guarantee for continued future funding.