\ Atmospheric Sciences Research Laboratory
rj? CHEMISTRY, MATHEMATICS, METEOROLOGY, MODELING, PHYSICS
PEER REVIEW REPORT
ON THE
IN-HOUSE RESEARCH PROGRAM BY THE
METEOROLOGY DIVISION
September 15-18, 1987
Panel
Ernest M. Agee, Chair
Raymond J. Deland
David EmmJtt
P.K. Misra
Anthony R. Olsen
Anthony D. Thrall
Office of Acid Deposition, Environmental Monitoring and Quality Assurance
Office of Research and Development
U.S. Environmental Protection Agency
Research Triangle Park, North Carolina 27711
-------
U.S. ENVIRONMENTAL PROTECTION AGENCY
ATMOSPHERIC SCIENCES RESEARCH LABORATORY
PEER REVIEW AND WORKSHOP MANAGEMENT SERVICES
Contract Number 68-02-4129
Project Officer
Ronald K. Patterson
Prepared by
Research and Evaluation Associates, Inc.
1030 15th Street, N.W., Suite 750
Washington, D.C. 20005
(202) 842-2200
727 Eastowne Drive, Suite 200A
Chapel Hill, N.C. 27514
(919) 493-1661
-------
TABLE OF CONTENTS
CHARGE TO REVIEW PANEL 2
GENERAL OVERVIEW 3
PROGRAM ASSESSMENT 6
Boundary Layer/Turbulence Research 7
Regional Oxidant Modeling 8
Regional Particulate Modeling 11
Regional Acid Deposition Modeling Area 12
Complex Terrain Dispersion Modeling 14
Wake Effects Research 15
Technology Transfer/Application 16
APPENDIX A: Peer Review Panel 18
APPENDIX B: Agenda for the Peer Review of the In-House
Research by the Meteorology Division 20
APPENDIX C: Comments on Model Evaluation 26
APPENDIX D: ASRL Process Evaluation for Meteorology
Division In-House Research Peer Review 29
APPENDIX E: Results of the EPA Participant Survey
Meteorology Division 33
APPENDIX F: ASRL Staff Response to Reviewers' Comments 37
APPENDIX G: The Laboratory Director's Review
Comments on the Panel Report and the ASRL
Staff Response 49
-------
EXECUTIVE SUMMARY
The Review Panel has reached a consensus on the following findings
and recommendations:
• Leadership of the Division is highly effective
t Scientific staff is competent
• Scientific morale is good
f Experimental fluid dynamics is excellent
• Computational quality control of model development is
exemplary
• Computer resources are inadequate
• External pressures are forcing premature release of
preliminary scientific findings
• In-house statistical expertise should be strengthened to
support model evaluation
t A program for advanced research model implementation is
strongly endorsed
-------
SECTION 1
INTRODUCTION
This is a report from the Peer Review Panel (see Appendix A for
list of panel members) organized to review the in-house research
activities of the Meteorology Division (MD) of the US Environmental
Protection Agency (EPA), Atmospheric Sciences Research Laboratory
(ASRL). Prior to meeting with the MD staff, the panel received three
volumes of selected research papers by the MD staff for preliminary
review. On September 15-18, 1987, the panel met at the Research
Triangle Park, NC, to hear 45 presentations by the MD staff on in-house
activities. Presentations focused on individual scientific
contributions to EPA programs rather than a review of activity as
project officers or monitors of extramural contracts.
The panel appreciates the efforts of the MD Director and individual
staff members in preparing well-organized preview material and
informative presentations. In general, the quality and organization
were equivalent to that of a scientific professional meeting. In
organizing this report, the panel chose to follow the general
scientific areas that were used to organize the presentations (see
meeting agenda in Appendix B).
Finally, the panel is grateful to Charlotte Coley (Research and
Evaluation Associates) and Ron Patterson (ASRL Peer Review Coordinator)
for assistance with the logistics in conducting the review and
preparing this report.
-------
SECTION 2
CHARGE TO REVIEW PANEL
The panel was provided with the Division Director's preview
materials outlining the mission of the Division's program. Principles
to guide the peer review process were stated in the Research and
Evaluation Associates, Inc. document Guide for Participants in the EPA-
ASRL Task/Project Peer Review Process provided by Charlotte Coley.
Further discussions with the MD Director at the beginning of the site
visit helped to clarify the charge to the review panel. Evaluation of
in-house research and support activities given in this report are based
on the following program objectives:
t Basic Research
• Model Development
• Applications and Evaluations
The panel has strived to evaluate the research presented without
the benefit of some important information such as CVs that describe and
quantify total up-to-date professional activities. Although the panel
was provided with detailed individual workplans, insufficient time was
available to ingest all this information.
The panel attempted to evaluate the in-house activity, realizing
that a sizeable portion of the comprehensive program effort resides
within the extramural community. Further, the panel has taken into
consideration that MD program activities (although residing within the
ORD) must respond to regulatory needs.
-------
SECTION 3
GENERAL OVERVIEW
The MD programs are conducted in a favorable in-house research
support environment. Generally, the scientific staff seems to be
enthusiastically engaged in their efforts, and morale within the
Division seems good. Administrative leadership is effective in
promoting this favorable professional environment. MD programs are an
important part of both national and international research efforts, and
the reputation of the staff brings respect and requests for applying
in-house scientific expertise to help address the needs of EPA's policy
analysis.
The experimental fluid dynamics is excellent and goes far in
advancing the MD program objectives. The computational quality control
of model development is exemplary and in itself represents an ideal
model for QC assessment that could be transferred outside the Division.
However, computer resources are inadequate for model development,
applications, and evaluation. Computer cycles seem to be highly
variable and inadequate, and thus introduce some degree of uncertainty
in carrying out the MD programs. Small computers are augmenting
computing needs, but a total spectrum of computing support is required
(from small work stations that stand alone or access other machines to
supercomputer model simulations for numerically intensive model
development and testing).
In-house statistical expertise should be strengthened to support
model evaluations. The activities known variously as "model
evaluation" and "model validation" deserve more attention by
environmental researchers, in general, and thus by the MD in its role
as a leader of environmental research. There seem to be essentially
two purposes of these activities: (1) to better understand atmospheric
processes, and thus to improve atmospheric models, by identifying and
diagnosing failures in model performance, and (2) to quantify the
reliability of models that are used either to decide regulatory issues
-------
or to guide regulatory policy. (See Comments on Model Evaluation given
in Appendix C.)
The Meteorology Division seems to be a frequent recipient of
requests from other offices and laboratories within the EPA that
require redirection of program FTEs. The Division appears to be
responding to these requests, but at the cost of delaying longer term
research needed by the Agency. In some instances (e.g. the requests to
run the ROM simulations for EPA constituents), scientific expertise is
steered away from model development to application runs. Some of these
"quick turnaround" demands for staff time seem to be appropriate uses
of expertise within the MD; however, the balance of short-term and
long-term activities should be defined more explicitly.
One of the major challenges to ORD Management would seem to be
finding and preserving a balance of staff time allocated to quick
response needs of the Agency on the one hand and longer-term needs on
the other. Although a good balance seems to have been achieved by the
MD, it also seems to the panel that ORD Management in general has a
responsibility to both the scientific staff and to the Agency to take
steps to guard against the erosion of the Agency's research capability.
A recommendation of the panel with respect to this issue is to
broaden the criteria used to evaluate the achievements of the
scientific staff. In general, the staff members are doing an excellent
job of communicating with their peers within their respective
disciplines through publications and presentations. But communication
across disciplines (horizontal communication) should be emphasized, for
in the relatively young field of environmental research,
interdisciplinary communication is less well established, although
extremely important.
Similarly, more of the staff should be encouraged to familiarize
themselves with the Agency's use of their research to help set policy
(vertical communication). This will help to ensure the relevance of
individual research within the MD. More importantly, it should
engender a vision and a coherent, internally developed agenda for the
MD's research. Even though the MD's tasks are ultimately determined
outside the Division, such an agenda would be an important reference
-------
for establishing the MD's research priorities. A more active role by
the MD in setting research priorities would help to guard against the
erosion of the MD's research capability by an overload of quick
response tasks.
Finally, the emerging program concept of Advanced Research Model
Implementation is strongly endorsed by the panel. Several existing
modeling efforts form a nucleus for this initiative. An additional
consideration should be given to nesting some of these models inside
the global climate models that presently exist within the research
community.
-------
SECTION 4
PROGRAM ASSESSMENT
To facilitate the achievement of consensus, the committee adopted an
internal evaluation procedure whereby each panel member assigned the
value of 5 (Excellent), 4 (Very Good), 3 (Good), 2 (Fair), or 1 (Poor)
for each of the following performance criteria:
• Quality of Science/Work
• Relevance to MD/EPA Missions
• Interaction with In-House Colleagues and Outside
Community
Emphasis was given to scientific accomplishment, such as reviewed
journal publications and conference/workshop publications and
presentations. Professional service to EPA and extramural activities
were also considered. In some instances, scientific support was the
focus of work activity rather than basic research or model development
and testing. Recognition was also given to the importance of applied
research. Although the scores are not provided in this report, the
panel directed itself to write program evaluations that were consistent
with composite numerical scores assigned to the performance criteria.
The panel was pleased with the consistency of the individual
evaluations. The reviews given below reflect more directly on some
presentations than others. Some individual comments are provided, but
not in detail for each individual presenter. Absence of detailed
technical comments should not be construed as positive or negative.
-------
Boundary Laver/Turbulence Research
The ASRL is actively involved in the following areas of boundary
layer and turbulence research:
1. Experimental investigation of atmospheric boundary layer
turbulence,
2. Innovative dispersion model development,
3. Evaluation and assessment of dispersion models.
The committee evaluated these areas of research in terms of their
quality, relevance to EPA goals, and interaction of the responsible
scientists with the outside scientific communities.
Experimental boundary layer research includes turbulent diffusion
experiments inside a convective boundary layer and urban and non-urban
boundary layers. The quality of research in these areas is excellent.
This research is very relevant to EPA objectives and the responsible
scientists have excellent interactions with outside communities.
New modeling approaches are being investigated using probability
density functions of vertical velocity in convective boundary layers.
For buoyant plumes, semi-empirical models have been developed. Again,
the quality of research in these areas is excellent, the work is highly
relevant, and there is very good interaction with outside communities.
Future work in this area should emphasize better characterization of
buoyant plume dispersion inside a convective boundary layer since the
probability density function approach is not expected to work well. The
quality of work in the application of models is very good, and the work
is highly relevant and interactive with the outside community. It is
good to see the efforts directed towards definition of atmospheric
stability with proper boundary layer velocity and length scales. The
sensitivity analyses of the models could be improved by using a joint
probability density function of the variables rather than varying them
independently.
The rotary spectral density application in the analyses of
turbulence data is not obviously relevant to EPA objectives even though
-------
the quality of the work is good. Such work may not enhance our
understanding of boundary layer turbulence significantly. The
interaction with the outside community is not very apparent.
Similarly, the analyses of turbulence data using superposition of
autocorrelation functions is not very relevant. The quality of the
work is fair and interaction with outside communities not very good.
Eulerian autocorrelation functions do not determine diffusion in a
turbulent flow. Relating Eulerian autocorrelation functions to
Lagrangian autocorrelation functions is not simple and unique. Also,
"true" Eulerian autocorrelation functions in the atmosphere exhibit
oscillation about the zero line thus making the estimate of time scales
questionable.
Regional Oxidant Modeling
This effort within the Division epitomizes the full spectrum of
program objectives ranging from modeling development to applications
and evaluation. The nucleus of staff spearheading this effort is
highly competent and is attempting to entrain appropriate activities
throughout the division. It seems that external pressures (outside of
ASRL/MD) are responsible in part for this program initiative. These
pressures also require model results that are difficult to achieve.
Imposed resolutions in the Regional Oxidant Model (ROM), as required to
be practical, are inconsistent with routine meteorological data sets.
For example, upper air weather observations are taken every 12 hours on
a mean spatial scale of 325 km. Model integrations are for one hour
over an 18x18 km grid. The ROM appears to be a zero-order model in
atmospheric dynamics (at best), whereas the chemistry of the 28
reactive species is treated in a more sophisticated manner. Further,
the raw input data of emissions are rather discretized and irregularly
available in space and time. The ROM is a pure diagnostic model, i.e.
output concentrations are used (after the fact) to evaluate the effects
of alternative strategies for managing air quality planning and
standards emission. It is not realistic to impose point concentrations
(observations and/or standards) on regional model plume dispersions.
8
-------
Skill scores and/or performance evaluation may objectively rate the ROM
concentrations satisfactorily when instances of point concentrations
actually fail miserably. New methods of evaluation should be
considered that weigh both spatial and temporal scales or features.
If the ROM is labeled as a zero-order modeling effort, then its
performance will be properly evaluated. More support of this effort
should be provided. Also, apparently none of the field programs
to-date have provided high resolution (space and time) rawinsonde data
sets. Since a diagnostic model approach is acceptable, more emphasis
should be placed on collecting a more representative data set for model
development and performance evaluation.
ROM development apparently needs more computing cycles. Algorithms
for the model are not fully understood (for example, incorporation of
the effect of clouds and radiative processes). The ROM model should
interact more with numerical weather prediction model efforts. Can
some nested grid simulations and use of trajectory model forecasts (by
NOAA-NWS) be considered? Trajectory models combined with chemical
reactions code may yield some useful results.
In several instances the scientific capability of the involved
scientists exceeds the scientific credibility of ROM. For example, it
has been hypothesized that receptor sites can be grouped into classes,
such that concentrations averaged over all sites in a given receptor
class at a given hour is a quantity that is approximately the same for
all realizations of the concentration ensemble. If this is true, then
with respect to the receptor class average (RCA), the concentration
field is a quasi-deterministic variable. Therefore, model simulations
of single realizations can be interpreted in the conventional
deterministic manner. Although this hypothesis has not yet been
tested, it is assumed for the present purposes that it is correct. It
is unscientific to proceed in this way, though it may be politically
and practically sound.
Biogenic emissions inventory development is essential for the
application of ROM. Currently, the work involves translation of known
emission information to an hourly emissions ROM grid. This requires
allocation of land use and biomass information to grid cells,
-------
derivation of relevant biomass volumes, and identification of species
emission factors. This approach appears appropriate for the intended
use and necessarily involves extrapolation and disaggregation of
existing information. The research has involved contact with forest
and crop researchers during the development of information sources, but
has not resulted in presentation of results to that community. The
latter is recommended as a complement to existing reports to the air
pollution modeling community. Uncertainty estimates were presented,
but seemed questionable.
Work has focused on ozone dry deposition over large areas as
determined from aircraft measurements. Data were collected through
extramural effort, but data analysis has occurred in-house. Aircraft
measurements suggest that ozone concentrations may be highly correlated
to organized convection. Ozone appears to be a good tracer of
convection, similar to moisture. Detrended temperature should be
spectrally decomposed to show the basic convective mode (BCM) or cloud
street mode. An approximate 4 km length scale is evident in the raw
data. The ozone spectrum presented indicates a 3.8 km spectral peak,
which accords well with temperature data. What about the spectrum for
the specific humidity q? Also, several papers exist in the literature
that show larger length scales of mesoscale convective organization in
a convective boundary layer, known as higher convective modes (HCM).
The typical aspect ratio for the BCM is 3 to 1, compared to an order of
magnitude increase for the larger HCMs. Inspection of the raw
concentration for ozone indicates a signal on the larger length scale
that is in phase with temperature trace. An effort should be made to
complete the analysis of these unique data sets by interpretation of
convective structures (for all fields) and treating ozone as a tracer.
This has some exciting possibilities.
The development and implementation of verification procedures for
the ROM will significantly enhance the current and future use of ROM.
Current computer science concepts have been integrated into EPA's
atmospheric modeling community. This integration not only results in
products of known quality, but also appears to be accomplished as a
natural part of the model development. There appears to be an active
10
-------
effort to publish this approach within the air pollution modeling
community. Although not directly presented during this peer review, it
is known that this experience has contributed to the formulation of
quality assurance procedures for the RADM model evaluation program.
The presence of the Data Systems Analysis Branch staffed with
knowledgeable application computer scientists is one key to the
Meteorology Division's ability to fulfill their mission. Its presence
allows atmospheric scientists and computer scientists to contribute
their skills to an interdisciplinary effort of model development,
evaluation, and application.
Very systematic and methodical quality control analysis is
performed with state of the art practices, from modular programming
down to file naming conventions. Professional systems programming
architecture could serve as a model for other similar unrelated model
developments. Feedback between the physicist(s) behind the model
development and programming quality control people was a little vague.
Scientists that develop the model know the inherent weaknesses.
Quality control should generate, independently, a similar set of
weaknesses. The need for more computing cycles is not abundantly
clear, but probably is the case. Funds should not be used to upgrade
antiquated computing equipment. What about vectorizing the code for
the ROM? Also, how much do QC checks slow down the ROM simulation? A
more coordinated plan is apparently needed for providing computing
facilities to ASRL/MD or even to all of EPA at Research Triangle Park.
Can metacode be saved, upon which variations of model predictions can
be made? Although briefly discussed, the ROM may be improved more by a
parallel implementation than by a vector implementation. Also, can EPA
use the supercomputer resources at the five national centers?
Regional Particulate Modeling
The Regional Particulate Model (RPM) program uses ROM and has some
of the same problems of weak scientific support and inadequate input
data, especially emissions. The work on different aspects of the
model, presented by four of the scientific staff, showed evidence of
11
-------
industrious efforts to provide what is needed with inadequate resources
of manpower and computing facilities, resulting in contributions that
were incomplete.
The cloud process module work may evolve into some good science.
However, this component of both the ROM and the RPM (or any regional
scale model) is not getting the attention it merits. The simplistic
approach presented is a reasonable application of the state-of-the-art
knowledge for simple cloud model formulation (primarily a thermodynamic
emphasis). Effort needs to be extended to properly treat the total
entrainment process between the cloud top and the overlying interfacial
layer. Cloud ensemble scale should be considered in accordance with
the ROM grid length.
The TTNEPH data may be all right for radiation studies but its
value in the wet deposition and cloud processing modules is very
questionable. Efforts should be made to incorporate the GOES imagery,
which is very useful for the ROM domain. The work on this project
epitomizes the vitality offered by young new staff members.
The MESOPUFFII modeling effort is probably important in principle,
but (very good) performance evaluation shows poor performance. The
CAPTEX data set appears adequate to test performance. Best results
were obtained for cases of no directional shear. Work should be
published in reviewed journals (not just an EPA report).
The evaluations of the Regional Lagrangian Model of Air Pollution
RELMAP has served to expose NAPAP emission inventory deficiencies.
Thoroughness and comprehensive knowledge were evident in the
presentation.
Regional Acid Deposition Modeling Area
MD in-house research activities related to regional acid deposition
modeling involve the International Sulfur Deposition Model Evaluation
(ISDME) project, Regional Acid Deposition Model (RADM) evaluation and
assessment projects, and cumulus cloud venting based on data from
VENTEX, CUVENT, NEROS, and NASA Langley field experiments. The
in-house activities reviewed comprise only a fraction of the project
12
-------
activities. Hence no attempt is made to review either past or planned
field experiments or the entire RADM model evaluation effort. Since
the United States portion of the ISDME was completed within the MD, it
will be more fully reviewed.
The ISDME work has placed the MD staff at the forefront of the
regional model evaluation scientific community. The ISDME Project
evaluated eleven linear-chemistry atmospheric models of sulfur
deposition using 1980 observational data. This effort is noteworthy as
being the first organized effort at regional model evaluation. Not
only has the work of MD staff contributed to an increase in knowledge
on model performance, but also to new approaches to model evaluation.
Although evaluation methods for regional scale models which are
acceptable to all scientists have not been formally established, the
ISDME scientists have employed a careful system of statistical
procedures to arrive at their conclusions. The effort to rank the
models' performance does not appear to aid the choice between models
for policy use and requires further research to be usable in that
context. In addition to the statistical evaluation, additional effort
is needed to outline conditions under which the models are or are not
applicable. For example, the simple Lagrangian models show the same
performance repeatedly over several years when annual averages are
compared. The ISDME analysis would be more useful if it had led to an
understanding of why the models show this type of performance. Use of
metrics based on spatial pattern is important, but requires
incorporation of spatial surface interpolation. Further research into
the applicabilities of kriging to surface concentration and deposition
fields is recommended.
Current in-house research focuses on design studies for surface
network, intra-spacing network and emissions data collection. The
design work relies heavily on kriging but does provide a basis for
quantitative design studies that focus on minimizing the evaluation
"error". The MD staff recognizes that kriging requires further
evaluations as a viable design and model evaluation tool and that other
evaluation methods should be explored.
13
-------
The cloud venting research focuses on the study of penetrative
convection, which leads to venting of the planetary boundary layer and
transport of pollutants. The results of this work have important
implications for both the ROM and RADM programs and should be
continued. The researcher is encouraged to establish stronger research
relationships with the cloud dynamics community. The field experiment
data are unique and deserve more detailed documentation of the
meteorology surrounding the events. The further classification and
determination of 6 factors for several types of cloud-topped
(fractionally) planetary boundary layer is recommended.
Overall, the MD staff involved in regional acid deposition modeling
are (1) performing research that is very relevant to EPA programs, (2)
actively interacting with the relevant scientific community, and (3)
completing research that is worthy of publication (and is being
published) in the peer-reviewed literature.
Complex Terrain Dispersion Modeling
Nearly 30% of the presentations were associated with in-house
efforts to support the Complex Terrain Dispersion Modeling (CTDM)
program. The panel was unanimous in its positive evaluation of the
quality of the experimental modeling activity. The ability and
willingness of the fluid modeling team to respond to the demands of a
regulatory agency and still be able to carry out basic research speaks
well for both the scientists and division management.
Since the panel was not briefed on the complete CTDM program, it
was unable to review the in-house research within the context of the
whole project. However, the following specific comments can be made:
1) Good, interesting results are obtained in the fluid modeling
laboratory and should continue to be published in JFM type journals.
The panel, however, questions whether the laboratory experiments are
sufficiently reliable simulations of atmospheric flows to be used to
set regulatory guidelines.
2) Inverted tows in stratified cases are adequate, however a deep
sheared flow layer is not present (as in the neutral wind tunnel case
14
-------
studies). Experimental fluid mechanics is done well, both in
laboratory techniques (flow visualization and measurement) and
application of theory. How these results are factored into the CTDM
was not explained well.
3) Relevance of model and laboratory results (even with good
agreement) do not necessarily apply to atmospheric boundary layer
flows. Stably stratified flows are useful, but do not address the
convective planetary boundary layers. Dynamic similarity between
numerical model/laboratory flow and atmospheric analog was not
established. It appears that a constant eddy viscosity value
arbitrarily replaces the molecular viscosity in the model momentum
equation. In reality, isotropic Fickian diffusion cannot be assumed.
Further, the spatial dependencies in nonlinear systems of equations
give rise to non-Gaussian dispersion. Again, the fluid mechanics work
is excellent but its extension to actual atmospheric dispersion is not
clearly explained.
Wake Effects Research
Although driven by regulatory requirements, the wake effects
research can be considered quite basic. The work on the
characterization of the complex turbulence in the wake of a
non-aerodynamic structure is a good example. The need for an improved
plume formulation is obvious and common to much of the theoretical and
physical modeling efforts at MD. Efforts to obtain better flow
visualization and, in particular, to quantify those observations are to
be commended. Illumination techniques and the statistical evaluation
of the resulting data need to be improved. New plume formulations call
for a departure from the traditional Gaussian approach. Any
fundamental advances in this area would have an impact not only on EPA
activities but also on the boundary layer turbulence research community
in general.
The work on auto exhaust dispersion is clearly associated with
EPA's need to model the impact of a major source of atmospheric
pollution. The experimental results presented during the review were
15
-------
based upon only a part of a more comprehensive program of experiments.
It is difficult therefore to judge what should or should not be done
since the work described was very incomplete in itself.
Technology Transfer/Application
This set of presentations, along with two presented as a part of
Boundary Layer/Turbulence Research, seemed to be a representative
selection of work by the Environmental Applications Branch (EAB). That
is, the presentations were consistent the function of the EAB.
The panel regards the UNAMAP to be of very good quality and to be a
most important function of the MD. Indeed, UNAMAP is one of the most
important and visible aspects of the EPA's air quality operations,
since regulatory applications (e.g., obtaining permits, establishing
emission limits, or demonstrating attainment of air quality standards)
typically require the use of a UNAMAP model.
The estimation of the reduction of soybean crops resulting from
various levels of ozone was presented. This analysis was requested by
the Office of Air Quality Planning and Standards. Current levels of
ozone at 320 farms growing soybeans were related to existing and
proposed national ambient air quality standards (NAAQS) for ozone. An
important finding from this research was that, contrary to greenhouse
studies, field studies of the effect of ozone on soybean growth
suggested that two parameters of soybean exposure to ozone (namely, the
duration and the concentration of the exposure) could be summarized by
a single index (effective mean ozone concentration). This work seems
to the panel to be highly responsive to the needs of EPA policy
analysis, and like previous work at the MD on the relationship between
concentration averaging times, may be widely used and cited. Due to
the importance of this work, it deserves further investigation and the
collaboration of other experts in meteorology, atmospheric chemistry,
plant physiology, and statistical inference.
The panel rated as very important the investigation of alternative
techniques for randomly sampling data to efficiently estimate long-term
(seasonal or annual) concentrations using the Point, Area, and
16
-------
Line-Source (PAL) algorithm. This work addresses a practical need for
computational efficiency, but, more significantly, explores
probabilistic modes of air quality analysis. This work should continue
and should be supported by a statistician who is an expert in the
techniques and inferential issues of data sampling schemes.
Finally, a preliminary diagnosis of the Turbulent Profile Sigmas
(TUPOS) dispersion model based on a comparison of calculated and
measured SFs concentrations obtained from the EPRI's Kincaid data base
was presented. The investigation will help to determine the degree to
which more detailed turbulence data improves model performance under
field conditions and is therefore regarded as extremely important by
the panel. As with other model diagnoses, sensitivity studies, and
reliability studies conducted by the Division, the statistical issues
are quite challenging (see Section 3 and Appendix C of this review) and
deserve the support of an expert statistician.
17
-------
APPENDIX A
PEER REVIEW PANEL
-------
EPA-ASRL Peer Review Panel
Meteorology Division
Research Triangle Park, North Carolina
Septenber 15-18, 1987
Ernest M. Agee
Chairman, Dept. of Earth and
Atmospheric Sciences
Purdue University
West Lafayette, IN 47907
(317) 494-3282
Raymond J. Deland
Raymond Deland & Associates
2 Collyer Drive
Ossining, N3f 10562
(914) 941-5622
David Etnmitt
Simpson Weather Associates, Inc.
809 E. Jefferson Street
Charlottesville, VA 22902
(804) 979-3571
P.K. Misra
Ministry of the Environment
125 Resources Road
East Wing
Rexdale, Ontario
Canada M9W 5L1
(416) 235-5771
Anthony R. Olsen
Battelle Pacific Northwest Program
P.O. Box 999
Richland, WA 99352
(509) 376-4265
Anthony D. Thrall
Environment Risk Assesntent Program
Electric Power Research Institute
3412 Eillville Avenue
Palo Alto, CA 94304
(415) 855-2627
19
-------
APPENDIX B
AGENDA FOR EPA-ASRL PEER REVIEW OF
METEOROLOGY AND ASSESSMENT DIVISION
SEPTEMBER 15-18, 1987
-------
AGENDA
Peer Review of the EPA-ASKL
Meteorology and Assessment Division
In-Hbuse Research Program
September 15-18, 1987
TIME TCPIC SPlMiK REF.f
Tuesday. Sept. 15 Classroom 3, Environmental
Research Center Research Triangle
Park, North Carolina
i
08:00 - 08:20 AM Opening Session - Panel Welcome Charlotte Coley
Peer Review Program Orientation
ASRL Peer Review Coordinator Ron Patterson
08:20 - 08:40 AM Executive Session in Meeting Al Ellison
Room for Panel and Staff, ASRL
Director
08:40 - 09:00 AM Coffee Break and Introductions of
Review Panel and ASRL Staff
09:00 - 09:10 AM Introduction by MD Director Frank Schiermeier 1
Boundary Laver/Itarbulence Research
09:10 - 09:40 AM Analysis and Parameterization Gary Briggs 2
of Convective Diffusion
Processes
09:40 - 10:00 AM Evaluation of Convective Scaling Tom Pierce 3
for Estimating Diffusion
10:00 - 10:30 AM Boundary Layer Turbulence Studies Jason Ching 4
in Urban and Non-Urban Areas
10:30 - 11:00 AM Rotary Spectral Analysis of Peter Finkelstein 5
Turbulence Measurements
11:00 - 11:30 AM Exponential-Sum-Fitting Tech- Steve Perry 6
niques for On-Site Turbulence
Analysis
11:30 - 12:30 AM Lunch
12:30 - 01:00 PM Executive Session for Review Panel
01:00 - 01:20 PM Meteorological Scaling in Applied John Irwin 7
Dispersion Modeling
21
-------
01:20 - 01:35 EM
01:35 - 02:20 PM
02:20 - 02:40 EM
02:40 - 03:00 EM
03:00 - 03:30 EM
03:30 - 04:00 EM
04:00 - 04:30 EM
04:30 - 05:00 EM
Wednesday. Sept. 16
08:00 - 08:20 AM
08:20 - 08:40 AM
08:40 - 08:50 AM
08:50 - 09:15 AM
09:15 - 09:40 AM
09:40 - 10:00 AM
Relation of Error Bounds of
Maxmium Concentration Estimates
to Meteorology Uncertainty
Regional Oxidant Modeling
Regional Oxidant Model (BCM)
Biogenic Emissions Inventory
Development for Regional Oxidant
Modeling
Break
Ozone Dry Deposition Over Large
Areas from Aircraft Measurements
Verification Procedures Applied
to the Regional Oxidant Model
Evaluation of Regional Oxidant
Model
Executive Session for Raview
Eanel and MD Division Director
John Irwin
Bob Lanb
Jim Reagan
Jim Godowitch
Joan Novak
Ken Schere
Frank Schiermeier
Particulate Modeling
Regional Particulate Model (EPM) John Clarke
doud Processes Module for Frank Binkowski
Regional Particulate Modeling
Utilization of X^NEXU. Data for
Regional Particulate Modeling
Russ Bullock
Sensitivity Analysis of MESOPDFF Jim Godowitch
n and Evaluation with CAETEX Data
Regional Lagrangian Model of Air Brian Eder
Polution (HELMVP) Sensitivity
Analysis and Evaluation for
Particulate Matter
Coffee Break and Introductions of
Review Panel and ASKL Presenters
Deposition Modeling
10:00 - 10:30 AM
International Sulfur Deposition
Model Evaluation (ISDME)
Terry dark
9
10
n
12
13
14
15
16
17
18
19
22
-------
10:30 - 11:00 AM Regional Acid Deposition Model Robin Dennis 20
(RADM) Evaluation Research Program
11:00 - 11:30 AM Cumulus Venting of Pollutants Jason Ching 21
from the Mixed Layer
11:30 - 12:30 PM Lunch
12:30 - 01:00 PM Executive Session for Review Panel
01:00 - 01:30 PM Acid Deposition Assessment Robin Dennis 22
Studies
Complex Terrain Dispersion Modeling
01:30 - 01:40 PM Overview of Complex Terrain Frank Schiermeier 23
Research
01:40 - 02:00 PM Complex Terrain Field Data Bases Larry Truppi 24
02:00 - 02:20 PM Fluid Modeling Facility Object- Bill Snyder 25
ivesf Facilities, Approach, and
Recent Research Accomplishments
02:20 - 02:40 PM Deformation of Plumes by Hills Bill Snyder 26
02:40 - 03:00 PM Break
03:00 - 03:20 PM Measurements of Streamline Traj- Roger Thompson 27
ectories in Stratified Flow Over
Isolated
03:20 - 03:35 PM Laboratory Measurements of Bill Snyder 28
Unsteadiness of Strongly Strat-
ified Flow Fields CDrag) Over
Two-dimensional Hills
03:35 - 03:50 PM Comparison of Numerical and Roger Thompson 29
Laboratory Experiments on Density-
Stratified Flow Over a Hill
03:50 - 04:10 PM Stratified Flow Over Ridges and Bob Lawson 30
Valleys flushing of Valleys)
04:10 - 04:30 PM Laboratory Validation of Flat- Bill Snyder 31
Dividing-Streamline-Surface
Approximation for Complex Terrain
Dispersion Models
04:30 - 05:00 PM Executive Session for Review Frank Schierneier
Panel and M5 Division Director
23
-------
Thursday, Sept. 17
08:00 - 08:20 AM Wind Direction Effects on Bill Snyder 32
Dispersion from Sources Downwind
of
08:20 - 08:50 AM Complex Terrain Amplification Bob Lawson 33
Factors for Various Stack Heights/
Locations
08:50 - 09:30 AM Diffusion from Low Level Sources Bert Eskridge 34
Under Extreme Stratification
09:30 - 09:50 AM Coffee Break and Introductions of
Review Panel and ASRL Presenters
Wake Effects Research
09:50 - 10:20 AM Theoretical Modeling and Eval- Alan Huber 35
uation of Building Wake Effects
10:20 - 10:40 AM Observations of Surface Flow Bob Lawson 36
Patterns Near Buildings Using
Flow Visualization
10:40 - 11:10 AM Video Image Techniques for Wind Alan Huber 37
Tunnel Measurements of Building
Wake Dispersion
11:10 - 11:30 AM Near-Wake Dispersion of Auto Roger Thompson 38
Exhaust
11:30 - 12:30 PM Lunch
12:30 - 01:00 PM Executive Session for Review Panel
Technology Transfer Applications
01:00 - 01:30 EM User's Network for Applied Bruce Turner 39
Modeling of Air Pollution
CUNAMAP)
01:30 - 01:50 EM Development and Application of Ralph Larsen 40
Air Quality Effects Models
01:50 - 02:10 PM Climatological Version of the Bill Petersen 41
Point Areas, and Line Source
Algorithm CPAL)
02:10 - 02:30 EM Evaluation of the TUPOS Dispersion Bruce Turner 42
Model - Preliminary Findings
02:30 - 02:50 EM Break
24
-------
02:50 - 03:10 EM Conparing Air Quality with Ralph Larsen 43
Present or Proposed Eegulatory
Standards
03:10 - 03:30 PM Rise and Dispersion of Merging Bill Snyder 44
Buoyant Plumes in a Stratified
Crossflow (Ocean Outfalls)
03:30 - 03:50 PM Dispersion of Dense Gas Over a Bill Snyder 45
Ramp
03:50 - 04:15 PM Advanced Model Operation and Frank Schierneier 46
Analysis
04:15 - 04:45 PM Executive Session for Review Panel Frank Schienneier
and MD Division Director
Friday. Sept. 18
08:30 - 10:00 AM. Reviewer Debriefing with ASRL Al Ellison
Director
10:10 AM Report Preparation
25
-------
APPENDIX C
COMMENTS ON MODEL EVALUATION
-------
COMMENTS ON MODEL EVALUATION
It seems useful to distinguish the following four types of model
evaluation:
(1) model diagnosis
(2) reliability estimation
(3) sensitivity analysis
(4) uncertainty analysis
Model diagnosis is the comparison of measurements to model
calculations so that inadequate assumptions or formulations embodied in
the model can be identified and investigated. The investigation often
leads to model improvements, so that model diagnosis is typically one
phase of model development. Such diagnoses are often informal data
analyses used to generate new hypotheses. Of course, if such an
informal data analysis suggests a major redirection of research or
model development, the diagnosis is worth checking more rigorously.
Estimating the reliability of a model requires more careful and
thorough statistical analyses than does model diagnosis. First, the
use or uses of the model for which the reliability analysis is being
conducted must be carefully defined. Then a means of estimating the
model's reliability must be determined. Finally, the degree to which
the results of analysis generalize to other conditions must be judged.
These issues are just beginning to be addressed by environmental
researchers and require the full attention of both statisticians and
atmospheric scientists.
Two modes of analysis intermediate to model diagnosis and
reliability estimation are sensitivity analysis and uncertainty
analysis. The goal of sensitivity analysis is to determine the
influence of model input variables or model coefficients on some aspect
of the model output. Like model diagnosis, the purpose of sensitivity
analysis is to better understand the model, at least under some
conditions. Such probing of the model is sometimes performed very
roughly by perturbing just one variable at a time to approximately
bound the plausible range of ambient concentrations. A more thorough
27
-------
analysis of model sensitivities may require the statistical design of
simultaneous perturbations of several variables.
Uncertainty analysis is the estimated attribution of the
discrepancies between measurements and calculations to, respectively,
errors in model input, errors in ambient measurements, model bias, and
random variation of the atmosphere for each set of conditions defined
by the model input variables. In statistical parlance this is called
variance components analysis (if characterization of conditional
variability is restricted to conditional means and variances).
Uncertainty analysis is relevant to model diagnosis. For example, it
can be used to determine whether the apparent poor performance of the
model under certain conditions can be plausibly attributed to
measurement errors or random atmospheric variation rather than model
bias. Uncertainty analysis also pertains to model reliability. That
is, the amount of evidence showing that the model "is" or "is not"
biased under certain conditions, determines the degree to which such
findings generalize beyond the study to other settings or sites.
Uncertainty analysis is extremely challenging since the atmosphere
rarely provides more than one replicate of the same set of model input
conditions. However, the use of either statistical data resampling
schemes or wind-tunnel data as a test-bed may yield useful information.
In all four types of "model evaluation", it is essential that the
evaluation team include or have access to appropriate statistical
expertise.
28
-------
APPENDIXD
ASRL PROCESS EVALUATION FOR METEOROLOGY DIVISION
IN-HOUSE RESEARCH PEER REVIEW
-------
ATMOSPHERIC SCIENCES RESEARCH LABORATORY
Process Evaluation for Meteorology Division
In-House Research Peer Review
The Atmospheric Sciences Research Laboratory (ASRL) of the US
Environmental Protection Agency convened a panel of scientific experts
on September 15 - 18, to review the in-house research of the
Meteorology and Assessment Division. The panel consisted of six
scientists. These reviewers were asked to evaluate the process
involved in preparing and implementing this specific meeting. Five
panelists completed the process evaluation.
The evaluation instrument was designed to assess the following
aspects of the process: 1) Preview Materials; 2) Process and
Logistical Information; and 3) the Review Meeting. A section was also
provided for reviewers to give their comments and recommendations. The
reviewers were instructed to respond to 15 items by circling numbers
from 1 to 5 (with 1 representing poor; 2-fair; 3-good; 4-very good; and
5-excellent).
Table 1 presents a summary of the reviewers' rating for the 15
items. No items were rated poor. All items received some "excellent"
ratings except for overall peer review process and adequacy of time for
executive sessions. Most categories were either rated "good" or
"excellent". Three items were rated fair: adequacy of time available
to preview, meeting purpose, and overall peer review process. Thirteen
items were rated excellent: written quality, technical quality,
utility for outside reviewer, adequacy of time available to preview,
meeting purpose, scheduling of meeting, time/preparation requirement of
reviewers, timeliness of meeting notification, timeliness of logistical
information, adequacy of time for discussion with EPA staff, quality
and utility of presentations, quality and utility of materials
disseminated, and support services and activities.
30
-------
TABLE 1. SUMMARY OF PROCESS EVALUATIONS
Review Categories
Preview Materials
1. Written Quality
2. Technical Duality
3. Utility for Outside Reviewer
4. Adequacy of Time Available
To Preview
SUB TOTALS
Process and Logistical Information
5. Meeting Purpose
6. Scheduling of Meeting
7. Reviewer Responsibilities:
Time/Preparation Reauirement
8. Overall Peer Review Process
9. Timeliness of Meeting
Notification
10. Timeliness of Logistical
Information
SUB TOTALS
Review Meeting
11. Adequacy of Time for
Discussion w/EPA Staff
12. Adequacy of Time for
Executive Session
13. Quality and Utility of
Presentations
14. Quality and Utility of
Materials Disseminated
15. Support Services and
Activities
SUB TOTALS
TOTALS
Number of Reviewers
Rating Each Item
Very
Poor Fair Good Good Excellent
1
1
1
1
2
3
1
1
?
7
1
3
1
1
1
9
3
4
1
1
1
10
21
1
2
1
2
6
1
1
1
3
6
1
1
1
3
15
4
3
3
1
11
1
3
1
4
4
13
2
3
4
3
12
36
31
-------
Is."
Comments:
§ "The organization and support of the peer review by R & E, the
Meteorology Division director and his staff is by far the best
of all prior ASRL reviews that I have participated in. They
are to be commended for their efforts. Reviews of in-house
research by a Division are new and require further
clarification. The review has elements of personnel action
review/appraisal, university tenure review, departmental
accreditation, and "traditional" EPA program reviews. It is
difficult for peer reviewers to define their task within the
existing framework."
t "Impossible to examine all preview material:
• "Entire process is too brutal."
t "Too many people to review with diverse activities."
t "The purpose of the review must be carefully considered. My
impression is that the review was foisted on ASRL by
Washington, DC. To their considerable credit, ASRL management
shared with us this perspective on the review. But I feel
that the use to the ASRL of such reviews can be better defined
and expressed to future panels, after some reflection by the
ASRL management."
Recommendati ons:
• "It would have been nice to have some one-on-one discussion
with selected staff."
• "Up-to-date vita should be provided."
0 "EPA MD organizational chart would have been useful."
t "A time to speak with key people on a one-to-one basis."
• "Provide organizational chart to all panel members in advance
of meeting. Also, as part of putting the research being
reviewed in perspective, it would have been useful to have had
a few pages describing the Meteorology Division's mission,
long-term objectives, and recent accomplishments - a
description at a level of detail somewhere between reference
item #1 and individual workplans."
32
-------
APPENDIX E
RESULTS OF THE EPA PARTICIPANT SURVEY
METEOROLOGY DIVISION
-------
Results of the EPA Participant Survey
Meteorology Division
In-House Research Peer Review
The Atmospheric Sciences Research Laboratory of the US
Environmental Protection Agency convened a panel of scientific experts
to review the in-house research done by the Meteorology and Assessment
Division. EPA personnel involved in the program review were surveyed
as to the effectiveness and usefulness of the peer review process to
them; and to identify problem areas that needed improving. Thirteen
surveys were completed.
The evaluation instrument was designed to assess the following
aspects of the process: 1) review format and logistics; 2) review
panel; and 3) your assessment of the peer review process. A section
was also provided for comments and suggestions. The participants were
instructed to respond to 10 items by circling numbers from 1 to 4 (with
1 representing very satisfied; 2-satisfied; 3-dissatisfied; and 4-very
dissatisfied). Table 2 presents a summary of the ratings for these 10
items. None of the participants were "very dissatisfied" with any
items, although nearly all items received at least one "dissatisfied"
rating. However, the majority of the ratings were in the "satisfied"
and "very satisfied" categories.
34
-------
TABLE 2. EPA PARTICIPANT SURVEY SUMMARY
Number of Participants Ratinq Each Item
REVIEW FORMAT &
LOGISTICS
1. Adequacy of time
for group discussion
with reviewers
2. Adequacy of time for
Proiect oresentations
3. Adequacy of time for
individual discussion
with reviewers
SUBTOTALS
REVIEW PANEL
4. Reviewers' familiarity
with oreview materials
5. Reviewers' demonstrated
knowledge of your
oroqram
6. Reviewers' expertise in
the field
7. Selection of reviewers
(Quality of the panel
as a reviewina unit)
SUBTOTALS
YOUR ASSESSMENT OF THE
PEER REVIEW PROCESS
8. Support service &
activities carried out
bv contractor
9. Objectivity & profes-
sionalism
10. Effectiveness as a
mechanism for OA
SUBTOTALS
TOTALS
Very
Satisfied
1
3
2
6
2
2
4
4
12
3
3
6
24
Satisfied
9
7
7
23
10
10
8
8
36
9
9
11
29
88
Dissatisfied
3
3
4
10
1
1
1
1
4
2
2
16
Very
Dissatisfied
35
-------
Comments;
• "The schedule was overly crowded; it should have been
stretched to 5 days, with so many presentations."
t "Two of the reviewers seemed familiar with air pollution
modeling and meteorology; the others were much less familiar
with this sub-field of meteorology. Although they may be
professional meteorologists, that is no guarantee they are
knowledgable in all aspects of the field.
• "This review was unusual in that so many presentations had to
be compressed into three days. However, it still seems that
most presenters had adequate time for interaction with
reviewers as long as they cooperated by allowing time for
questions."
• "As is usually the case, there was insufficient time allowed
for the large number of topics presented."
t "Very disappointed in not having an opportunity to set up
slides before my presentation because the group did not break
for coffee as scheduled."
• "I find that the handouts (copies of visual aids) are
distracting. The audience begins leafing through the material
instead of paying attention to the speaker. I believe the
handouts should be done away with."
• "Did not believe the process has much benefit for anyone
except it provides a defense if attacked. In other words, we
can say 'well, the peer reviewers like our program, why are
you attacking my budget'. Otherwise, the effort is in essence
of no use."
36
-------
APPENDIX F
ASRL STAFF RESPONSE TO REVIEWERS' COMMENTS
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
ATMOSPHERIC SCIENCES RESEARCH LABORATORY
RESEARCH TRIANGLE PARK
NORTH CAROLINA 27711
MEMORANDUM
DATE: January 15, 1988
SUBJECT: Response to Peer Review Comments on Meteorology and
Assessment Division In-House Research Programs
FROM: Francis A. Schiermeier, Director
Meteorology and Assessment Division, ASRL
TO: Ronald K. Patterson
ASRL Peer Review Coordinator
The peer reviewers' report on the Meteorology and Assessment Division (MD)
in-house research programs is both comprehensive and enlightening. We appreciate
their significant efforts, considering that forty-five technical presentations
were directed at the reviewers during the three-day schedule.
The report contains comments and recommendations on program strengths and
weaknesses, on the adequacy of personnel staffing and computer facilities, on
maintaining the critical balance between research and applications, and on the
Division's role in establishing research priorities. The responses provided
here are addressed to critiques, recommendations, and reviewer questions in all
of these areas. Responses are not provided for the complimentary statements
contained in the report although they have been noted with appreciation.
Comments, pp. 3. 17. and Executive Summary; Repeated recommendations on the
need to strengthen in-house statistical expertise to support model evaluations.
Response: We agree. We are currently recruiting a Ph.D. statistician for the
Division but our efforts are hampered by a temporary NOAA hiring freeze.
Comment. D. 4; The Meteorology Division seems to be a frequent recipient of
requests from other offices and laboratories within the EPA that require redi-
rection of program FTE's. The Division appears to be responding to these
requests, but at the cost of delaying longer term research needed by the Agency.
In some instances...scientific expertise is steered away from model development
to application runs. Some of these "quick, turnaround" demands for staff time
seem to be appropriate uses of expertise within the MD; however, the balance of
short-term and long-term activities should be defined more explicitly.
Response: We agree with the ideal of a clearly defined balance between research
and applications in the Division but we must also realize that the EPA is a reg-
ulatory Agency. As such, it is subject to frequent crises-oriented mandates for
redirection of research efforts imposed by Congress and the Administration.
38
-------
Comment, pp. 4-5: One of the major challenges to ORD management would seem to
be finding and preserving a balance of staff time allocated to quick response
needs of the Agency on the one hand and longer-term needs on the other. Although
a good balance seems to have been achieved by the MD, it also seems to the panel
that ORD management in general has a responsibility to both the scientific staff
and to the Agency to take steps to guard against the erosion of the Agency's
research capability... A more active role by the MD is setting research priori-
ties would help to guard against the erosion of the MD's research capability by
an overload of quick response tasks.
Response; ORD management executes a series of steps each year in the budget
planning process that includes the ranking of existing research programs and new
research initiatives (prepared by Laboratory scientists). These priorities are
negotiated by research committees that are composed of both ORD and regulatory
program personnel. Unfortunately, this deliberate planning process is then
subject to the disruptive redirections described in the previous response. It
is thus up to us to sort out these conflicting priorities to achieve the "good
balance (that) seems to have been achieved by the MD."
Comment, pp. 4-5: More importantly, it should engender a vision and a coherent,
internally developed agenda for the MD's research. Even though the HD's tasks
are ultimately determined outside the Division, such an agenda would be an
important reference for establishing the MD's research priorities.
Response: We feel that we do have such an ongoing agenda for the Division's
research, and that this agenda is frequently used in sorting out conflicting
priorities when they arise.
Comment, p. 5: Finally, the emerging program concept of Advanced Research Model
Implementation is strongly endorsed by the panel.
Response; The establishment of the ASRL Research Modeling Facility is our
attempt to prevent further erosion of the Division's research capability by not
allowing scientific expertise to be diluted by model applications at the expense
of ongoing model development and evaluation.
Comment. DP. 7-8; The rotary spectral density application in the analyses of
turbulence data is not obviously relevant to EPA objectives even though the
quality of the work is good. Such work may not enhance our understanding of
boundary layer turbulence significantly. The interaction with the outside com-
munity is not very apparent.
Response: ¥e take strong exception to the implication of non-relevance. Dif-
fusion models, which are the Division's most important product, are basically
the application of turbulence theory to practical problems. Since the develop-
ment of turbulence theory (and its mathematical treatment) is far from "solved",
it is appropriate for us to spend some of our effort in refining the basis for
diffusion models. Whether the work will be successful or not is, of course, an
open question as it is with a great deal of true research. While it may not have
been made apparent to the reviewers, the spectral research has been reviewed
very favorably by personnel of the NOAA's Wave Propagation Laboratory.
39
-------
Comment, p. 8; Similarly, the analyses of turbulence data using superposition
of autocorrelation functions is not very relevant. The quality of the work is
fair and interaction with outside communities not very good. Eulerian autocor-
relation functions do not determine diffusion in a turbulent flow. Relating
Eulerian autocorrelation functions to Lagrangian autocorrelation functions is
not simple and unique. Also "true" Eulerian autocorrelation functions in the
atmosphere exhibit oscillation about the zero line thus making the estimate of
time scales questionable.
Response: Concerning the relationship between the Eulerian and Lagrangian
autocorrelation functions, the interpretation of Eulerian autocorrelation func-
tions as an indication of the turbulent eddy structure is certainly not new;
Sutton (1953) mentions this topic. Although the Eulerian autocorrelation func-
tion does not directly determine turbulent diffusion, the literature is replete
with attempts to better estimate and interpret the form of this function; some
examples are Hisra (1979) and several cited by Moore et al. (1985). Much has
been published (for example, Hanna 1981, 1986; Weber et al. 1982) showing the
usefulness of estimates of Lagrangian statistics (such as integral time scales)
from Eulerian statistics for applications in diffusion modeling. Regarding the
contact with the outside community, the autocorrelation study was presented (and
received favorable comment) at the AHS Fifth Joint Conference on Applications of
Air Pollution Meteorology (Dahai 1986).
Comment, pp. 8-9: The Regional Oxidant Model (ROM) appears to be a zero-order
model in atmospheric dynamics (at best)... The ROM is a pure diagnostic model,
i.e., output concentrations are used (after the fact) to evaluate the effects of
alternative strategies for managing air quality planning and standards emission.
ROM development apparently needs more computing cycles. Algorithms for the model
are not fully understood (for example, incorporation of the effect of clouds and
radiative processes).
Response: While the reviewers acknowledge the purely diagnostic use of the ROM
for assessing emissions control strategies on a regional basis, there seems to
be some confusion on the nature of the model itself. There are inferences to
the use of better predictive techniques for the atmospheric dynamics portion of
the model. The wind fields used in the ROM are derived from observational data
and physical principles using a new technique described in Lamb and Hati (1987)
that permits explicit treatment of the uncertainty in describing atmospheric
motion. This method permits full utilization of dynamical principles in con-
structing flow fields for the ROM, but only a portion of the capabilities of the
technique have been implemented to date. In addition, flow in the night-time
radiation inversion is simulated using a submodel based on solutions of the
shallow water equations.
Therefore, if the reference to a "zero-order" model pertains to the flow field
description used in the ROM, it is incorrect. The wind field specifications are
at least "first-order" accurate in their satisfaction of dynamical constraints.
In addition, the ROM incorporated schemes for describing vertical mass flux,
including cumulus cloud fluxes, horizontal transport, deposition, subgrid scale
chemistry effects, etc., that were designed specifically to describe regional
scale transport and chemistry phenomena. Unfortunately, there was not enough
time in the peer review to describe the numerous facets of the ROM model in any
detail.
40
-------
Comment, p. 9: Since a diagnostic model approach is acceptable, more emphasis
should be placed on collecting a more representative data set for model develop-
ment and performance evaluation.
Response; We agree. The regional ozidant program has never been funded suffi-
ciently to mount its own field evaluation study. However, as a last resort, we
plan to Join in the field study for the Regional Acid Deposition Model (RADH) to
acquire some appropriate measurement data for evaluation of the ROM.
Comment, p. 9; The ROM model should interact more with numerical weather pre-
diction model efforts.
Response: Driving the transport portions of air quality models with the output
of numerical weather prediction models is a relatively new concept, now being
used by a few larger scale models such as the RADM and ADOM for the assessment
of acid deposition. This approach is incompatible with the basic premise con-
cerning the stochastic nature of atmospheric motion that underlies the method of
Lamb and Hati used by the ROM. The integration of four-dimensional data assimi-
lation into prognostic wind models is more suited to our needs and it may be
explored in the course of implementing our own wind analysis procedure.
Comment, p. 9; ...it has been hypothesized that receptor sites can be grouped,
such that concentrations averaged over all sites in a given receptor class at a
given hour is a quantity that is approximately the same for all realizations of
the concentration ensemble. If this is true, then with respect to the receptor
class average, the concentration field is a quasi-deterministic variable. There-
fore, model simulations of single realizations can be interpreted in the conven-
tional deterministic manner. Although this hypothesis has not yet been tested,
it is assumed for the present purposes that it is correct. It is unscientific
to proceed in this way, though it may be politically and practically sound.
Response; The ROM was developed for use in a probabilistic mode. That is, the
model output concentrations for a given receptor site should take the form of an
expected probability distribution of concentrations, reflecting the stochastic
nature of the interpolated flow fields used by the model. The development of
the probabilistic flow model has lagged far behind the development on the other
components of the ROM system. We therefore now have a fully working model except
for this component. Rather than hold up the entire system while awaiting the
completion of the probabilistic flow model (which may yet take considerable
time to complete), we have chosen to proceed in the quasi-deterministic manner
described where observations at aggregates of receptor sites may be matched with
corresponding aggregates of model predictions. While it is true that this
approach necessitates the use of a-priori assumptions on how the receptors may
be grouped, we feel that while imperfect, it is a better route than either put-
ting the model on hold while awaiting the final flow model or using the model in
a fully deterministic manner which would be completely inappropriate.
Comment. D. 9-10; Biogenic emissions inventory development is essential for the
application of ROM... The research has involved contact with forest and crop
researchers during the development of information sources, but has not resulted
in presentation of results to that community. The latter is recommended as a
complement to existing reports to the air pollution modeling community. Uncer-
tainty estimates were presented but seemed questionable.
41
-------
Response: He agree. A journal article describing the in-house effort on devel-
opment of a biogenic hydrocarbon emissions inventory is in progress, although
transmittal of results to the forest and crop researchers has been ongoing. He
also agree with the reviewers that the uncertainty estimates are questionable.
The intention of the presentation was to provide uncertainty estimates for bio-
genic emissions using the same methodology currently used by the National Acid
Precipitation Assessment Program (NAPAP) for the anthropogenic emissions inven-
tory. Thus, at least the relative uncertainties of the biogenic and anthropo-
genic emissions could be compared. NAPAP recognizes the weakness of the current
methodology but funding for improvements in this area is limited.
Comment, p. 10; Work has focused on ozone dry deposition over large areas as
determined from aircraft measurements... An effort should be made to complete
the analysis of these unique data sets by interpretation of convective struc-
tures (for all fields) and treating ozone as a tracer. This has some exciting
possibilities.
Response; In-house analysis of the aircraft turbulence ozone measurements has
indeed been focused on the dry deposition of ozone as noted by the reviewers.
The major purpose of the turbulence aircraft flight program during the Northeast
Regional Oxidant Study (NEROS) was to obtain experimental data about the spatial
variation and temporal behavior of ozone dry deposition over representative land
use types. The in-house research to date was intentionally designed to study the
variability of ozone fluxes and deposition velocities for different land use
areas as functions of time/height in support of the existing in-house regional
oxidant model program. There are certainly other interesting aspects of this
data set, such as venting of ozone by convective cloud elements and vertical
ozone transport over urban areas; these topics have been examined through an
extramural effort that was not part of this peer review. Other reviewer-
suggested enhancements to the program would be interesting and productive to
accomplish, but resources are limited and we must focus on program needs.
Comment, D. 11; Feedback between the physicist (s) behind the model development
and programming quality control people was a little vague. Scientists that
develop the model know the inherent weaknesses. Quality control should gener-
ate, independently, a similar set of weaknesses.
Response: Interaction between the model developer. Bob Lamb, and the quality
control staff is significant even though this aspect was not stressed in the
presentation. Any suspect data is immediately brought to the attention of Dr.
Lamb who further directs an assessment of the cause and effect of the anomalous
data. Summary graphics such as contours of the maximum daily ozone and the QC
tracer species are provided to both Dr. Lamb and to Ken Schere, the model evalu-
ator, on a routine basis.
- P- 11: Hhat about vectorizing the code for the ROM?... Although
briefly discussed, the ROM may be improved more by a parallel implementation
than by a vector implementation.
Response: About one year ago, IBM personnel made a serious attempt to vectorize
the ROM code. Benchmark executions showed QO_ improvement in performance of the
vectorized code. Subsequent studies performed by the Research Triangle Insti-
tute (RTI) explained why ROM is not effectively vectorized, but more likely to
42
-------
benefit from a parallel processing implementation. Therefore, a cooperative
research program with RTI is currently implementing a loosely coupled multipro-
cessor architecture for the ROM.
Comment, p. 11; A more coordinated plan is apparently needed for providing
computing facilities to ASRL/MD or even to all of EPA at Research Triangle Park.
Also, how much do QC checks slow down the ROM simulation?
Response: Since the EPA has acquired an additional IBM 3090 E computer, the
computing power of the Agency appears to be sufficient for current and antici-
pated ROM applications. ADP timeshare funding for ROM applications is always a
limiting factor in the number of ROM executions. However, the requirement for
consistent QC checks on the numerous files produced for ROM input is also a
limiting factor in the number of new episode periods to be run. Execution and
QC of three days of model inputs can require up to a week. This processing
involves approximately 60 hours of VAX 785 CPU. Fortunately, the time required
to prepare for a control strategy execution is typically less than a day.
Comment, p. 11: Also, can EPA use the supercomputer resources at the five
national centers?
Response: ASRL is also working closely with the EPA National Computer Center to
evaluate the need for supercomputer resources within the EPA. A recent study by
RTI has concluded that the EPA should acquire shared access or its own (mini)
supercomputer at least to explore the potential of the vector supercomputer for
some of its future environmental modeling. However, current OMB funding regula-
tions restrict the use of research dollars for ADP expenditures. Creative solu-
tions may be necessary to use external supercomputer resources.
Comment, p. 12: The cloud process module work may evolve into good science...
The simplistic approach presented is a reasonable application of the state-of-
the-art knowledge for simple cloud model formulation (primarily a thermodynamic
emphasis). Effort needs to be extended to properly treat the total entrainment
process between the cloud top and the overlying inter facial layer.
Response: We agree that the current approach to the cloud process module is
simplistic; what was presented was an initial in-house effort. We are currently
exploring the use of a non-hydrostatic three-dimensional cloud model for examin-
ing some of the most important questions such as cloud top and lateral entrain-
ment.
Comment. P. 12: The TTNEPH data may be alright for radiation studies but its
value in the wet deposition and cloud processing modules is very questionable.
Efforts should be made to incorporate the GOES imagery, which is very useful for
the ROM domain.
Response; He agree. However, the GOES imagery for the 1980 test scenario for
the Regional Particulate Model (RPM) were not available in sufficient complete-
ness to allow determinations of cloud base and top heights to be made, in addi-
tion to the normal areal coverage that most applications require. The RTNEPH
data sets already contain this vertical cloud information, as estimated by the
Air Force through a best possible method approach.
43
-------
Recent correspondence with the NWS National Environmental Satellite Data and
Information Service concerning the GOES NEXT generation of satellites shows
promise that, in the coming decade, sufficiently complete data sets will be
archived to allow subsequent estimations of vertical cloud definition to be made
by our air quality models.
Comment. p. 12; The MESOPUFF II modeling-effort is probably important in prin-
ciple, but (very good) performance evaluation shows poor performance. The CAP-
TEX data set appears adequate to test performance. Best results were obtained
for cases of no directional shear. Work should be published in reviewed jour-
nals (not just an EPA report).
Response: The in-house effort to evaluate and test the MESOPUFF II regional
episodic model with the CAPTEX data base consisted of: (1) an operational evalu-
ation in which the model was applied with all default features; (2) diagnostic
model runs to explore differences with alternative wind fields and dispersion
features in the model; and (3) model sensitivity test runs which primarily
focused on variations in SOx from exercising model options and changing key
parameters in the dry deposition and chemistry modules. The presentation at the
peer review showed just a sampling of some of the many statistical evaluation
results and graphical displays of observed and model tracer plume paterns. The
analysis and interpretation of model results were aimed at identifying reasons
for model overprediction and spatial displacements between the respective plume
positions. While complete results of this task are being documented in an EPA
report, there are plans to publish the notable results in other publications as
recommended.
Comment, p. 13; In addition to the statistical evaluation, additional effort is
needed to outline conditions under which the models are or are not applicable.
For example, the simple Lagrangian models show the same performance repeatedly
over several years when annual averages are compared. The ISDHE analysis would
be more useful if it had led to an understanding of why the models show this
type of performance.
Response: The goals of the International Sulfur Deposition Model Evaluation
(ISDME) focused on the performance of eleven long-term models for each season of
one year. These evaluation periods were commensurate with the periods for which
the models were designed to simulate the processes. However, to identify the
conditions under which these models are or are not applicable, one first must
investigate the behavior of the models for periods much shorter than a season,
i.e., for a week or less. Although this was once an objective of the ISDME,
participating modelers did not submit results for periods considerably less than
their interpretation of the appropriate modeling period. Nonetheless, an in-
house investigation is currently underway to characterize the behavior of one of
these models for daily periods within one season of the ISDME evaluation year.
From these results, we should be able to determine those meteorological situa-
tions for which the model is or is not applicable.
Comment, p. 14: The cloud venting research focuses on the study of penetrative
convection, which leads to venting of the planetary boundary layer and transport
of pollutants. The results of this vork have important implications for both
the ROM and RADM programs and should be continued.
44
-------
Response; We agree. However, NAPAP budget cuts not only prevented further
field evaluation studies but even prematurely terminated analysis of field
measurement data that had been previously collected.
Comment, p. 14: Good, interesting results are obtained in the fluid modeling
laboratory and should continue to be published in JFM type journals. The panel,
however, questions whether the laboratory experiments are sufficiently reliable
simulations of atmospheric flows to be used to set regulatory guidelines.
Response; We believe the proper answer to the reviewers' question is, "not if
taken in isolation." The results of laboratory experiments have not been used,
in and of themselves, to set regulatory guidelines. Many of the studies at the
Fluid Modeling Facility (FMF) have been done upon request from the Office of Air
Quality Planning and Standards (OAQPS). It is within the purview of the OAQPS
to utilize the laboratory results along with many other inputs in setting such
guidelines. As a prime example, the dividing-streamline concept arose from FMF
laboratory experiments. This concept was validated through the complex terrain
field program, then implemented in the Complex Terrain Dispersion Model (CTDH).
It is now up to the OAQPS to establish how the results of the CTDM program,
including the results of the laboratory experiments, are to be used in the regu-
latory process.
Comment, pp. 14-15: Inverted tows in stratified cases are adequate, however a
deep sheared flow layer is not present (as in the neutral wind tunnel case
studies). Experimental fluid mechanics is done well, both in laboratory tech-
niques (flow visualization and measurement) and application of theory. How
these results are factored into the CTDM was not explained well.
Response: The primary value of the work in the stratified towing tank is in
providing fundamental understanding of the physical processes involved in stra-
tified flows. Much can be learned concerning the structure of stratified flows
even in the absence of velocity shear layers. Many aspects of these flows are
dominated by the density stratification rather than the velocity shear. Earlier
work (not reported to the review panel) showed that under very strongly strati-
fied conditions, the shear layer is unimportant. The incorporation of labora-
tory results (and complementary field results) into the CTDM was done by the
contractor and therefore was not discussed in detail at this in-house review.
Comment, p. 15: Stably stratified flows are useful, but do not address the
convective planetary boundary layer.
Response: The addition of a convective tank at the Fluid Modeling Facility is
presently in progress. The most important question to be answered concerning
stratified flows in complex terrain is "where is the plume", e.g., impinging on
the windward slope, going over the top, or being directed around the side. The
question of diffusion about the plume centerline is generally of secondary
importance.
Comment. D. 15: Although driven by regulatory requirements, the wake effects
research (non-aerodynamic structures) can be considered quite basic... Efforts
to obtain better flow visualization and, in particular, to quantify those obser-
vations are to be commended. Illumination techniques and the statistical evalu-
ation of the resulting data need to be improved.
45
-------
Response: He agree. Research on the study of diffusion in building wakes is
intended to provide a balance between the immediate needs for regulatory appli-
cation and the need to develop a solid research base. Improvements to the illu-
mination techniques have been made, and more thorough statistical evaluations of
the resulting data are to be pursued once the relation between video image
intensity and vertically integrated concentration is firmly established.
Comment, pp. 15-16: The work on auto exhaust dispersion is clearly associated
with EPA's need to model the impact of a major source of atmospheric pollution.
It is difficult therefore to judge what should or should not be done since the
work described was very incomplete in itself.
Response; The presentation of the results from the study of dispersion in
automobile wakes was indeed only a part of a larger project but, as an example
of an in-house "wake effects" study, it was included in the peer review. No
further work in this area is planned for the near future.
Appendix C. pp. 27-28: Comments on Model Evaluation
Response: The four categories described by the peer review panel are the major
elements pertaining to application of a model to better understand and charac-
terize its behavior, performance and capabilities (as distinct from applications
of the model to answer a specific policy question). While we might quibble some
about the details of the descriptions under each category, possibly seeing them
a bit more interconnected than presented by the review panel, we tend to agree.
The main point is that a good evaluation requires a team with members who are
good at posing informative and useful tests of the model as well as members who
are good at quantifying and objectively interpreting those tests, the latter
usually being someone with statistical expertise. We believe that our present
actions are in concert with the panel recommendation. He have included persons
with statistical experience as part of the International Sulfur Deposition Model
Evaluation and have statisticians on the model evaluation planning team for the
Regional Acid Deposition Model (RADM) evaluation.
Reviewer Recommendations. D. 32; Provide organizational chart to all panel mem-
bers in advance of meeting. Also... it would have been useful to have had a few
pages describing the Meteorology Division's mission...
Response: These materials were indeed provided as indicated in the first
sentence on page 2 of the peer review report.
Reviewee Comment. D. 36; Two of the reviewers seemed familiar with air pollu-
tion modeling and meteorology; the others were much less familiar with this sub-
field of meteorology. Although they may be professional meteorologists, that is
no guarantee they are knowledgable in all aspects of the field.
Response: Per my instructions to the ASRL Peer Review Coordinator, the panel
was constituted by representatives from the disciplines of statistics, microme-
teorology, systems analysis, fluid modeling, and meteorological modeling. This
mixture was needed to ensure that at least one reviewer was knowledgable in each
of the technical presentations.
46
-------
Comments, DP. 32 and 3fi: Lack of adequate time for presentations.
EfiSponge; As noted in some of my preceding responses, the time allotments for
presentations were occasionally too limited to accommodate complete descriptions
of individual in-house research efforts. For example, the Regional Oxidant Model
(ROM) and supporting components were presented in two and one-half hours, where-
as the ROM was allotted a full day in an ASRL project-oriented peer review in
January 1987. In adhering to this stringent schedule, therefore, presenters
were only able to describe the basic tenents of their research projects at the
expense of discussing ancillary topics such as relevancy to the Agency's mis-
sion, publication of results, and involvement with the scientific community at
large. The peer review report shows that, in most cases, the reviewers took
this limitation into consideration.
References
Dahai, X., S. G. Perry, and P. L. Finkelstein, 1986: Multi-Scaling and Exponen-
tial Fitting in Autocorrelation Analysis. Extended Abstracts, Fifth Joint Con-
ference on Applications of Air Pollution Meteorology, AMS, pp. 348-351.
Hanna, S. R., 1981: Lagrangian and Eulerian Time-Scale Relations in the Daytime
Boundary Layer. Journal of Applied Meteorology, 20:242-249.
Hanna, S. R., 1986: Spectra of Concentration Fluctuations - The Two Time Scales
of a Meandering Plume. Atmospheric Environment, 20:1131-1137.
Lamb, R. G. and S. K. Hati, 1987: The Representation of Atmospheric Motion in
Models of Regional-Scale Air Pollution. Journal of Climate and Applied Meteor-
ology, 26:837-846.
Misra, P. K., 1979: The Auto-Correlation Function of the Vertical Velocity in
the Low-Frequency Range. Preprints, Fourth Symposium on Turbulence, Diffusion,
and Air Pollution, AMS, pp. 41-45.
Moore, G. E., M. Liu, and L. Shi, 1985: Estimates of Integral Time Scales from
a 100-M Meteorological Tower at a Plains Site. Boundary-Layer Meteorology,
31:349-368.
Sutton, 0. G., 1953: Hicrometeorology. McGraw-Hill Book Company, New York, NY,
333 pp.
Weber, A. H., J. S. Irwin, W. B. Petersen, J. J. Mathis, and J. P. Kahler, 1982:
Spectral Scales in the Atmospheric Boundary Layer. Journal of Applied Meteorol-
ogy, 21:1622-1632.
47
-------
APPENDIXG
THE LABORATORY DIRECTOR'S REVIEW COMMENTS
ON THE PANEL REPORT AND THE ASRL STAFF RESPONSE
-------
m)
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
ATMOSPHERIC SCIENCES RESEARCH LABORATORY
RESEARCH TRIANGLE PARK
NORTH CAROLINA 2771 1
DATE: February 1, 1988
SUBJECT: Meteorology Division In-house Program
FROM: Jack H. Shreffler /V^ H S
Deputy Director, AS^L/(MD-59)
TO: Ronald K. Patterson
TPRO, ASRL (MD-59)
I have read the Peer Review report on the in-house research program
of the ASRL Meteorology Division and the response by its Director, Francis
A. Schiermeier. The report is one of the most complimentary I have re-
viewed and the Director's response is complete and adequate in all respects.
Moreover, the Summary of Process Evaluations by reviewers indicates a very
good preparation and execution of the peer review meeting, for which Research
and Evaluation Associates, Inc. and Project Officer Ron Patterson deserve
credit.
Management should not get involved in fixing things that are not broken.
cc: A. Ellison
49
-------
\
§ UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
ATMOSPHERIC SCIENCES RESEARCH LABORATORY
<*/„ ,-0"- RESEARCH TRIANGLE PARK
1 ""& NORTH CAROLINA 2771 1
DATE: February 11, 1988
SUBJECT: Release of Peer Review Panel Report: In-House
Research Program By The Meteorology Division
FROM: Ronald K. Patterson, Peer Review Coordinator
Atmospheric Sciences Research Laboratory (MD-59)
TO: H. Matthew Bills, Acting Director
Office of Acid Deposition, Environmental Monitoring
and Quality Assurance (RD-680)
THRU: Alfred H. Ellison, Director
Atmospheric Sciences Research Laboratory (MD-59)
The enclosed peer review panel report has been reviewed and released
by our Laboratory. Comments by ASRL participants and our Laboratory
Director are included in the report under Appendices F and G, respectively.
We find the comments from our participating staff to be complete and appro-
priate.
This peer review evaluated only the in-house research program under
the ASRL Meteorology Division. A total of 45 technical presentations
were made during the four day peer review. The enclosed peer review
panel report is highly complimentary of the Division's leadership and the
competence of the technical staff. The review panel was impressed by the
high degree of excellence exemplified in the areas of experimental fluid
dynamics and computational quality control. However, the review panel
expressed concern regarding inadequate computer resources and the external
pressures exerted on the staff to prematurely release scientific findings.
The review panel also discussed the need to strengthen in-house statisti-
cal expertise.
We are distributing copies of this report and cover letter to the
individuals listed below. The ASRL peer review contractor, Research and
Evaluation Associates, Inc., has been instructed to forward a copy of
this final released version of this report to each peer review panelist.
Enclosure
cc: Erich Bretthauer (RD-672) Courtney Riordan (RD-682)
William Keith (RD-680) Deran Pashayan (RD-680)
Morris Altschuler (RD-674) \#4rbert Wiser (ANR-443)
Elenora Karicher (RD-680) Jack Shreffler (MD-59)
Basil Dimitriades (MD-59) William Wilson (MD-59)
Frank Schiermeier (MD-80) Meteorology Division Staff (MD-80)
------- |