\  Atmospheric Sciences Research Laboratory
     rj?  CHEMISTRY, MATHEMATICS, METEOROLOGY, MODELING, PHYSICS
           PEER REVIEW REPORT
                   ON THE
IN-HOUSE RESEARCH PROGRAM BY THE
         METEOROLOGY  DIVISION
               September 15-18, 1987

                      Panel
                Ernest M. Agee, Chair
                 Raymond J. Deland
                   David EmmJtt
                    P.K. Misra
                 Anthony R. Olsen
                 Anthony D. Thrall
       Office of Acid Deposition, Environmental Monitoring and Quality Assurance
                Office of Research and Development
                U.S. Environmental Protection Agency
              Research Triangle Park, North Carolina 27711

-------
      U.S. ENVIRONMENTAL PROTECTION AGENCY
   ATMOSPHERIC SCIENCES RESEARCH LABORATORY
PEER REVIEW AND WORKSHOP MANAGEMENT SERVICES
                Contract Number 68-02-4129
                     Project Officer
                    Ronald K. Patterson
                       Prepared by
           Research and Evaluation Associates, Inc.
                1030 15th Street, N.W., Suite 750
                   Washington, D.C. 20005
                      (202) 842-2200
                727 Eastowne Drive, Suite 200A
                   Chapel Hill, N.C. 27514
                      (919) 493-1661

-------
                           TABLE OF CONTENTS

CHARGE TO REVIEW PANEL  	   2
GENERAL OVERVIEW  	   3
PROGRAM ASSESSMENT  	   6
          Boundary Layer/Turbulence Research  	   7
          Regional Oxidant Modeling 	   8
          Regional Particulate Modeling 	  11
          Regional Acid Deposition Modeling Area 	  12
          Complex Terrain Dispersion Modeling  	  14
          Wake Effects Research  	  15
          Technology Transfer/Application  	  16
APPENDIX A:  Peer Review Panel    	18
APPENDIX B:  Agenda for the  Peer  Review of  the In-House
             Research by the Meteorology Division    	  20
APPENDIX C:  Comments on Model  Evaluation    	  26
APPENDIX D:  ASRL Process Evaluation for Meteorology
             Division In-House  Research Peer  Review    	  29
APPENDIX E:  Results of the  EPA Participant Survey
             Meteorology Division 	  33
APPENDIX F:  ASRL Staff Response  to Reviewers' Comments    	  37
APPENDIX G:  The  Laboratory  Director's  Review
             Comments on the Panel Report  and the ASRL
             Staff  Response	49

-------
                           EXECUTIVE SUMMARY

    The Review Panel has reached  a  consensus  on the following findings
and recommendations:
    •    Leadership of the Division is highly effective
    t    Scientific staff is competent
    •    Scientific morale is good
    f    Experimental fluid dynamics is excellent
    •    Computational  quality control of model development is
         exemplary
    •    Computer resources are inadequate
    •    External pressures are forcing premature release of
         preliminary scientific findings
    •    In-house statistical expertise should be strengthened to
         support model  evaluation
    t    A program for advanced research model implementation is
         strongly endorsed

-------
                               SECTION 1

                              INTRODUCTION

    This is a report from the Peer Review Panel (see Appendix A for
list of panel members) organized to review the in-house research
activities of the Meteorology Division (MD) of the US Environmental
Protection Agency (EPA), Atmospheric Sciences Research Laboratory
(ASRL).  Prior to meeting with the MD staff, the panel received three
volumes of selected research papers by the MD staff for preliminary
review.  On September 15-18, 1987, the panel met at the Research
Triangle Park, NC, to hear 45 presentations by the MD staff on in-house
activities.  Presentations focused on individual scientific
contributions to EPA programs rather than a review of activity as
project officers or monitors of extramural contracts.
    The panel appreciates the efforts of the MD Director and individual
staff members in preparing well-organized preview material and
informative presentations.  In general, the quality and organization
were equivalent to that of a scientific professional meeting.  In
organizing this report, the panel chose to follow the general
scientific areas that were used to organize the presentations (see
meeting agenda in Appendix B).
    Finally, the panel is grateful to Charlotte Coley (Research and
Evaluation Associates) and Ron Patterson (ASRL Peer Review Coordinator)
for assistance with the logistics in conducting the review and
preparing this report.

-------
                               SECTION 2

                         CHARGE TO  REVIEW  PANEL

    The  panel  was  provided  with  the  Division  Director's  preview
materials  outlining  the  mission  of the  Division's program.   Principles
to  guide  the peer  review  process  were  stated  in  the Research  and
Evaluation Associates, Inc. document Guide for Participants in the EPA-
ASRL  Task/Project  Peer  Review  Process  provided  by  Charlotte  Coley.
Further  discussions  with the MD Director at  the  beginning  of the site
visit helped  to  clarify  the charge to the review panel.  Evaluation of
in-house research and support activities given in this report are based
on the following program objectives:
    t    Basic Research
    •    Model Development
    •    Applications and Evaluations
    The  panel has  strived  to evaluate the  research presented without
the benefit of some  important information such as CVs  that describe and
quantify total  up-to-date professional  activities.   Although the panel
was provided  with detailed individual workplans, insufficient time was
available  to  ingest  all  this  information.
    The  panel attempted to evaluate the in-house  activity, realizing
that  a  sizeable  portion of  the  comprehensive  program effort resides
within  the extramural community.    Further,  the panel  has  taken into
consideration that  MD program activities (although residing within the
ORD) must  respond to regulatory  needs.

-------
                               SECTION 3

                           GENERAL OVERVIEW

    The  MD programs  are conducted  in a  favorable in-house  research
support  environment.    Generally,  the scientific  staff  seems  to  be
enthusiastically  engaged  in  their  efforts,  and  morale  within  the
Division  seems  good.    Administrative   leadership  is  effective  in
promoting this favorable professional  environment.  MD  programs  are an
important part of both national and international  research efforts, and
the reputation  of the staff  brings  respect  and requests  for  applying
in-house scientific expertise to help address the needs  of EPA's  policy
analysis.
    The  experimental   fluid   dynamics is  excellent  and  goes  far  in
advancing the MD program objectives.  The computational  quality control
of  model  development  is  exemplary and  in  itself represents  an  ideal
model for QC assessment that could be transferred outside the Division.
    However, computer resources are  inadequate  for model development,
applications,  and  evaluation.    Computer  cycles  seem  to be  highly
variable and inadequate,  and thus introduce  some degree of uncertainty
in  carrying out the MD  programs.    Small   computers   are  augmenting
computing needs,  but  a total  spectrum of computing support is required
(from small work stations  that stand alone or access other machines to
supercomputer   model   simulations  for   numerically   intensive   model
development and  testing).
    In-house  statistical  expertise  should  be strengthened  to support
model   evaluations.      The   activities   known   variously  as  "model
evaluation"   and   "model   validation"   deserve   more   attention  by
environmental  researchers,  in general, and thus by  the MD in its role
as  a  leader of  environmental  research.   There  seem  to be essentially
two purposes of  these activities:   (1) to better understand atmospheric
processes,  and thus  to  improve  atmospheric  models,  by  identifying and
diagnosing  failures  in model  performance,   and  (2)  to  quantify the
reliability of models that  are used either to decide regulatory issues

-------
or to guide regulatory policy.  (See Comments on Model  Evaluation given
in Appendix C.)
    The  Meteorology  Division seems  to  be  a  frequent  recipient  of
requests  from  other  offices  and  laboratories  within  the  EPA  that
require  redirection  of  program  FTEs.    The  Division appears  to  be
responding to  these requests,  but at the cost  of delaying  longer term
research needed by the Agency.  In some instances (e.g. the  requests to
run the  ROM simulations  for  EPA  constituents),  scientific expertise is
steered away from model development to application runs.   Some of these
"quick turnaround"  demands for staff time seem to be  appropriate uses
of  expertise  within  the MD;  however,  the balance  of short-term  and
long-term activities should be defined more explicitly.
    One  of  the major challenges  to  ORD Management would  seem  to  be
finding  and  preserving  a balance of  staff  time allocated  to  quick
response needs  of the Agency on the  one  hand  and longer-term needs on
the other.  Although  a good  balance  seems to have been achieved by the
MD,  it  also  seems  to  the panel  that ORD Management in general  has  a
responsibility  to  both the  scientific staff and  to  the Agency to take
steps to guard  against the erosion of the Agency's research  capability.
    A  recommendation of  the panel with  respect  to this  issue  is  to
broaden  the   criteria  used  to   evaluate   the  achievements  of  the
scientific staff.    In general, the staff members are  doing an  excellent
job  of   communicating  with  their  peers  within  their   respective
disciplines through  publications  and  presentations.  But  communication
across disciplines  (horizontal communication) should  be emphasized, for
in   the   relatively   young  field  of   environmental    research,
interdisciplinary  communication  is   less well  established,   although
extremely important.
    Similarly,  more of  the  staff  should  be encouraged to  familiarize
themselves with the Agency's use  of their research  to help set policy
(vertical communication).   This will help  to  ensure  the relevance  of
individual  research  within   the   MD.     More   importantly,   it  should
engender a vision  and a  coherent, internally developed agenda  for the
MD's research.   Even though  the  MD's tasks are  ultimately determined
outside the Division,  such  an agenda would  be an important  reference

-------
for establishing the  MD's  research  priorities.   A more  active  role  by
the MD  in  setting  research priorities would help to  guard  against  the
erosion  of  the MD's  research  capability  by  an  overload  of  quick
response tasks.
    Finally,  the  emerging  program  concept  of Advanced  Research  Model
Implementation  is  strongly endorsed  by the  panel.   Several  existing
modeling efforts  form a nucleus  for this  initiative.   An  additional
consideration  should  be  given to nesting  some  of these models  inside
the  global   climate  models that  presently exist  within the  research
community.

-------
                               SECTION 4

                           PROGRAM ASSESSMENT

To facilitate  the achievement of  consensus,  the committee  adopted  an
internal  evaluation  procedure whereby  each  panel  member  assigned  the
value of  5  (Excellent),  4  (Very  Good),  3  (Good),  2  (Fair), or 1 (Poor)
for each of the following performance criteria:
         •    Quality of Science/Work
         •    Relevance to MD/EPA Missions
         •    Interaction with In-House Colleagues and Outside
              Community

    Emphasis was  given to scientific  accomplishment,  such as reviewed
journal   publications    and   conference/workshop    publications   and
presentations.   Professional  service  to  EPA  and extramural  activities
were  also considered.   In some  instances, scientific  support  was  the
focus of  work  activity rather than basic research or model development
and testing.   Recognition was also  given to  the importance of applied
research.   Although  the scores are not provided in  this report,  the
panel directed  itself to write program evaluations that were consistent
with  composite numerical  scores  assigned to  the performance criteria.
The   panel   was  pleased  with   the  consistency  of  the   individual
evaluations.   The  reviews given  below reflect more  directly  on some
presentations  than  others.  Some individual comments are  provided, but
not  in  detail  for  each  individual  presenter.   Absence  of detailed
technical  comments  should  not  be construed  as  positive or  negative.

-------
Boundary Laver/Turbulence Research

    The ASRL  is  actively involved in  the  following areas of  boundary
layer and turbulence research:

    1.   Experimental investigation of atmospheric boundary layer
         turbulence,
    2.   Innovative dispersion model  development,
    3.   Evaluation and assessment of dispersion models.

    The committee  evaluated  these areas  of research in terms  of their
quality,  relevance to  EPA  goals, and  interaction of the responsible
scientists with the outside scientific communities.
    Experimental boundary  layer research  includes  turbulent  diffusion
experiments inside a  convective  boundary layer  and urban  and  non-urban
boundary layers.   The quality  of research  in  these areas  is excellent.
This  research is  very  relevant  to EPA objectives  and the responsible
scientists have excellent interactions with outside communities.
    New  modeling  approaches  are being investigated  using probability
density  functions  of vertical velocity  in convective boundary layers.
For buoyant plumes, semi-empirical models  have  been developed.  Again,
the quality of research in these areas is  excellent, the work  is highly
relevant,  and there is very good interaction with outside communities.
Future  work  in  this area should  emphasize  better characterization of
buoyant  plume dispersion inside  a convective boundary  layer  since the
probability density function approach  is not expected to work well. The
quality of work  in the  application of models is very good, and the work
is  highly relevant and  interactive with the  outside  community.  It is
good  to  see  the  efforts directed  towards definition of atmospheric
stability  with  proper  boundary  layer velocity  and length  scales.   The
sensitivity analyses of the models could  be  improved by  using a joint
probability density function of the variables rather than varying them
independently.
    The   rotary  spectral   density  application  in  the  analyses  of
turbulence data  is not  obviously relevant  to EPA  objectives even though

-------
the  quality of  the work  is  good.    Such work  may  not  enhance  our
understanding  of   boundary   layer  turbulence  significantly.     The
interaction with the outside community is not  very apparent.
    Similarly,  the  analyses  of turbulence data using  superposition  of
autocorrelation functions  is not  very  relevant.    The quality of  the
work is  fair and interaction with  outside communities not very  good.
Eulerian  autocorrelation  functions  do  not  determine  diffusion  in  a
turbulent   flow.     Relating  Eulerian   autocorrelation  functions   to
Lagrangian  autocorrelation  functions  is not simple and unique.   Also,
"true"  Eulerian  autocorrelation  functions in  the atmosphere  exhibit
oscillation about the zero line thus making the estimate of time scales
questionable.

Regional Oxidant Modeling

    This  effort  within  the Division epitomizes  the  full  spectrum of
program  objectives   ranging  from modeling development to applications
and  evaluation.    The  nucleus of  staff  spearheading this  effort  is
highly  competent and  is  attempting to  entrain  appropriate activities
throughout  the division.   It seems that external  pressures (outside of
ASRL/MD)  are responsible in  part  for this program  initiative.   These
pressures  also require  model results  that  are difficult  to achieve.
Imposed  resolutions  in the Regional Oxidant Model  (ROM), as required to
be  practical,  are  inconsistent  with  routine  meteorological  data sets.
For example, upper  air weather observations are taken  every 12 hours on
a  mean  spatial scale  of  325 km.  Model  integrations  are  for one hour
over  an  18x18 km grid.   The  ROM  appears to  be a zero-order model in
atmospheric dynamics  (at  best),   whereas the   chemistry  of  the  28
reactive  species  is treated in a  more  sophisticated  manner.   Further,
the raw  input  data  of emissions  are rather discretized and irregularly
available in space  and  time.  The  ROM is a pure diagnostic model, i.e.
output concentrations are used (after the fact) to evaluate the effects
of  alternative  strategies  for  managing  air  quality  planning  and
standards emission.  It is not realistic to impose point concentrations
(observations  and/or standards) on  regional  model  plume dispersions.
                                   8

-------
Skill scores and/or performance evaluation may objectively rate the ROM
concentrations  satisfactorily  when instances  of point  concentrations
actually  fail  miserably.     New  methods  of   evaluation  should  be
considered that weigh both spatial  and temporal  scales or features.
    If  the  ROM is  labeled  as a zero-order modeling effort,  then its
performance will  be properly evaluated.   More  support of  this  effort
should  be  provided.    Also,   apparently  none  of  the  field  programs
to-date have provided high resolution  (space  and time)  rawinsonde data
sets.   Since  a diagnostic model  approach  is  acceptable,  more emphasis
should be placed on collecting a more representative data set for model
development and performance evaluation.
    ROM development apparently needs more computing cycles.  Algorithms
for  the model  are not fully understood  (for  example,  incorporation of
the  effect  of clouds and radiative  processes).  The  ROM model  should
interact  more  with  numerical  weather  prediction  model  efforts.   Can
some nested grid  simulations  and use  of trajectory model  forecasts (by
NOAA-NWS)  be  considered?   Trajectory models  combined  with  chemical
reactions code may yield  some  useful results.
     In  several  instances the  scientific  capability  of  the  involved
scientists  exceeds  the  scientific  credibility of ROM.   For example, it
has  been  hypothesized that  receptor  sites can be grouped into classes,
such that concentrations averaged over all sites  in  a  given receptor
class at  a  given  hour is a quantity that is approximately the same for
all  realizations  of the concentration ensemble.  If this is true, then
with  respect  to  the receptor class  average  (RCA),  the concentration
field is  a quasi-deterministic variable.  Therefore, model simulations
of   single   realizations  can  be  interpreted  in  the  conventional
deterministic  manner.    Although  this  hypothesis  has  not  yet been
tested, it  is  assumed for the present purposes that it is correct.  It
is  unscientific to proceed  in this  way,  though  it may be politically
and  practically sound.
     Biogenic   emissions   inventory  development  is  essential  for the
application of ROM.   Currently,  the  work involves translation of  known
emission  information to  an  hourly emissions  ROM grid.   This requires
allocation  of  land  use  and  biomass  information  to  grid   cells,

-------
derivation of  relevant  biomass volumes, and  identification  of species
emission factors.   This approach appears appropriate for  the  intended
use  and  necessarily  involves  extrapolation  and  disaggregation  of
existing information.   The research  has involved contact with  forest
and crop researchers during the development of information  sources, but
has not  resulted in  presentation  of results  to  that community.   The
latter is  recommended as a complement  to  existing reports  to the air
pollution  modeling  community.   Uncertainty estimates were  presented,
but seemed questionable.
    Work  has  focused on  ozone dry  deposition  over large  areas  as
determined  from aircraft measurements.   Data  were  collected through
extramural effort,  but  data analysis has occurred  in-house.   Aircraft
measurements suggest  that ozone concentrations may be highly correlated
to  organized  convection.    Ozone  appears  to  be  a good  tracer  of
convection,  similar  to  moisture.    Detrended  temperature  should  be
spectrally decomposed to  show the  basic convective mode  (BCM)  or cloud
street mode.   An approximate 4 km length scale  is  evident  in the raw
data.   The ozone spectrum  presented  indicates  a  3.8 km spectral peak,
which accords  well  with temperature data.   What about the spectrum for
the specific  humidity q?  Also, several papers exist in the literature
that  show  larger length scales of mesoscale convective organization in
a  convective  boundary  layer, known as  higher  convective  modes (HCM).
The typical  aspect  ratio  for  the BCM  is  3 to  1, compared to  an order of
magnitude  increase  for  the  larger  HCMs.    Inspection  of  the  raw
concentration  for ozone indicates  a  signal  on  the larger length  scale
that  is  in phase with  temperature  trace.   An effort should be made to
complete  the  analysis  of  these unique data  sets  by interpretation of
convective  structures (for all fields)  and treating ozone as  a tracer.
This  has some  exciting  possibilities.
    The  development  and  implementation  of  verification  procedures for
the ROM  will  significantly enhance the  current and future use of ROM.
Current  computer   science  concepts  have  been  integrated  into   EPA's
atmospheric  modeling community.  This  integration  not  only results in
products  of known  quality, but also  appears to be  accomplished  as  a
natural  part of the  model  development.   There appears to be  an active
                                   10

-------
effort  to publish  this  approach within  the  air pollution  modeling
community.  Although not directly presented during this peer review, it
is  known  that this  experience  has contributed  to the  formulation of
quality assurance procedures for the RADM model evaluation program.
    The  presence of the  Data   Systems  Analysis  Branch staffed  with
knowledgeable  application  computer  scientists  is  one  key  to  the
Meteorology Division's  ability to  fulfill  their mission.  Its presence
allows  atmospheric   scientists  and computer  scientists  to  contribute
their  skills  to an interdisciplinary  effort  of model  development,
evaluation, and application.
    Very   systematic  and  methodical  quality   control   analysis  is
performed  with state  of the art  practices,   from  modular  programming
down  to  file naming  conventions.   Professional  systems  programming
architecture  could  serve as a model  for  other similar unrelated model
developments.    Feedback  between the  physicist(s)  behind  the  model
development  and  programming quality  control  people was a little vague.
Scientists that  develop  the   model  know  the   inherent  weaknesses.
Quality  control  should  generate,  independently,  a  similar  set of
weaknesses.    The need for more  computing cycles  is  not  abundantly
clear,  but probably is  the case.   Funds should not  be  used  to  upgrade
antiquated computing equipment.   What about  vectorizing the code for
the ROM?  Also,  how much do QC  checks slow down the  ROM simulation?   A
more  coordinated  plan  is  apparently  needed   for  providing  computing
facilities to ASRL/MD or even to  all of  EPA at Research Triangle  Park.
Can metacode be  saved,  upon which variations  of model  predictions can
be  made?  Although  briefly  discussed, the ROM  may be  improved more by  a
parallel  implementation than by  a vector implementation. Also,  can EPA
use the supercomputer resources  at the five national  centers?

Regional  Particulate Modeling

     The Regional Particulate Model (RPM) program uses  ROM  and has some
of  the same  problems  of weak scientific support and  inadequate  input
data,   especially  emissions.   The work on  different  aspects  of the
model,  presented by four of the  scientific  staff,  showed evidence  of
                                   11

-------
industrious efforts to provide what is needed with inadequate resources
of manpower  and  computing facilities, resulting  in  contributions  that
were incomplete.
    The cloud  process module work may  evolve into  some  good science.
However, this  component of both the  ROM and the RPM  (or any regional
scale model)  is  not  getting  the  attention it merits.   The simplistic
approach presented  is a reasonable application  of the state-of-the-art
knowledge for simple  cloud model formulation (primarily a thermodynamic
emphasis).   Effort needs  to be extended  to properly treat  the total
entrainment process between the cloud top and the overlying interfacial
layer.   Cloud ensemble  scale should  be considered  in  accordance  with
the ROM grid length.
    The  TTNEPH data  may be  all   right  for radiation studies  but its
value  in  the wet  deposition  and cloud  processing  modules  is  very
questionable.  Efforts should be  made to incorporate the GOES  imagery,
which  is very useful for the  ROM domain.   The work  on this  project
epitomizes  the vitality offered by young new  staff members.
    The  MESOPUFFII  modeling effort is probably important in  principle,
but  (very  good)  performance  evaluation shows  poor performance.   The
CAPTEX  data  set  appears adequate  to test performance.   Best  results
were  obtained  for  cases of no  directional  shear.   Work  should  be
published  in reviewed journals (not just an EPA  report).
    The  evaluations  of the  Regional  Lagrangian  Model of Air Pollution
RELMAP  has  served  to  expose  NAPAP  emission  inventory deficiencies.
Thoroughness  and   comprehensive  knowledge   were   evident   in  the
presentation.

Regional Acid Deposition Modeling  Area

    MD  in-house  research activities related to regional  acid  deposition
modeling involve the International  Sulfur Deposition Model  Evaluation
 (ISDME)  project, Regional Acid Deposition Model (RADM)  evaluation and
assessment  projects,  and  cumulus cloud  venting  based on  data  from
VENTEX,  CUVENT,  NEROS, and  NASA   Langley  field  experiments.   The
in-house activities  reviewed comprise  only  a  fraction of the project
                                   12

-------
activities.  Hence no attempt  is made  to  review either past or planned
field experiments  or the entire RADM  model  evaluation effort.   Since
the United States portion of the ISDME was  completed within the MD,  it
will be more fully reviewed.
    The  ISDME  work  has  placed the MD staff at  the forefront  of  the
regional model  evaluation scientific community.   The   ISDME   Project
evaluated  eleven   linear-chemistry   atmospheric   models  of   sulfur
deposition using 1980 observational data.   This  effort is noteworthy as
being the  first organized  effort  at  regional  model  evaluation.   Not
only has the work  of MD staff contributed  to an  increase in  knowledge
on model performance, but  also to  new  approaches to  model  evaluation.
Although  evaluation  methods  for   regional  scale  models  which  are
acceptable to  all  scientists  have not been formally  established,  the
ISDME   scientists  have  employed   a   careful   system  of  statistical
procedures to  arrive at their conclusions.    The  effort to  rank  the
models'  performance  does not  appear to aid the choice  between  models
for  policy use  and  requires  further  research to  be  usable in  that
context.   In addition to the statistical  evaluation, additional  effort
is  needed  to  outline conditions under  which the  models  are or are not
applicable.   For example,  the simple  Lagrangian models  show  the same
performance  repeatedly  over  several  years when  annual  averages  are
compared.  The  ISDME analysis  would be more useful  if it had led to an
understanding of why the models  show this type  of performance.  Use of
metrics   based  on   spatial   pattern   is   important,   but   requires
incorporation of spatial surface  interpolation.  Further research into
the  applicabilities  of  kriging to  surface concentration and deposition
fields  is  recommended.
     Current  in-house research  focuses on  design studies  for surface
network,  intra-spacing  network  and  emissions  data  collection.   The
design  work relies  heavily on  kriging but does  provide  a  basis  for
quantitative  design   studies  that  focus  on minimizing  the evaluation
"error".    The  MD  staff  recognizes   that kriging  requires  further
evaluations as a viable  design and model evaluation tool and that other
evaluation methods should be explored.
                                   13

-------
    The  cloud  venting  research focuses  on the  study of  penetrative
convection, which leads to venting of the  planetary  boundary layer and
transport  of pollutants.   The  results of this  work have  important
implications  for  both  the  ROM  and  RADM  programs  and  should  be
continued.  The researcher is encouraged to establish stronger research
relationships with the  cloud dynamics community.   The field experiment
data  are  unique  and  deserve  more  detailed  documentation  of  the
meteorology  surrounding the events.   The further  classification  and
determination  of   6   factors   for   several  types   of   cloud-topped
(fractionally) planetary boundary layer is recommended.
    Overall, the MD staff involved in regional  acid deposition modeling
are (1)  performing research that  is  very  relevant to EPA  programs, (2)
actively  interacting  with the  relevant  scientific  community,  and (3)
completing  research  that  is   worthy of   publication  (and  is  being
published) in the peer-reviewed literature.

Complex Terrain Dispersion Modeling

    Nearly  30%  of  the  presentations  were  associated with  in-house
efforts  to  support  the Complex  Terrain  Dispersion  Modeling  (CTDM)
program.   The panel  was unanimous  in  its  positive  evaluation  of the
quality  of  the  experimental   modeling  activity.    The   ability  and
willingness  of  the fluid modeling team to respond to the demands of a
regulatory  agency  and still  be able  to carry out  basic research speaks
well for both the  scientists and division  management.
    Since  the  panel   was  not  briefed on the complete CTDM program, it
was unable to review the in-house research within  the context of the
whole project.  However,  the following specific comments can be made:
    1)   Good,  interesting  results are obtained  in  the fluid modeling
laboratory  and  should  continue  to be published  in  JFM type journals.
The panel, however,  questions whether the laboratory experiments are
sufficiently  reliable  simulations of atmospheric flows to be  used to
set regulatory guidelines.
    2)    Inverted  tows  in stratified cases are adequate, however a deep
sheared  flow layer is  not present (as  in  the neutral wind tunnel case
                                   14

-------
studies).    Experimental  fluid  mechanics   is   done   well,   both  in
laboratory  techniques   (flow  visualization   and  measurement)   and
application of  theory.   How these  results are factored  into  the CTDM
was not explained well.
    3)   Relevance  of model  and laboratory  results  (even with  good
agreement)  do   not  necessarily  apply  to atmospheric  boundary  layer
flows.   Stably  stratified  flows are  useful, but  do  not  address  the
convective  planetary  boundary  layers.    Dynamic  similarity  between
numerical  model/laboratory  flow   and  atmospheric  analog   was  not
established.     It  appears  that   a   constant   eddy  viscosity  value
arbitrarily  replaces   the  molecular viscosity  in  the  model  momentum
equation.   In  reality, isotropic Fickian  diffusion  cannot  be  assumed.
Further,  the  spatial  dependencies  in  nonlinear  systems of  equations
give rise to non-Gaussian dispersion.   Again,  the fluid mechanics work
is excellent but its  extension to  actual  atmospheric dispersion is not
clearly explained.

Wake Effects Research

    Although  driven   by  regulatory   requirements,  the  wake  effects
research   can    be   considered  quite  basic.      The   work   on  the
characterization  of  the   complex  turbulence  in  the   wake  of   a
non-aerodynamic structure  is a good example.   The need for an  improved
plume  formulation is  obvious and common to much of the theoretical and
physical  modeling  efforts  at  MD.   Efforts  to  obtain  better flow
visualization and,  in  particular, to quantify those observations  are to
be  commended.    Illumination  techniques and  the statistical evaluation
of the resulting data  need to  be improved.  New plume formulations call
for   a  departure  from  the   traditional   Gaussian  approach.    Any
fundamental advances  in  this area  would have an impact not only  on EPA
activities but  also on the boundary layer  turbulence research community
in general.
    The  work on auto  exhaust dispersion  is clearly  associated with
EPA's  need  to  model  the   impact  of  a  major  source  of  atmospheric
pollution.   The experimental  results  presented  during  the review were
                                   15

-------
based upon only a part of  a  more  comprehensive  program of experiments.
It is difficult  therefore to judge  what  should or should not  be  done
since the work described was very incomplete in  itself.

Technology Transfer/Application

    This set  of  presentations,  along with  two  presented as a  part  of
Boundary  Layer/Turbulence  Research,  seemed  to  be  a  representative
selection of work by the Environmental  Applications Branch (EAB).   That
is, the presentations were consistent the  function of the EAB.
    The panel regards the UNAMAP to be of  very good quality and  to  be a
most important function  of the MD.  Indeed, UNAMAP  is  one of the  most
important  and visible  aspects of the EPA's  air quality  operations,
since regulatory  applications  (e.g.,  obtaining  permits,  establishing
emission limits, or  demonstrating  attainment  of air quality  standards)
typically require the use of a UNAMAP model.
    The  estimation  of  the reduction of  soybean crops  resulting  from
various  levels of ozone  was  presented.  This  analysis was requested by
the  Office of Air Quality Planning  and Standards.   Current levels  of
ozone  at  320 farms  growing  soybeans  were  related  to existing  and
proposed national ambient  air quality  standards (NAAQS) for  ozone.  An
important  finding  from this  research was  that,  contrary to  greenhouse
studies,  field  studies  of  the   effect  of  ozone  on  soybean  growth
suggested  that two parameters of soybean exposure to ozone (namely, the
duration and  the concentration  of the  exposure) could be summarized by
a  single index (effective mean ozone concentration).   This  work seems
to  the  panel to  be  highly  responsive  to  the needs  of  EPA policy
analysis,  and like previous  work at  the MD on the relationship between
concentration averaging  times, may  be  widely used and  cited.   Due to
the  importance of this  work,  it deserves  further investigation  and the
collaboration of  other experts  in meteorology,  atmospheric  chemistry,
plant physiology, and statistical  inference.
    The  panel rated  as  very important the investigation of alternative
techniques for randomly sampling data to efficiently estimate long-term
(seasonal  or annual)   concentrations  using  the  Point,  Area,   and
                                   16

-------
Line-Source (PAL) algorithm.  This work addresses a practical need  for
computational   efficiency,   but,   more   significantly,  explores
probabilistic modes of air quality analysis.  This work  should continue
and should  be supported  by a  statistician  who  is  an  expert  in  the
techniques and inferential issues of data  sampling schemes.
    Finally,  a  preliminary diagnosis of  the Turbulent  Profile  Sigmas
(TUPOS)  dispersion model  based  on  a  comparison  of  calculated  and
measured SFs  concentrations obtained from the EPRI's Kincaid data  base
was presented.  The investigation will help  to determine the degree to
which  more  detailed turbulence  data  improves  model  performance  under
field  conditions  and  is  therefore  regarded  as  extremely important  by
the panel.   As  with  other model  diagnoses,  sensitivity studies,  and
reliability studies conducted  by the Division,  the statistical  issues
are quite challenging (see Section 3 and  Appendix  C of  this  review)  and
deserve the support of an expert statistician.
                                   17

-------
   APPENDIX A
PEER REVIEW PANEL

-------
      EPA-ASRL Peer Review Panel
         Meteorology Division
Research Triangle Park, North Carolina
         Septenber 15-18, 1987
         Ernest M. Agee
         Chairman, Dept. of Earth and
           Atmospheric Sciences
         Purdue University
         West Lafayette, IN  47907
          (317) 494-3282
         Raymond J. Deland
         Raymond Deland & Associates
         2 Collyer Drive
         Ossining, N3f  10562
          (914) 941-5622
         David Etnmitt
         Simpson Weather Associates, Inc.
         809 E. Jefferson Street
         Charlottesville, VA  22902
          (804) 979-3571
         P.K. Misra
         Ministry of  the Environment
         125 Resources Road
         East Wing
         Rexdale, Ontario
         Canada M9W  5L1
          (416)  235-5771
         Anthony R. Olsen
         Battelle Pacific Northwest Program
         P.O. Box 999
         Richland, WA  99352
          (509) 376-4265
         Anthony D. Thrall
         Environment Risk  Assesntent Program
         Electric Power Research Institute
         3412  Eillville Avenue
         Palo  Alto, CA 94304
          (415) 855-2627
                 19

-------
           APPENDIX B



AGENDA  FOR EPA-ASRL  PEER REVIEW OF



METEOROLOGY AND ASSESSMENT DIVISION



       SEPTEMBER 15-18,  1987

-------
                                  AGENDA

                        Peer Review of the EPA-ASKL
                    Meteorology and Assessment Division
                         In-Hbuse Research Program
                           September 15-18, 1987
TIME                          TCPIC                      SPlMiK          REF.f


Tuesday. Sept. 15    Classroom 3, Environmental
                     Research Center Research Triangle
                     Park, North Carolina
                                        i
08:00 - 08:20 AM    Opening Session - Panel Welcome    Charlotte Coley

                     Peer Review Program Orientation

                     ASRL Peer Review Coordinator       Ron Patterson

08:20 - 08:40 AM    Executive Session in Meeting       Al Ellison
                     Room for Panel and Staff, ASRL
                     Director

08:40 - 09:00 AM    Coffee Break and Introductions of
                     Review Panel and ASRL Staff

09:00 - 09:10 AM    Introduction by MD Director        Frank Schiermeier   1

                     Boundary Laver/Itarbulence Research

09:10 - 09:40 AM    Analysis and Parameterization      Gary Briggs         2
                     of Convective Diffusion
                     Processes

09:40 - 10:00 AM    Evaluation of Convective Scaling   Tom Pierce          3
                     for Estimating Diffusion

10:00 - 10:30 AM    Boundary Layer Turbulence Studies  Jason Ching         4
                     in Urban and Non-Urban Areas

10:30 - 11:00 AM    Rotary  Spectral Analysis of        Peter Finkelstein   5
                     Turbulence Measurements

11:00 - 11:30 AM    Exponential-Sum-Fitting Tech-      Steve Perry         6
                     niques  for On-Site Turbulence
                     Analysis

11:30 - 12:30 AM    Lunch

12:30 - 01:00 PM    Executive Session for Review Panel

01:00 - 01:20 PM    Meteorological Scaling  in Applied  John Irwin          7
                     Dispersion Modeling

                                21

-------
01:20 - 01:35 EM
01:35 - 02:20 PM

02:20 - 02:40 EM



02:40 - 03:00 EM

03:00 - 03:30 EM


03:30 - 04:00 EM


04:00 - 04:30 EM


04:30 - 05:00 EM


Wednesday.  Sept.  16



08:00 - 08:20 AM

08:20 - 08:40 AM


08:40 - 08:50 AM


08:50 - 09:15 AM


09:15 - 09:40 AM
 09:40 - 10:00 AM
Relation of Error Bounds of
Maxmium Concentration Estimates
to Meteorology Uncertainty

Regional Oxidant Modeling

Regional Oxidant Model (BCM)

Biogenic Emissions Inventory
Development for Regional Oxidant
Modeling

Break

Ozone Dry Deposition Over Large
Areas from Aircraft Measurements

Verification Procedures Applied
to the Regional Oxidant Model

Evaluation of Regional Oxidant
Model

Executive Session for Raview
Eanel and MD Division Director
                                  John Irwin
                                  Bob Lanb

                                  Jim Reagan
                                  Jim Godowitch


                                  Joan Novak


                                   Ken Schere


                                  Frank Schiermeier
         Particulate Modeling

Regional Particulate Model (EPM)    John Clarke

doud Processes Module for         Frank Binkowski
Regional Particulate Modeling

Utilization of X^NEXU. Data for
Regional Particulate Modeling
                                   Russ Bullock
 Sensitivity Analysis of MESOPDFF   Jim Godowitch
 n and Evaluation with CAETEX Data

 Regional Lagrangian Model of Air   Brian Eder
 Polution (HELMVP) Sensitivity
 Analysis and Evaluation for
 Particulate Matter

 Coffee Break and Introductions  of
 Review Panel and ASKL Presenters
                                    Deposition Modeling
 10:00 - 10:30 AM
International Sulfur Deposition
Model Evaluation (ISDME)
                                    Terry dark
                                                       9

                                                       10
                                                       n

                                                       12

                                                       13
                                                       14

                                                       15


                                                       16


                                                       17


                                                       18
                                                                             19
                                 22

-------
10:30 - 11:00 AM     Regional Acid Deposition Model      Robin Dennis       20
                     (RADM) Evaluation Research Program

11:00 - 11:30 AM     Cumulus Venting of Pollutants      Jason Ching         21
                     from the Mixed Layer

11:30 - 12:30 PM     Lunch

12:30 - 01:00 PM     Executive Session for Review Panel

01:00 - 01:30 PM     Acid Deposition Assessment         Robin Dennis        22
                     Studies

                     Complex Terrain Dispersion Modeling

01:30 - 01:40 PM     Overview of Complex Terrain        Frank Schiermeier   23
                     Research

01:40 - 02:00 PM     Complex Terrain Field Data Bases   Larry Truppi        24

02:00 - 02:20 PM     Fluid Modeling Facility Object-    Bill Snyder         25
                     ivesf Facilities, Approach, and
                     Recent Research Accomplishments

02:20 - 02:40 PM     Deformation of Plumes by Hills     Bill Snyder         26

02:40 - 03:00 PM     Break

03:00 - 03:20 PM     Measurements of Streamline Traj-   Roger Thompson      27
                     ectories in Stratified Flow Over
                     Isolated
03:20 - 03:35 PM     Laboratory Measurements of         Bill Snyder         28
                     Unsteadiness of Strongly Strat-
                     ified Flow Fields CDrag) Over
                     Two-dimensional Hills

03:35 - 03:50 PM     Comparison of Numerical and        Roger Thompson      29
                     Laboratory Experiments on Density-
                     Stratified Flow Over a Hill

03:50 - 04:10 PM     Stratified Flow Over Ridges and    Bob Lawson          30
                     Valleys  flushing of Valleys)

04:10 - 04:30 PM     Laboratory Validation of Flat-     Bill Snyder         31
                     Dividing-Streamline-Surface
                     Approximation for Complex Terrain
                     Dispersion Models

04:30 - 05:00 PM     Executive Session for Review       Frank Schierneier
                     Panel and M5 Division Director

                                 23

-------
Thursday, Sept. 17

08:00 - 08:20 AM     Wind Direction Effects on          Bill Snyder         32
                     Dispersion from Sources Downwind
                     of
08:20 - 08:50 AM     Complex Terrain Amplification      Bob Lawson          33
                     Factors for Various Stack Heights/
                     Locations

08:50 - 09:30 AM     Diffusion from Low Level Sources   Bert Eskridge       34
                     Under Extreme Stratification

09:30 - 09:50 AM     Coffee Break and Introductions of
                     Review Panel and ASRL Presenters

                     Wake Effects Research

09:50 - 10:20 AM     Theoretical Modeling and Eval-     Alan Huber          35
                     uation of Building Wake Effects

10:20 - 10:40 AM     Observations of Surface Flow       Bob Lawson          36
                     Patterns Near Buildings Using
                     Flow Visualization

10:40 - 11:10 AM     Video Image Techniques for Wind    Alan Huber          37
                     Tunnel Measurements of Building
                     Wake Dispersion

11:10 - 11:30 AM     Near-Wake Dispersion of Auto       Roger Thompson      38
                     Exhaust

11:30 - 12:30 PM     Lunch

12:30 - 01:00 PM     Executive Session for Review Panel

                     Technology Transfer Applications

01:00 - 01:30 EM     User's Network for Applied         Bruce Turner        39
                     Modeling of Air Pollution
                      CUNAMAP)

01:30 - 01:50 EM     Development and Application of     Ralph Larsen        40
                     Air Quality Effects Models

01:50 - 02:10 PM     Climatological Version of the      Bill Petersen       41
                     Point Areas, and Line Source
                     Algorithm  CPAL)

02:10 - 02:30 EM     Evaluation of the TUPOS Dispersion  Bruce Turner       42
                     Model - Preliminary Findings

02:30 - 02:50 EM     Break

                                24

-------
02:50 - 03:10 EM     Conparing Air Quality with         Ralph Larsen        43
                     Present or Proposed Eegulatory
                     Standards

03:10 - 03:30 PM     Rise and Dispersion of Merging     Bill Snyder         44
                     Buoyant Plumes in a Stratified
                     Crossflow (Ocean Outfalls)

03:30 - 03:50 PM     Dispersion of Dense Gas Over a     Bill Snyder         45
                     Ramp

03:50 - 04:15 PM     Advanced Model Operation and       Frank Schierneier   46
                     Analysis

04:15 - 04:45 PM     Executive Session for Review Panel  Frank Schienneier
                     and MD Division Director

Friday. Sept. 18

08:30 - 10:00 AM.     Reviewer Debriefing with ASRL      Al Ellison
                     Director

10:10 AM             Report Preparation
                                 25

-------
        APPENDIX C



COMMENTS ON MODEL EVALUATION

-------
                     COMMENTS ON MODEL EVALUATION
    It seems useful  to distinguish the  following  four types  of model
evaluation:
    (1)  model  diagnosis
    (2)  reliability estimation
    (3)  sensitivity analysis
    (4)  uncertainty analysis
    Model  diagnosis   is  the   comparison   of  measurements  to  model
calculations so that inadequate assumptions or formulations embodied in
the model  can  be  identified  and investigated.   The investigation often
leads  to model  improvements, so that model diagnosis  is  typically one
phase  of model development.   Such diagnoses  are  often  informal  data
analyses  used  to generate  new hypotheses.    Of  course,  if  such  an
informal  data  analysis  suggests  a  major  redirection of  research  or
model  development, the diagnosis is worth checking more rigorously.
    Estimating  the reliability  of a model  requires  more  careful  and
thorough  statistical  analyses  than does model diagnosis.   First,  the
use or uses of the  model  for which the reliability analysis is being
conducted  must be carefully defined.   Then a  means of estimating the
model's  reliability  must  be determined.    Finally, the degree to which
the results  of analysis generalize to  other conditions must be  judged.
These  issues  are  just beginning to  be  addressed  by  environmental
researchers  and  require the full  attention of  both statisticians and
atmospheric  scientists.
    Two  modes  of  analysis   intermediate   to model  diagnosis  and
reliability   estimation  are   sensitivity   analysis  and  uncertainty
analysis.    The  goal   of  sensitivity  analysis  is   to  determine  the
influence  of model input variables or model coefficients  on some aspect
of the model output.   Like model diagnosis, the purpose  of sensitivity
analysis   is  to  better understand  the  model,  at   least under  some
conditions.    Such  probing  of  the model  is  sometimes  performed very
roughly  by  perturbing  just one  variable at  a time  to approximately
bound  the  plausible range of ambient concentrations.   A more  thorough
                                   27

-------
analysis of  model  sensitivities  may require the statistical  design  of
simultaneous perturbations of several  variables.
    Uncertainty   analysis   is   the  estimated   attribution  of   the
discrepancies between  measurements  and calculations  to,  respectively,
errors in model input, errors  in ambient measurements,  model  bias,  and
random variation  of  the  atmosphere  for each set of  conditions defined
by the model  input variables.   In  statistical parlance this is  called
variance  components   analysis   (if  characterization  of  conditional
variability   is   restricted   to   conditional  means   and  variances).
Uncertainty  analysis  is  relevant  to model  diagnosis.   For example,  it
can be  used  to determine whether the apparent poor  performance  of the
model   under  certain  conditions  can  be  plausibly   attributed  to
measurement  errors or random  atmospheric  variation  rather  than model
bias.   Uncertainty analysis  also pertains to model  reliability.  That
is,  the amount  of evidence showing  that  the model  "is" or  "is  not"
biased  under certain  conditions, determines the degree  to  which  such
findings  generalize  beyond  the  study  to  other  settings   or  sites.
Uncertainty  analysis  is extremely challenging  since  the  atmosphere
rarely  provides more than one  replicate  of the same set of model input
conditions.   However,  the  use  of  either statistical  data resampling
schemes or wind-tunnel data as a test-bed may yield useful information.
    In  all  four types of "model evaluation",  it is essential that the
evaluation   team  include or  have  access  to  appropriate statistical
expertise.
                                   28

-------
                    APPENDIXD





ASRL PROCESS EVALUATION FOR METEOROLOGY DIVISION



          IN-HOUSE  RESEARCH  PEER REVIEW

-------
               ATMOSPHERIC SCIENCES RESEARCH LABORATORY
              Process Evaluation for Meteorology Division
                     In-House Research Peer Review

    The  Atmospheric  Sciences  Research  Laboratory  (ASRL)   of  the  US
Environmental  Protection Agency convened a  panel  of scientific  experts
on  September  15  -  18,   to  review  the   in-house  research  of  the
Meteorology  and  Assessment  Division.   The  panel   consisted  of  six
scientists.    These  reviewers  were  asked  to  evaluate  the  process
involved  in  preparing  and  implementing this  specific meeting.   Five
panelists completed the process evaluation.
    The  evaluation  instrument  was designed  to  assess  the  following
aspects  of  the  process:    1)  Preview Materials;   2)  Process  and
Logistical Information; and 3)  the  Review  Meeting.   A section was also
provided for reviewers to give their comments and recommendations.  The
reviewers  were  instructed  to respond to  15 items  by circling  numbers
from 1 to  5 (with  1 representing poor; 2-fair; 3-good; 4-very good; and
5-excellent).
    Table  1  presents  a summary  of the reviewers'  rating  for  the 15
items.   No items were rated  poor.  All  items received some  "excellent"
ratings  except for overall peer review process and  adequacy of time for
executive  sessions.    Most   categories  were  either  rated  "good"  or
"excellent".   Three items  were rated fair:  adequacy of time available
to preview, meeting purpose,  and  overall peer review  process.  Thirteen
items  were  rated  excellent:    written  quality,   technical  quality,
utility  for outside  reviewer,  adequacy of time  available to preview,
meeting  purpose, scheduling  of  meeting, time/preparation requirement of
reviewers, timeliness of meeting  notification, timeliness of logistical
information,  adequacy  of  time for discussion with EPA  staff,  quality
and   utility   of  presentations,   quality  and  utility  of  materials
disseminated,  and  support  services  and activities.
                                   30

-------
TABLE 1.  SUMMARY OF PROCESS EVALUATIONS
Review Categories
Preview Materials
1. Written Quality
2. Technical Duality
3. Utility for Outside Reviewer
4. Adequacy of Time Available
To Preview
SUB TOTALS
Process and Logistical Information
5. Meeting Purpose
6. Scheduling of Meeting
7. Reviewer Responsibilities:
Time/Preparation Reauirement
8. Overall Peer Review Process
9. Timeliness of Meeting
Notification
10. Timeliness of Logistical
Information
SUB TOTALS
Review Meeting
11. Adequacy of Time for
Discussion w/EPA Staff
12. Adequacy of Time for
Executive Session
13. Quality and Utility of
Presentations
14. Quality and Utility of
Materials Disseminated
15. Support Services and
Activities
SUB TOTALS
TOTALS
Number of Reviewers
Rating Each Item
Very
Poor Fair Good Good Excellent






















1
1
1


1


2






3


1
1
?
7
1
3
1
1
1
9
3
4
1
1
1
10
21
1
2
1
2
6
1
1
1
3


6

1
1

1
3
15
4
3
3
1
11
1
3
1

4
4
13
2

3
4
3
12
36
                  31

-------
                                              Is."
  Comments:


§   "The organization and support of the peer review by R & E,  the
    Meteorology Division director and his staff is by far the best
    of all  prior  ASRL reviews that  I  have  participated  in.   They
    are to  be commended for  their efforts.  Reviews  of in-house
    research   by   a   Division  are   new  and   require   further
    clarification.   The review  has elements of  personnel  action
    review/appraisal,   university   tenure   review,   departmental
    accreditation,  and  "traditional" EPA program reviews.   It is
    difficult  for peer reviewers to define  their task within  the
    existing framework."

t   "Impossible to examine all preview material:

•   "Entire process is too brutal."

t   "Too many  people  to review with diverse activities."

t   "The purpose  of the review must be  carefully considered.   My
    impression is   that   the  review   was  foisted  on  ASRL  by
    Washington, DC.   To their considerable credit, ASRL management
    shared  with  us  this  perspective  on  the review.   But  I feel
    that the use  to the ASRL  of such reviews can  be better defined
    and expressed to future  panels, after  some  reflection by the
    ASRL management."
Recommendati ons:


•   "It  would have  been  nice to have  some  one-on-one discussion
    with selected  staff."

•   "Up-to-date vita  should  be provided."

0   "EPA MD organizational chart would  have been useful."

t   "A time to  speak  with  key people on a one-to-one basis."

•   "Provide  organizational  chart  to all  panel members in advance
    of  meeting.   Also,  as  part  of putting  the  research  being
    reviewed  in perspective,  it would have been useful to have had
    a  few pages   describing  the Meteorology  Division's  mission,
    long-term  objectives,   and   recent   accomplishments   -   a
    description  at a level  of  detail  somewhere between reference
    item #1 and individual workplans."
                                   32

-------
               APPENDIX E
RESULTS OF THE EPA PARTICIPANT SURVEY
         METEOROLOGY  DIVISION

-------
                 Results of the EPA Participant Survey
                          Meteorology  Division
                      In-House Research Peer Review

         The   Atmospheric   Sciences   Research  Laboratory  of  the  US
Environmental  Protection  Agency convened a panel  of scientific experts
to review  the  in-house research done  by the Meteorology and Assessment
Division.   EPA personnel  involved in the  program review were surveyed
as to  the effectiveness  and  usefulness of the peer  review process  to
them;  and  to  identify problem  areas  that needed  improving.   Thirteen
surveys were completed.
         The evaluation instrument was designed to assess the following
aspects  of the process:   1)   review format  and  logistics;  2)   review
panel;  and 3)   your  assessment  of the peer review process.  A section
was also  provided  for comments and suggestions.  The participants were
instructed to  respond to  10  items  by  circling numbers from  1 to 4  (with
1 representing very  satisfied; 2-satisfied;  3-dissatisfied; and 4-very
dissatisfied).  Table 2 presents a summary of the ratings  for these 10
items.   None  of  the participants were "very dissatisfied"  with any
items,  although nearly all  items  received at least one  "dissatisfied"
rating.   However,  the majority  of the ratings were in  the  "satisfied"
and "very  satisfied"  categories.
                                   34

-------
TABLE 2.  EPA PARTICIPANT SURVEY SUMMARY
          Number of Participants  Ratinq  Each  Item
REVIEW FORMAT &
LOGISTICS
1. Adequacy of time
for group discussion
with reviewers
2. Adequacy of time for
Proiect oresentations
3. Adequacy of time for
individual discussion
with reviewers
SUBTOTALS
REVIEW PANEL
4. Reviewers' familiarity
with oreview materials
5. Reviewers' demonstrated
knowledge of your
oroqram
6. Reviewers' expertise in
the field
7. Selection of reviewers
(Quality of the panel
as a reviewina unit)
SUBTOTALS
YOUR ASSESSMENT OF THE
PEER REVIEW PROCESS
8. Support service &
activities carried out
bv contractor
9. Objectivity & profes-
sionalism
10. Effectiveness as a
mechanism for OA
SUBTOTALS
TOTALS
Very
Satisfied
1
3
2
6
2
2
4
4
12
3
3

6
24
Satisfied
9
7
7
23
10
10
8
8
36
9
9
11
29
88
Dissatisfied
3
3
4
10
1
1
1
1
4


2
2
16
Very
Dissatisfied














            35

-------
 Comments;
•    "The   schedule   was  overly  crowded;   it  should  have  been
     stretched to 5 days, with so many presentations."

t    "Two  of  the  reviewers  seemed  familiar  with  air  pollution
     modeling  and  meteorology; the others  were much  less familiar
     with  this  sub-field  of  meteorology.    Although  they  may  be
     professional  meteorologists,  that  is  no guarantee  they  are
     knowledgable in  all aspects  of the field.

•    "This  review was unusual  in that so many presentations had to
     be compressed  into three days.  However,  it  still  seems that
     most   presenters  had  adequate  time  for  interaction  with
     reviewers  as  long as  they cooperated  by allowing  time  for
     questions."

•    "As  is usually  the case,  there  was  insufficient time allowed
     for the large number  of topics presented."

t    "Very  disappointed  in  not  having  an  opportunity  to  set  up
     slides  before  my presentation because the group  did not break
     for coffee  as  scheduled."

•    "I  find  that   the  handouts   (copies   of visual  aids)  are
     distracting.  The audience  begins leafing  through the material
     instead  of paying attention to the  speaker.    I  believe the
     handouts  should  be done away with."

•    "Did  not  believe  the  process  has  much benefit  for  anyone
     except it provides a defense if attacked.  In other words, we
     can  say  'well,  the peer reviewers like  our  program,  why are
     you attacking  my budget'.   Otherwise,  the effort  is  in  essence
     of no  use."
                                    36

-------
                APPENDIX F



ASRL STAFF RESPONSE TO REVIEWERS' COMMENTS

-------
                UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                         ATMOSPHERIC SCIENCES RESEARCH LABORATORY
                                 RESEARCH TRIANGLE PARK
                                 NORTH CAROLINA 27711
MEMORANDUM

DATE:     January  15,  1988

SUBJECT:  Response  to  Peer  Review Comments  on Meteorology and
          Assessment Division In-House Research  Programs

FROM:     Francis  A. Schiermeier, Director
          Meteorology  and Assessment Division, ASRL

TO:       Ronald K. Patterson
          ASRL  Peer Review  Coordinator

     The  peer  reviewers' report on the Meteorology  and Assessment Division (MD)
in-house research  programs  is both comprehensive and enlightening. We appreciate
their  significant  efforts,  considering that forty-five technical presentations
were directed at the reviewers during the three-day  schedule.

     The report contains comments and recommendations on program  strengths  and
weaknesses,  on the   adequacy of personnel staffing and computer facilities, on
maintaining the critical balance between research and applications, and  on  the
Division's  role   in   establishing  research priorities.  The responses provided
here are addressed  to  critiques, recommendations,  and reviewer questions in  all
of  these  areas.   Responses  are not provided  for  the complimentary statements
contained in the report although they have  been  noted with appreciation.

Comments, pp. 3. 17. and Executive Summary;   Repeated  recommendations  on  the
need to strengthen  in-house statistical expertise to support model evaluations.

Response:   We  agree.   We  are currently recruiting  a Ph.D. statistician for the
Division but our efforts are  hampered by a  temporary NOAA hiring freeze.

Comment. D. 4;  The Meteorology Division seems to be a  frequent  recipient  of
requests  from  other  offices and laboratories within the EPA that require redi-
rection of program  FTE's.  The  Division appears to be  responding  to  these
requests, but at the cost of  delaying longer term research needed by the Agency.
In some instances...scientific expertise is steered  away from model  development
to  application runs.    Some of these "quick, turnaround" demands for staff time
seem to be appropriate  uses of expertise within  the  MD; however, the balance  of
short-term and  long-term activities should  be defined more explicitly.

Response:  We agree with the  ideal of a clearly  defined balance between research
and applications in the Division but we must also realize that the EPA is a reg-
ulatory Agency.  As such, it  is subject to  frequent  crises-oriented mandates for
redirection of  research efforts imposed by  Congress  and the Administration.

                                        38

-------
Comment, pp. 4-5:   One  of  the  major  challenges to ORD management  would  seem to
be finding and preserving  a  balance  of  staff  time allocated  to  quick  response
needs of the Agency on  the one hand  and longer-term needs on the other. Although
a good balance seems to have been achieved  by the MD, it also seems to the panel
that ORD management in  general has a responsibility to both the scientific staff
and to the Agency  to take  steps to guard against the  erosion  of  the  Agency's
research capability...   A  more active role  by the MD is setting research priori-
ties would help  to guard against the erosion  of the MD's research capability  by
an overload of quick response  tasks.

Response;   ORD  management  executes  a series of steps each year in the budget
planning process that includes the ranking  of existing research programs and new
research  initiatives (prepared by Laboratory scientists).  These priorities are
negotiated by research  committees that  are  composed of both ORD  and  regulatory
program  personnel.   Unfortunately,  this  deliberate  planning process is then
subject to the disruptive  redirections  described in the previous  response.   It
is  thus  up to  us to sort out these conflicting priorities to achieve the "good
balance (that) seems to have been achieved  by the MD."

Comment, pp. 4-5:   More importantly, it should engender a vision and a coherent,
internally  developed  agenda  for the MD's  research.  Even though the HD's tasks
are ultimately determined  outside the Division,  such  an  agenda  would  be  an
important reference for establishing the MD's research priorities.

Response:   We   feel  that  we do have  such an ongoing agenda  for the Division's
research, and that this agenda is frequently  used   in  sorting  out  conflicting
priorities when  they arise.

Comment, p. 5:   Finally, the emerging program concept of Advanced Research Model
Implementation  is  strongly endorsed  by the  panel.

Response;  The establishment of the   ASRL  Research  Modeling  Facility  is  our
attempt  to prevent further  erosion of the  Division's research capability by not
allowing scientific expertise  to be  diluted by model applications at the expense
of ongoing model development and evaluation.

Comment. DP. 7-8;    The  rotary  spectral density  application  in  the analyses of
turbulence data  is not obviously relevant to  EPA   objectives  even   though  the
quality  of  the  work  is good.  Such work may  not enhance our understanding of
boundary layer  turbulence significantly.  The interaction  with the outside  com-
munity  is not very apparent.

Response:  ¥e take strong exception to the   implication  of non-relevance.  Dif-
fusion  models,  which are the Division's most important   product,  are   basically
the  application of turbulence theory to practical  problems.   Since  the develop-
ment of turbulence theory (and its mathematical  treatment)  is  far from "solved",
it  is  appropriate for us to spend some of our  effort  in  refining the basis for
diffusion models.   Whether the work will be successful or  not  is, of course,  an
open question as it is with a great deal of true research.  While  it  may not have
been made apparent to  the reviewers, the spectral   research has  been  reviewed
very favorably  by  personnel  of the NOAA's Wave Propagation Laboratory.
                                        39

-------
Comment, p. 8;  Similarly,  the  analyses  of   turbulence  data using superposition
of autocorrelation  functions  is not  very relevant.  The quality of the  work  is
fair  and  interaction with  outside communities not very good.  Eulerian autocor-
relation functions  do not determine  diffusion  in  a  turbulent  flow.   Relating
Eulerian   autocorrelation  functions  to Lagrangian autocorrelation functions is
not simple and unique.   Also  "true"  Eulerian autocorrelation  functions  in  the
atmosphere exhibit  oscillation about the  zero line thus making the estimate of
time scales questionable.

Response:  Concerning the  relationship  between  the  Eulerian  and  Lagrangian
autocorrelation   functions, the interpretation of Eulerian autocorrelation func-
tions as an indication of the turbulent eddy structure  is  certainly  not  new;
Sutton  (1953) mentions this  topic.   Although  the Eulerian autocorrelation func-
tion does  not directly determine turbulent  diffusion, the literature is  replete
with  attempts  to  better estimate and interpret the form of this function; some
examples are Hisra  (1979) and several cited by Moore et al.  (1985).   Much  has
been  published   (for  example, Hanna 1981, 1986; Weber et al. 1982) showing the
usefulness of estimates of  Lagrangian statistics  (such as integral time  scales)
from  Eulerian statistics for applications  in  diffusion modeling.  Regarding the
contact with the  outside community,  the autocorrelation study was presented (and
received favorable  comment) at the AHS Fifth Joint Conference on Applications of
Air Pollution Meteorology (Dahai 1986).

Comment, pp. 8-9:   The Regional Oxidant Model  (ROM) appears  to be  a  zero-order
model   in  atmospheric dynamics (at best)... The ROM is a pure diagnostic model,
i.e., output concentrations are used (after the  fact) to evaluate the effects of
alternative strategies for  managing  air quality planning and standards emission.
ROM development apparently  needs more computing cycles. Algorithms  for the model
are not fully understood (for example, incorporation of the  effect of clouds and
radiative  processes).

Response:  While  the reviewers acknowledge  the purely diagnostic use of the  ROM
for  assessing  emissions control strategies on a  regional basis, there seems to
be some confusion on the nature of  the model itself.  There  are  inferences  to
the  use of better  predictive techniques for the  atmospheric dynamics portion of
the model. The wind fields used in  the ROM are  derived from observational  data
and  physical principles using a new technique described  in  Lamb and Hati  (1987)
that permits explicit treatment of  the  uncertainty   in  describing  atmospheric
motion.    This  method  permits full utilization  of  dynamical principles  in con-
structing  flow  fields for the ROM,  but only a  portion of  the capabilities  of the
technique  have   been  implemented  to date.  In addition,  flow  in the night-time
radiation  inversion is simulated using a submodel   based  on solutions  of  the
shallow water equations.

Therefore,  if   the reference to a  "zero-order" model pertains  to the flow field
description used  in the ROM,  it is  incorrect.   The wind  field specifications are
at  least  "first-order" accurate in their satisfaction of dynamical  constraints.
In addition,  the  ROM incorporated schemes for   describing  vertical  mass   flux,
including  cumulus  cloud fluxes, horizontal transport, deposition,  subgrid scale
chemistry  effects,  etc., that were  designed specifically   to describe  regional
scale   transport   and  chemistry phenomena.  Unfortunately,  there was not  enough
time  in the peer  review to describe the numerous facets of the  ROM  model  in any
detail.

                                        40

-------
Comment, p. 9:   Since  a  diagnostic  model  approach  is acceptable, more emphasis
should be placed on  collecting  a more representative data set for model develop-
ment and performance evaluation.

Response;   We agree.   The  regional ozidant program  has never been funded suffi-
ciently to mount its own field  evaluation study.  However, as a last resort,  we
plan to Join  in  the  field study for the  Regional  Acid Deposition Model (RADH) to
acquire some  appropriate measurement data for  evaluation of the ROM.

Comment, p. 9;   The  ROM model should interact  more with  numerical  weather pre-
diction model efforts.

Response:   Driving  the transport portions  of  air quality models with the output
of numerical  weather prediction models is a relatively new  concept,  now  being
used  by  a few  larger scale models such as the RADM and ADOM for the assessment
of acid deposition.  This approach is incompatible with the basic  premise  con-
cerning the stochastic nature of atmospheric motion  that underlies the method of
Lamb and Hati used by  the ROM.   The integration of four-dimensional data assimi-
lation  into  prognostic wind   models is more suited to our needs and it may be
explored in the  course of implementing our  own wind  analysis procedure.

Comment, p. 9;   ...it  has been  hypothesized that  receptor sites can be  grouped,
such  that concentrations averaged over  all sites in a given receptor class at a
given hour is a  quantity that  is approximately the same for all realizations  of
the  concentration ensemble.  If this is true, then  with respect to the receptor
class average, the concentration field is a quasi-deterministic variable. There-
fore, model simulations of  single realizations can be interpreted in the conven-
tional deterministic manner. Although this hypothesis has not yet been  tested,
it  is  assumed  for  the present purposes that  it  is  correct.  It is unscientific
to proceed in this way, though  it may be politically and practically sound.

Response;  The ROM was developed for use in a  probabilistic mode.  That is,  the
model output  concentrations for a given receptor  site should take the form of an
expected probability distribution of concentrations, reflecting  the  stochastic
nature  of  the   interpolated  flow fields used by the model.  The development of
the probabilistic flow model has lagged far behind the development on the  other
components of the ROM  system. We therefore  now have  a fully working model except
for this component.  Rather than hold up the entire  system  while  awaiting  the
completion  of   the  probabilistic  flow model (which may yet take  considerable
time to complete), we  have  chosen to proceed in the  quasi-deterministic  manner
described where  observations at aggregates  of  receptor sites may be matched with
corresponding aggregates of model predictions.   While  it  is  true  that  this
approach  necessitates  the use of a-priori assumptions on how the receptors may
be grouped, we feel  that while  imperfect, it is a better route than either  put-
ting the model on hold while awaiting the final flow model or using the model in
a fully deterministic  manner which would be completely inappropriate.

Comment. D. 9-10;  Biogenic emissions inventory development is essential for the
application   of   ROM...  The research has  involved  contact with forest and crop
researchers during the development of information sources, but has not  resulted
in  presentation of  results  to that community.  The latter is recommended as a
complement to existing reports  to the air pollution  modeling  community.  Uncer-
tainty estimates were  presented but seemed  questionable.

                                        41

-------
Response:  He agree.   A  journal  article describing the in-house effort on devel-
opment of a biogenic hydrocarbon emissions  inventory is  in  progress,  although
transmittal  of  results to  the  forest and  crop researchers has been ongoing.  He
also agree with  the reviewers  that  the uncertainty estimates  are  questionable.
The  intention of  the  presentation  was to provide uncertainty estimates for bio-
genic emissions  using  the same methodology  currently used by the  National  Acid
Precipitation  Assessment Program (NAPAP) for  the anthropogenic emissions inven-
tory.  Thus, at  least  the relative  uncertainties of the biogenic  and  anthropo-
genic emissions  could  be compared.   NAPAP recognizes the weakness of the current
methodology but  funding  for  improvements  in this area is limited.

Comment, p. 10;  Work  has focused on ozone  dry deposition over  large  areas  as
determined  from  aircraft measurements...   An effort should be made to complete
the analysis of  these  unique data sets by interpretation  of  convective  struc-
tures  (for  all fields) and treating ozone as a tracer.  This has some exciting
possibilities.

Response;  In-house analysis of  the aircraft turbulence ozone  measurements  has
indeed   been   focused   on the dry deposition of ozone as noted by the reviewers.
The major purpose  of  the turbulence aircraft flight program during the Northeast
Regional Oxidant Study (NEROS) was to obtain experimental data about the spatial
variation and  temporal behavior  of ozone  dry deposition over representative land
use types. The  in-house  research to date  was intentionally designed to study the
variability of  ozone  fluxes  and  deposition  velocities  for  different  land  use
areas  as  functions  of  time/height in  support of  the existing in-house regional
oxidant  model  program.  There are certainly other  interesting  aspects  of  this
data  set,  such  as   venting of ozone  by convective cloud elements and vertical
ozone transport over  urban areas; these  topics have  been  examined  through  an
extramural  effort  that  was  not  part  of  this peer review.  Other reviewer-
suggested enhancements to the program would be  interesting  and  productive  to
accomplish, but resources are limited and we must  focus on program needs.

Comment, D. 11;  Feedback between the physicist (s) behind the model  development
and programming quality control  people  was   a  little  vague.    Scientists  that
develop  the   model  know the inherent weaknesses.  Quality control should gener-
ate, independently,  a similar set of weaknesses.

Response:  Interaction between the model  developer.  Bob Lamb,  and  the  quality
control  staff  is  significant  even though this  aspect  was not stressed  in the
presentation.   Any suspect data is immediately brought to the attention  of  Dr.
Lamb  who  further  directs an assessment of  the cause  and  effect  of the anomalous
data.  Summary graphics such as contours  of the  maximum daily ozone and   the  QC
tracer species are provided to both Dr.  Lamb and to  Ken Schere,  the model evalu-
ator, on a routine basis.
        -  P-  11:  Hhat about vectorizing  the  code  for  the  ROM?...    Although
briefly  discussed,  the  ROM  may be improved more by a parallel  implementation
than  by a vector implementation.

Response:  About one year ago, IBM personnel made a serious attempt to  vectorize
the   ROM code.   Benchmark executions showed QO_ improvement in performance  of  the
vectorized code.  Subsequent studies performed by the Research  Triangle  Insti-
tute   (RTI)   explained why ROM is not effectively vectorized, but  more  likely to

                                        42

-------
benefit from a parallel  processing   implementation.   Therefore,  a  cooperative
research  program  with RTI  is  currently  implementing a loosely coupled multipro-
cessor architecture  for  the ROM.

Comment, p. 11;  A more  coordinated  plan  is   apparently  needed  for  providing
computing facilities to  ASRL/MD or even  to all of  EPA at Research Triangle Park.
Also, how much do  QC checks slow down the  ROM  simulation?

Response:  Since the EPA has acquired an additional IBM  3090  E  computer,  the
computing  power   of the Agency appears  to be  sufficient for current and antici-
pated ROM applications.   ADP timeshare funding for ROM applications is always  a
limiting  factor   in the number of ROM executions.  However, the requirement for
consistent QC checks on  the numerous files produced for  ROM  input  is  also  a
limiting  factor   in the number of new episode periods to be run.  Execution and
QC of three days of  model inputs can require up to  a  week.   This  processing
involves  approximately  60  hours of  VAX  785 CPU.   Fortunately, the time required
to prepare for a control strategy execution is typically less than a day.

Comment, p. 11:  Also, can  EPA use   the  supercomputer  resources  at  the  five
national centers?

Response:  ASRL is also  working closely  with the EPA National Computer Center to
evaluate the need  for supercomputer  resources  within the EPA.  A recent study by
RTI  has  concluded   that the EPA should acquire shared access or its own  (mini)
supercomputer at least to explore the potential of the vector supercomputer  for
some of its future environmental modeling. However, current OMB funding regula-
tions restrict the use of research dollars for ADP expenditures.  Creative solu-
tions may be necessary to use external supercomputer resources.

Comment, p. 12:  The cloud   process   module work may evolve  into good science...
The simplistic approach  presented is a reasonable  application of  the  state-of-
the-art  knowledge for simple cloud  model  formulation  (primarily a thermodynamic
emphasis).  Effort needs to be extended  to properly treat the total  entrainment
process between the  cloud top and the overlying inter facial  layer.

Response:   We  agree that  the current approach  to the cloud process module is
simplistic; what was presented was an initial  in-house effort.  We are currently
exploring the use  of a non-hydrostatic three-dimensional cloud model for examin-
ing some of the most important questions such  as cloud top and lateral  entrain-
ment.

Comment. P. 12:  The TTNEPH data may be  alright   for  radiation studies but its
value in the wet deposition and cloud processing modules is  very  questionable.
Efforts should be  made to incorporate the  GOES imagery, which is very useful for
the ROM domain.

Response;  He agree.  However, the GOES  imagery for the 1980 test  scenario  for
the  Regional Particulate Model (RPM) were not available in  sufficient complete-
ness to allow determinations of cloud base and top heights to be made, in  addi-
tion  to  the  normal areal coverage that  most applications  require.  The  RTNEPH
data sets already  contain this vertical  cloud  information, as estimated  by  the
Air Force through  a  best possible method approach.


                                        43

-------
Recent  correspondence   with  the  NWS National  Environmental Satellite Data and
Information Service  concerning the GOES  NEXT  generation  of  satellites  shows
promise  that,   in   the   coming  decade,  sufficiently complete data sets will be
archived to allow subsequent estimations  of  vertical cloud definition to be made
by our air quality models.

Comment. p. 12;  The MESOPUFF II  modeling-effort is  probably important in prin-
ciple, but (very good) performance evaluation  shows poor performance.  The  CAP-
TEX  data  set appears adequate to test performance.  Best results were obtained
for cases of  no  directional  shear.  Work  should  be published in  reviewed  jour-
nals (not just an EPA report).

Response:   The  in-house  effort  to  evaluate and test the MESOPUFF II regional
episodic model with  the  CAPTEX data base  consisted of:  (1) an operational evalu-
ation  in  which the model  was applied with  all  default features; (2) diagnostic
model runs to explore differences with alternative wind  fields  and  dispersion
features  in  the  model;  and  (3) model sensitivity  test runs which primarily
focused on variations in SOx from exercising  model  options  and  changing  key
parameters in the dry deposition and chemistry modules.  The presentation at the
peer review showed just  a sampling of  some of  the many  statistical  evaluation
results  and  graphical displays of observed  and  model tracer plume paterns.  The
analysis and  interpretation of model results were aimed at  identifying  reasons
for  model overprediction and spatial  displacements between the respective plume
positions.  While complete  results of  this task  are being documented in  an  EPA
report,  there are plans to  publish the notable  results in other publications as
recommended.

Comment, p. 13;  In  addition to the statistical  evaluation, additional effort is
needed  to  outline  conditions under which the models are or are not applicable.
For example,  the simple  Lagrangian models show the same  performance  repeatedly
over  several years  when annual averages  are compared.  The ISDHE analysis would
be more useful if it had led to an understanding of why  the  models  show  this
type of performance.

Response:   The  goals   of   the International  Sulfur Deposition Model Evaluation
(ISDME) focused  on the performance of  eleven long-term  models for each season of
one year.  These evaluation periods were  commensurate with the periods for which
the models were  designed to  simulate the  processes.  However,  to  identify  the
conditions  under  which  these models are or  are not applicable, one first must
investigate the  behavior of the models for periods much shorter than  a  season,
i.e.,  for  a  week   or  less.  Although this was once an objective of the ISDME,
participating modelers did  not submit  results  for periods considerably less than
their  interpretation of  the appropriate modeling period.  Nonetheless, an in-
house investigation  is currently underway to characterize the behavior of one of
these  models  for daily periods within one  season of the ISDME evaluation year.
From these results,  we should be able  to  determine those  meteorological  situa-
tions for which  the  model is or is not applicable.

Comment, p. 14:  The cloud  venting research  focuses on the study of penetrative
convection, which leads  to  venting of  the planetary boundary layer and transport
of  pollutants.   The results of this vork  have important implications for both
the ROM and RADM programs and should be continued.


                                        44

-------
Response;  We agree.   However,  NAPAP budget  cuts  not  only  prevented  further
field  evaluation   studies   but  even   prematurely  terminated analysis of field
measurement data that  had been  previously collected.

Comment, p. 14:  Good,  interesting  results are obtained in  the  fluid  modeling
laboratory and should  continue  to be published in JFM type journals.  The panel,
however, questions  whether  the  laboratory experiments are sufficiently  reliable
simulations of atmospheric  flows  to be  used  to set regulatory guidelines.

Response;   We  believe the proper  answer to the reviewers' question is, "not if
taken in isolation."   The results of laboratory experiments have not been  used,
in  and of themselves,  to set regulatory guidelines.  Many of the studies at the
Fluid Modeling Facility (FMF) have  been done upon request from the Office of Air
Quality  Planning   and Standards  (OAQPS).  It  is within the purview of the OAQPS
to utilize the laboratory results along with many other inputs in  setting  such
guidelines.   As a  prime example, the dividing-streamline concept arose from FMF
laboratory experiments.  This concept was validated through the complex  terrain
field  program, then  implemented  in the Complex Terrain Dispersion Model (CTDH).
It is now up to the OAQPS to establish  how the  results  of  the  CTDM  program,
including the results  of the laboratory experiments, are to be used in the regu-
latory process.

Comment, pp. 14-15:  Inverted tows  in stratified cases are adequate,  however  a
deep  sheared  flow  layer   is  not present (as in the neutral wind tunnel case
studies).  Experimental fluid mechanics is done well, both in  laboratory  tech-
niques  (flow  visualization and  measurement)  and application of theory.  How
these results are factored  into the CTDM was not explained well.

Response:  The primary value of the work in  the stratified  towing  tank  is  in
providing  fundamental understanding of the  physical processes involved in stra-
tified flows.  Much can be  learned  concerning  the structure of stratified  flows
even  in  the absence  of velocity shear layers.  Many aspects of these flows are
dominated by the density stratification rather than the velocity shear.  Earlier
work  (not reported to the  review panel) showed that under very strongly strati-
fied conditions, the  shear  layer  is unimportant.  The incorporation  of  labora-
tory  results  (and  complementary  field results) into the CTDM was done by the
contractor and therefore was not  discussed in  detail at this in-house review.

Comment, p. 15:  Stably stratified  flows are useful,  but  do  not  address  the
convective planetary  boundary layer.

Response:   The  addition of a  convective tank at the Fluid Modeling Facility is
presently in progress.   The most  important question to  be  answered  concerning
stratified  flows in  complex terrain is "where is the plume", e.g., impinging on
the windward slope, going over  the  top, or being directed around the side.   The
question  of  diffusion about  the plume   centerline is generally of secondary
importance.

Comment. D. 15:  Although driven  by regulatory requirements,  the  wake  effects
research  (non-aerodynamic  structures)  can be  considered quite basic...  Efforts
to obtain better flow  visualization and, in  particular, to quantify those obser-
vations are to be commended. Illumination techniques and the statistical evalu-
ation of the resulting data need  to be  improved.

                                        45

-------
Response:  He agree.  Research  on  the  study of diffusion in  building  wakes  is
intended  to provide  a  balance  between the immediate needs for regulatory appli-
cation and the need to  develop  a solid research base.  Improvements to the illu-
mination techniques have  been made,  and more  thorough statistical evaluations of
the resulting data are  to be pursued  once  the  relation  between  video  image
intensity and vertically  integrated  concentration  is firmly established.

Comment, pp. 15-16:   The  work on auto   exhaust  dispersion is clearly associated
with EPA's need  to model  the impact  of a major source of atmospheric  pollution.
It  is  difficult therefore to  judge what should or should not be done since the
work described was very incomplete in  itself.

Response;  The presentation of  the results  from   the  study  of  dispersion  in
automobile  wakes  was  indeed only a part of  a larger project but, as an example
of an in-house "wake  effects" study, it was included in  the  peer  review.   No
further work  in  this  area is planned for the  near  future.

Appendix C. pp.  27-28:   Comments on Model Evaluation

Response:   The  four  categories described by  the peer review panel are the major
elements pertaining  to  application of  a model to better understand  and  charac-
terize  its behavior,  performance and capabilities  (as distinct from applications
of the  model  to  answer  a specific  policy question).  While we might quibble some
about   the details of the descriptions under  each  category, possibly  seeing them
a bit more interconnected than  presented by the  review panel, we  tend to agree.

The main point  is  that  a good evaluation requires  a  team with  members  who  are
good  at posing  informative and useful tests  of  the  model as well as  members who
are good at quantifying and objectively interpreting   those  tests,   the   latter
usually  being   someone  with statistical expertise. We believe  that  our present
actions are  in concert  with the panel  recommendation.  He have included  persons
with statistical experience as  part of the International Sulfur  Deposition Model
Evaluation and have  statisticians  on the model evaluation planning  team  for  the
Regional Acid Deposition Model   (RADM)  evaluation.

Reviewer Recommendations. D. 32;  Provide organizational chart to all panel mem-
bers in advance  of  meeting.  Also... it would  have  been useful to have had  a  few
pages describing the Meteorology Division's  mission...

Response:    These  materials  were  indeed  provided  as   indicated in  the first
sentence on  page 2  of the peer  review  report.

Reviewee Comment.  D.  36;  Two of  the   reviewers seemed  familiar with air  pollu-
tion modeling and meteorology;  the others were much less  familiar with  this sub-
field of meteorology.  Although they may be  professional meteorologists, that  is
no guarantee  they are knowledgable  in all aspects  of the  field.

Response:    Per   my   instructions to the ASRL Peer Review Coordinator,  the panel
was constituted  by representatives  from the  disciplines  of  statistics,   microme-
teorology,   systems  analysis, fluid modeling, and meteorological modeling.  This
mixture was  needed to ensure that at least one reviewer  was  knowledgable  in each
of the  technical presentations.


                                        46

-------
Comments, DP.  32 and  3fi:   Lack  of  adequate  time  for presentations.

EfiSponge;   As  noted in  some of my  preceding  responses, the time allotments for
presentations  were  occasionally too  limited to accommodate complete descriptions
of individual  in-house  research efforts.  For example, the Regional Oxidant Model
(ROM) and supporting  components were presented in  two and one-half hours, where-
as  the  ROM   was allotted a full  day in  an ASRL project-oriented peer review in
January 1987.   In adhering to this  stringent  schedule,  therefore,  presenters
were  only  able to describe the basic tenents of  their research projects at the
expense of discussing ancillary topics such as relevancy to  the  Agency's  mis-
sion,  publication  of  results, and  involvement  with the scientific community at
large.  The peer review report  shows that,  in  most cases,  the  reviewers  took
this limitation into  consideration.


References

Dahai, X., S.  G. Perry, and P.  L.  Finkelstein, 1986:  Multi-Scaling and Exponen-
tial Fitting  in Autocorrelation Analysis.   Extended Abstracts, Fifth Joint  Con-
ference on Applications of Air  Pollution  Meteorology, AMS, pp. 348-351.

Hanna, S. R.,  1981:  Lagrangian and  Eulerian Time-Scale Relations in the Daytime
Boundary Layer.  Journal  of Applied  Meteorology, 20:242-249.

Hanna, S. R.,  1986:  Spectra of Concentration  Fluctuations - The Two Time Scales
of a Meandering Plume.  Atmospheric  Environment, 20:1131-1137.

Lamb,  R.  G.  and S.  K. Hati,  1987:   The  Representation of Atmospheric Motion in
Models of Regional-Scale  Air Pollution.   Journal of Climate and Applied  Meteor-
ology, 26:837-846.

Misra,  P.  K., 1979:  The Auto-Correlation Function of the Vertical Velocity in
the Low-Frequency Range.   Preprints, Fourth Symposium on Turbulence,  Diffusion,
and Air Pollution,  AMS, pp. 41-45.

Moore,  G. E.,  M. Liu,  and L. Shi, 1985:   Estimates of Integral Time Scales from
a 100-M Meteorological  Tower at a  Plains  Site.   Boundary-Layer  Meteorology,
31:349-368.

Sutton, 0. G.,  1953:   Hicrometeorology.   McGraw-Hill Book Company, New York, NY,
333 pp.

Weber, A. H.,  J. S. Irwin, W. B. Petersen,  J.  J. Mathis, and J. P. Kahler, 1982:
Spectral Scales in  the  Atmospheric Boundary Layer. Journal of Applied Meteorol-
ogy, 21:1622-1632.
                                        47

-------
                    APPENDIXG



   THE  LABORATORY  DIRECTOR'S  REVIEW  COMMENTS



ON THE PANEL REPORT AND THE ASRL STAFF RESPONSE

-------
m)
                UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                        ATMOSPHERIC SCIENCES RESEARCH LABORATORY
                               RESEARCH TRIANGLE PARK
                                NORTH CAROLINA 2771 1
DATE:     February 1, 1988

SUBJECT:  Meteorology Division In-house Program

FROM:     Jack H. Shreffler  /V^ H S
          Deputy Director, AS^L/(MD-59)

TO:       Ronald K. Patterson
          TPRO, ASRL (MD-59)

     I have read the Peer Review report on  the in-house research program
of the ASRL Meteorology Division and  the response  by its Director, Francis
A. Schiermeier.  The report is one of the most complimentary I have re-
viewed and the Director's response is complete and adequate in all respects.
Moreover, the Summary of Process Evaluations  by  reviewers indicates a very
good preparation and execution of the peer  review  meeting, for which Research
and Evaluation Associates, Inc. and Project Officer Ron Patterson deserve
credit.

     Management should not get involved in  fixing  things that are not broken.

cc:  A. Ellison
                                     49

-------
       \
       §          UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                           ATMOSPHERIC SCIENCES RESEARCH LABORATORY
<*/„   ,-0"-                           RESEARCH TRIANGLE PARK
  1 ""&                              NORTH CAROLINA 2771 1


     DATE:      February  11,  1988

     SUBJECT:   Release of Peer Review Panel Report:   In-House
               Research  Program By The Meteorology Division

     FROM:      Ronald K.  Patterson,  Peer  Review Coordinator
               Atmospheric  Sciences  Research  Laboratory  (MD-59)

     TO:        H.  Matthew Bills,  Acting Director
               Office of Acid  Deposition,  Environmental  Monitoring
                and Quality Assurance (RD-680)
    THRU:      Alfred  H.  Ellison,  Director
               Atmospheric  Sciences  Research  Laboratory  (MD-59)


         The enclosed  peer review panel report has  been reviewed  and released
    by our  Laboratory.   Comments  by  ASRL  participants and  our  Laboratory
    Director are included  in the report under Appendices  F and G, respectively.
    We find the comments from our participating staff to  be complete and appro-
    priate.

         This  peer  review  evaluated only the in-house  research program under
    the ASRL  Meteorology  Division.    A  total  of  45  technical presentations
    were made  during  the  four  day  peer review.   The  enclosed  peer   review
    panel report is highly complimentary of  the  Division's leadership  and the
    competence of the  technical  staff.  The  review  panel was  impressed by the
    high degree  of  excellence exemplified in the areas  of experimental fluid
    dynamics and  computational  quality  control.   However,  the  review panel
    expressed  concern  regarding inadequate computer resources  and  the external
    pressures  exerted  on the staff to prematurely release scientific findings.
    The review panel also  discussed the need to  strengthen in-house statisti-
    cal expertise.

         We are  distributing  copies of  this report and cover letter  to the
    individuals  listed below.   The  ASRL  peer review contractor,  Research and
    Evaluation Associates,  Inc., has  been  instructed  to  forward a  copy of
    this final released  version of this  report to  each  peer  review panelist.

    Enclosure

    cc:  Erich Bretthauer  (RD-672)        Courtney  Riordan (RD-682)
         William Keith (RD-680)           Deran  Pashayan (RD-680)
         Morris  Altschuler (RD-674)      \#4rbert Wiser  (ANR-443)
         Elenora Karicher  (RD-680)        Jack Shreffler (MD-59)
         Basil Dimitriades (MD-59)        William Wilson (MD-59)
         Frank Schiermeier (MD-80)        Meteorology Division Staff (MD-80)

-------