United States
                    Environmental Protection
                    Agency   	
Atmospheric Research and Exposure
Assessment Laboratory
Research Triangle Park, NC 27711
                    Research and Development
EPA/600/SR-92/112  Aug.  1992
^ EPA      Project Summary
                    Protocols   for  Evaluating  Oxidant
                    Mechanisms for Urban  and
                    Regional  Models
                    H.E. Jeffries, M.W. Gery, W.P. L. Carter,
                      A task force of chamber operators
                    and modelers was assembled to ad-
                    dress needs raised at a prior workshop
                    on the procedures and practices that
                    should be  followed  when evaluating
                    photochemical reaction mechanisms for
                    their suitability for use in EPA's urban
                    and regional air quality models.  In ad-
                    dition to the task force, two workshops
                    were held that were attended by re-
                    searchers in the field and at which is-
                    sues raised by the task force were dis-
                    cussed and guidance was sought.
                    Based on our work and the community
                    input, we describe how to create a pro-
                    tocol for the evaluation of photochemi-
                    cal reaction mechanisms.  Rather than
                    prescribe a set of actions, we present
                    instead criteria that influence decisions
                    without specifying what those decisions
                    must be.  These specify what the evalu-
                    ator must consider, what is and is not
                    relevant, and what must be reported as
                    the basis of decisions made.
                      Based  on general scientific  prin-
                    ciples, we describe five characteristics
                    that reaction mechanisms  must have if
                    they are acceptable.  Our approach is
                    based on a complex argument  in the
                    form of a cascaded inference chain
                    showing  how  to proceed  to establish
                    that a candidate mechanism might ex-
                    hibit all five characteristics. The evi-
                    dence in this argument is chamber data.
                    The report  details the elements that
                    must be considered and describes the
                    general content of reports the evalua-
                    tor must produce. We conclude that,
                    because of  data incompleteness prob-
                    lems, it is  not presently possible to
 make a compelling case for accepting
 a mechanism for ambient air use, only
 to vindicate its use as having been
 evaluated the best that can be currently
 done, given the available data,,
  This Project Summary was developed
 by  EPA's Atmospheric Research and
 Exposure Assessment Laboratory, Re-
 search Triangle Park, NC, to announce
 key findings of the research project
 that is fully documented in a separate
 report of the same title (see Project
 Report ordering information at back).

 Introduction
  The EPA  must be able to defend  its
 control policies and regulations. For ozone
 and other oxidants these policies depend
 upon the application of mathematical mod-
 els  of emissions, atmospheric transport,
 and chemical transformations to forecast
 the  effects of proposed controls.  An im-
 portant element in this  defense  is the
 evaluation of the accuracy and reliability
 of these models.  Because  the models
 are large and complex, the first approach
 to establishing their overall accuracy is to
 assess independently the accuracy of each
 major component in the models, i.e., emis-
 sions, transport, and chemistry. Not only
 can  the accuracy of the  atmospheric
 chemical transformation component  of
 these models be evaluated independently
 of the emissions and transport terms, such
 independent testing is the least ambigu-
 ous method.  The need for such testing
 raises the need for standards to compare
 with the model and for a method by which
 the  comparisons can best be done. Pre-
 vious work  sponsored by  EPA  (see,
                                                                    Printed on Recycled Paper

-------
'Workshop on Evaluation/Documentation
of Chemical Mechanisms," EPA/600/9-87/
024,1987) established that both standards
and methods acceptable  to the scientific
community exist.  Provided it was of high
quality, smog chamber data was recom-
mended as the best standard.  Provided
that efforts were made to treat the prob-
lems caused by the influence of the cham-
ber and for coping with the scarcity and
quality of  measurement data,  and even
though it varied significantly from modeler
to modeler, the then-existing mechanism
evaluation practice was recommended as
the best comparison process. The work-
shop recommended that  a task force  of
chamber operators and modelers be as-
sembled to address these problems. The
work described in this summary is part of
the effort  of this task force.  During the
project, the task force held two workshops
which were attended  by researchers  in
the field.  In these, issues raised  by the
task force were discussed and  guidance
was sought.   Based  on our  work and
communily input, we describe how to cre-
ate a protocol for the evaluation of photo-
chemical  reaction mechanisms that are
candidates for use in EPA urban and re-
gional oxidant models.
  Based on general scientific  principles,
we present a  set of characteristics that
reaction mechanisms  must have  if they
are to be acceptable. A central assertion
here is that  mechanism acceptance  is
more than just establishing agreement of
the model's predictions with observations.
We also recognize that any set of objec-
tive  criteria may be insufficient to deter-
mine in full any algorithm for mechanism
choice. Therefore, rather than prescribe a
detailed set of actions  (i.e., a cookbook),
we present instead criteria that influence
decisions  without specifying what  those
decisions must be.  Such criteria specify:
(1) what each evaluator must consider in
reaching his decisions, (2)  what he may
and may  not consider relevant, and (3)
what he can legitimately be required  to
report as the basis for his choices.  Our
goal Is to  have the evaluation be a com-
plete and persuasive  argument and this
requires a clear chain of reasoning based
on credible and complete  evidence.
  Our approach is based on a complex
argument  in the form of a cascaded infer-
ence chain showing how  to proceed from
kinetics data and chamber data  to the
necessary claims that allow  use  of the
mechanism in EPA models.   A second
part of the inference chain addresses cred-
ibility issues for the kinetics and chamber
data.  Further, we provide general war-
 rants for why this reasoning chain is legiti-
 mate.
   Our argument  is structured around es-
 tablishing that an acceptable mechanism
 has five general characteristics:  (1) con-
 sistency with currently accepted theories
 and facts applicable to these theories; (2)
 accuracy,  that is, its predictions are in
 demonstrated agreement with observations
 from experiments and  such agreement
 does rjot arise from compensating errors;
 (3)  simplicity,  that  is,  the mechanism
 should explain  events in terms of neces-
 sary causal forces rather than empirical
 generalizations;  (4)  scope, that is, the
 mechanism's  predictions  should  extend
 beyond the particular observations it was
 initially designed to explain;  and (5)  fertil-
ity, that is, the  mechanism is expected to
 predict novel phenomena that  were not
 part of the set to be originally explained.
 In an evaluation,  data that are relevant to
 each  of  these characteristics  are as-
 sembled  to construct lower-order claims
 (e.g., the  mechanism is well-formulated)
 which in turn are  assembled into higher-
 order and more abstract claims that lead
 to acceptance or rejection of the mecha-
 nism.
   After the first chapter that presents the
 mechanism characteristics, the argument,
 and a general description  of the process,
 the  report is divided into five substantive
 chapters and a conclusions chapter.  One
 chapter defines the terminology and meth-
 ods used and presents an overview of the
 operations.  Four chapters  describe the
 elements that must be  considered in an
 evaluation and  provide specification of the
 content of reports that must be produced
 by the evaluator.  These are:  (a) reports
 on the mechanism,   (b)  reports  on the
 chamber data, (c) reports on the  simula-
 tions, and (d)  reports on the evaluation
 process.
 Names and Definitions
   In such a complex procedure involving
 models, data, and operations, it is neces-
 sary to agree on formal terms and to have
 formal definitions.  For example, the first
 part of the definition of a principal mecha-
 nism is "a complete mechanism for an
 organic compound or mixtures of organic
 compounds  combined with  oxides of ni-
 trogen and air that all chambers and the
 ambient air  are thought to have in  com-
 mon at present."  The rest of the definition
 describes what is included in and what is
 not included in a mechanism.   Unfortu-
 nately, principal mechanisms are not what
 are tested  with  chamber data.  This  is
 because each chamber has a perhaps
 unique set of wall-mediated reactions with
 reactants and products in common with
 the principal mechanism, and thus, these
 wall reactions influence the chemistry; their
 effects must be included in any simulation
 if it is to describe accurately the chamber
 observations.  Thus,  auxiliary  reaction
 mechanisms, one for each chamber, must
 be added to the principal mechanism and
 it is  this combined mechanism  that is
 tested with chamber data.  Other terms
 and definitions given  in this chapter are
 related to the chamber operations models
 and the  comparison of observations and
 predictions.

 Reporting Mechanisms
   For an  evaluator to test a principal
 mechanism using our procedures, certain
-, information-aboutthe principal mechanism
 must be known and reported.  First, the
 evaluator must report that he has an ex-
 ecuting  version of the mechanism and
 that he  can operate the  mechanism  as
 intended by the developer.  Next, he must
 report that he has assured that the mecha-
 nism is "well-formed." Thus, he must have
 knowledge  of the development protocol,
 knowledge of the supporting data used in
 formulating the mechanism, and he must
 document that he understands the formu-
 lation and condensation rationale used by
 the  developer.  Further, he must deter-
 mine the basis for believing that the prin-
 cipal mechanism's formulation is indepen-
 dent of the chamber data used in its de-
 velopment.   If the evaluator determines
 that the  mechanism suffers from flaws in
 its formulation, he must decide and report
 (justify) one of three courses of action: (1)
 decide to act as if the problem would not
 influence the mechanism's predictions and
 proceed; (2) decide that the mechanism
 needs to  be "fixed"  before proceeding,
 deviate  from  the protocol to  fix  the
 mechanism's formulation, and then pro-
 ceed; or (3) decide" that the mechanism's
 flaws are so bad that it cannot be fixed
 and, therefore,  stop  the evaluation  be-
 cause any exhibited accuracy by a funda-
 mentally flawed mechanism is  meaning-
 less.
   Assuming that  he chooses to proceed,
 the evaluator must select chamber-depen-
 dent auxiliary mechanisms for each cham-
 ber that  will be used and he must assure
 (and document) that  these mechanisms
 are also as well-formed as we know how
 to make them.

 Reporting Chamber Data
   The primary evidence that would sup-
 port the  claims leading to mechanism ac-
 ceptance are chamber data.  For cham-
 ber data to be evidence in these matters,

-------
they must exist and they must be relevant
to the claims being made, i.e., the cham-
ber data must be capable of causing the
evaluator to change  his mind  about the
validity of the claims.
  Thus, the evaluator must first report the
chamber data needed to conduct a com-
pelling evaluation.  That is, one in which
the evidence (if  it existed)  would permit
only one reasonable  interpretation about
the acceptability  of the mechanism.  The
second condition to  be reported is the
existence of the needed data; and the
third condition is the  relevance of  any
existing  data to the  case  being made.
The foundations  of these arguments rest
on the credibility of the chamber data used,
and if the evaluator is to produce a suc-
cessful evaluation, a reviewer of the evalu-
ation must be convinced that the chamber
data used in the comparisons are believ-
able. Therefore, the  fourth condition the
evaluator must  report is chamber  data
credibility and why he believes it has this
quality.   The evaluator then must report
the extent to which the available, relevant,
credible chamber data meet the data need.
Where there is  a significant shortfall in
available data, the evaluator must report
on  how  he will  continue to conduct an
evaluation when  only part of the chamber
data that is needed is available.
  Once the chamber data  are  selected,
the evaluator must prepare the appropri-
ate summary observations for comparison
with simulation predictions in subsequent
work.

Reporting Simulations
  For  each selected experiment to be
simulated, the evaluator must build a simu-
lation  solver  input  file containing the
model's  representations of the chamber
conditions.  First, species  present in the
chamber experiment  must be represented
in the simulation input file.   For explicit
species in the model and in the chamber
this is no problem. For chamber species
that are represented as some type of gen-
eralized  model  species, the  appropriate
transformations must  be applied.  For spe-
cies thought to  be present in the cham-
ber, but that were not  measured, the evalu-
ator must  report his algorithms for esti-
mating these values.   Data on light inten-
sity must also  be converted into actinic
flux for each experiment.  The  evaluator
must report the algorithms he uses for this
calculation. Next, photolysis rates for the
photolyzing species   in the mechanism
must be calculated from the actinic flux,
and these algorithms must be reported.
Finally, the simulation's representation of
chamber temperature, dilution, and water
vapor must be  created  and again the
evaluator must report the algorithms used
to produce these representations from the
observed chamber data.  The simulation
solver input files must have internal docu-
mentation for their values and  the  files
must be available for distribution to inter-
ested parties.
  After the simulations are run, the evalu-
ator must compute and report the corre-
sponding summary measures for the pre-
dictions as was done for the observations.
He then must calculate the absolute and
relative errors in amplitudes and in times
to events.  Next, he must display these
errors on plots that  show results on a
chamber by chamber basis.  Other re-
quirements for these plots are described
that help detect internal compensating er-
rors  among the  parts of the principal
mechanism.  All simulations with large
errors must be plotted as concentration
time profile plots of observations and pre-
dictions and the evaluator must offer ex-
planations for why the agreements were
so poor.

Reporting the Evaluation
  The  evaluator would now  have all the
raw evidence at hand and he  must build a
case to accept  or reject the  mechanism.
Three major obstacles stand in the way of
making compelling cases:  (1)  Because the
theories  to support  a chamber-auxiliary
mechanism are much weaker than those
for a  principal mechanism and  because
there  are too few chamber characteriza-
tion data, the evaluator may have found it
necessary to use an  auxiliary mechanism
that was more "tuned" than it was formu-
lated.  Thus, there is the real possibility
that  any demonstrated accuracy of the
combined mechanism is caused by com-
pensation between inaccurate components
in both the auxiliary  mechanism  and the
principal  mechanism.  (2) The chamber
data are incomplete  in the necessary in-
puts to simulate accurately an experiment
and these  inputs must be estimated by
the evaluator. If there is bias  in the choice
of unmeasured but necessary simulation
inputs, then again there is the real possi-
bility any demonstrated accuracy of the
combined mechanism is caused by com-
pensating errors between the  principal
mechanism and the simulation inputs. (3)
The chamber data are incomplete in their
coverage of  single species  and simple
mixtures.  Thus, the  accuracy of mecha-
nisms  for some  reacting species in the
principal mechanism cannot  be tested at
all.  Strategies  for treating these  difficul-
ties are  discussed.  The evaluator  must
report his choices and procedures for ad-
dressing these problems.
  The evaluator must make judgments
about each experiment he simulated and
must  record these judgments in a table
form described in the text. He must report
these judgments by chamber and by ex-
perimental class.  Finally, he must make
summary statements about the five char-
acteristics that acceptable mechanisms
must have.

Conclusions
  We have  presented criteria that influ-
ence  decisions without specifying what
those decisions must be; we have com-
pletely described all the elements  the
evaluator must consider in performing an
evaluation; we have made clear those is-
sues that  are relevant to  an evaluation;
and we have been explicit about what the
evaluator must report as the basis of the
decisions he makes.  We have presented
a process for evaluating reaction mecha-
nisms that is based on sound  scientific
principles, draws  on classical  logic and
formal arguments, and considers practical
difficulties.  It is based upon evolving cur-
rent practice, but represents a significant
improvement over existing evaluations.
  Using this report,  an evaluator and a
sponsor can produce  a specific set of
agreements—a protocol—for an evalua-
tion of a   mechanism  using a  set of
chamber data.  As an aid in this process,
the Appendix  contains a summary of all
the necessary reports.
  Chamber data from UNO and from UCR
that meet  the criteria given in this report
will be available as part of this  project
later in  1992.  Additional chamber  data
from the chambers operated by the Com-
monwealth Scientific  and Industrial  Re-
search Organization in Sydney, Australia,
are being documented and quality assured
now and should be available by the  end
of 1993.
  Because  of the  data  incompleteness
problem that is described in Chapter 6,
however,  it  is presently  not possible to
make a compelling case to accept a chemi-
cal reaction  mechanism as accurately de-
scribing urban chemical transformations.
In fact,  the  chamber evidence  may not
allow even a presumptive standard to be
used. For many species of interest, there
is pnly missing evidence. Therefore, until
relevant chamber data  are  made avail-
able, the evaluator and the sponsor must
be satisfied  with vindicating  the use of a
mechanism,  that is, to claim  that  the
mechanism  has been evaluated the best
we can currently do given the  available
data.
                                                                                      •U.S. Government Printing Office: 1992 — 648-080/60049

-------
   H.E Jeffries Is with the University of North Carolina, Chapel Hill, NC 27599. M. W.
     Gary is with Atomospheric Research Associates, Boston, MA 02116. W. P. L
     Carter Is with the University of California, Riverside, CA 92522.
   Marcla C. Dodge is the EPA Project Officer (see  below).
   The complete report, entitled "Protocols for Evaluating Oxidant Mechanisms for
     Urban and Regional Models," (Order No. PB92-205 848/AS; Cost: $26.00;
     subject to change) will be available only from:
           National Technical Information Service
           5285 Port Royal Road
           Springfield, VA 22161
           Telephone: 703-487-4650
   The EPA Project Officer can be contacted at:
           Atmospheric Research and Exposure Assessment Laboratory
           U.S. Environmental Protection Agency
           Research Triangle Park, NC 27711
Untied States
Environmental Protection Agency
Center for Environmental Research Information
Cincinnati, OH 45268

Official Business
Penalty for Private Use
$300
     BULK RATE
POSTAGE & FEES PAID
         EPA
   PERMIT No. G-35
EPA/600/SR-92/112

-------