United States
Environmental Protection
Agency
*«;.'
Atmospheric Research and
Exposure Assessment Laboratory -, fc/ *
Research Triangle Park NC 27711 f / >'
Research and Development
EPA/600/S3-89/082 Apr. 1990
f/EPA Project Summary
The Role of Grid-Based,
Reactive Air Quality Modeling in
Policy Analysis: Perspectives
and Implications, as
Drawn from a Case Study
Philip M. Roth, Charles L. Blanchard, and Steven D. Reynolds
A primary objective of this study is
to improve the understanding of the
role of performance quality in
determining a model's acceptability
and usefulness to the policy maker,
and thus to aid in developing
soundly-based expectations of the
modeling process. The vehicle for
pursuing this objective Is examining
the historical evolution of the Urban
Airshed Model (UAM) - a grid-based
photochemical model, whose basic
formulation Is similar to the Regional
Acid Deposition Model (RADM) - and
its application to policy analysis in
the South Coast Air Basin of
California. A derivative objective Is to
draw implications from the findings
to aid In appraising the merits of
future pursuits, notably the mounting
of comprehensive field programs to
support the evaluation of regional
acid deposition models.
In the technical report, we: (1)
describe the UAM, (2) examine Its
predictive capability thrugh scrutiny
of historical performance statistics,
(3) assess its degree of acceptance
in the scientific and regulatory com-
munities, based on information and
viewpoints solicited through ques-
tionnaires and interviews, (4)
examine the influence on the policy-
making community of the UAM
studies and, (5) based on these
"findings," attempt to develop a
perspective on the expectations of
current regional acid deposition
models.
This Project Summary was
developed by EPA's Atmospheric
Research and Exposure Assessment
Laboratory, Research Triangle Park,
NC, to announce key findings of the
research project that Is fully
documented In a separate report of
the same title (see Project Report
ordering Information at back).
Introduction
During the 1980s the National Acid
Precipitation Assessment Program sup-
ported the development of the Regional
Acid Deposition Model (RADM). The
technical community urged that sound
performance evaluation be a part of the
overall development process and, con-
sequently, that a comprehensive data
collection program be supported finan-
cially. They also expressed concerns that
policy makers may hold overly optimistic
expectations of performance of the
RADM at its current stage of develop-
ment and of time-to-availability of the
model for unrestricted use.
One means of developing a realistic
portrayal of the model evaluation and
acceptance process is to examine the
experiences of the past. In 1970,
Systems Applications Inc. began devel-
opment of the Urban Airshed Model
(UAM), a grid-based photochemical
-------
model of ozone formation on an urban
scale. A recent version of that original
model is now in widespread use. The
history of the UAM's continuing develop-
ment, sporadic evaluation, and frequent
application is of particular relevance
since the DAM is a complex, grid-based
"chemical" model similar in basic
formulation to the RADM.
A primary objective of this study is to
gain an improved understanding of the
importance of quality of performance in
determining a model's acceptability and
usefulness to the policy maker. We
hoped to learn some lessons from the
DAM experience and present them for
others to consider in planning and
executing RADM-related activities. The
vehicle for pursuing this objective is to
examine the historical evolution of the
(JAM and its application to policy analysis
in the South Coast Air Basin of California
over a period of years. A derivative
objective is to draw implications from the
findings that may be of value in
appraising the merits of future pursuits,
notably the mounting of expensive field
programs to support the evaluation of
regional acid deposition models. The
technical report offers insight gained from
the study of one model's use that is
intended to aid the policy maker in
developing soundly-based expectations
of the modeling process.
Procedure
In this study we surveyed both the
performance evaluation history of the
UAM and the degree of acceptance
achieved by the model in the policy-
making and scientific communities. We
first examined and interpreted
performance statistics for the model over
the history of its development. We then
interviewed three categories of interested
and informed parties: (1) members of
the contemporaneous scientific commu-
nity whose pursuits led them to become
familiar with the UAM and its
development history, (2) industry repre-
sentatives, and (3) regulatory agency
staff and policy makers who have had to
assess the merits of using the results of
the UAM. The broad questions to be
answered by these interviews were:
Was the model acceptable?
Did its degree of acceptability change
over time?
How was model acceptability related
to model performance?
Based on the findings of these efforts, we
developed a "subjective history" of the
perception of model performance and of
model acceptance.
Results and Discussion
Analysis of time trends in performance
statistics for the UAM suggest that (1) the
tendency toward underprediction of
ozone concentrations decreased with
time over fifteen years of testing, but the
range in estimates of bias at a particular
time were high and remained relatively
unchanged with time, and (2) the average
error in prediction decreased with time,
but even in recent times displayed a
significant range in variation.
Analysis of interviews of thirty
individuals who have been associated
with UAM-use in policy analysis in the
South Coast Air Basin suggests that (1)
performance of the UAM was "good
enough for estimates to be considered"
in the policy context, but not "good
enough to provide a high degree of
confidence in the model," (2) increased
acceptance of the UAM was not related
to improved performance with time (or to
perceived performance), (3) interviewees'
increased acceptance of the model was
related to improved understanding of the
model's capabilities and limitations, (4)
interviewees are prepared to accept mod-
el use in assessing the relative effects of
alternative control scenarios, but not in
quantitative evaluation of specific mea-
sures, and (5) interviewees favor in-
creased emphasis on acquisition of better
input data and more comprehensive data
bases for supporting model evaluation.
Conclusions
Modeling is most appropriately
conceived as a continuing process of
model development, collection of data,
testing, diagnostic analysis, modification
and improvement of the model, and
repetition of the cycle. Performance eval-
uation is an essential, though not the
only, element in establishing confidence
in a models' value. Sensitivity analyses
are considered as important as per-
formance evaluations for assessing the
accuracy and reliability of a model.
Acceptance of a model by regulatory
and industrial communities appears to be
linked as much or more to understanding
the model as it is to the results of per-
formance evaluations. Key ingredients in
gaining familiarity with a model and
establishing an understanding of the
model's strengths and limitations are:
dialogues with modelers, in lay terms,
about the model - its formulation,
limitations, and attributes.
discussions of results of performance
evaluations, including at the stage of
diagnostic assessment, when possible
problems and "cures" were conte
plated.
discussions of sensitivity analysi
when model behavior could be si
jectively compared with expectatic
based on individual knowledge and 1
degree of conformity with the
expectations could be examined.
For complex atmospheric modeli
systems, experience suggests that 1
process of acceptance requires seve
cycles of development and testing a
that the overall time required to achie
model acceptance, based on quality
performance, may be fifteen to twei
years. While this admittedly subject!
time estimate may not be welcomi
knowledge gained during the process
development, data acquisition, and di<
nostic analysis is generally of cons
erable value.
-------
Philip M. Roth is a private consultant in San Anselmo, CA 94960; Charles L
Blanchard is with Princeton University, Princeton, NJ 08544; and Steven D.
Reynolds is with Systems Applications, Inc., San Rafael, CA 94903.
Robin L Dennis is the EPA Project Officer (see below).
The complete report, entitled "The Role of Grid-Based, Reactive Air Quality
Modeling in Policy Analysis: Perspectives and Implications, as Drawn from
a Case Study," (Order No. PB 90-187 2041 AS; Cost: $23.00, subject to
change) will be available only from:
National Technical Information Service
5285 Port Royal Road
Springfield, VA 22161
Telephone: 703-487-4650
The EPA Project Officer can be contacted at:
Atmospheric Research and Exposure Assessment Laboratory
U.S. Environmental Protection Agency
Research Triangle Park NC 27711
United States
Environmental Protection
Agency
Center for Environmental Research
Information
Cincinnati OH 45268
/^"-^\ us-°FHCIAL MAIL
Av 7\ *~"
/O -A\ 'ENALTV
MAT-3'90 "I Ofi
4 / 'SE S300
~'' ''3 METER
6090444
1UPOSTAGE
s 0 2 S ,
Official Business
Penalty for Private Use $300
EPA/600/S3-89/082
PEOTECflOB
------- |