j A *
1®
U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
Improving Air Quality
EPA Can Strengthen Its
Process for Revising Air Quality
Dispersion Models that Predict
Impact of Pollutant Emissions
Report No. 18-P-0241	September 5, 2018
Maximum Modeled Concentration

-------
Report Contributors:
Bao Chuong
Kevin Good
Renee McGhee-Lenart
James Hatfield
Stacey Garfinkle
Abbreviations
AERMOD
American Meteorological Society/EPA Regulatory Model
AQMG
Air Quality Modeling Group
CFR
Code of Federal Regulations
EPA
U.S. Environmental Protection Agency
NAAQS
National Ambient Air Quality Standards
OAQPS
Office of Air Quality Planning and Standards
OIG
Office of Inspector General
QAPP
Quality assurance project plan
SOPs
Standard operating procedures
S02
Sulfur dioxide
Cover Image: AERMOD modeling and pollutant emissions imagery. (EPA and OIG images)
Are you aware of fraud, waste or abuse in an
EPA program?
EPA Inspector General Hotline
1200 Pennsylvania Avenue, NW (2431T)
Washington, DC 20460
(888) 546-8740
(202) 566-2599 (fax)
OIG Hotline@epa.gov
Learn more about our OIG Hotline.
EPA Office of Inspector General
1200 Pennsylvania Avenue, NW (2410T)
Washington, DC 20460
(202) 566-2391
www.epa.gov/oiq
Subscribe to our Email Updates
Follow us on Twitter @EPAoig
Send us your Project Suggestions

-------
£
<
^eDsv^	U.S. Environmental Protection Agency	18-P-0241
*	Office of Inspector General	Septembers, 2018
^ At a Glance
° VV *
PRCtf*-
Why We Did This Project
We conducted this audit to
assess the effectiveness of the
U.S. Environmental Protection
Agency's (EPA's) process for
reviewing and approving air
quality dispersion models it
recommends for use by state,
local and tribal air pollution
control agencies.
Air quality dispersion models
predict the air quality impact of
pollutants released into the
atmosphere. The EPA's review
and approval process for
designating preferred models is
outlined in Appendix W of
40 CFR Part 51. The goal of
this process is to identify the
best-performing model as the
preferred model, and the
appendix lists preferred
models. Appendix W was
originally promulgated in 1978
and most recently revised in
2017. The American
Meteorological Society/EPA
Regulatory Model (AERMOD)
is the EPA preferred model for
most regulatory uses listed in
Appendix W. AERMOD
predicts the air quality impact
of pollutants from sources up to
50 kilometers downwind, and
was first designated as a
preferred model in the 2005
revision of Append ixW.
This report addresses the
following:
• Improving air quality.
Send all inquiries to our public
affairs office at (202) 566-2391
or visit www.epa.gov/oiq.
EPA Can Strengthen Its Process for
Revising Air Quality Dispersion Models
that Predict Impact of Pollutant Emissions
Air quality estimates
derived from air quality
dispersion models are
used to make important
decisions to protect
public health, such as
setting emissions limits.
What We Found
Although the agency has prepared guidance on the
recommended procedures for reviewing the
development and evaluation of new air quality
dispersion models, similar guidance is not available
for model revisions. The development of standard
operating procedures (SOPs) and quality
assurance project plans (QAPPs) or equivalent
documents for model revisions could assure
consistent application of quality assurance and
quality control activities.
From 2006 through 2016, the EPA issued 12 Model Change Bulletins revising
AERMOD with enhancements, bug fixes and/or miscellaneous changes to
improve the model. The 12th Model Change Bulletin was associated with the
2017 Appendix W revisions that included adding new regulatory uses to
AERMOD. However, the quality assurance and control activities undertaken for
these revisions were not as extensive as what EPA guidance recommends for
new model development and evaluation. For example, the agency used peer-
reviewed journal articles to satisfy peer-review requirements, while AERMOD
received a panel peer review when it was developed. In one instance, the agency
proposed a new regulatory option for AERMOD, which lacked peer-review
literature and later needed additional evaluation. Development of SOPs, as well
as QAPPs or equivalent documents, could assure that consistent and
appropriate quality control and assurance activities are conducted when revising
preferred models by helping assure that the predicted results are of sufficient
quality. This is especially important because AERMOD is used by all 50 states,
as well as tribes and territories, to predict air quality impacts for regulatory
purposes under the Clean Air Act.
Recommendations and Planned Agency Corrective Actions
We made four recommendations to the Assistant Administrator for Air and
Radiation. These recommendations involved developing SOPs to guide and
document its process for reviewing and approving revisions to preferred air
quality dispersion models, developing QAPPs or equivalent documents to
describe results of systematic planning for air quality dispersion model revisions,
updating the Office of Air Quality Planning and Standards' Quality Management
Plan, and training staff. The agency agreed with our recommendations and
provided acceptable corrective actions.
Listing of OIG reports.

-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
THE INSPECTOR GENERAL
September 5, 2018
MEMORANDUM
SUBJECT: EPA Can Strengthen Its Process for Revising Air Quality Dispersion Models
that Predict Impact of Pollutant Emissions
Report No. 18-P-0241
FROM: Arthur A. Elkins Jr.
TO:
William Wehrum, Assistant Administrator
Office of Air and Radiation
This is a final report on the subject audit conducted by the Office of Inspector General (OIG) of the
U.S. Environmental Protection Agency (EPA). The project number for this audit was OPE-FY17-0016.
This report contains findings that describe the problems the OIG has identified and corrective actions the
OIG recommends.
In accordance with EPA Manual 2750, your office provided acceptable corrective actions and milestone
dates in response to OIG recommendations. All recommendations are resolved and no final response to
this report is required. However, if you submit a response, it will posted on the OIG's website, along
with our memorandum commenting on your response. Your response should be provided as an Adobe
PDF file that complies with accessibility requirements of Section 508 of the Rehabilitation Act of 1973,
as amended. The final response should not contain data that you do not want to be released to the public;
if your response contains such data, you should identify the data for redaction or removal along with
corresponding justification.
We will post this report to our website at www.epa.gov/oig.

-------
EPA Can Strengthen Its Process for
Revising Air Quality Dispersion Models
that Predict Impact of Pollutant Emissions
18-P-0241
Table of C
Chapters
1	Introduction		1
Purpose		1
Background		1
Responsible Office		5
Scope and Methodology		5
2	Controls Should Be Strengthened to Improve
Review and Approval Process for Preferred Models		7
Model Evaluation Determines the Quality of a Model		7
Model Evaluation Activities Varied for Model Revisions		8
AQMG Lacks SOPs for Review and Approval Process for
Revising Preferred Models 		10
AQMG Did Not Develop a QAPP When It Revised AERMOD		11
Development of SOPs and QAPPs Provides Better Assurance
that Modeling Results Are of Sufficient Quality		12
Conclusion		13
Recommendations		13
Agency Comments and OIG Evaluation		14
Status of Recommendations and Potential Monetary Benefits		15
Appendices
A Agency Comments on Draft Report and OIG Evaluation	 16
B Distribution	 21

-------
Chapter 1
Introduction
Purpose
We conducted this audit to assess the effectiveness of the U.S. Environmental
Protection Agency's (EPA's) process for reviewing and approving air quality
dispersion models that it recommends for use by state, local and tribal air
pollution control agencies.
Background
Air quality models are tools that predict the fate1 of pollutants upon their release
into the atmosphere. Air quality modeling uses include:
•	Estimating the impact of emissions from new sources and from
modifications to existing sources on the surrounding air quality.
•	Predicting future pollutant concentrations from multiple sources after
implementation of a new regulatory program.
Air quality dispersion modeling uses mathematical formulations to characterize
the atmospheric processes that disperse a pollutant emitted by a source. Based on
emissions and meteorological inputs, a dispersion model can be used to predict
concentrations at selected downwind locations.
The Clean Air Act includes the following requirements related to air quality models:
•	Section 103 requires the EPA to conduct a program that includes the
development of methods for modeling of air pollutants.
•	Section 165(e)(3)(D) requires the EPA to adopt regulations specifying
with reasonable particularity models to be used to comply with the act's
Prevention of Significant Deterioration requirements. The EPA has met
this requirement through publication of the Guideline on Air Quality
Models as Appendix W to 40 CFR Part 51.
•	Section 320 requires the EPA to conduct a conference on air quality
modeling at least once every 3 years. The EPA has held 11 such
conferences since 1977.
1 Fate refers to the predicted future state of pollutants. The concentration of pollutants may become diluted downwind of
the source, or undergo chemical and/or physical transformations after being released into the atmosphere.
18-P-0241
1

-------
Process of Modeling for Environmental Decision-Making
Modeling for environmental decision-making follows the four basic phases shown
in Figure 1:
Figure 1: Environmental decision-making phases
Problem Identification
This phase identifies the problem that the model is intended to address and
establishes modeling objectives.
Model Development
This phase develops the conceptual model reflecting the underlying science of
the process being modeled and develops a computational model to represent
the underlying science in a computer program.
Model Evaluation
This phase tests the model to determine whether model expressions are
correctly encoded in the computer program, and to test the model outputs by
comparing them with empirical data.
Model Application
The model is run and the results analyzed to inform a decision.
Source: EPA Office of Inspector General (OIG) image developed from the EPA's Guidance
on the Development, Evaluation, and Application of Environmental Models (CREM guidance),
EPA/100/K-09/003 (2009).
Appendix W Provides Guidelines for Consistent Application of Models
To provide consistency in the application of air quality models for regulatory
purposes in accordance with Section 165(e)(3)(D) of the Clean Air Act, the EPA
publishes the Guideline on Air Quality Models, which is codified as Appendix W
to 40 CFR Part 51, and is known simply as Appendix W.
Appendix W was originally promulgated in 1978 and was most recently revised in
2017. Appendix W is used by the EPA, states, tribes and industry to prepare and
review permits for new or modified sources of air pollution. State and tribal air
agencies also use Appendix W to revise their plans detailing strategies for
reducing emissions and improving air quality known as State or Tribal
18-P-0241
2

-------
Implementation Plans. Appendix A of Appendix W includes a list of those models
that the EPA has determined are "preferred" for the various regulatory uses.
Appendix W also defines the process that the EPA uses to determine the best
performing model, which is then designated as the agency's preferred model for
air quality modeling analyses. As noted in Appendix W, the EPA has developed
some models suitable for regulatory application, while other models have been
submitted by private developers for possible inclusion in Appendix W.
Appendix W further notes that refined models that are preferred and required by
the EPA for particular applications are to go through the necessary peer scientific
reviews and model performance evaluation exercises, including statistical
measures of model performance in comparison with measured air quality data.
When a single model is found to perform better than others, it is recommended for
application as a preferred model and listed in Appendix W.
AERMOD a Preferred Model Listed in Appendix W
Appendix W lists three2 preferred air quality models. Of those three, the American
Meteorological Society/EPA Regulatory Model (AERMOD) is the one used for the
majority of regulatory uses identified in Appendix W.
AERMOD is a steady-state Gaussian plume
dispersion model3 that predicts the transport and
dispersion of pollutants up to 50 kilometers
downwind of pollutant sources. AERMOD resulted
from a joint effort by the American Meteorological
Society and the EPA. The effort was initiated to build
upon earlier model developments and provide a
state-of-the-art dispersion model for regulatory
applications.
In 2005, the EPA designated AERMOD as a preferred model for determining air
quality impacts of air pollutant emissions from stationary sources, such as point,
volume and area sources.4 Figure 2 provides a timeline of the events concerning
AERMOD.
2	The other two preferred air quality dispersion models are the Complex Terrain Dispersion Model Plus Algorithms
for Unstable Situations and the Offshore and Coastal Dispersion Model.
3	A Gaussian plume dispersion model assumes that the pollutant concentration distribution in the emission plume is
a normal probability distribution.
4	According to the Air Quality Modeling Group, stationary sources can be characterized in AERMOD as point,
volume or area sources. Point sources are well-defined stacks, chimneys or pipe vents that are considered to have a
single point of release. Area sources are typically low-level, non-buoyant releases considered to represent uniform
fugitive emissions over a large area. Examples of area sources are storage piles and lagoons. Volume sources could
be considered similar to area sources except that they have a vertical dimension at the time of the release due to
buoyancy (temperature), turbulence or the nature of the release. Examples of volume sources are building roof
monitors and drop points from loaders.
AERMOD Modeling Uses
AERMOD is used to model
the impact on air quality from
sources that emit a variety of
pollutants regulated by the
EPA. These include common
pollutants, such as carbon
monoxide, lead, sulfur
dioxide, nitrogen dioxide and
primary particulate matter;
and hazardous air pollutants,
also known as air toxics.
18-P-0241
3

-------
Figure 2: The Timeline for the AERMOD Model
Final rule promulgated with AERMOD
designated as a preferred model
(replacing ISCST3 model) in Appendix W.
2002
1998
1996
2000
2004
2006
2008
2010
2016 2018
2012 2014
Initial version of
AERMOD
released (1995)
Rule proposed
to replace
ISCST3 model
with AERMOD
in Appendix W.
Final rule promulgated
revising Appendix W that
included updates to
AERMOD.
Rule proposed to revise
Appendix W that included
updates to AERMOD.
Modeling conference held to
discuss the development of
proposed revisions to
Appendix W with stakeholders.
EPA conducted formal peer
review and model evaluations
to inform the 2000 proposed
rule.
EPA issued 12 Model Change Bulletins from 2006 to 2016 that included
enhancements and bug fixes to AERMOD. Consequence analyses were
performed for revisions that involved formulation or code changes.
Note: Modeling evaluations were conducted between 1994 and 2005 and between 2013 and 2017.
Modeling conferences were held in 1995, 2000, 2005, 2008, 2012 and 2015. During the 2000 and 2015
modeling conferences, the EPA provided an opportunity for public comment on the proposed Appendix W
revisions.
ISCST3: Industrial Source Complex - Short Term Model.
Source: OIG analysis of EPA documents pertaining to the development and revision of AERMOD.
AERMOD Uses
States and local air pollution control agencies rely heavily on AERMOD to make
regulatory decisions on such matters as:
•	New Source Review/Prevention of Significant Deterioration permit
applications and State and Tribal Implementation Plans. AERMOD is used
in all 50 states for conducting compliance demonstrations under the New
Source Review permit program.
•	Designating areas as in attainment or nonattainment of the 1-hour sulfur
dioxide (SO2) National Ambient Air Quality Standards (NAAQS). On
August 21, 2015, the EPA issued the final Data Requirements Rule5 for the
2010 1-Hour SO2 Primary NAAQS to inform the designations process. This
gave air pollution control agencies the flexibility to characterize air quality
through monitoring or modeling of ambient SO2 levels in areas with large
sources of SO2 emissions. According to the EPA, 41 states, two territories
and one tribal nation are using AERMOD to model SO2 sources in their
jurisdiction to satisfy the requirements of the Data Requirement Rules.
5 Data Requirements Rule for the 2018 1-Hour SO2 Primary NAAQS Final Rule, 80 Fed. Reg. 51,052
(Aug. 21, 2015).
18-P-0241
4

-------
• Federally supported highway and transportation project activities. The
2017 revision to Appendix W replaced the CALINE3 model with
AERMOD as the preferred model for mobile
sources. There is a 3-year transition period that
ends on January 17, 2020, for transportation
conformity determinations. Any modeling
analyses started before the end of this 3-year
transition period with CALINE3 can be
completed after the end of the transition peri od.
Also, the 2017 revision to Appendix W replaced
the Buoyant Line and Point Source model with
AERMOD as the preferred model for buoyant,
elevated line sources,6 such as aluminum
reduction plants.
Responsible Office
The EPA office primarily responsible for the review and approval of air quality
dispersion models is the Air Quality Modeling Group in the Office of Air Quality
Planning and Standards (OAQPS) within the EPA's Office of Air and Radiation.
Scope and Methodology
We conducted our performance audit from June 2017 through June 2018,
in accordance with generally accepted government auditing standards. Those
standards require that we obtain sufficient, appropriate evidence to provide a
reasonable basis for our findings and conclusions based on our objective. We
believe the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objective.
Our audit focused primarily on the revi ew and approval process for AERMOD
because AERMOD is used for the most regulatory purposes compared to the other
two preferred models. Our audit included a review of the process used by the EPA
to revise AERMOD in the Appendix W revisions in 2017. We also reviewed the
EPA's 12 Model Change Bulletins, which included bug fixes, enhancements,
and/or miscellaneous changes to AERMOD. We also reviewed the Appendix W
revisions concerning AERMOD.
To address our objective, we identified and reviewed applicable statutes,
regulations, policies and guidance. To help us determine what actions the EPA
has taken to evaluate AERMOD as well as the adherence to the EPA's Quality
System, we obtained and reviewed Appendix W; AERMOD evaluations;
6 Examples of line sources are roadways and streets along which there are well-defined movements of motor
vehicles, and lines of roof vents or stacks such as in aluminum refineries.
Highway traffic. AERMOD is the preferred model for
estimating the impact of mobile source emissions on
air quality. (U.S. Department of Transportation
Federal Highway Administration photo)
18-P-0241
5

-------
AERMOD consequence analysis reports; AERMOD Technical Support
Documents; NAAQS Technical Assistance Documents; modeling guidance;
modeling conference transcripts; clarification memorandums; EPA Quality
System policy, procedures and guidance documents; and the regulatory docket
associated with the 2017 Appendix W revisions. We compared these documents
to the model development and revision processes described by the agency and
found in the EPA's model development and evaluation guidance documents.
We interviewed EPA staff and managers in the OAQPS, Region 1, Region 4
(which covers North Carolina), Region 5 (which covers Indiana), and a Region 10
water quality modeler. In addition, we interviewed an EPA Office of
Environmental Information manager and staff to learn more about the EPA's
Quality System and how it relates to the development and revisions of air quality
dispersion models. We interviewed a former EPA employee to gain an
understanding of EPA actions to evaluate AERMOD. We also interviewed staff
from the North Carolina Department of Environmental Quality, the Indiana
Department of Environmental Management, and the Association of Air Pollution
Control Agencies.
To assess internal controls, we reviewed EPA policies and guidance on the EPA's
Quality System, including:
•	Policy and Program Requirements for the Mandatory Agency-wide
Quality System, EPA Order CIO 2105.0 (2000).
•	EPA Quality Manual for Environmental Programs, CIO 2105-P-01-0 (2000).
•	Guidance for Quality Assurance Project Plans for Modeling,
EPA QA/G-5M (2002).
•	Guidance on Systematic Planning Using the Data Quality Objectives
Process, EPA QA/G-4 (2006).
•	Guidance for Preparing Standard Operating Procedures (SOPs),
EPA QA/G-6 (2007).
•	U.S. Environmental Protection Agency Quality Policy, EPA CIO Order
2106.0 (2008).
•	Procedure for Quality Policy, CIO 2106-P-01.0 (2008).
•	Guidance on the Development, Evaluation, and Application of
Environmental Models (CREM guidance), EPA/100/K-09/003 (2009).
•	The OAQPS Quality Management Plan (2015).
We compared these Quality System policy, procedure and guidance documents to
the agency's activities described in interviews, as well as the documents produced
during the AERMOD development and revision processes. We also analyzed
these documents to determine the Quality System requirements that the agency
had to follow during model development and revision processes.
We did not independently evaluate or assess the accuracy of AERMOD.
18-P-0241
6

-------
Chapter 2
Controls Should Be Strengthened to Improve
Review and Approval Process for Preferred Models
Although the agency has prepared guidance on the recommended procedures for
developing and evaluating new air quality dispersion models, similar guidance is
not available to define the process for model revisions. The model evaluation
activities conducted to assess the revisions to AERMOD were not always as
extensive as what EPA guidance requires for new models. The EPA could
strengthen the model revision process by developing:
•	Standard operating procedures (SOPs) to assure consistency in the
development, evaluation and approval of revisions to existing models.
•	Model-specific quality assurance project plans (QAPPs) or equivalent
documents defining the activities that should be conducted to assure the
desired quality of results when developing or revising the preferred model.
Without SOPs and QAPPs, the agency lacks the assurance that all necessary
technical activities have been completed to produce a preferred model that generates
results of sufficient quality for its intended uses. This is especially important because
AERMOD is used by all 50 states, as well as tribes and territories, to predict air
quality impacts for regulatory purposes under the Clean Air Act.
Model Evaluation Determines the Quality of a Model
As described in the EPA's CREM guidance, model evaluation is the process for
generating information over the project's life cycle that helps determine whether a
model and its results are of sufficient quality to serve as the basis for a decision.
Model quality is an attribute that is meaningful only within the context of a specific
application. Information from a model evaluation helps to answer the following
questions:
•	How have sound science principles been addressed during model
development?
•	How is the model choice supported by the quantity and quality of
available data?
•	How well does the model approximate the real system of interest?
•	How well does the model perform the specified task while meeting the
objectives established under quality assurance project planning?
18-P-0241
7

-------
Model Evaluation Activities Varied for Model Revisions
As described in the EPA's CREM guidance, the recommended components of the
model evaluation process include:
a.	Credible, objective peer review.
b.	QAPP and data quality assessment.
c.	Qualitative and/or quantitative model corroboration
d.	Sensitivity and uncertainty analyses.
We noted that the process for evaluating AERMOD revisions was not as extensive
as that recommended for new model development. For example, AERMOD
received a panel peer review when it was developed. However, the EPA's Air
Quality Modeling Group (AQMG) used peer-reviewed journal articles to support
AERMOD revisions. In one instance, the EPA proposed a new regulatory option
for AERMOD that had not been peer reviewed prior to its proposal. Specifically,
the EPA proposed a change in Appendix W concerning a low-wind option in
AERMOD to address issues with the model's tendency to over-predict during low
wind conditions. In its response to public comments on the proposed rule, EPA
acknowledged it lacked published, peer-reviewed literature with supporting model
evaluations that fully addressed the concerns for the low-wind option literature and
acknowledged public commenter concerns regarding the available model
evaluations of this low wind option. As a result, the
EPA did not include this proposed revision in the final
2017 version of Appendix W.
AERMOD was corroborated with 14 field databases
before the Appendix W revisions were finalized in
2017. Initial development of AERMOD in the
1990s included corroborating the model with
17 different field databases with varying conditions.
Fourteen of these 17 field databases were used to
corroborate the 2017 Appendix W revisions.
From 2006 to 2016, the EPA issued 12 Model Change Bulletins to revise
AERMOD. Model Change Bulletin 12 was associated with the Appendix W
revisions in 2017. Nine of the bulletins contained enhancements.7 Enhancements
add new functionality or change existing functionality in a way that makes
AERMOD more efficient, usable and useful.
During the first 10 times AERMOD was undergoing revisions, AQMG conducted
consequence analyses for all but three AERMOD revisions. According to AQMG,
the three AERMOD versions that did not undergo consequence analysis did not
1
These enhancements included options for estimating the conversion of nitrogen oxide emissions to ambient
nitrogen dioxide concentrations, and allowing users to specify background ozone concentrations.
Power plant in Poca, West Virginia. (EPA photo)
18-P-0241
8

-------
contain formulation updates or code changes that would affect modeled
concentrations. Consequence analyses assess the change in predicted pollutant
concentrations between the revised and current AERMOD versions.
Table 1 summarizes the revisions to AERMOD from 2006 to 2016.
Table 1: Changes made to AERMOD summarized in 12 Model Change Bulletins
Model Change

Number of changes by type
Bulletin




number
Date
Bug fixes3
Enhancements15'd
Miscellaneous0
1
12/07/2006
9
6
5
2
1/26/2007
9
0
3
3
10/19/2009
24
13
31
4
2/28/2011
7
12
6
5
4/13/2011
1
0
1
6
12/19/2011
5
0
0
7
2/29/2012
3
2
2
8
12/10/2012
6
2
2
9
12/16/2013
10
3
5
10
5/14/2014
21
5
8
11
6/30/2015
12
7
1
12
8/03/2016
10
6
1
Totals

117
56
65
Note a: A bug fix is a fix to a software bug, which is an error, flaw, failure or fault in a computer
program or system that causes it to produce an incorrect or unexpected result, or to behave in
unintended ways.
Note b: An enhancement adds new functionality or changes existing functionality in a way that
makes the software application more efficient, usable and useful.
Note c: Miscellaneous changes are other changes that are neither bug fixes nor enhancements,
such as including potential downwash effects for stack heights that equal or exceed the EPA
formula height.
Note d: New options that represent scientific formulation changes have been added as Beta
options as different versions of the model have been released over the years. Beta options were
included in the "enhancements" section of Model Change Bulletins.
Source: OIG analysis of EPA's Model Change Bulletins for AERMOD.
The extent of quality control and assurance activities needed for a model revision
project could reasonably vary based on the nature of the revision. However, due
to a lack of documentation describing (1) the basic procedures for revisions and
(2) the results of systematic planning describing the detailed model evaluation
activities needed for a specific model's revisions, there is a lack of assurance that
the appropriate steps were taken to assure the desired quality of the revision.
18-P-0241
9

-------
AQMG Lacks SOPs for Review and Approval Process for
Revising Preferred Models
The AQMG lacks SOPs for the review and approval process for revising preferred
models. In addition to AERMOD, the EPA oversees revisions to two other
preferred air quality dispersion models: the Complex Terrain Dispersion Model
Plus Algorithms for Unstable Situations and the Offshore and Coastal Dispersion
Model. These models were developed in the 1980s and are used infrequently
compared to AERMOD. According to the AQMG manager, it is likely that one of
these models will be incorporated into AERMOD at a later date.
Since AERMOD has been frequently revised, the AQMG manager indicated that
the agency would benefit from the development of SOPs to assure consistency in
the process. These SOPs could lay out what procedures need to be completed
when a model is revised, such as:
•	Determining whether the revision requires a QAPP.
•	Determining whether consequence analyses need to be completed.
•	Determining what type of model evaluations are needed for revision.
•	Determining when and what type of peer review is needed.
•	Determining how code will be verified.
The Guidance for Preparing Standard Operating Procedures (SOPs), developed
by the EPA's Office of Environmental Information, states that the development
and use of SOPs are an integral part of a successful quality system, as they
provide staff with the information to perform a job properly and facilitate
consistency in the quality and integrity of a product or end-result. SOPs detail the
regularly recurring work processes to be followed within an organization. SOPs
document the way activities are to be performed to facilitate conformance to
technical and quality system requirements, and to support data quality. Further,
The OAQPS Quality Management Plan states that SOPs should ensure consistent
conformity with organizational practices; serve as training aids; provide ready
reference of proper procedures; reduce work effort; reduce data error occurrences;
and improve data comparability, credibility and defensibility.
The EPA has started to develop SOPs for changing AERMOD's source code.
In March 2017, the AQMG developed a draft AERMOD System Update
Checklist to provide a process to complete before changing the AERMOD source
code. While this checklist does provide SOPs for changing the model code when
the model is revised, additional SOPs are needed to define the general process of
developing and revising models.
18-P-0241
10

-------
AQMG Did Not Develop a QAPP When It Revised AERMOD
The AQMG did not develop a QAPP or similar planning documents describing
the results of systematic planning to guide AERMOD revisions. These plans
would include a description of the extent and type of model evaluation activities
needed to determine the quality of the model revision based on the results of
systematic planning. We also noted that a QAPP was not prepared to describe the
systematic planning and model evaluation process when AERMOD was initially
developed in the 1990s. Such a plan, with supplemental language added as
necessary, could have served as a guide for future enhancements to the model.
The EPA's I'olicy and Program Requirements for the Mandatory Agency-wide
Quality System establishes the minimum requirements for quality systems
supporting EPA environmental programs that encompass the collection,
evaluation and use of environmental data by or for the EPA, which would be
applicable to the development and revision of environmental models. The
agency's Quality System requirements include the use of a systematic planning
approach to develop performance criteria for all work covered by the agencywide
Quality System, and the development of approved QAPPs or equivalent
documents for all applicable projects and tasks involving environmental data.
The OAQPS Quality Management Plan8 requires the use of QAPPs for modeling.
The OAQPS uses a graded approach to determine the appropriate level for its
projects: Categories I to IV. Category I projects include monitoring, modeling
and/or analyses involving Prevention of Significant Deterioration. AERMOD is
used in modeling for Prevention of Significant Deterioration permit applications.
OAQPS personnel noted that they developed numerous documents in support of
revisions to AERMOD, and that, in their view, these documents are equivalent to a
QAPP and demonstrate evidence that their process complies with EPA
requirements for systematic planning. These documents included the following:
•	EPA's Guideline on Air Quality Models.
•	AERMOD User's Guide.
•	AERMOD Model Formulation and Evaluation Document.
•	AERMOD Implementation Guide.
•	Consequence analyses.
•	Model Change Bulletins.
•	Technical Support Documents.
•	Option-specific evaluation documents.
•	Evaluation databases.
8 The EPA Quality Manual for Environmental Programs requires each EPA organization unit to document its quality
system in a quality management plan.
18-P-0241
11

-------
The EPA's Policy and Program Requirements for the Mandatory Agency-wide
Quality System allows the use of equivalent planning documents in lieu of a
QAPP as long as the documents are defined in an office's quality management
plan. However, The OAQPS Quality Management Plan does not state whether the
above-listed documents are equivalent to a QAPP and does not define the
systematic planning process used to revise AERMOD. Thus, it was not clear from
the documentation we reviewed how the EPA's process for revising AERMOD
complies with the systematic planning process required by the EPA's Quality
System.
Development of SOPs and QAPPs Provides Better Assurance that
Modeling Results Are of Sufficient Quality
The development of SOPs and QAPPs to guide model revisions can better assure
that the model produces data sufficient for its intended use. Air quality dispersion
models generate results that are used to make regulatory decisions. Therefore, it is
important the models do not considerably over- or underestimate ambient air
concentrations.
A potential impact of models that overestimate ambient air concentrations is:
•	Establishment of facility emission limits that are more stringent than
required. This in turn could increase facility construction costs to
implement pollution controls that may not be needed.
Potential impacts of models that underestimate ambient air concentrations include:
•	Establishment of facility emissions limits that are not stringent enough to
sufficiently protect the public from exposure to harmful air pollutants.
•	Inaccurate determinations that areas are complying with ambient air
quality standards. This would result in the public being exposed to harmful
air pollutants in concentrations above what the EPA considers safe, thus
increasing public health risk.
The EPA received comments expressing concern as to whether AERMOD was
sufficiently evaluated for some of its proposed uses during the 2015 comment
period for the proposed rulemaking to revise Appendix W. For example, the
American Association of State Highway and Transportation Officials
recommended that the EPA consider the results from other peer-reviewed studies
in the literature and perform additional model comparison studies for a range of
conditions before replacing one model with another. The EPA responded in the
preamble to the 2017 final rulemaking on revisions to Appendix W that it has
reviewed the available literature and conducted its own analysis that demonstrates
18-P-0241
12

-------
AERMOD provides superior performance to that of CALINE3 for refined
applications
Conclusion
SOPs are intended to minimize variation and promote quality through consistent
implementation of a process or procedure within the organization. Due to the
frequency of revisions and enhancements to existing air quality dispersion models,
SOPs would help assure that the development, evaluation and approval of these
revisions meet minimum requirements for this process. At the project level, QAPPs
document the results of systematic planning for an environmental data project to
assure the project produces results of sufficient quality for the intended use. For
models, this planning, which should be included in a QAPP, should address the
nature and extent of model evaluation activities needed to determine whether the
model produces the desired quality of results.
Assuring the consistency and quality of air quality dispersion model revisions for
AERMOD is important because the model is used by all 50 states, as well as tribes
and territories, to predict air quality impacts for regulatory purposes under the
Clean Air Act. AERMOD is also used to assess compliance with ambient air
quality standards for SO2 designation decisions and to evaluate the impact of
emission control strategies for State and Tribal Implementation Plans.
Recommendations
We recommend that the Assistant Administrator for Air and Radiation:
1.	Develop standard operating procedures for the review and approval
process for revising preferred air quality dispersion models.
2.	Develop a quality assurance project plan or equivalent documents
describing the results of systematic planning before developing a new air
quality dispersion model or undertaking any significant revisions in the
future to existing preferred air quality dispersion models, which are
codified in Appendix A to Appendix W of 40 CFR Part 51.
3.	Revise the Office of Air Quality Planning and Standards' Quality
Management Plan to state whether the agency is developing quality
assurance project plans or equivalent documents to meet EPA Quality
System requirements for developing or revising preferred air quality
dispersion models.
4.	Train the Air Quality Modeling Group staff concerning the standard
operating procedures of preferred air quality dispersion model review and
approval and EPA Quality System requirements.
18-P-0241
13

-------
Agency Comments and OIG Evaluation
The agency concurred with the recommendations and provided acceptable planned
corrective actions and completion dates. All recommendations are resolved with
correction actions pending. In addition to a response to our recommendations, the
agency provided technical comments on the draft report. Based on the agency
response and technical comments received, we revised the report where
appropriate.
The agency's comments are in Appendix A.
18-P-0241
14

-------
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
Rec.
No.
No.
Subject
Status1
Action Official
13 Develop standard operating procedures for the review and
approval process for revising preferred air quality dispersion
models.
Assistant Administrator for
Air and Radiation
Planned
Completion
Date
Potential
Monetary
Benefits
(in $000s)
9/30/18
13 Develop a quality assurance project plan or equivalent
documents describing the results of systematic planning before
developing a new air quality dispersion model or undertaking any
significant revisions in the future to existing preferred air quality
dispersion models, which are codified in Appendix A to
Appendix W of 40 CFR Part 51.
13 Revise the Office of Air Quality Planning and Standards' Quality
Management Plan to state whether the agency is developing
quality assurance project plans or equivalent documents to meet
EPA Quality System requirements for developing or revising
preferred air quality dispersion models.
13 Train the Air Quality Modeling Group staff concerning the
standard operating procedures of preferred air quality dispersion
model review and approval and EPA Quality System
requirements.
Assistant Administrator for 3/31120
Air and Radiation
Assistant Administrator for 3/31 /20
Air and Radiation
Assistant Administrator for 9/30/19
Air and Radiation
1 C = Corrective action completed.
R = Recommendation resolved with corrective action pending.
U = Recommendation unresolved with resolution efforts in progress.
18-P-0241	15

-------
Appendix A
Agency Comments on Draft Report
and OIG Evaluation
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
July 5, 2018
OFFICE OF
AIR AND RADIATION
MEMORANDUM
SUBJECT: Response to Office of Inspector General Draft Report, "EPA Can Strengthen Its
Process for Revising Air Quality Dispersion Models That Predict Impact of
Pollutant Emissions" - Project No. OPE-FY17-0016
INTRODUCTION
EPA's Office of Air and Radiation (OAR) welcomes the opportunity to review and comment on
the Office of Inspector General's (OIG) draft report titled EPA Can Strengthen Its Process for
Revising Air Quality Dispersion Models That Predict Impact of Pollutant Emissions (Draft
Report). In general, we appreciate the observations and recommendations the OIG has provided
in the Draft Report that underscore the need for more formal development and documentation of
quality assurance (QA) measures and standard operating procedures (SOPs) for the review and
approval of new or revision of existing preferred air quality dispersion models. We believe that
such improvements will strengthen the agency's air quality modeling program.
As means of background, we wish to note that our air quality modeling program stems from the
statutory requirements of Section 165 of the Clean Air Act (CAA), which states that "The
Administrator... shall specify with reasonable particularity each air quality model or models to
be used under specific sets of conditions for the purposes of this part..." To satisfy this
congressional mandate, EPA established the Guideline on Air Quality Models (Appendix W to
40 CFR part 51, or Guideline) in 1978, which includes the review and approval approach that
18-P-0241	16
FROM: William L. Wehrum
Assistant Administrator
TO:
Kevin Christensen, Assistant Inspector General
Office of Audit and Evaluation

-------
EPA must take in determining preferred models for use in regulatory air quality programs under
the CAA and a summary of EPA's promulgated preferred models. Section 320 of the CAA also
requires EPA to conduct a conference on air quality models at least once every 3 years to ensure
ongoing formal public engagement, review, and comment on the existing preferred air quality
models and future air quality model development needs necessary for various regulatory
applications and compliance demonstrations.
EPA's preferred dispersion air quality model for many applications is the American
Meteorological Society/EPA Regulatory Model (AERMOD) Modeling System, which was
promulgated in 2005 after extensive peer-review and formal public review and comment. EPA
established the applicability and suitability of AERMOD through federal rulemaking under the
Action Development Process (ADP) that included full documentation and clear demonstration
that the modeling system met the criteria as specified for preferred and alternative models in the
Guideline. Aside from usability enhancements that make AERMOD more efficient for regulatory
applications and "bug fixes" that address identified computer coding errors, the only significant
revision of AERMOD occurred in 2017 through the federal rulemaking process including public
review and comment, and based upon peer-reviewed scientific research of those formulation
changes.
The Air Quality Modeling Group (AQMG) in OAR's Office of Air Quality Planning and
Standards (OAQPS) has been responsible for the Guideline since the late 1970s and its periodic
revisions that include development and updates to preferred air quality models. Based on this
long-standing role and responsibility, the AQMG manager has served as the OAR representative
on EPA's Council for Regulatory Environmental Modeling (CREM) and assisted in the
development of the 2002 Guidance for Quality Assurance Project Plans for Modeling and the
2009 Guidance on Development, Evaluation, and Application of Environmental Models. These
guidance documents and others related to procedures for model development and updates have
benefited from the AQMG's experience under the Guideline and the approaches in determining
preferred air quality models. This background discussion is intended to highlight our statutory
directive, current practice, and long-standing role in ensuring that EPA's air quality modeling
program meets the needs under the CAA and adheres to appropriate and necessary QA measures
and SOPs.
RESPONSES TO RECOMMENDATIONS
Consistent with our interest in continuously improving our air quality modeling program, OAR
welcomes the observations and recommendations OIG has provided in the Draft Report. OAR's
responses to OIG's specific recommendations follow.
OIG Recommendation 1:
Develop standard operating procedures for the review and approval process for revising
preferred air quality dispersion models.
EPA Response 1:
Throughout the development history of AERMOD, EPA has followed a systematic approach to
model development that seeks to ensure that model enhancements and updates are technically
18-P-0241
17

-------
and scientifically appropriate. This approach was acknowledged by the OIG report as they
examined and reported on the panel peer review for the initial model promulgation, the peer
reviewed journal articles, model evaluations based on field datasets, and the consequence
analyses performed for each model update that resulted in changed concentrations. However,
EPA acknowledges that these procedures were not codified in written SOPs or other documents,
but were instead historically completed at the discretion of AQMG's Dispersion Modeling Team
(DMT).
In response to this recommendation, EPA proposes the following three actions:
1.	EPA will develop internal SOPs for logging bugs, enhancements, and planned
formulation updates, handling code, evaluating model code changes, and acquiring
approval for releasing updated code to the public.
2.	As part of the SOPs, EPA will include a management review and approval process, which
will include a management approval form. This approval form will be archived internally
to document adherence to systematic planning for model development and update
activities.
3.	EPA will provide a general summary of the internal SOPs along with a broad explanation
of our approach to model updates in an upcoming release of a report titled AERMOD
System Development and Update Plan.
Planned Completion Date:
1.	Complete. EPA completed formal documentation of a set of SOPs earlier this year. While
these are continuously being evaluated for improvements, they were formally adhered to
as part of EPA's release of AERMOD version 18081 on April 24, 2018.
2.	Complete. Our current SOPs include management review and signoff of model code prior
to release. This was included with the release of AERMOD version 18081 and the review
and approval form has been archived internally.
3.	In progress. Expected release date of September 30, 2018. A draft of the AERMOD
System Development and Update Plan is currently under review by the AQMG Manager.
We expect to update this plan annually to reflect changes in new model releases and new
directions in model improvement efforts for future updates to the model.
OIG Response #1: The agency concurred with the recommendation and provided acceptable
planned corrective actions and completion dates. Recommendation 1 is resolved.
OIG Recommendation 2:
Develop a quality assurance project plan or equivalent documents describing the results of
systematic planning before developing a new air quality dispersion model or undertaking any
significant revisions in the future to existing preferred air quality dispersion models, to include
any revisions to preferred models in EPA's Guideline on Air Quality Models, which is codified
as Appendix A to Appendix W of 40 CFR part 51.
EPA Response 2:
In response to this recommendation, EPA proposes the following action:
18-P-0241
18

-------
1.	EPA will develop the AERMOD System Development and Update Plan. The plan serves
several functions. In addition to generally describing the SOPs for model development,
the plan will provide detailed descriptions of the model development and update process
outlined in EPA's Guideline which relies upon EPA's ADP requirements for the federal
rulemaking process. The ADP process provides a robust process and documentation that
ensures quality of its regulatory actions such that the model development and update
process meets EPA's Quality System requirements.
2.	As noted in the OIG report, EPA provides extensive documentation on model
performance, function, and application (e.g., the AERMOD User's Guide, the AERMOD
Formulation and Evaluation Document, and the AERMOD Implementation Guide). We
believe these documents provide the documentation necessary to meet EPA's Quality
System requirements. The connections between these documents and these requirements
will be spelled out in updates to the OAQPS QMP (see response to recommendation 3).
Planned Completion Date:
1.	In progress. Expected release date of September 30, 2018.
2.	In progress. Please refer to planned completion of this task under recommendation 3.
OIG Response #2: The agency provided two corrective actions to address the
recommendation. We met with the agency to clarify its corrective actions. During our meeting,
AQMG managers committed to documenting the systematic planning process for developing
air quality dispersion models and revising existing preferred air quality dispersion models in
its revised Quality Management Plan as well as its AERMOD System Development and Update
Plan. We accept the EPA's corrective actions as meeting the intent of our recommendation.
Recommendation 2 is resolved.
OIG Recommendation 3:
Revise the OAQPS QMP to state whether the agency is developing quality assurance project
plans or equivalent documents to meet EPA Quality System requirements for developing or
revising preferred air quality dispersion models.
EPA Response 3:
In response to this recommendation, EPA proposes the following action:
1. The AQMG Manager will coordinate with the OAQPS QA Manager to modify the
OAQPS QMP so that it clearly states how the process for developing and revising
preferred air quality models is conducted and adheres to EPA Quality System
requirements.
Planned Completion Date:
1. In progress. The OAQPS QMP was last issued in 2015 and is revised every 5 years. EPA
will provide the requested update to the QMP with its currently scheduled release by
March 31, 2020.
OIG Response #3: The agency concurred with the recommendation and provided acceptable
planned corrective actions and completion dates. Recommendation 3 is resolved.
18-P-0241
19

-------
OIG Recommendation 4:
Train the Air Quality Modeling Group staff concerning the standard operating procedures of
preferred air quality dispersion model review and approval and EPA Quality System
requirements.
EPA Response 4:
OAR agrees with this recommendation and recognizes the importance of ensuring the AQMG
staff (now and into the future) are fully aware of and adhere to the group's internal SOPs for
model development and updates and are fully understanding of EPA's Quality System
requirements and related guidance for model development and updates.
In response to this recommendation, EPA proposes the following action:
1. The AQMG Manager will work with staff in the DMT to define the annual review of
SOPs by the team and coordinate with the OAQPS QA Manager to identify the
appropriate training materials on the EPA Quality System requirements.
Planned Completion Date:
1. In progress. Definition of training materials will be defined by September 30, 2018,
codified in the FY2019 Performance Appraisal and Recognition System (PARS) for each
staff in the DMT, and completed by September 30, 2019.
OIG Response #4: We met with the agency to clarify the corrective action. During our
meeting, the AQMG manager committed to AQMG staff receiving SOP training. This
requirement will be included in AQMG staffs Fiscal Year 2019 Performance Appraisal and
Recognition System agreements. The AQMG manager also committed to work with OAQPS'
Central Operations and Resources Office to look for training courses on EPA's Quality
System that are available and appropriate to also include as part of the staffs Fiscal Year
2019 Performance Appraisal and Recognition System agreements. The identified courses will
be required training. We accept the EPA's corrective actions as meeting the intent of our
recommendation. Recommendation 4 is resolved.
If you have any questions regarding this response, please contact Tyler Fox, Group Leader,
AQMG, OAQPS, at (919) 541-5562, or at fox.tyler@epa.gov.
cc: Peter Tsirigotis
Richard Wayland
Betsy Shaw
Marc Vincent
Tyler Fox
George Bridgers
Mike Jones
18-P-0241
20

-------
Appendix B
Distribution
The Administrator
Deputy Administrator
Special Advisor, Office of the Administrator
Chief of Staff
Assistant Administrator for Air and Radiation
Agency Follow-Up Official (the CFO)
Agency Follow-Up Coordinator
General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Career Deputy Assistant Administrator for Air and Radiation
Director, Office of Air Quality Planning and Standards, Office of Air and Radiation
Director, Office of Continuous Improvement, Office of the Administrator
Audit Follow-Up Coordinator, Office of the Administrator
Audit Follow-Up Coordinator, Office of Air and Radiation
Audit Follow-Up Coordinator, Office of Air Quality Planning and Standards,
Office of Air and Radiation
18-P-0241
21

-------