A PERFORMANCE EVALUATION OF THE
2004 RELEASE OF MODELS-3 CMAQ
Brian K. Eder, Shaocai Yu*	EPA/600/A-04/081
1.	INTRODUCTION
The Clean Air Act and its Amendments require that the U.S. Environmental Protection
Agency (EPA) establish National Ambient Air Quality Standards for 03 and particulate
matter and to assess current and future air quality regulations designed to protect human
health and welfare. Air quality models, such as EPA's Models-3 Community Multi-scale
Air Quality (CMAQ) model, provide one of the most reliable tools for performing such
assessments. CMAQ simulates air concentrations and deposition of numerous pollutants
on a myriad of spatial and temporal scales to support both regulatory assessment as well
as scientific studies conducted by research institutions. In order characterize its
performance and to build confidence in the air quality regulatory community, CMAQ,
like any model, needs to be evaluated using observational data. Accordingly, this
evaluation compares concentrations of various species (S04, N03, PM2 5, NH4 EC, OC,
and 03 (not available at press time)), simulated by CMAQ with data collected by the
Interagency Monitoring of PROtected Visual Environments (IMPROVE) network, the
Clean Air Status and Trends Network (CASTNet) and the Speciated Trends Network
(STN).
2.	CMAQ GENERAL DESCRIPTION
CMAQ is an Eulerian model that simulates the atmospheric and surface processes
affecting the transport, transformation and deposition of air pollutants and their
precursors [Byun and Ching, 1999], CMAQ follows first principles and employs a "one
atmosphere" philosophy that tackles the complex interactions among multiple
atmospheric pollutants and between regional and urban scales. Pollutants considered
within CMAQ include tropospheric ozone, particulate matter and airborne toxics, as well
as acidic and nutrient species. The model also calculates visibility parameters.
1
* Brian Eder®, Shaocai Yu#, NERL, U.S. Environmental Protection Agency, RTP NC, 27711,
USA. # On Assignment from Science and Technology Corp., VA 23666, USA. ®On assignment
rom Air Resources Laboratory, National Oceanic and Atmospheric Administration, RTP, NC
27711, USA.

-------
2
B.K. EDER ETAL.
2.1. CMAQ Simulation Attributes
This evaluation focused on a full, one year simulation of (2001) using the 2004 release of
CMAQ. The modeling domain covered the contiguous U.S. using a 36 km grid
resolution and a 14-layer vertical resolution (set on a sigma coordinate). The simulation
used the CB-IV gas-phase chemistry mechanism. The meteorological fields were derived
from MM5, the Fifth-Generation Pennsylvania State University/National Center for
Atmospheric Research (NCAR) Mesoscale Model. Emissions of gas-phase S02, CO,
NO, N02, NH3, and VOC were based on EPA's 2001 National Emissions Inventory.
Primary anthropogenic PM25 emissions were separated into different species including
particle S04, N03, OC, EC. Emissions of HC, CO, NOx, and particulate matter from
cars, trucks, and motorcycles are based on MOBILE6, while biogenic emissions were
obtained from BEIS 3.12.
3. OBSERVATIONAL DATA
3.1.	IMPROVE
IMPROVE is a collaborative monitoring effort governed by a steering committee
comprised of Federal, regional and State organizations designed to: (1) establish current
visibility and aerosol conditions; (2) identify the chemical species and emission sources
responsible for visibility degradation; and (3) document long-term visibility trends at
over 100 remote locations nationwide. A majority of the sites located in the western
United States (Figure 1). IMPROVE monitors collect 24-hr integrated samples every
third day (midnight to midnight LST). Given CMAQ's one year simulation and
IMPROVE's sampling schedule, a total of 115 days were available for comparison.
IMPROVE species used in this evaluation include PM2 5, S04, N03, EC and OC.
3.2.	CASTNet
The Clean Air Status and Trends Network evolved from EPA's National Dry Deposition
Network (NDDN) in 1990. The concentration data are collected at predominately rural
sites, the majority of which are in the eastern United States, using an open-faced, 3-stage
filter pack. The filter packs, which are exposed for 1-week intervals (i.e., Tuesday to
Tuesday) at a flow rate of 1.5 liters per minute (3.0 liters per minute for western sites),
utilize a Teflon filter for collection of the particulate species. Again, given CMAQ's one
year simulation period and CASTNet's weekly sampling schedule, a total of 51 weekly
observations were available from a total of 73 sites. CASTNet species used in this
evaluation include: S04, N03, and NH4.
3.3.	STN
The more recently established Speciated Trends Network, developed by EPA, follows the
protocol of the IMPROVE network (i.e. every third day collection) with the exception
that most of the sites are found in urban areas. The main objectives of the STN are to:
provide annual and seasonal spatial characterization of aerosols; provide air quality

-------
A PERFORMANCE EVALUATION OF THE 2004 RELEASE OF MODELS-3 CMAQ
3
trends analysis; and track the progress of control programs. The number of STN sites
available during 2001 varied as the new network was being deployed. STN species used
in this evaluation include: S04, N03, NH4 and PM2 5.
Figure 1. Station location map of the networks used in the evaluation.
4. STATISTICS
Because of the noted differences in sampling protocols, evaluation statistics were
calculated separately for each network. Monitored values were assigned to the CMAQ
grid cells in which the fell without interpolation. On the rare occasion when more than
one monitor was located within a 36 km grid cell, the average of the monitors would be
used to represent that grid cell. In addition to general summary statistics (not shown),
two measures of model bias: the Mean Bias (MB) and the Normalized Mean Bias (NMB)
and two measures of model error: the Root Mean Square Error (RMSE) and Normalized
Mean Error (NME) were calculated as seen below:
1 N
MB = 2 (Model- Obs)
N
I (Model - Obs)
NMB
1
N
¦100%
I (Obs)
1
N
£ (Model - Obs)'
£ Model- Obs
RMSE
NME
1
N
¦100%
I (Obs)
Scatter plots of monthly aggregated concentrations of CMAQ and observations are also
provided for each network and specie, with two-to-one reference lines (Figures 2 - 6).

-------
4
5. RESULTS
B.K. EDER ETAL.
Examination of the scatter plots and tables reveals that CMAQ varies in its ability to
accurately simulate the various species. Simulations of S04 are by far the best (Table 1).
Correlation coefficients (Pearson's) associated with each data set are high, ranging from
0.78 (STN) to 0.92 (CASTNet) and the vast majority of the aggregated monthly
simulations are within a factor of two of the observations (Fig. 2). The bias is small, with
the NMBs ranging from - 4% (CASTNet) to 6% (STN). The errors are relatively small
as well, with NMEs ranging from 24% (CASTNet) to 43% (STN). The model generally
performs better in eastern locations as opposed to western locations (not shown) - likely
a result of greater experience inherent in CMAQ and its predecessors in simulating
eastern locations.
Table 1. S04 statistics

CASTNET
IMPROVE
STN
n
3,737
13,447
6,970
r
0.92
0.86
0.78
MB (ppb)
-0.12
0.04
0.22
NMB (%)
-4.0
2.0
6.0
RMSE (ppb)
1.12
1.29
2.32
NME (%)
24.0
40.0
43.0
16
14
12
10
~a3
-o
1 8
6
4
2
0
6
Figure 2. Scatterplot of S04 concentrations (with 2:1, 1:1, 0.5:1 ratios lines).



CMAQ simulations of N03 are not nearly as good as those for S04. Correlations are
lower, ranging from 0.42 (STN) to 0.73 (CASTNet) and the NMBs are larger, ranging

-------
A PERFORMANCE EVALUATION OF THE 2004 RELEASE OF MODELS-3 CMAQ
5
from - 5% (STN) to 27% (IMPROVE)). The NMEs are much larger, ranging from 76%
(CASTNet) to 102% (IMPROVE). When examined over space, the NMEs exhibit little
if any difference from one area of the United States to the next. This is not the case for
the NMB, however, as CMAQ tends to over predict in the eastern U.S. and under predict
in the western United States (with a few exceptions).
Table 2. N03 statistics

CASTNet
IMPROVE
STN
n
3,763
13,398
6,130
r
0.73
0.59
0.42
MB (ppb)
0.26
0.13
-0.08
NMB (%)
26.0
27.0
-5.0
RMSE (ppb)
1.17
1.05
2.88
NME (%)
76.0
102.0
80.0
N03-East (CASTNet)
N03-West (CASTNet)
N03-east (IMPROVE)
N03-west (IMPROVE)
N03-East (STN)
N03-West (STN)
"+Monthly mean
. (2001)
Observation
Figure 3. Scatterplot of N03 concentrations (with 2:1, 1:1, 0.5:1 ratios lines).
The quality of NH4 simulations is similar to, but not quite as good as that of S04. Most
aggregated monthly simulations are within a factor of two of the observations and the
correlations range from 0.58 (STN) to 0.82 (CASTNet). The NMBs are positive and
small (7% for CASTNet and 25% for STN), with the majority of over prediction
occurring in the eastern U.S. The NMEs range from 34% (CASTNet) to 66% (STN) with
the error, for the most part, being equally distributed over the U.S.

-------
6
B.K. EDER ETAL.
Table 3. NH4 statistics.

CASTNet
STN
n
3,737
6,970
r
0.82
0.58
MB (ppb)
0.08
0.32
NMB (%)
7.0
25.0
RMSE (ppb)
0.56
1.29
NME (%)
34.0
66.0
8
7
6
5
ai
3
2
1
0
01 2345678
Observation
Figure 4. Scatterplot of NH4 concentrations (with 2:1, 1:1, 0.5:1 ratios lines).
The quality of OC and EC simulations are similar and fairly poor, with correlations of
0.46 for EC and 0.34 for OC. Note that many monthly aggregated simulations fall
outside the factor of two lines (especially for OC and especially for western stations).
For OC the NMB and NME are 34% and 82%, respectively, while for EC they are -2%
and 62%, respectively. This relatively poor performance is not surprising, given the
crude physical representation of organics within the CMAQ aerosol component and the
uncertain emission inventories of organics. There is a marked spatial difference with
both species in that CMAQ tends to perform considerably better in the eastern U.S. (most
NMEs < 50%) when compared to the western U.S. (most NMEs > 50%). OC NMBs are
also considerably larger in the western U.S.
NH4-East (CAST Met)
NH4-West (CASTNet)
NH4-East (STN)
NH4-West (STN)
Monthly mean
(2001)

-------
A PERFORMANCE EVALUATION OF THE 2004 RELEASE OF MODELS-3 CMAQ
Table 4. EC statistics.	Table 5. OC statistics.
7

IMPROVE
n
13,441
r
0.46
MB (ppb)
-0.01
NMB (%)
-2.4
RMSE(ppb)
0.28
NME (%)
61.6

IMPROVE
n
13,427
r
0.34
MB (ppb)
0.38
NMB (%)
34.5
RMSE (ppb)
1.77
NME (%)
82.6
14
OC-east (IMPROVE)
OC-west (IMPROVE)
12
10
Monthly mean
(2001)
8
6
4
2
OC (n g C rrf3)
0
0
2
4
6
8
10
12
14
EC-east (IMPROX/E)
EC-west (IMPROVE)
Monthly mean
(2001)
0
2
3
4
Obseivation	Observation
Figure 5. Scatterplot of EC (left panel) and OC (right panel) concentrations (with 2:1,
1:1, 0.5:1 ratios lines).
Simulations of PM25 concentrations (which are composites of the other species), are
fairly good as the majority of the simulations lie within a factor of two of the
observations. The correlations range from 0.52 (STN) to 0.68 (IMPROVE). The NMBs
are small and similar (7- 9%) and the NMEs range from 47 -50%. As with the other
species, the model does somewhat better in the eastern U.S. as opposed to the western
U.S.
Table 6. PM25 statistics

IMPROVE
STN
n
13,217
6,419
r
0.68
0.52
MB (ppb)
0.54
0.97
NMB (%)
9.0
7.0
RMSE (ppb)
4.45
9.21
NME (%)
50.0
47.0

-------
8
B.K. ~EOERETAL.
60
PM2.5-East (STN/
PM2.5-West (ScCN)
PM2.5-east (IMPROVE)
PM2.5-we& (IMPROVE)
Monthly mean
(2001)
50
40
a)
~G
o 30
20
Figure 6. Scatterplot of PM25
concentrations (with 2:1, 1:1, 0.5:1
ratios lines).
10
o
0
10
20 30 40
Observation
50
6. SUMMARY
This performance evaluation of the 2004 release of CMAQ reveals that the model's
ability to accurately simulate the various species continues to improve, especially for
N03 concentrations, which have improved markedly since an earlier evaluation (Mebust
et al., 2003). Both S04 and NH4 continue to be well simulated by the model, as does
PM25. Although simulations of the carbon species are somewhat deficient,
improvements in both OC and EC simulations are expected with future releases of
CMAQ as the scientific community's understanding of these species matures. Potential
areas of research into the sources of the deficiencies identified in this evaluation include
uncertainties in emissions inventories, imperfect representation of the meteorological
fields, a as well as an incomplete understanding of aerosol dynamics in the CMAQ
aerosol component.
REFERENCES
Byun, D.W. and J.K.S. Ching, Science algorithms of the EPA Models-3 Community Multi-scale Air
Quality (CMAQ) modeling system, EPA-600/R-99/030, US EPA, US Government Printing Office, Washington
D.C., 1999.
Mebust, M., Eder, B., Binkowski, B. And S. Roselle, Model-3 CMAQ model aerosol component, 2.
Model evaluation. JGR Vol. 108, No. D6, 2003.
The authors would like to thank members of NERL's MEARB branch, especially Alfreida Torian and Steven
Howard for the processing of the model and observation datasets.
The research presented here was performed under the Memorandum of Understanding between the U.S.
Environmental Protection Agency (EPA) and the U.S. Department of Commerce's National Oceanic and
Atmospheric Administration (NOAA) and under agreement number DW13921548. Although it has been
reviewed by EPA and NOAA and approved for publication, it does not necessarily reflect their policies or
ACKNOWLEDGEMENTS
DISCLAIMER
views.

-------