A PERFORMANCE EVALUATION OF THE 2004 RELEASE OF MODELS-3 CMAQ Brian K. Eder, Shaocai Yu* EPA/600/A-04/081 1. INTRODUCTION The Clean Air Act and its Amendments require that the U.S. Environmental Protection Agency (EPA) establish National Ambient Air Quality Standards for 03 and particulate matter and to assess current and future air quality regulations designed to protect human health and welfare. Air quality models, such as EPA's Models-3 Community Multi-scale Air Quality (CMAQ) model, provide one of the most reliable tools for performing such assessments. CMAQ simulates air concentrations and deposition of numerous pollutants on a myriad of spatial and temporal scales to support both regulatory assessment as well as scientific studies conducted by research institutions. In order characterize its performance and to build confidence in the air quality regulatory community, CMAQ, like any model, needs to be evaluated using observational data. Accordingly, this evaluation compares concentrations of various species (S04, N03, PM2 5, NH4 EC, OC, and 03 (not available at press time)), simulated by CMAQ with data collected by the Interagency Monitoring of PROtected Visual Environments (IMPROVE) network, the Clean Air Status and Trends Network (CASTNet) and the Speciated Trends Network (STN). 2. CMAQ GENERAL DESCRIPTION CMAQ is an Eulerian model that simulates the atmospheric and surface processes affecting the transport, transformation and deposition of air pollutants and their precursors [Byun and Ching, 1999], CMAQ follows first principles and employs a "one atmosphere" philosophy that tackles the complex interactions among multiple atmospheric pollutants and between regional and urban scales. Pollutants considered within CMAQ include tropospheric ozone, particulate matter and airborne toxics, as well as acidic and nutrient species. The model also calculates visibility parameters. 1 * Brian Eder®, Shaocai Yu#, NERL, U.S. Environmental Protection Agency, RTP NC, 27711, USA. # On Assignment from Science and Technology Corp., VA 23666, USA. ®On assignment rom Air Resources Laboratory, National Oceanic and Atmospheric Administration, RTP, NC 27711, USA. ------- 2 B.K. EDER ETAL. 2.1. CMAQ Simulation Attributes This evaluation focused on a full, one year simulation of (2001) using the 2004 release of CMAQ. The modeling domain covered the contiguous U.S. using a 36 km grid resolution and a 14-layer vertical resolution (set on a sigma coordinate). The simulation used the CB-IV gas-phase chemistry mechanism. The meteorological fields were derived from MM5, the Fifth-Generation Pennsylvania State University/National Center for Atmospheric Research (NCAR) Mesoscale Model. Emissions of gas-phase S02, CO, NO, N02, NH3, and VOC were based on EPA's 2001 National Emissions Inventory. Primary anthropogenic PM25 emissions were separated into different species including particle S04, N03, OC, EC. Emissions of HC, CO, NOx, and particulate matter from cars, trucks, and motorcycles are based on MOBILE6, while biogenic emissions were obtained from BEIS 3.12. 3. OBSERVATIONAL DATA 3.1. IMPROVE IMPROVE is a collaborative monitoring effort governed by a steering committee comprised of Federal, regional and State organizations designed to: (1) establish current visibility and aerosol conditions; (2) identify the chemical species and emission sources responsible for visibility degradation; and (3) document long-term visibility trends at over 100 remote locations nationwide. A majority of the sites located in the western United States (Figure 1). IMPROVE monitors collect 24-hr integrated samples every third day (midnight to midnight LST). Given CMAQ's one year simulation and IMPROVE's sampling schedule, a total of 115 days were available for comparison. IMPROVE species used in this evaluation include PM2 5, S04, N03, EC and OC. 3.2. CASTNet The Clean Air Status and Trends Network evolved from EPA's National Dry Deposition Network (NDDN) in 1990. The concentration data are collected at predominately rural sites, the majority of which are in the eastern United States, using an open-faced, 3-stage filter pack. The filter packs, which are exposed for 1-week intervals (i.e., Tuesday to Tuesday) at a flow rate of 1.5 liters per minute (3.0 liters per minute for western sites), utilize a Teflon filter for collection of the particulate species. Again, given CMAQ's one year simulation period and CASTNet's weekly sampling schedule, a total of 51 weekly observations were available from a total of 73 sites. CASTNet species used in this evaluation include: S04, N03, and NH4. 3.3. STN The more recently established Speciated Trends Network, developed by EPA, follows the protocol of the IMPROVE network (i.e. every third day collection) with the exception that most of the sites are found in urban areas. The main objectives of the STN are to: provide annual and seasonal spatial characterization of aerosols; provide air quality ------- A PERFORMANCE EVALUATION OF THE 2004 RELEASE OF MODELS-3 CMAQ 3 trends analysis; and track the progress of control programs. The number of STN sites available during 2001 varied as the new network was being deployed. STN species used in this evaluation include: S04, N03, NH4 and PM2 5. Figure 1. Station location map of the networks used in the evaluation. 4. STATISTICS Because of the noted differences in sampling protocols, evaluation statistics were calculated separately for each network. Monitored values were assigned to the CMAQ grid cells in which the fell without interpolation. On the rare occasion when more than one monitor was located within a 36 km grid cell, the average of the monitors would be used to represent that grid cell. In addition to general summary statistics (not shown), two measures of model bias: the Mean Bias (MB) and the Normalized Mean Bias (NMB) and two measures of model error: the Root Mean Square Error (RMSE) and Normalized Mean Error (NME) were calculated as seen below: 1 N MB = 2 (Model- Obs) N I (Model - Obs) NMB 1 N ¦100% I (Obs) 1 N £ (Model - Obs)' £ Model- Obs RMSE NME 1 N ¦100% I (Obs) Scatter plots of monthly aggregated concentrations of CMAQ and observations are also provided for each network and specie, with two-to-one reference lines (Figures 2 - 6). ------- 4 5. RESULTS B.K. EDER ETAL. Examination of the scatter plots and tables reveals that CMAQ varies in its ability to accurately simulate the various species. Simulations of S04 are by far the best (Table 1). Correlation coefficients (Pearson's) associated with each data set are high, ranging from 0.78 (STN) to 0.92 (CASTNet) and the vast majority of the aggregated monthly simulations are within a factor of two of the observations (Fig. 2). The bias is small, with the NMBs ranging from - 4% (CASTNet) to 6% (STN). The errors are relatively small as well, with NMEs ranging from 24% (CASTNet) to 43% (STN). The model generally performs better in eastern locations as opposed to western locations (not shown) - likely a result of greater experience inherent in CMAQ and its predecessors in simulating eastern locations. Table 1. S04 statistics CASTNET IMPROVE STN n 3,737 13,447 6,970 r 0.92 0.86 0.78 MB (ppb) -0.12 0.04 0.22 NMB (%) -4.0 2.0 6.0 RMSE (ppb) 1.12 1.29 2.32 NME (%) 24.0 40.0 43.0 16 14 12 10 ~a3 -o 1 8 6 4 2 0 6 Figure 2. Scatterplot of S04 concentrations (with 2:1, 1:1, 0.5:1 ratios lines). CMAQ simulations of N03 are not nearly as good as those for S04. Correlations are lower, ranging from 0.42 (STN) to 0.73 (CASTNet) and the NMBs are larger, ranging ------- A PERFORMANCE EVALUATION OF THE 2004 RELEASE OF MODELS-3 CMAQ 5 from - 5% (STN) to 27% (IMPROVE)). The NMEs are much larger, ranging from 76% (CASTNet) to 102% (IMPROVE). When examined over space, the NMEs exhibit little if any difference from one area of the United States to the next. This is not the case for the NMB, however, as CMAQ tends to over predict in the eastern U.S. and under predict in the western United States (with a few exceptions). Table 2. N03 statistics CASTNet IMPROVE STN n 3,763 13,398 6,130 r 0.73 0.59 0.42 MB (ppb) 0.26 0.13 -0.08 NMB (%) 26.0 27.0 -5.0 RMSE (ppb) 1.17 1.05 2.88 NME (%) 76.0 102.0 80.0 N03-East (CASTNet) N03-West (CASTNet) N03-east (IMPROVE) N03-west (IMPROVE) N03-East (STN) N03-West (STN) "+Monthly mean . (2001) Observation Figure 3. Scatterplot of N03 concentrations (with 2:1, 1:1, 0.5:1 ratios lines). The quality of NH4 simulations is similar to, but not quite as good as that of S04. Most aggregated monthly simulations are within a factor of two of the observations and the correlations range from 0.58 (STN) to 0.82 (CASTNet). The NMBs are positive and small (7% for CASTNet and 25% for STN), with the majority of over prediction occurring in the eastern U.S. The NMEs range from 34% (CASTNet) to 66% (STN) with the error, for the most part, being equally distributed over the U.S. ------- 6 B.K. EDER ETAL. Table 3. NH4 statistics. CASTNet STN n 3,737 6,970 r 0.82 0.58 MB (ppb) 0.08 0.32 NMB (%) 7.0 25.0 RMSE (ppb) 0.56 1.29 NME (%) 34.0 66.0 8 7 6 5 ai 3 2 1 0 01 2345678 Observation Figure 4. Scatterplot of NH4 concentrations (with 2:1, 1:1, 0.5:1 ratios lines). The quality of OC and EC simulations are similar and fairly poor, with correlations of 0.46 for EC and 0.34 for OC. Note that many monthly aggregated simulations fall outside the factor of two lines (especially for OC and especially for western stations). For OC the NMB and NME are 34% and 82%, respectively, while for EC they are -2% and 62%, respectively. This relatively poor performance is not surprising, given the crude physical representation of organics within the CMAQ aerosol component and the uncertain emission inventories of organics. There is a marked spatial difference with both species in that CMAQ tends to perform considerably better in the eastern U.S. (most NMEs < 50%) when compared to the western U.S. (most NMEs > 50%). OC NMBs are also considerably larger in the western U.S. NH4-East (CAST Met) NH4-West (CASTNet) NH4-East (STN) NH4-West (STN) Monthly mean (2001) ------- A PERFORMANCE EVALUATION OF THE 2004 RELEASE OF MODELS-3 CMAQ Table 4. EC statistics. Table 5. OC statistics. 7 IMPROVE n 13,441 r 0.46 MB (ppb) -0.01 NMB (%) -2.4 RMSE(ppb) 0.28 NME (%) 61.6 IMPROVE n 13,427 r 0.34 MB (ppb) 0.38 NMB (%) 34.5 RMSE (ppb) 1.77 NME (%) 82.6 14 OC-east (IMPROVE) OC-west (IMPROVE) 12 10 Monthly mean (2001) 8 6 4 2 OC (n g C rrf3) 0 0 2 4 6 8 10 12 14 EC-east (IMPROX/E) EC-west (IMPROVE) Monthly mean (2001) 0 2 3 4 Obseivation Observation Figure 5. Scatterplot of EC (left panel) and OC (right panel) concentrations (with 2:1, 1:1, 0.5:1 ratios lines). Simulations of PM25 concentrations (which are composites of the other species), are fairly good as the majority of the simulations lie within a factor of two of the observations. The correlations range from 0.52 (STN) to 0.68 (IMPROVE). The NMBs are small and similar (7- 9%) and the NMEs range from 47 -50%. As with the other species, the model does somewhat better in the eastern U.S. as opposed to the western U.S. Table 6. PM25 statistics IMPROVE STN n 13,217 6,419 r 0.68 0.52 MB (ppb) 0.54 0.97 NMB (%) 9.0 7.0 RMSE (ppb) 4.45 9.21 NME (%) 50.0 47.0 ------- 8 B.K. ~EOERETAL. 60 PM2.5-East (STN/ PM2.5-West (ScCN) PM2.5-east (IMPROVE) PM2.5-we& (IMPROVE) Monthly mean (2001) 50 40 a) ~G o 30 20 Figure 6. Scatterplot of PM25 concentrations (with 2:1, 1:1, 0.5:1 ratios lines). 10 o 0 10 20 30 40 Observation 50 6. SUMMARY This performance evaluation of the 2004 release of CMAQ reveals that the model's ability to accurately simulate the various species continues to improve, especially for N03 concentrations, which have improved markedly since an earlier evaluation (Mebust et al., 2003). Both S04 and NH4 continue to be well simulated by the model, as does PM25. Although simulations of the carbon species are somewhat deficient, improvements in both OC and EC simulations are expected with future releases of CMAQ as the scientific community's understanding of these species matures. Potential areas of research into the sources of the deficiencies identified in this evaluation include uncertainties in emissions inventories, imperfect representation of the meteorological fields, a as well as an incomplete understanding of aerosol dynamics in the CMAQ aerosol component. REFERENCES Byun, D.W. and J.K.S. Ching, Science algorithms of the EPA Models-3 Community Multi-scale Air Quality (CMAQ) modeling system, EPA-600/R-99/030, US EPA, US Government Printing Office, Washington D.C., 1999. Mebust, M., Eder, B., Binkowski, B. And S. Roselle, Model-3 CMAQ model aerosol component, 2. Model evaluation. JGR Vol. 108, No. D6, 2003. The authors would like to thank members of NERL's MEARB branch, especially Alfreida Torian and Steven Howard for the processing of the model and observation datasets. The research presented here was performed under the Memorandum of Understanding between the U.S. Environmental Protection Agency (EPA) and the U.S. Department of Commerce's National Oceanic and Atmospheric Administration (NOAA) and under agreement number DW13921548. Although it has been reviewed by EPA and NOAA and approved for publication, it does not necessarily reflect their policies or ACKNOWLEDGEMENTS DISCLAIMER views. ------- |