Additional Analyses of the Monte Carlo Model Developed for the Determination of PEMS Measurement Allowances for Gaseous Emissions Regulated Under the Heavy-Duty Diesel Engine In-Use Testing Program United States Environmental Protection Agency ------- Additional Analyses of the Monte Carlo Model Developed for the Determination of PEMS Measurement Allowances for Gaseous Emissions Regulated Under the Heavy-Duty Diesel Engine In-Use Testing Program Assessment and Standards Division Office of Transportation and Air Quality U.S. Environmental Protection Agency Prepared for EPA by Southwest Research Institute EPA Contract No. EP-C-05-018 Work Assignment No. 2-6 v>EPA NOTICE This technical report does not necessarily represent final EPA decisions or positions. It is intended to present technical analysis of issues using data generated in the associated test program. The purpose in the release of such reports is to facilitate the exchange of technical information and to inform the public of technical developments which may form the basis for a final EPA decision, position, or regulatory action. United States EPA420-R-07-010 Environmental Protection . ^ „„., Agency August 2007 ------- EXECUTIVE SUMMARY This report documents the program conducted by Southwest Research Institute® (SwRI), on behalf of the U.S. Environmental Protection Agency (EPA), the objective of which was to perform additional analyses on the Portable Emissions Measurement Systems (PEMS) Monte Carlo simulation models in order to determine validation of the three emissions (BSNOx, BSNMHC and BSCO) using three calculation methods. Several steady-state error surfaces were modified based on recommendations from the Heavy-Duty In-Use Testing (HDIUT) Steering Committee. These included modifications to the steady-state NOx, exhaust flow rate, CO and CO2 error surfaces. Several reference NTE events were run which produced validation results from all three emissions and all three calculation methods. Measurement allowances were computed for two of the simulation strategies based on using 50 reference NTE events. The Mod 1 strategy involved changing the steady-state CO, CO2 and exhaust flow rate error surfaces to eliminate bias while changing the steady-state NOx to a level independent error surface including all the test data. The Mod 2 strategy was the same as Mod 1 except the steady-state NOx error surface was also changed to a level independent error surface but excluded several questionable low NOx values from one of the test engines. The measurement allowance values by calculation method determined at the conclusion of these analyses are summarized in Table 1. This report details the process used to determine the measurement values reported in Table 1. TABLE 1. MEASUREMENT ALLOWANCES FOR MOD 1 AND MOD 2 Pollutant NOX NMHC CO Calculation Method 1 2 3 1 2 3 1 2 3 Mod 1 Measurement Allowance, g/hp-hr 0.232 0.178 0.192 0.015 0.014 0.014 0.268 0.258 0.270 Mod 2 Measurement Allowance, g/hp-hr 0.211 0.151 0.171 0.014 0.014 0.014 0.266 0.250 0.262 SwRI Report 03.12859.06 ------- LIST OF ACRONYMS Brake-Specific BS California Air Resources Board CARB Center for Environmental Research & Technology CE-CERT Code of Federal Regulations CFR Electronic Flow Meter EFM Empirical Distribution Function EDF Engine Control Module ECM Engine Manufacturer's Association EMA Environmental Protection Agency EPA Heavy Duty In-Use Testing HDIUT Heavy Heavy Duty HHD Light Heavy Duty LHD Median Absolute Deviation MAD Medium Heavy Duty MHD Memorandum of Agreement MO A Mobile Emissions Laboratory MEL Not To Exceed NTE Portable Emissions Measurement System PEMS Root Mean Square RMS SEMTECH-DS SN G05-SDS04 PEMS 1 SEMTECH-DS SN G05-SDS02 PEMS 2 SEMTECH-DS SN G05-SDS03 PEMS 3 SEMTECH-DS SN G05-SDS01 PEMS 4 SEMTECH-DS SN D06-SDS01 PEMS 5 SEMTECH-DS SN D06-SDS06 PEMS 6 SEMTECH-DS SN F06-SDS02 PEMS 7 Southwest Research Institute SwRI Standard Deviation SD SwRI Report 03.12859.06 11 ------- TABLE OF CONTENTS EXECUTIVE SUMMARY i LIST OF ACRONYMS ii LIST OF FIGURES iv LIST OF TABLES vi 1.0 INTRODUCTION 1 2.0 ERROR SURFACE MODIFICATIONS 2 2.1 Steady-State Exhaust Flow Rate Error Surface 2 2.2 Steady-State CO2 Error Surface 3 2.3 Steady-State CO Error Surface 4 2.4 Steady-State NOX Error Surface 5 3.0 REFERENCE NTE EVENTS 9 4.0 CE-CERT MEASUREMENTS 13 5.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE EVENTS USING MOD 1, MOD 2 AND MOD 3 14 6.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE EVENTS USING MODE 25 7.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE EVENTS USING MOD 1 35 8.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE EVENTS USING MOD 2 45 9.0 MEASUREMENT ALLOWANCE CALCULATIONS 56 10.0 VALIDATION SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS.... 77 11.0 FULL MODEL SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS ... 81 12.0 REFERENCES 83 SwRI Report 03.12859.06 iii ------- LIST OF FIGURES 1 Revised Error Surface for Steady-State Exhaust Flow Rate 3 2 Revised Error Surface for Steady-State CC>2 4 3 Revised Error Surface for Steady-State CO 5 4 Revised Error Surface Mod 1 for Steady-State NOX 6 5 Revised Error Surface Mod 2 for Steady-State NOX 7 6 Revised Error Surface Mod 3 for Steady-State NOX 8 7 Distribution of Ideal NOx (g/kW-hr) Method 1 for 23 Selected Ref NTE Events and 195 Ref NTE Events 11 8 Distribution of Ideal NOx (g/kW-hr) Method 1 for 50 Selected Ref NTE Events and 195 Ref NTE Events 12 9 Validation EOF Plots Using 23 Reference NTE Events for NOX Method 1 Mods 1,2 and 3 16 10 Validation EOF Plots Using 23 Reference NTE Events for NOX Method 2 for Mods 1,2 and 3 17 11 Validation EOF Plots Using 23 Reference NTE Events for NOX Method 3 for Mods 1,2 and 3 18 12 Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 1 for Mods 1,2 and 3 19 13 Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 2 for Mods 1,2 and 3 20 14 Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 3 for Mods 1,2 and 3 21 15 Validation EOF Plots Using 23 Reference NTE Events for CO Method 1 for Mods 1,2 and 3 22 16 Validation EOF Plots Using 23 Reference NTE Events for CO Method 2 Mods 1,2 and 3 23 17 Validation EOF Plots Using 23 Reference NTE Events for CO Method 3 for Mods 1,2 and 3 24 18 Validation EOF Plots Using 23 Reference NTE Events for NOX Method 1 Mod B 26 19 Validation NTE Events Using 23 Reference NTE Events for NOX Method 2 Mod B. .27 20 Validation EOF Plots Using 23 Reference NTE Events for NOX Method 3 Mod B 28 21 Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 1 Mod B .. 29 22 Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 2 Mod B .. 3 0 23 Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 3 Mod B .. 31 24 Validation EOF Plots Using 23 Reference NTE Events for CO Method 1 Mod B 32 25 Validation EOF Plots Using 23 Reference NTE Events for CO Method 2 Mod B 33 26 Validation EOF Plots Using 23 Reference NTE Events for CO Method 3 Mod B 34 27 Validation EOF Plots Using 50 Reference NTE Events for NOX Method 1 Mod 1 36 28 Validation EOF Plots Using 50 Reference NTE Events for NOX Method 2 Mod 1 37 29 Validation EOF Plots Using 50 Reference NTE Events for NOX Method 3 Mod 1 38 30 Validation EOF Plots Using 50 Reference NTE Events for NMHC Method 1 Mod 1...39 31 Validation EOF Plots Using 50 Reference NTE Events for NMHC Method 2 Mod 1.. .40 32 Validation EOF Plots Using 50 Reference NTE Events for NMHC Method 3 Mod 1.. .41 33 Validation EOF Plots Using 50 Reference NTE Events for CO Method 1 Mod 1 42 34 Validation EOF Plots Using 50 Reference NTE Events for CO Method 2 Mod 1 43 SwRI Report 03.12859.06 iv ------- 3 5 Validation EOF Plots Using 50 Reference NTE Events for CO Method 3 Mod 1 44 36 Validation EOF Plots Using 50 Reference NTE Events for NOX Method 1 Mod 2 46 37 Validation EOF Plots Using 50 Reference NTE Events for NOX Method 2 Mod 2 47 38 Validation EOF Plots for 50 Reference NTE Events for NOX Method 3 Mod 2.. .48 39 Validation EOF Plots for 50 Reference NTE Events for NMHC Method 1 Mod 2 49 40 Validation EOF Plots for 50 Reference NTE Events for NMHC Method 2 Mod 2 50 41 Validation EOF Plots Using 50 Reference NTE Events for NMHC Method 3 Mod 2... 51 42 Validation EOF Plots Using 50 Reference NTE Events for CO Method 1 Mod2 52 43 Validation EOF Plots Using 50 reference NTE Events for CO Method 2 Mod 2.53 44 Validation EOF Plots Using 50 Reference NTE Events for CO Method 3 Mod 2 54 45 Regression Plot of 95th Percentile Delta BSNOX Versus Ideal BSNOX for Method 1 Modi 57 46 Regression Plot of 95th Percentile Delta BSNOX Versus Ideal BSNOX for Method 2 Modi 58 47 Regression Plot of 95th percentile Delta B SNOX Versus Ideal B SNOX for Method 3 Modi 59 48 Regression Plot of 95th percentile Delta BSNMHC Versus Ideal BSNMHC for Method 1 Modi 60 49 Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for Method2Modl 61 5 0 Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for Method 3 Modi 62 51 Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 1 Modi 63 52 Regression Plot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 2 Modi 64 5 3 Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 3 Modi 65 54 Regression Plot of 95th Percentile Delta BSNOx Versus Ideal BSNOx for Method 1 Mod2 66 55 Regression Plot of 95th Percentile Delta BSNOX Versus Ideal BSNOX for Method 2 Mod2 67 56 Regression Plot of 95th Percentile Delta BSNOX Versus Ideal BSNOX for Method 3 Mod2 68 5 7 Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for Method !Mod2 69 5 8 Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for Method2Mod2 70 5 9 Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for Method 3 Mod 2 71 60 Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 1 Mod 2 72 61 Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 2 Mod 2 73 62 Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 3 Mod 2 74 SwRI Report 03.12859.06 ------- LIST OF TABLES Table Page 1 Measurement Allowances For MOD 1 and Mod 2 i 2 Revised Error Surfaces Used in Monte Carlo Simulations 8 3 Reference NTE Events Used in Monte Carlo Simulations 10 4 Descriptive Statistics for Ideal BSNOX (g/kW-hr) for Various NTE Subsets 12 5 Convergence criteria by Emission 14 6 BSNOX Validation Results Based on 23 Reference NTE Events 15 7 Summary of Validation Results 55 8 Measurement Error at Threshold for BSNOX Using Regression and Median Methods for Method 1 Mod 1 57 9 Measurement Error at Threshold for BSNOX Using Regression and Median Methods for Method 2 Mod 1 58 10 Measurement Error at Threshold for BSNOX Using Regression and Median Methods for Method 3 Mod 1 59 11 Measurement Error at Threshold for BSNMHC Using Regression and Median Methods for Method 1 Mod 1 60 12 Measurement Error at Threshold for BSNMHC Using Regression and Median Methods for Method 2 Mod 1 61 13 Measurement Error at Threshold for BSNMHC Using Regression and Median Methods for Method 3 Mod 1 62 14 Measurement Error at Threshold for B SCO Using Regression and Median Methods for Method 1 Mod 1 63 15 Measurement Error at Threshold for B SCO Using Regression and Median Methods for Method 2 Mod 1 64 16 Measurement Error at Threshold for B SCO Using Regression and Median Methods for Method 3 Mod 1 65 17 Measurement Error at Threshold for BSNOX Using regression and Median methods for Method 1 Mod 2 66 18 Measurement Error at Threshold for BSNOX Using Regression and Median methods for Method 2 Mod 2 67 19 Measurement Error at Threshold for BSNOX Using Regression and Median Methods for Method 3 Mod 2 68 20 Measurement Error at Threshold for BSNMHC Using Regression and Median Methods for Method 1 Mod 2 69 21 Measurement Error at Threshold for BSNMHC Using Regression and Median Methods for Method 2 Mod 2 70 22 Measurement Error at Threshold for BSNMHC Using Regression and Median Methods for Method 3 Mod 2 71 23 Measurement Error at Threshold for B SCO Using Regression and Median Methods for Method 1 Mod 2 72 24 Measurement Error at Threshold for B SCO Using Regression and Median Methods for Method 2 Mod 2 73 SwRI Report 03.12859.06 vi ------- 25 Measurement Error at Threshold for B SCO Using Regression and Median Methods for Method 3 Mod 2 74 26 Summary of Measurement Errors at Respective Threshold (%) 75 27 Summary of Measurement Allowance, g/hp-hr 76 28 Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte Carlo validation Simulations for BSNOX 78 29 Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte Carlo validation Simulations for BSNMHC 79 30 Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte Carlo validation Simulations for BSCO 80 31 Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte Carlo full model Simulations for BSNOX 82 SwRI Report 03.12859.06 vii ------- 1.0 INTRODUCTION Southwest Research Institute completed a Portable Emissions Measurement System (PEMS) program on behalf of the U.S. Environmental Protection Agency (EPA) in early April 2007. The purpose of this project was to determine the brake-specific (BS) measurement allowances for the gaseous pollutants regulated under the Heavy-Duty In-Use Testing (HDIUT) program [Ref 1]. The study was performed under cooperation between the EPA, the California Air Resources Board (CARB), and the Engine Manufacturer's Association (EMA). All efforts during this program were conducted under the direction of a joint body, the HDIUT Measurement Allowance Steering Committee, referred to in this report simply as the Steering Committee. The program consisted of modeling various PEMS measurement errors using a statistical Monte Carlo modeling simulation. The simulation results were used to generate the brake- specific measurement allowances based on three calculation methods for BS emissions of NOX, NMHC and CO. To confirm the results of the simulation, a SEMTECH-DS PEMS was operated in-use with the CE-CERT Mobile Emission Laboratory (MEL) over several routes in California. The differences between the PEMS and MEL gaseous emission measurements were used to validate the SwRI Monte Carlo modeling simulation results. Calculation Methods 2 (BSFC based) and 3 (ECM Fuel Specific) did not validate for BSNOX based on the CE-CERT MEL emissions and the Monte Carlo PEMS simulations. Given this result, EPA contracted with SwRI in April 2007 to conduct additional analyses to identify possible causes as to why the BSNOX did not validate. The results of these analyses are included in this report. SwRI Report 03.12859.06 1 of 83 ------- 2.0 ERROR SURFACE MODIFICATIONS Since calculation Methods 2 and 3 did not validate for BSNOX after the Monte Carlo model simulation runs during the original PEMS study, the associated Steering Committee chose to modify key error surfaces in order to determine if such alterations would effect the validation of Methods 2 and 3. Over the course of several meetings, the Steering Committee brainstormed to generate alternate error surface processing methods. The Committee decided to modify Steady-State CO, CO2, Exhaust Flow Rate, and NOX, as these surfaces had significant influence on the Model results based on the sensitivity analyses. In reprocessing the error surfaces, the Steering Committee agreed that it would be necessary to remove any biases that were recorded in the SwRI laboratory. It was assumed that these bias errors were due to the limited number of PEMS units and comparative observations, and that if more PEMS and Sensors Inc. exhaust flow meters were tested the biases may have been eliminated. Therefore, to remove the biases the original steady-state error surfaces were transformed to be a symmetric error surface with the 50th percentile errors set to zero and the 5th and 95th percentile errors modified to be a mirror image of one another. Two analysis methods were used to generate the revised error surfaces. If the recorded error data showed an emissions level dependency, an envelope was generated to encompass the extreme 5th or 95th percentile error data. The envelope segments with the largest absolute error were then mirrored to generate a symmetric error surface. A more rigorous analysis was used to reprocess level independent error surfaces. 2.1 Steady-State Exhaust Flow Rate Error Surface Figure 1 shows the reprocessed, level-dependent steady-state exhaust flow rate error surface as well as the exhaust flow error surface data that was used in the original MC model runs. The largest errors recorded during laboratory testing were positive deltas measured during Engine 3 testing with the 3-inch EFMs. The 3-inch EFM errors defined the 95th percentile error contours for the original error surface as well as the reprocessed error surface. The difference between the two 95l percentile contours was due to a correction of the 3-inch EFM calibrations when the exhaust flow rate error surface was reprocessed. SwRI and Sensors Inc. had recalibrated the 3-inch flow meters using data generated at SwRI. The SwRI calibration increased the slope multiplier by approximately 4 % for each 3-inch flow meter. Later in the program, Sensors Inc. discovered the 3-inch EFMs were likely not operating correctly on the SwRI flow stand due to EFM pressures below ambient levels. Therefore, each 3-inch flow meter calibration was corrected to the original coefficients generated by Sensors. The corrected 3-inch flow meter data generated the 95th percentile contour shown in Figure 1. Because the 95th percentile errors were larger than the 5th percentile errors, the 95th percentile data was mirrored to generate the 5th percentile contour in the reprocessed error surface. SwRI Report 03.12859.06 2 of 83 ------- -95th percentile • Previous 95th -50th percentile (median) • Previous 50th -5th percentile • Previous 5th Lab Reference Mean Exhaust Flow Rate [% of Max] FIGURE 1. REVISED ERROR SURFACE FOR STEADY-STATE EXHAUST FLOW RATE 2.2 Steady-State CO2 Error Surface th Figure 2 shows the reprocessed, level-dependent CC>2 error surface. The 95 percentile th contour represents the envelope that was created to encompass the original 95 percentile error data. Line segments were used to connect the most extreme points from the original 95 th -th th percentile data. The 95 percentile contour was mirrored to produce the 5 percentile error data, with the 50th percentile deltas set to zero. The reprocessed error surface was meant to encompass all possible PEMS CC>2 measurement errors, not just those measured at SwRI. SwRI Report 03.12859.06 3 of 83 ------- -95th percentile • Previous 95th -50th percentile (median) • Previous 50th -5th percentile • Previous 5th 0.8 0.6 " 0.4 0.2 0.0 c o o CM O O 40 -0.2 -0.4 -0.6 -0.8 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 90 Lab Reference Mean CO2 Concentration [%] FIGURE 2. REVISED ERROR SURFACE FOR STEADY-STATE CO2 2.3 Steady-State CO Error Surface Figure 3 shows the revised steady-state error surface for CO. Because the CO error data showed no level dependence, a statistical calculation process, developed by Bill Martin from Cummins Inc., was used to generate the reprocessed CO error surface. The total standard deviation of the pooled CO error data was calculated by combining the variance of the PEMS mean delta standard deviation about zero and the PEMS pooled estimate of repeatability standard deviation. The calculation of the total standard deviation is shown below. ^U total \ repeatability ~~ - mean, zero repeatability IV S7) 2 / j ^-^indiv,perns n SDindiV,pems = standard deviation of each PEMS calculated over 20 repeats of each steady-state point n = 10 steady-state points per engine * 3 engines * 3 PEMS = 90 SwRI Report 03.12859.06 4 of 83 ------- SD, n-l PEMSindiv,mean = mean delta of each PEMS calculated over 20 repeats of each steady-state point Shown below, the multiplier for a 90% confidence interval assuming a normal distribution for the errors was applied to the total standard deviation (48.84 ppm = 0.004882%) to generate the constant 5th and 95th percentile error values. 5* % = -1.645 * SDtotal = - 0.008% 95*% = 1.645 * SDtotal = + 0.008% •95th percentile • Previous 95th •50th percentile (median) • Previous 50th -5th percentile • Previous 5th 0.012 0.010 0.008 - - 4M»- 0.006 0.004 5. 0.002 LU Q. •4*4- « «» 0.000 o.opoo -0.002 -0.004 -0.006 -0.008 \ - - | -0.010 0.0005 0.0010 0.0015 0.0020 0.0 )25 Lab Reference Mean CO Concentration [%] FIGURE 3. REVISED ERROR SURFACE FOR STEADY-STATE CO 2.4 Steady-State NOX Error Surface For reprocessing the original steady-state NOX, three different methods were used to generate three NOX concentration error surfaces. For each of the three modified NOX error surfaces, the original set of Engine 2 steady-state data was added to the original pooled data set. The Engine 2 steady-state data was collected in a repeat run in the original study due to PEMS failures that caused a large time gap in the original data. The time gap was associated with a NOX concentration shift. Since the NOX shift would have caused problems with the steady-state SwRI Report 03.12859.06 5 of 83 ------- variance correction of the transient data the repeat run data was used. However, the original Engine 2 delta data was later determined to be valid and thus was included in the modified data set to increase the PEMS delta observations. The first NOX error surface modification (Mod 1) followed a procedure similar to the -state CO error surface. Using the complete NOX delta data set, the 5th and 95th percentile error values were calculated with the total standard deviation (15.08 ppm). The results of the first NOX erro ±24.80 ppm. steady-state CO error surface. Using the complete NOX delta data set, the 5th and 95th percentile >8l , first NOX error surface modification are shown in Figure 4 with the 5l and 95l percentiles set to 40 20 - Q. Q. I -20 H re m § -40 O 5 LU 0. -80 -100 •95th percentile • Previous 95th -50th percentile (median) • Previous 50th •5th percentile • Previous 5th ,- .2.00 300 400 500 600 Lab Reference Mean NOx Concentration [ppm] FIGURE 4. REVISED ERROR SURFACE MOD 1 FOR STEADY-STATE NOX The second NOX error surface modification (Mod 2) was similar to the first modification except six outlying low delta values recorded during Engine 3 testing were removed from the pooled data set. During Engine 3 steady-state testing, PEMS 1 and 6 showed extremely low deltas at high NOX concentration levels. PEMS 4 also showed low biases at high NOX levels; however, the biases were not as large as those measured for the other PEMS. Therefore, PEMS 1 and 6 delta measurements at the three highest NOX levels measured during engine testing were removed from the data set. This resulted in a total standard deviation equal to 9.39 ppm and the -th -th 5 and 95 percentile deltas equal to ±15.45 ppm. The results of this analysis are shown in Figure 5. SwRI Report 03.12859.06 6 of 83 ------- -95th percentile • Previous 95th -50th percentile (median) • Prefious 50th -5th percentile • Previous 5th -100 Lab Reference Mean NOx Concentration [ppm] FIGURE 5. REVISED ERROR SURFACE MOD 2 FOR STEADY-STATE NOX Shown in Figure 6, the third NOX error surface modification (Mod 3) combined the level- dependent and level-independent analysis methods. Below 311 ppm, the delta data was assumed to be level independent. Therefore, a statistical method similar to that used for CO was used for the NOX delta data below 311 ppm. The total standard deviation for the level-independent data was 9.41 ppm and the resulting 5th and 95th percentiles were ± 15.48 ppm. Above 311 ppm, the original 5l percentile profile was used and mirrored to the 95th percentile to generate a symmetric error surface. SwRI Report 03.12859.06 7 of 83 ------- -95th percentile • Previous 95th -50th percentile (median) • Previous 50th -5th percentile • Previous 5th 100 80 60 40 20 0 -20 -40 -60 -80 -100 Lab Reference Mean NOx Concentration [ppm] FIGURE 6. REVISED ERROR SURFACE MOD 3 FOR STEADY-STATE NOX In addition to running the MC simulations with the three different changes to the steady- state NOx error surface, it was useful to run a simulation with the original steady-state NOx error surface and only include the revised steady-state exhaust flow rate and steady-state CO2 error surfaces. This set of simulations is called the Mod B runs. Table 2 provides a summary of the four MC simulation runs made in this study with the revised error surfaces that were included in each run. All the other error surfaces that are not listed in Table 2 are the same as those run in the original PEMS study. TABLE 2. REVISED ERROR SURFACES USED IN MONTE CARLO SIMULATIONS Steady-State Exhaust Flow Rate Steady-State CO2 Steady-State CO Steady- State NOX Mod 1 Steady- State NOX Mod 2 Steady-State NOX Mod 3 X X X X X X X X X X X X X X SwRI Report 03.12859.06 8 of 83 ------- 3.0 REFERENCE NTE EVENTS The Monte Carlo simulation results from the original study included a reference data set consisting of 195 NTE events gathered from a number of sources. These included five engine manufacturers, SwRI transient lab tests and pre-pilot CE-CERT data. The investigations performed in this modified program included only a subset of the original 195 reference NTE events. Simulations from three different subsets of the original NTE events were run to study the validation of the emissions by the three calculation methods using the revised error surfaces. Table 3 lists the NTE events along with their ideal BSNOX values selected for each simulation. The various NTE subsets are described as follows: • 13 Reference Events - These are the same 13 NTE events chosen during the original study to investigate the error surface sensitivities due to bias and variance. These events were selected to bound the BSNOX threshold of 2.6820 g/kW-hr. These 13 NTE events were used to examine the error surface sensitivities for both the validation and full model simulation runs. • 23 Reference Events - Ten additional NTE events were added to the original 13 NTE events described above in order to increase the sample size for generating the validation EDF plots using the drift corrected CE-CERT data. • 50 Reference Events - After the model validation was confirmed for the three emissions across the three calculation methods, Mods 1 and 2 were chosen for continued analysis. Twenty-seven additional reference NTE events were simulated with the Monte Carlo model in order to estimate the measurement allowances for each emission. The selection of the additional 27 reference NTE events was based on maintaining a similar distribution of ideal BSNOX values for Method 1 established from the original study containing 195 reference NTE events. Figure 7 shows comparison histograms of the Method 1 BSNOX values for the 23 reference NTE events (lower histogram) and the 195 reference NTE events (upper histogram) while Figure 8 compares the Method 1 BSNOX histograms for the 50 reference NTE events (lower histogram) and the 195 reference NTE events (upper histogram). Descriptive statistics of BSNOX for these NTE subsets are detailed in Table 4. SwRI Report 03.12859.06 9 of 83 ------- TABLE 3. REFERENCE NTE EVENTS USED IN MONTE CARLO SIMULATIONS 1 3 4 7 11 16 20 22 23 25 29 37 38 40 43 44 46 51 57 63 65 66 67 69 71 82 86 87 89 92 96 99 103 115 125 127 136 139 146 148 157 4.0713 3.0668 3.5832 5.2516 1.8583 2.3511 3.7245 4.7916 3.1483 5.4061 5.5261 5.4511 0.0250 4.1675 1.1473 1.0730 2.6958 2.8299 4.3382 2.6670 2.7437 3.9378 5.8600 3.0257 6.6867 2.4569 1.7132 1.5207 2.2566 1.6041 1.6224 1.8147 1.9186 1.3854 2.3214 3.3005 2.6782 2.4018 2.3053 1.9985 3.4666 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X SwRI Report 03.12859.06 10 of 83 ------- 160 162 163 168 176 177 179 191 193 2.0773 2.1405 2.5908 1.5859 1.9671 1.7507 2.2023 6.0815 5.7521 X X X X X X X X X X X X 40 30 10 sr 0 10 20 Ideal NOx Method 1 195RefNTE 02468 Ideal NOx Method 1 23 Ref NTE FIGURE 7. DISTRIBUTION OF IDEAL NOX (G/KW-HR) METHOD 1 FOR 23 SELECTED REF NTE EVENTS AND 195 REF NTE EVENTS SwRI Report 03.12859.06 11 of 83 ------- Ideal NOx Method 1 195 Ref NTE Events 40 30 Ł 20 0) =3 10 D" Ł 0 LL 10 20 02468 Ideal NOx Method 1 50 Ref NTE Events FIGURE 8. DISTRIBUTION OF IDEAL NOX (G/KW-HR) METHOD 1 FOR 50 SELECTED REF NTE EVENTS AND 195 REF NTE EVENTS TABLE 4. DESCRIPTIVE STATISTICS FOR IDEAL BSNOX (G/KW-HR) FOR VARIOUS NTE SUBSETS Minimum Maximum Mean Median Standard Deviation 0.0249 5.5261 2.8983 2.6782 1.3671 0.0249 6.6867 3.0068 2.6289 1.5175 0.0249 7.1927 3.0071 2.6033 1.4807 SwRI Report 03.12859.06 12 of 83 ------- 4.0 CE-CERT MEASUREMENTS The CE-CERT on-road measurements collected during the original PEMS study included delta BS emissions for all three emissions and three calculation methods. As part of this study to investigate all possible reasons why BSNOX did not validate for Methods 2 and 3, the 100 NTE events from the CE-CERT data were examined for correctness by EPA. Since the CE-CERT data had not been drift corrected in the original PEMS study, EPA performed drift corrections on all three emissions for the 100 CE-CERT NTE events. As a result, several CE-CERT NTE events did not pass the drift check criteria and, therefore, were not included in the simulation performed for this study. In addition, time alignment problems were found in a few of the 100 CE-CERT NTE events and these were also excluded from the simulation. After deleting the NTE events due to drift check or time alignment problems, the on-road delta BS emissions were calculated from 81 NTE events for BSNOX and 87 NTE events for BSNMHC and BSCO. These remaining CE-CERT NTE events were all drift corrected. In contrast, the original PEMS program did not use drift correction in the CE-CERT on-road NTE events. SwRI Report 03.12859.06 13 of 83 ------- 5.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE EVENTS USING MOD 1, MOD 2 AND MOD 3 The Monte Carlo simulations performed using the 23 reference NTE events listed in Table 3 included modifications for the error surfaces listed in Table 2. All the error surfaces from the original MC simulations were used except those listed as revised in Table 2. Each of the 23 reference NTE events were run using the error surfaces revised for Mod 1 for either 10,000 or 30,000 trials. If the ideal BSNOX for an NTE event was less than 2.68204 g/kW-hr then the simulation was run for 10,000 trials. Otherwise, the NTE was simulated using 30,000 trials. Once the MC simulations were completed using the Mod 1 revised error surfaces, the same 23 NTE events were simulated a second time with the error surface modifications for Mod 2. Lastly, the 23 reference NTE events were simulated using the error surface modifications for Mod 3. In Mod 1, Mod 2, and Mod 3 simulation runs, all three emissions converged within 1% of the emissions threshold value by all three calculation methods. Table 5 lists the convergence criteria for all three brake-specific emissions. Thus, all 23 reference NTE events met the convergence criteria. TABLE 5. CONVERGENCE CRITERIA BY EMISSION BSNOx BSNMHC BSCO 0.02682 0.00282 0.26015 From the MC simulations, the validation 5th and 95th percentile delta BS emissions were extracted from the output files for each of the 23 reference NTE events. These delta emissions were then plotted as empirical distribution functions (EDF) to form a validation interval for the on-road data. Also plotted was the EDF computed from the on-road CE-CERT NTE events. Figure 9 through Figure 11 represent the validation plots for the BSNOX for calculation methods 1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th percentile EDFs for the Mod 1, Mod 2, and Mod 3 runs. Figure 12 through Figure 17 depict the validation plots for the BSNMHC and BSCO simulation runs, respectively. The validation criteria set by the Steering Committee for the original study was used in the modified program. It included the following criteria: • At least 90% of the CE-CERT emissions deltas must be within the 5th and 95th percentiles of the MC validation cumulative emissions deltas. • No more than 10% of the CE-CERT emissions deltas may fall less than the 5th percentile or greater than the 95th percentile. This may indicate that the model is biased low or high. • Validation must be shown for all three calculation methods. SwRI Report 03.12859.06 14 of 83 ------- A summary of the BSNOX validation conclusions based on the 23 reference NTE events is provided in Table 6. All BSNOX, BSNMHC and BSCO emissions validated for all three calculation methods and all three Mod runs. TABLE 6. BSNOx VALIDATION RESULTS BASED ON 23 REFERENCE NTE EVENTS Mod 1 Mod 2 Mod3 • All CE-CERT data > 5th percentile • All CE-CERT data < 95th percentile •VALID • All CE-CERT data > 5th percentile • All CE-CERT data < 95th percentile •VALID • All CE-CERT data > 5th percentile • All CE-CERT data < 95th percentile •VALID • All CE-CERT data > 5th percentile • 1% of CE-CERT data > 95th percentile •VALID • All CE-CERT data > 5th percentile • 1% of CE-CERT data > 95th percentile •VALID • All CE-CERT data > 5th percentile • All CE-CERT data < 95th percentile •VALID • All CE-CERT data > 5th percentile • All CE-CERT data < 95th percentile •VALID • All CE-CERT data > 5th percentile • 1% of CE-CERT data > 95th percentile •VALID • All CE-CERT data > 5th percentile • All CE-CERT data < 95th percentile •VALID SwRI Report 03.12859.06 15 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas Error Models Mod 1, Mod 2 and Mod 3 NCK (g/kW—hr) Method 1 100 en a. rt D Ł D O 80- 60 40- 20- 0 —0.60 -O.45 -0,30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 Delta NCK FIGURE 9. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NOX METHOD 1 MODS 1, 2 AND 3 SwRI Report 03.12859.06 16 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas Error Models Mod 1, Mod 2 and Mod 3 NCK (s^kW-hr) Method 2 100 80- en a. O N 23 ^^^^^^^^^^^^^^m ^^^^^^^^^^^^^^m 95ti % Mod 2 Delta -0.60 -Q.45 -0,30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 FIGURE 10. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NOX METHOD 2 FOR MODS 1, 2 AND 3 SwRI Report 03.12859.06 17 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected SI CE—CERT Deltas Error Models Mod 1, Mod Ł and Mod 3 NOx (g/kW-hr) Method 3 E o 100- 80- 60- 51h % Mod 1 De 1a 5th W Mod 2 Delta 5th % Mod 3 Delta 951h M Mod 1 Delta N 23 ^^^^^^^^^^^^ ^^^^^^^^^^^^^^H 95th W Mod 2 Delta N 23 ^^^^^^^^^^^^^^M ^^^^^^^^^^^^^^H 95th % Mod 3 Delta -0,60 -0.45 -0,30 -0.15 0.00 0.15 0.30 0,45 0.60 0,75 0.90 Delta NCK g/kW—hr 4C" 20 < FIGURE 11. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NOX METHOD 3 FOR MODS 1, 2 AND 3 SwRI Report 03.12859.06 18 of 83 ------- en a. rt D Ł D O 100 80- 60 40- 20- 0 Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Models Mod 1, Mod 2 and Mod 3 NMHC (g/kW—hr) Method 1 |51h % Mod 1 Delta —0.04 -O.03 -0,02 -0,01 0.00 0.01 0.02 0,03 0,04 0,05 0,06 Delta NMHC g/kW-hr FIGURE 12. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NMHC METHOD 1 FOR MODS 1, 2 AND 3 SwRI Report 03.12859.06 19 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Models Mod 1, Mod 2 and Mod 3 NMHC (g/kW-hr) Method 2 100 en a. 80- 60 rt u E 40- o 20- —0.03 -0,01 0.00 0.01 Delta NMHC g/kW-hr 0,02 0.03 0.04 FIGURE 13. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NMHC METHOD 2 FOR MODS 1, 2 AND 3 SwRI Report 03.12859.06 20 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Models Mod 1, Mod 2 and Mod 3 NMHC (g/kW-hr) Method 3 100 en a. 80- 60 rt u E 40- o 20- —0.03 -0,01 0.00 0.01 Delta NMHC g/kW-hr 0,02 0.03 0.04 FIGURE 14. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NMHC METHOD 3 FOR MODS 1, 2 AND 3 SwRI Report 03.12859.06 21 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Models Mod 1, Mod 2 and Mod 3 CO (gykW-hr) Method 1 100 80- en a. O 9E>th % Mod 1 DeHa 23 ^m ^m 95th % Mod 2 Delta -0,60 —0,46 —0,30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1,05 Delta CO a/kW-hr FIGURE 15. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR CO METHOD 1 FOR MODS 1, 2 AND 3 SwRI Report 03.12859.06 22 of 83 ------- en a. rt D Ł D O 100 80- Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Models Mod 1, Mod 2 and Mod 3 CO (g/RW—hr) Method 2 5th % Mod 1 Delta 9E>th % Mod 1 De ta 23 ^a ^H 95th % Mod 2 Delta -0,60 —0,46 —0,30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1,05 Delta CO a/kW-hr FIGURE 16. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR CO METHOD 2 MODS 1, 2 AND 3 SwRI Report 03.12859.06 23 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Models Mod 1, Mod 2 and Mod 3 CO (g/RW—hr) Method 3 100 80- en a. O 9E>th % Mod 1 DeHa 23 ^a ^H 95th % Mod 2 Delta -0,60 —0,46 —0,30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1,05 Delta CO a/kW-hr FIGURE 17. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR CO METHOD 3 FOR MODS 1, 2 AND 3 SwRI Report 03.12859.06 24 of 83 ------- 6.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE EVENTS USING MOD B Since all three methods validated using the 23 reference NTE events by all three modification runs for BSNOX, BSNMHC and BSCO, the question arose as to which of the revised error surfaces had the most influence in the validation. Three possible explanations included (1) the steady-state CC>2 bias had been eliminated, (2) the change in the steady-state NOx error values, and (3) the bias in the steady-state exhaust flow rate had been eliminated. To study these possible explanations, none of the revised SSNOX error surfaces or the SSCO error surface was used in the simulations. These error surfaces were set to their original format in the original PEMS study. Therefore, only the steady-state exhaust flow rate and the steady-state CC>2 error surfaces were revised and the other remaining validation error surfaces were set to their original definitions for running the 23 reference NTE events for the Mod B MC simulation. This set of simulations is referred to as the 'Mod B' runs. Again, the simulations were performed at 10,000 trials for NTE events with ideal BSNOx values less than 2.68204 g/kW-hr and at 30,00 trials otherwise. All 23 reference NTE events met the convergence criteria listed in Table 5. From the Mod B MC simulations, the validation 5th and 95th percentile delta BS emissions were extracted from the output files for each of the 23 reference NTE events. These delta emissions were then plotted as empirical distribution functions (EDF) to form a validation interval for the on-road data. Also plotted was the EDF computed from the on-road CE-CERT NTE events. Figure 18 through Figure 20 represent the validation plots for the BSNOX for calculation methods 1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th percentile EDFs for the Mod B runs and the on-road CE-CERT data. Figure 21 through Figure 26 depict the validation plots for the BSNMHC and BSCO simulation runs, respectively. All BSNOx, BSNMHC and BSCO emissions validated for all three calculation methods for the Mod B runs. Conclusions made from the results of the Mod B analysis were that the steady-state NOx error surface changes did not have as much of an effect on the validation as the bias elimination in the steady-state CO2 and steady-state exhaust flow rate error surfaces. SwRI Report 03.12859.06 25 of 83 ------- c u m u I a t v e P e r c e n t Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas Error Model Mod B NCK (g/kW-hr) Method 1 100 5th % Mod B Delta -0.60 -0,45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 Delta NQx o/kW-hr FIGURE 18. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NOX METHOD 1 MOD B SwRI Report 03.12859.06 26 of 83 ------- c u m u I a t i v e P e r c e n t Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected SI CE—CERT Deltas Error Model Mod B NQx (g/kW-hr) Method 2 100 -0,60 -0.45 -0.30 -0.15 0.00 FIGURE 19. VALIDATION NTE EVENTS USING 23 REFERENCE NTE EVENTS FOR NOX METHOD 2 MOD B SwRI Report 03.12859.06 27 of 83 ------- c u m u I a t v e p e r c e n t 100' Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected SI CE—CERT Deltas Error Model Mod B NOK (g/kW—hr) Method 3 5th W Mod B Delta 5th % Mod B Ddta -0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 Delta NOx a/kW-hr FIGURE 20. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NOX METHOD 3 MOD B SwRI Report 03.12859.06 28 of 83 ------- c u m u I a t i v e P e r c e n t Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas Error Model Mod B NMHC (a/kW^nr) Method 1 100 60 40' 20- 0 -0.04 -0.03 -0.02 -0.01 0.00 0.01 0.02 0.03 0.04 0.05 0.06 Delta NMHC g/kW-hr FIGURE 21. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NMHC METHOD 1 MOD B SwRI Report 03.12859.06 29 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas Error Model Mod B NMHC (g/kW-hr) Method 2 100 C u m u I a t i v e P e r c e n t 60 40' 20- -0.03 -0,02 -0,01 0.00 0.01 Delta NMHC g/kW-hr 0.02 0,03 0.04 FIGURE 22. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NMHC METHOD 2 MOD B SwRI Report 03.12859.06 30 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas Error Model Mod B NMHC (g/kW-hr) Method 3 100 C u m u I a t i v e P e r c e n t 60 40' 20- -0.03 -0,02 -0,01 0.00 0.01 Delta NMHC g/kW-hr 0.02 0,03 0.04 FIGURE 23. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NMHC METHOD 3 MOD B SwRI Report 03.12859.06 31 of 83 ------- c u m u I a t i v e P e r c e n t 100 Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Model Mod B CO (g/kW-hr) Method 1 5th % Mod B Delta 95th »i Mod B De ta -0,60 -0,45 -0.30 -0.15 0,00 0.15 0.30 0,45 0.60 0,75 0,90 1.05 1.20 Delta CO g/kW-hr FIGURE 24. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR CO METHOD 1 MOD B SwRI Report 03.12859.06 32 of 83 ------- c u m u I a t i v e P e r c e n t Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Model Mod B CO (g/RW^nr) Method 2 100 5th % Mod B Delta 95th »i Mod B De ta -0,60 -0,45 -0.30 -0.15 0,00 0.15 0.30 0,45 0.60 0,75 0,90 1.05 1.20 Delta CO g/kW-hr FIGURE 25. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR CO METHOD 2 MOD B SwRI Report 03.12859.06 33 of 83 ------- c u m u I a t i v e P e r c e n t 100 Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Model Mod B CO (g/RW^nr) Method 3 5th % Mod B Delta 95th »i Mod B De ta -0,60 -0,45 -0,30 -0.15 0,00 0.15 0.30 0,45 0.60 0,75 0,90 1.05 1.20 Delta CO g/kW-hr FIGURE 26. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR CO METHOD 3 MOD B SwRI Report 03.12859.06 34 of 83 ------- 7.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE EVENTS USING MOD 1 Based on the information provided in the validation plots for the Mod 1, Mod 2, Mod 3 and Mod B runs, changes made to the various steady-state error surfaces resulted in the validation of the MC model for all three emissions and all three calculation methods for each of the four Mod runs using the 23 reference NTE events. EPA chose to continue the simulation runs using only the Mod 1 and Mod 2 revised error surfaces by running an additional 27 reference NTE events (total = 50 reference NTE events) through the MC model. These two Mods were chosen because they both represented steady-state NOx error surfaces that were level independent (as compared to the Mod 3 steady-state error surface that represented a combination of level dependent and level independent NOx errors). The results from these 50 simulations were used to calculate the measurement allowances provided in Section 9.0 of this report. This section details the results of the validation based on the 50 reference NTE events run with Mod 1. The Mod 1 simulations were performed at 10,000 trials for NTE events with ideal BSNOx values less than 2.68204 g/kW-hr and at 30,00 trials otherwise. All 50 reference NTE events met the convergence criteria listed in Table 5. From the Mod 1 MC simulations, the validation 5th and 95th percentile delta BS emissions were extracted from the output files for each of the 50 reference NTE events. These percentiles were then plotted as empirical distribution functions (EDF) to form a validation interval for the on-road data. Also plotted was the EDF computed from the on-road CE-CERT NTE events. Figure 27 through Figure 29 represent the validation plots for the BSNOX for calculation methods 1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th percentile EDFs for the Mod 1 runs and the on-road CE-CERT data. Figure 30 through Figure 35 depict the validation plots for the BSNMHC and BSCO simulation runs, respectively. All BSNOx, BSNMHC and BSCO emissions validated for all three calculation methods for the Mod 1 runs with the 50 reference NTE events. SwRIReport03.12859.06 35 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas Error Model Mod 1 NCK (g/kW—hr) Method 1 100' 80 60 < 40' 20 C u m u I a t v e p e r c e n t 0 -0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 Delta NQx s/kW-hr FIGURE 27. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NOX METHOD 1 MOD 1 SwRI Report 03.12859.06 36 of 83 ------- c u m u I a t i v e P e r c e n t Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected SI CE—CERT Deltas Error Model Mod 1 NQx (g/kW-hr) Method 2 100 5th % Mod B Delta -0,60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 Delta NQx g/kW—hr FIGURE 28. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NOX METHOD 2 MOD 1 SwRI Report 03.12859.06 37 of 83 ------- c u m u I a t i v e P e r c e n t Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas Error Model Mod 1 NQx (g/kW-hr) Method 3 100 5th % Mod B Delta -0,60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 Delta NQx g/kW—hr FIGURE 29. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NOX METHOD 3 MOD 1 SwRI Report 03.12859.06 38 of 83 ------- c u m u I a t i v e P e r c e n t Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas Error Model Mod 1 NMHC (a/kW^nr) Method 1 100 60 40' 20- 0 -0.04 -0.03 -0.02 -0.01 0.00 0.01 0.02 0.03 0.04 0.05 0.06 Delta NMHC g/kW-hr FIGURE 30. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NMHC METHOD 1 MOD 1 SwRI Report 03.12859.06 39 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Model Mod 1 NMHC (g/kW-hr) Method 2 100 C u m u I a t i v e P e r c e n t 60 40' 20- 15th M Mud B Delta I -0.03 -0,02 -0,01 0.00 0.01 Delta NMHC g/kW-hr 0.02 0,03 0.04 FIGURE 31. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NMHC METHOD 2 MOD 1 SwRI Report 03.12859.06 40 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Model Mod 1 NMHC (g/kW-hr) Method 3 100 C u m u I a t i v e P e r c e n t 60 40' 20- -0.03 -0,02 -0,01 0.00 0.01 Delta NMHC g/kW-hr 0.02 0,03 0.04 FIGURE 32. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NMHC METHOD 3 MOD 1 SwRI Report 03.12859.06 41 of 83 ------- c u m u I a t i v e P e r c e n t 100 Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas Error Model Mod 1 CO (g/kW-hr) Method 1 5th % Mod B Delta 95th % Mod 1 De 1 -0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1.05 Delta CO Q/kW-hr FIGURE 33. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR CO METHOD 1 MOD 1 SwRI Report 03.12859.06 42 of 83 ------- c u m u I a t i v e P e r c e n t Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas Error Model Mod 1 CO (g/RW^nr) Method 2 100 5th % Mod B Delta 95th % Mod 1 De 1 -0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1.05 Delta CO Q/kW-hr FIGURE 34. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR CO METHOD 2 MOD 1 SwRI Report 03.12859.06 43 of 83 ------- c u m u I a t i v e P e r c e n t Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas Error Model Mod 1 CO (g/RW^nr) Method 3 100 5th % Mod B Delta 95th % Mod 1 De 1 -0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1.05 Delta CO Q/kW-hr FIGURE 35. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR CO METHOD 3 MOD 1 SwRI Report 03.12859.06 44 of 83 ------- 8.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE EVENTS USING MOD 2 This section details the results of the validation based on the 50 reference NTE events run with Mod 2. EPA chose to continue simulations using the Mod 1 and Mod 2 revised error surfaces by running an additional 27 reference NTE events (total = 50 reference NTE events) through the MC model. These two Mods were chosen because they both represented steady- state NOx error surfaces that were level independent (as compared to the Mod 3 steady-state error surface that represented a combination of level dependent and level independent NOX errors). The Mod 2 simulations were performed at 10,000 trials for NTE events with ideal BSNOx values less than 2.68204 g/kW-hr and at 30,00 trials otherwise. All 50 reference NTE events met the convergence criteria listed in Table 5. From the Mod 2 MC simulations, the validation 5th and 95th percentile delta BS emissions were extracted from the output files for each of the 50 reference NTE events. These percentile delta emissions were then plotted as empirical distribution functions (EDF) to form a validation interval for the on-road data. Also plotted was the EDF computed from the on-road CE-CERT NTE events. Figure 36 through Figure 38 represent the validation plots for the BSNOX for calculation methods 1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th percentile EDFs for the Mod 2 runs and the on-road CE-CERT data. Figure 39 through Figure 44 depict the validation plots for the BSNMHC and BSCO simulation runs, respectively. All BSNOx, BSNMHC and BSCO emissions validated for all three calculation methods for the Mod 2 runs with the 50 reference NTE events. SwRI Report 03.12859.06 45 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas Error Model Mod 2 NCK (g/kW—hr) Method 1 100' 80 60 < 40' 20 C u m u I a t v e p e r c e n t 95ih % Mod 2 Delta I 0 -0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 Delta NQx s/kW-hr FIGURE 36. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NOX METHOD 1 MOD 2 SwRI Report 03.12859.06 46 of 83 ------- c u m u I a t v e p e r c e n t Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas Error Model Mod 2 NOK (g/kW—hr) Method 2 100' 5th W Mod 2 Delta 99th % Mod 2 Delta -0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 Delta NOx a/kW-hr FIGURE 37. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NOX METHOD 2 MOD 2 SwRI Report 03.12859.06 47 of 83 ------- c u m u I a t v e p e r c e n t Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas Error Model Mod 2 NOK (g/kW—hr) Method 3 100' 5th W Mod 2 Delta 99th % Mod 2 Delta -0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 Delta NOx a/kW-hr FIGURE 38. VALIDATION EDF PLOTS FOR 50 REFERENCE NTE EVENTS FOR NOX METHOD 3 MOD 2 SwRI Report 03.12859.06 48 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Model Mod 2 NMHC (g/KW-hr) Method 1 100' 80 60 < 40' 20 C u m u I a t v e p e r c e n t 0 -0.04 -0.03 -0.02 -0.01 0.00 0.01 0.02 0.03 0.04 0.05 0.06 Delta NMHC gykW-hr FIGURE 39. VALIDATION EDF PLOTS FOR 50 REFERENCE NTE EVENTS FOR NMHC METHOD 1 MOD 2 SwRI Report 03.12859.06 49 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Model Mod 2 NMHC (g/kW-hr) Method 2 100' 80 60 < 40' 20 C u m u I a t v e p e r c e n t 0 -0.04 -0.03 -0.02 -0.01 0.00 0.01 0.02 0.03 0.04 0.05 0.06 Delta NMHC gykW-hr FIGURE 40. VALIDATION EDF PLOTS FOR 50 REFERENCE NTE EVENTS FOR NMHC METHOD 2 MOD 2 SwRI Report 03.12859.06 50 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Model Mod 2 NMHC (g/kW-hr) Method 3 100' 80 60 < 40' 20 C u m u I a t v e p e r c e n t 15th M Mod 2 Delta I -0.03 -0,02 -0,01 0.00 0.01 Delta NMHC gykW-hr 0.02 0.03 0,04 FIGURE 41. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NMHC METHOD 3 MOD 2 SwRI Report 03.12859.06 51 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Model Mod 2 CO (g/kW-hr) Method 1 C u m u I a t v e p e r c e n t 100' 80 60 5th W Mod 2 Delta -0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1.05 Delta CO g/kW-hr FIGURE 42. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR CO METHOD 1 MOD 2 SwRI Report 03.12859.06 52 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Model Mod 2 CO (g/RW—hr) Method 2 c u m u I a t v e p e r c e n t 100' 80 60 5th W Mod 2 Delta -0.60 -Ci.45 -O.30 -0.15 0,00 0.15 0,30 0,45 0,60 0.75 0.90 1,05 Delta CO a/kW-hr FIGURE 43. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR CO METHOD 2 MOD 2 SwRI Report 03.12859.06 53 of 83 ------- Validation Analysis 5th and 95th Percentile Deltas Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas Error Model Mod 2 CO (g/RW—hr) Method 3 c u m u I a t v e p e r c e n t 100' 80 60 5th W Mod 2 Delta -0.60 -Ci.45 -O.30 -0.15 0,00 0.15 0,30 0,45 0,60 0.75 0.90 1,05 Delta CO a/kW-hr FIGURE 44. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR CO METHOD 3 MOD 2 Table 7 provides a summary of all the validation results for the various MC simulations as described in the above sections of this report. All emissions and all calculation methods validated using the Mod 1 and Mod 2 runs with 50 reference NTE events while the Mod 3 and Mod B runs validated using 23 reference NTE events. SwRI Report 03.12859.06 54 of 83 ------- TABLE 7. SUMMARY OF VALIDATION RESULTS BSNOx Method 1 BSNOx Method 2 BSNOx Method 3 BSNMHC Method 1 BSNMHC Method 2 BSNMHC Method 3 BSCO Method 1 BSCO Method 2 BSCO Method 3 Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated Validated SwRI Report 03.12859.06 55 of 83 ------- 9.0 MEASUREMENT ALLOWANCE CALCULATIONS As detailed in the original PEMS study, the measurement error allowances were computed using both a regression method and a median method to determine the measurement allowance. The procedure was applied to the simulation data for the 50 reference NTE events from the Mod 1 and Mod 2 runs for each of the three emissions and all three calculation methods. Figure 45 contains a regression plot of the 95th percentile delta BSNOX values (using Method 1 Mod 1) versus the ideal BSNOX values for the 50 reference NTE events. Included within the plot is the equation for the fitted regression line and the R-square (R2) value. Table 8 includes a comparison of the results of the regression method based on Figure 45 and the median method (described in Ref 1). Under the heading "Regression Method" in the table, it is shown that the R-square criterion is not met by the data (R-square must be > 0.90). Thus, the median method must be used. Under the heading "Median Method" in the table, the measurement error at the BSNOX threshold, based on using the median of the fifty 95th percentile delta BSNOX values, is 11.5932% when expressed as a percent of the threshold value of 2.68204 g/kW-hr. Similar regression plots and measurement error tables are provided in the remaining part of this section for the Mod 1 and Mod 2 results based on the 50 reference NTE events. Figure 45 through Figure 47 and Table 8 through Table 10 provide results for BSNOX Mod 1. Figure 48 through Figure 50 and Table 11 through Table 13 provide results for BSNMHC Mod 1, and Figure 51 through Figure 53 and Table 14 through Table 16 provide results for BSCO Mod 1. Mod 2 results for BSNOX can be found in Figure 54 through Figure 56 and Table 17 through Table 19, BSNMHC results are shown in Figure 57 through Figure 59 and Table 20 through Table 22 and BSCO results are shown in Figure 60 through Figure 62 and Table 23 through Table 25. SwRI Report 03.12859.06 56 of 83 ------- NOx g/kW-hr Method 1 Mod 1 0.90 -, 0.80 0.70 .c | 0.60 x „ -,. O 0.50 z TO o 0.40 Q ^ 0.30 u> OT 0.20 - 0.10 0.00 c 50 Ref NTE Events With Time Alignment Adjustment * ^ » ,S * *^ 4 ** ^^ * * |