Additional Analyses of the Monte Carlo
Model Developed for the Determination
of PEMS Measurement Allowances for
Gaseous Emissions Regulated Under
the Heavy-Duty Diesel Engine In-Use
Testing Program
United States
Environmental Protection
Agency
-------
Additional Analyses of the Monte Carlo
Model Developed for the Determination
of PEMS Measurement Allowances for
Gaseous Emissions Regulated Under
the Heavy-Duty Diesel Engine In-Use
Testing Program
Assessment and Standards Division
Office of Transportation and Air Quality
U.S. Environmental Protection Agency
Prepared for EPA by
Southwest Research Institute
EPA Contract No. EP-C-05-018
Work Assignment No. 2-6
v>EPA
NOTICE
This technical report does not necessarily represent final EPA decisions or
positions. It is intended to present technical analysis of issues using data
generated in the associated test program. The purpose in the release of such
reports is to facilitate the exchange of technical information and to inform the
public of technical developments which may form the basis for a final EPA
decision, position, or regulatory action.
United States EPA420-R-07-010
Environmental Protection . ^ „„.,
Agency August 2007
-------
EXECUTIVE SUMMARY
This report documents the program conducted by Southwest Research Institute® (SwRI),
on behalf of the U.S. Environmental Protection Agency (EPA), the objective of which was to
perform additional analyses on the Portable Emissions Measurement Systems (PEMS) Monte
Carlo simulation models in order to determine validation of the three emissions (BSNOx,
BSNMHC and BSCO) using three calculation methods.
Several steady-state error surfaces were modified based on recommendations from the
Heavy-Duty In-Use Testing (HDIUT) Steering Committee. These included modifications to the
steady-state NOx, exhaust flow rate, CO and CO2 error surfaces. Several reference NTE events
were run which produced validation results from all three emissions and all three calculation
methods.
Measurement allowances were computed for two of the simulation strategies based on
using 50 reference NTE events. The Mod 1 strategy involved changing the steady-state CO, CO2
and exhaust flow rate error surfaces to eliminate bias while changing the steady-state NOx to a
level independent error surface including all the test data. The Mod 2 strategy was the same as
Mod 1 except the steady-state NOx error surface was also changed to a level independent error
surface but excluded several questionable low NOx values from one of the test engines. The
measurement allowance values by calculation method determined at the conclusion of these
analyses are summarized in Table 1. This report details the process used to determine the
measurement values reported in Table 1.
TABLE 1. MEASUREMENT ALLOWANCES FOR MOD 1 AND MOD 2
Pollutant
NOX
NMHC
CO
Calculation
Method
1
2
3
1
2
3
1
2
3
Mod 1 Measurement
Allowance, g/hp-hr
0.232
0.178
0.192
0.015
0.014
0.014
0.268
0.258
0.270
Mod 2 Measurement
Allowance, g/hp-hr
0.211
0.151
0.171
0.014
0.014
0.014
0.266
0.250
0.262
SwRI Report 03.12859.06
-------
LIST OF ACRONYMS
Brake-Specific BS
California Air Resources Board CARB
Center for Environmental Research
& Technology CE-CERT
Code of Federal Regulations CFR
Electronic Flow Meter EFM
Empirical Distribution Function EDF
Engine Control Module ECM
Engine Manufacturer's Association EMA
Environmental Protection Agency EPA
Heavy Duty In-Use Testing HDIUT
Heavy Heavy Duty HHD
Light Heavy Duty LHD
Median Absolute Deviation MAD
Medium Heavy Duty MHD
Memorandum of Agreement MO A
Mobile Emissions Laboratory MEL
Not To Exceed NTE
Portable Emissions Measurement System PEMS
Root Mean Square RMS
SEMTECH-DS SN G05-SDS04 PEMS 1
SEMTECH-DS SN G05-SDS02 PEMS 2
SEMTECH-DS SN G05-SDS03 PEMS 3
SEMTECH-DS SN G05-SDS01 PEMS 4
SEMTECH-DS SN D06-SDS01 PEMS 5
SEMTECH-DS SN D06-SDS06 PEMS 6
SEMTECH-DS SN F06-SDS02 PEMS 7
Southwest Research Institute SwRI
Standard Deviation SD
SwRI Report 03.12859.06
11
-------
TABLE OF CONTENTS
EXECUTIVE SUMMARY i
LIST OF ACRONYMS ii
LIST OF FIGURES iv
LIST OF TABLES vi
1.0 INTRODUCTION 1
2.0 ERROR SURFACE MODIFICATIONS 2
2.1 Steady-State Exhaust Flow Rate Error Surface 2
2.2 Steady-State CO2 Error Surface 3
2.3 Steady-State CO Error Surface 4
2.4 Steady-State NOX Error Surface 5
3.0 REFERENCE NTE EVENTS 9
4.0 CE-CERT MEASUREMENTS 13
5.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE EVENTS
USING MOD 1, MOD 2 AND MOD 3 14
6.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE EVENTS
USING MODE 25
7.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE EVENTS
USING MOD 1 35
8.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE EVENTS
USING MOD 2 45
9.0 MEASUREMENT ALLOWANCE CALCULATIONS 56
10.0 VALIDATION SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS.... 77
11.0 FULL MODEL SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS ... 81
12.0 REFERENCES 83
SwRI Report 03.12859.06 iii
-------
LIST OF FIGURES
1 Revised Error Surface for Steady-State Exhaust Flow Rate 3
2 Revised Error Surface for Steady-State CC>2 4
3 Revised Error Surface for Steady-State CO 5
4 Revised Error Surface Mod 1 for Steady-State NOX 6
5 Revised Error Surface Mod 2 for Steady-State NOX 7
6 Revised Error Surface Mod 3 for Steady-State NOX 8
7 Distribution of Ideal NOx (g/kW-hr) Method 1 for 23 Selected Ref NTE
Events and 195 Ref NTE Events 11
8 Distribution of Ideal NOx (g/kW-hr) Method 1 for 50 Selected Ref NTE
Events and 195 Ref NTE Events 12
9 Validation EOF Plots Using 23 Reference NTE Events for NOX Method 1
Mods 1,2 and 3 16
10 Validation EOF Plots Using 23 Reference NTE Events for NOX Method 2 for
Mods 1,2 and 3 17
11 Validation EOF Plots Using 23 Reference NTE Events for NOX Method 3 for
Mods 1,2 and 3 18
12 Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 1
for Mods 1,2 and 3 19
13 Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 2
for Mods 1,2 and 3 20
14 Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 3
for Mods 1,2 and 3 21
15 Validation EOF Plots Using 23 Reference NTE Events for CO Method 1 for
Mods 1,2 and 3 22
16 Validation EOF Plots Using 23 Reference NTE Events for CO Method 2 Mods
1,2 and 3 23
17 Validation EOF Plots Using 23 Reference NTE Events for CO Method 3 for
Mods 1,2 and 3 24
18 Validation EOF Plots Using 23 Reference NTE Events for NOX Method 1 Mod B 26
19 Validation NTE Events Using 23 Reference NTE Events for NOX Method 2 Mod B. .27
20 Validation EOF Plots Using 23 Reference NTE Events for NOX Method 3 Mod B 28
21 Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 1 Mod B .. 29
22 Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 2 Mod B .. 3 0
23 Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 3 Mod B .. 31
24 Validation EOF Plots Using 23 Reference NTE Events for CO Method 1 Mod B 32
25 Validation EOF Plots Using 23 Reference NTE Events for CO Method 2 Mod B 33
26 Validation EOF Plots Using 23 Reference NTE Events for CO Method 3 Mod B 34
27 Validation EOF Plots Using 50 Reference NTE Events for NOX Method 1 Mod 1 36
28 Validation EOF Plots Using 50 Reference NTE Events for NOX Method 2 Mod 1 37
29 Validation EOF Plots Using 50 Reference NTE Events for NOX Method 3 Mod 1 38
30 Validation EOF Plots Using 50 Reference NTE Events for NMHC Method 1 Mod 1...39
31 Validation EOF Plots Using 50 Reference NTE Events for NMHC Method 2 Mod 1.. .40
32 Validation EOF Plots Using 50 Reference NTE Events for NMHC Method 3 Mod 1.. .41
33 Validation EOF Plots Using 50 Reference NTE Events for CO Method 1 Mod 1 42
34 Validation EOF Plots Using 50 Reference NTE Events for CO Method 2 Mod 1 43
SwRI Report 03.12859.06 iv
-------
3 5 Validation EOF Plots Using 50 Reference NTE Events for CO Method 3 Mod 1 44
36 Validation EOF Plots Using 50 Reference NTE Events for NOX Method 1 Mod 2 46
37 Validation EOF Plots Using 50 Reference NTE Events for NOX Method 2 Mod 2 47
38 Validation EOF Plots for 50 Reference NTE Events for NOX Method 3 Mod 2.. .48
39 Validation EOF Plots for 50 Reference NTE Events for NMHC Method 1 Mod 2 49
40 Validation EOF Plots for 50 Reference NTE Events for NMHC Method 2 Mod 2 50
41 Validation EOF Plots Using 50 Reference NTE Events for NMHC Method 3 Mod 2... 51
42 Validation EOF Plots Using 50 Reference NTE Events for CO Method 1 Mod2 52
43 Validation EOF Plots Using 50 reference NTE Events for CO Method 2 Mod 2.53
44 Validation EOF Plots Using 50 Reference NTE Events for CO Method 3 Mod 2 54
45 Regression Plot of 95th Percentile Delta BSNOX Versus Ideal BSNOX for Method 1
Modi 57
46 Regression Plot of 95th Percentile Delta BSNOX Versus Ideal BSNOX for Method 2
Modi 58
47 Regression Plot of 95th percentile Delta B SNOX Versus Ideal B SNOX for Method 3
Modi 59
48 Regression Plot of 95th percentile Delta BSNMHC Versus Ideal BSNMHC for
Method 1 Modi 60
49 Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for
Method2Modl 61
5 0 Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for
Method 3 Modi 62
51 Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 1
Modi 63
52 Regression Plot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 2
Modi 64
5 3 Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 3
Modi 65
54 Regression Plot of 95th Percentile Delta BSNOx Versus Ideal BSNOx for Method 1
Mod2 66
55 Regression Plot of 95th Percentile Delta BSNOX Versus Ideal BSNOX for Method 2
Mod2 67
56 Regression Plot of 95th Percentile Delta BSNOX Versus Ideal BSNOX for Method 3
Mod2 68
5 7 Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for
Method !Mod2 69
5 8 Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for
Method2Mod2 70
5 9 Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for
Method 3 Mod 2 71
60 Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 1
Mod 2 72
61 Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 2
Mod 2 73
62 Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 3
Mod 2 74
SwRI Report 03.12859.06
-------
LIST OF TABLES
Table Page
1 Measurement Allowances For MOD 1 and Mod 2 i
2 Revised Error Surfaces Used in Monte Carlo Simulations 8
3 Reference NTE Events Used in Monte Carlo Simulations 10
4 Descriptive Statistics for Ideal BSNOX (g/kW-hr) for Various NTE Subsets 12
5 Convergence criteria by Emission 14
6 BSNOX Validation Results Based on 23 Reference NTE Events 15
7 Summary of Validation Results 55
8 Measurement Error at Threshold for BSNOX Using Regression and Median
Methods for Method 1 Mod 1 57
9 Measurement Error at Threshold for BSNOX Using Regression and Median
Methods for Method 2 Mod 1 58
10 Measurement Error at Threshold for BSNOX Using Regression and Median
Methods for Method 3 Mod 1 59
11 Measurement Error at Threshold for BSNMHC Using Regression and Median
Methods for Method 1 Mod 1 60
12 Measurement Error at Threshold for BSNMHC Using Regression and Median
Methods for Method 2 Mod 1 61
13 Measurement Error at Threshold for BSNMHC Using Regression and Median
Methods for Method 3 Mod 1 62
14 Measurement Error at Threshold for B SCO Using Regression and Median
Methods for Method 1 Mod 1 63
15 Measurement Error at Threshold for B SCO Using Regression and Median
Methods for Method 2 Mod 1 64
16 Measurement Error at Threshold for B SCO Using Regression and Median
Methods for Method 3 Mod 1 65
17 Measurement Error at Threshold for BSNOX Using regression and Median
methods for Method 1 Mod 2 66
18 Measurement Error at Threshold for BSNOX Using Regression and Median
methods for Method 2 Mod 2 67
19 Measurement Error at Threshold for BSNOX Using Regression and Median
Methods for Method 3 Mod 2 68
20 Measurement Error at Threshold for BSNMHC Using Regression and Median
Methods for Method 1 Mod 2 69
21 Measurement Error at Threshold for BSNMHC Using Regression and Median
Methods for Method 2 Mod 2 70
22 Measurement Error at Threshold for BSNMHC Using Regression and Median
Methods for Method 3 Mod 2 71
23 Measurement Error at Threshold for B SCO Using Regression and Median
Methods for Method 1 Mod 2 72
24 Measurement Error at Threshold for B SCO Using Regression and Median
Methods for Method 2 Mod 2 73
SwRI Report 03.12859.06 vi
-------
25 Measurement Error at Threshold for B SCO Using Regression and Median
Methods for Method 3 Mod 2 74
26 Summary of Measurement Errors at Respective Threshold (%) 75
27 Summary of Measurement Allowance, g/hp-hr 76
28 Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte
Carlo validation Simulations for BSNOX 78
29 Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte
Carlo validation Simulations for BSNMHC 79
30 Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte
Carlo validation Simulations for BSCO 80
31 Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte
Carlo full model Simulations for BSNOX 82
SwRI Report 03.12859.06 vii
-------
1.0 INTRODUCTION
Southwest Research Institute completed a Portable Emissions Measurement System
(PEMS) program on behalf of the U.S. Environmental Protection Agency (EPA) in early April
2007. The purpose of this project was to determine the brake-specific (BS) measurement
allowances for the gaseous pollutants regulated under the Heavy-Duty In-Use Testing (HDIUT)
program [Ref 1]. The study was performed under cooperation between the EPA, the California
Air Resources Board (CARB), and the Engine Manufacturer's Association (EMA). All efforts
during this program were conducted under the direction of a joint body, the HDIUT
Measurement Allowance Steering Committee, referred to in this report simply as the Steering
Committee.
The program consisted of modeling various PEMS measurement errors using a statistical
Monte Carlo modeling simulation. The simulation results were used to generate the brake-
specific measurement allowances based on three calculation methods for BS emissions of NOX,
NMHC and CO. To confirm the results of the simulation, a SEMTECH-DS PEMS was operated
in-use with the CE-CERT Mobile Emission Laboratory (MEL) over several routes in California.
The differences between the PEMS and MEL gaseous emission measurements were used to
validate the SwRI Monte Carlo modeling simulation results.
Calculation Methods 2 (BSFC based) and 3 (ECM Fuel Specific) did not validate for
BSNOX based on the CE-CERT MEL emissions and the Monte Carlo PEMS simulations. Given
this result, EPA contracted with SwRI in April 2007 to conduct additional analyses to identify
possible causes as to why the BSNOX did not validate. The results of these analyses are included
in this report.
SwRI Report 03.12859.06 1 of 83
-------
2.0 ERROR SURFACE MODIFICATIONS
Since calculation Methods 2 and 3 did not validate for BSNOX after the Monte Carlo
model simulation runs during the original PEMS study, the associated Steering Committee chose
to modify key error surfaces in order to determine if such alterations would effect the validation
of Methods 2 and 3. Over the course of several meetings, the Steering Committee brainstormed
to generate alternate error surface processing methods. The Committee decided to modify
Steady-State CO, CO2, Exhaust Flow Rate, and NOX, as these surfaces had significant influence
on the Model results based on the sensitivity analyses.
In reprocessing the error surfaces, the Steering Committee agreed that it would be
necessary to remove any biases that were recorded in the SwRI laboratory. It was assumed that
these bias errors were due to the limited number of PEMS units and comparative observations,
and that if more PEMS and Sensors Inc. exhaust flow meters were tested the biases may have
been eliminated. Therefore, to remove the biases the original steady-state error surfaces were
transformed to be a symmetric error surface with the 50th percentile errors set to zero and the 5th
and 95th percentile errors modified to be a mirror image of one another. Two analysis methods
were used to generate the revised error surfaces. If the recorded error data showed an emissions
level dependency, an envelope was generated to encompass the extreme 5th or 95th percentile
error data. The envelope segments with the largest absolute error were then mirrored to generate
a symmetric error surface. A more rigorous analysis was used to reprocess level independent
error surfaces.
2.1 Steady-State Exhaust Flow Rate Error Surface
Figure 1 shows the reprocessed, level-dependent steady-state exhaust flow rate error
surface as well as the exhaust flow error surface data that was used in the original MC model
runs. The largest errors recorded during laboratory testing were positive deltas measured during
Engine 3 testing with the 3-inch EFMs. The 3-inch EFM errors defined the 95th percentile error
contours for the original error surface as well as the reprocessed error surface. The difference
between the two 95l percentile contours was due to a correction of the 3-inch EFM calibrations
when the exhaust flow rate error surface was reprocessed. SwRI and Sensors Inc. had
recalibrated the 3-inch flow meters using data generated at SwRI. The SwRI calibration
increased the slope multiplier by approximately 4 % for each 3-inch flow meter. Later in the
program, Sensors Inc. discovered the 3-inch EFMs were likely not operating correctly on the
SwRI flow stand due to EFM pressures below ambient levels. Therefore, each 3-inch flow meter
calibration was corrected to the original coefficients generated by Sensors. The corrected 3-inch
flow meter data generated the 95th percentile contour shown in Figure 1. Because the 95th
percentile errors were larger than the 5th percentile errors, the 95th percentile data was mirrored to
generate the 5th percentile contour in the reprocessed error surface.
SwRI Report 03.12859.06 2 of 83
-------
-95th percentile
• Previous 95th
-50th percentile (median)
• Previous 50th
-5th percentile
• Previous 5th
Lab Reference Mean Exhaust Flow Rate [% of Max]
FIGURE 1. REVISED ERROR SURFACE FOR STEADY-STATE EXHAUST FLOW
RATE
2.2 Steady-State CO2 Error Surface
th
Figure 2 shows the reprocessed, level-dependent CC>2 error surface. The 95 percentile
th
contour represents the envelope that was created to encompass the original 95 percentile error
data. Line segments were used to connect the most extreme points from the original 95
th
-th
th
percentile data. The 95 percentile contour was mirrored to produce the 5 percentile error data,
with the 50th percentile deltas set to zero. The reprocessed error surface was meant to encompass
all possible PEMS CC>2 measurement errors, not just those measured at SwRI.
SwRI Report 03.12859.06
3 of 83
-------
-95th percentile
• Previous 95th
-50th percentile (median)
• Previous 50th
-5th percentile
• Previous 5th
0.8
0.6
" 0.4
0.2
0.0
c
o
o
CM
O
O
40
-0.2
-0.4
-0.6
-0.8
4.5
5.0
5.5
6.0
6.5
7.0
7.5
8.0
8.5
90
Lab Reference Mean CO2 Concentration [%]
FIGURE 2. REVISED ERROR SURFACE FOR STEADY-STATE CO2
2.3 Steady-State CO Error Surface
Figure 3 shows the revised steady-state error surface for CO. Because the CO error data
showed no level dependence, a statistical calculation process, developed by Bill Martin from
Cummins Inc., was used to generate the reprocessed CO error surface. The total standard
deviation of the pooled CO error data was calculated by combining the variance of the PEMS
mean delta standard deviation about zero and the PEMS pooled estimate of repeatability standard
deviation. The calculation of the total standard deviation is shown below.
^U total \
repeatability ~~ - mean, zero
repeatability
IV S7) 2
/ j ^-^indiv,perns
n
SDindiV,pems = standard deviation of each PEMS calculated over 20 repeats of each steady-state
point
n = 10 steady-state points per engine * 3 engines * 3 PEMS = 90
SwRI Report 03.12859.06
4 of 83
-------
SD,
n-l
PEMSindiv,mean = mean delta of each PEMS calculated over 20 repeats of each steady-state point
Shown below, the multiplier for a 90% confidence interval assuming a normal
distribution for the errors was applied to the total standard deviation (48.84 ppm = 0.004882%)
to generate the constant 5th and 95th percentile error values.
5* % = -1.645 * SDtotal = - 0.008%
95*% = 1.645 * SDtotal = + 0.008%
•95th percentile
• Previous 95th
•50th percentile (median)
• Previous 50th
-5th percentile
• Previous 5th
0.012
0.010
0.008 - - 4M»-
0.006
0.004
5. 0.002
LU
Q.
•4*4-
« «»
0.000
o.opoo
-0.002
-0.004
-0.006
-0.008 \ - - |
-0.010
0.0005
0.0010
0.0015
0.0020
0.0 )25
Lab Reference Mean CO Concentration [%]
FIGURE 3. REVISED ERROR SURFACE FOR STEADY-STATE CO
2.4 Steady-State NOX Error Surface
For reprocessing the original steady-state NOX, three different methods were used to
generate three NOX concentration error surfaces. For each of the three modified NOX error
surfaces, the original set of Engine 2 steady-state data was added to the original pooled data set.
The Engine 2 steady-state data was collected in a repeat run in the original study due to PEMS
failures that caused a large time gap in the original data. The time gap was associated with a
NOX concentration shift. Since the NOX shift would have caused problems with the steady-state
SwRI Report 03.12859.06
5 of 83
-------
variance correction of the transient data the repeat run data was used. However, the original
Engine 2 delta data was later determined to be valid and thus was included in the modified data
set to increase the PEMS delta observations.
The first NOX error surface modification (Mod 1) followed a procedure similar to the
-state CO error surface. Using the complete NOX delta data set, the 5th and 95th percentile
error values were calculated with the total standard deviation (15.08 ppm). The results of the
first NOX erro
±24.80 ppm.
steady-state CO error surface. Using the complete NOX delta data set, the 5th and 95th percentile
>8l ,
first NOX error surface modification are shown in Figure 4 with the 5l and 95l percentiles set to
40
20 -
Q.
Q.
I -20 H
re
m
§ -40
O
5
LU
0.
-80
-100
•95th percentile
• Previous 95th
-50th percentile (median)
• Previous 50th
•5th percentile
• Previous 5th
,- .2.00
300
400
500
600
Lab Reference Mean NOx Concentration [ppm]
FIGURE 4. REVISED ERROR SURFACE MOD 1 FOR STEADY-STATE NOX
The second NOX error surface modification (Mod 2) was similar to the first modification
except six outlying low delta values recorded during Engine 3 testing were removed from the
pooled data set. During Engine 3 steady-state testing, PEMS 1 and 6 showed extremely low
deltas at high NOX concentration levels. PEMS 4 also showed low biases at high NOX levels;
however, the biases were not as large as those measured for the other PEMS. Therefore, PEMS
1 and 6 delta measurements at the three highest NOX levels measured during engine testing were
removed from the data set. This resulted in a total standard deviation equal to 9.39 ppm and the
-th
-th
5 and 95 percentile deltas equal to ±15.45 ppm. The results of this analysis are shown in
Figure 5.
SwRI Report 03.12859.06
6 of 83
-------
-95th percentile
• Previous 95th
-50th percentile (median)
• Prefious 50th
-5th percentile
• Previous 5th
-100
Lab Reference Mean NOx Concentration [ppm]
FIGURE 5. REVISED ERROR SURFACE MOD 2 FOR STEADY-STATE NOX
Shown in Figure 6, the third NOX error surface modification (Mod 3) combined the level-
dependent and level-independent analysis methods. Below 311 ppm, the delta data was assumed
to be level independent. Therefore, a statistical method similar to that used for CO was used for
the NOX delta data below 311 ppm. The total standard deviation for the level-independent data
was 9.41 ppm and the resulting 5th and 95th percentiles were ± 15.48 ppm. Above 311 ppm, the
original 5l percentile profile was used and mirrored to the 95th percentile to generate a
symmetric error surface.
SwRI Report 03.12859.06
7 of 83
-------
-95th percentile
• Previous 95th
-50th percentile (median)
• Previous 50th
-5th percentile
• Previous 5th
100
80
60
40
20
0
-20
-40
-60
-80
-100
Lab Reference Mean NOx Concentration [ppm]
FIGURE 6. REVISED ERROR SURFACE MOD 3 FOR STEADY-STATE NOX
In addition to running the MC simulations with the three different changes to the steady-
state NOx error surface, it was useful to run a simulation with the original steady-state NOx error
surface and only include the revised steady-state exhaust flow rate and steady-state CO2 error
surfaces. This set of simulations is called the Mod B runs. Table 2 provides a summary of the
four MC simulation runs made in this study with the revised error surfaces that were included in
each run. All the other error surfaces that are not listed in Table 2 are the same as those run in
the original PEMS study.
TABLE 2. REVISED ERROR SURFACES USED IN MONTE CARLO SIMULATIONS
Steady-State Exhaust Flow
Rate
Steady-State CO2
Steady-State CO
Steady- State NOX Mod 1
Steady- State NOX Mod 2
Steady-State NOX Mod 3
X
X
X
X
X
X
X
X
X
X
X
X
X
X
SwRI Report 03.12859.06
8 of 83
-------
3.0 REFERENCE NTE EVENTS
The Monte Carlo simulation results from the original study included a reference data set
consisting of 195 NTE events gathered from a number of sources. These included five engine
manufacturers, SwRI transient lab tests and pre-pilot CE-CERT data. The investigations
performed in this modified program included only a subset of the original 195 reference NTE
events. Simulations from three different subsets of the original NTE events were run to study the
validation of the emissions by the three calculation methods using the revised error surfaces.
Table 3 lists the NTE events along with their ideal BSNOX values selected for each
simulation. The various NTE subsets are described as follows:
• 13 Reference Events - These are the same 13 NTE events chosen during the original
study to investigate the error surface sensitivities due to bias and variance. These events
were selected to bound the BSNOX threshold of 2.6820 g/kW-hr. These 13 NTE events
were used to examine the error surface sensitivities for both the validation and full model
simulation runs.
• 23 Reference Events - Ten additional NTE events were added to the original 13 NTE
events described above in order to increase the sample size for generating the validation
EDF plots using the drift corrected CE-CERT data.
• 50 Reference Events - After the model validation was confirmed for the three emissions
across the three calculation methods, Mods 1 and 2 were chosen for continued analysis.
Twenty-seven additional reference NTE events were simulated with the Monte Carlo
model in order to estimate the measurement allowances for each emission. The selection
of the additional 27 reference NTE events was based on maintaining a similar distribution
of ideal BSNOX values for Method 1 established from the original study containing 195
reference NTE events.
Figure 7 shows comparison histograms of the Method 1 BSNOX values for the 23
reference NTE events (lower histogram) and the 195 reference NTE events (upper histogram)
while Figure 8 compares the Method 1 BSNOX histograms for the 50 reference NTE events
(lower histogram) and the 195 reference NTE events (upper histogram). Descriptive statistics of
BSNOX for these NTE subsets are detailed in Table 4.
SwRI Report 03.12859.06 9 of 83
-------
TABLE 3. REFERENCE NTE EVENTS USED IN MONTE CARLO SIMULATIONS
1
3
4
7
11
16
20
22
23
25
29
37
38
40
43
44
46
51
57
63
65
66
67
69
71
82
86
87
89
92
96
99
103
115
125
127
136
139
146
148
157
4.0713
3.0668
3.5832
5.2516
1.8583
2.3511
3.7245
4.7916
3.1483
5.4061
5.5261
5.4511
0.0250
4.1675
1.1473
1.0730
2.6958
2.8299
4.3382
2.6670
2.7437
3.9378
5.8600
3.0257
6.6867
2.4569
1.7132
1.5207
2.2566
1.6041
1.6224
1.8147
1.9186
1.3854
2.3214
3.3005
2.6782
2.4018
2.3053
1.9985
3.4666
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
SwRI Report 03.12859.06
10 of 83
-------
160
162
163
168
176
177
179
191
193
2.0773
2.1405
2.5908
1.5859
1.9671
1.7507
2.2023
6.0815
5.7521
X
X
X
X
X
X
X
X
X
X
X
X
40
30
10
sr
0
10
20
Ideal NOx Method 1 195RefNTE
02468
Ideal NOx Method 1 23 Ref NTE
FIGURE 7. DISTRIBUTION OF IDEAL NOX (G/KW-HR) METHOD 1 FOR 23
SELECTED REF NTE EVENTS AND 195 REF NTE EVENTS
SwRI Report 03.12859.06
11 of 83
-------
Ideal NOx Method 1 195 Ref NTE Events
40
30
Ł 20
0)
=3 10
D"
Ł 0
LL
10
20
02468
Ideal NOx Method 1 50 Ref NTE Events
FIGURE 8. DISTRIBUTION OF IDEAL NOX (G/KW-HR) METHOD 1 FOR 50
SELECTED REF NTE EVENTS AND 195 REF NTE EVENTS
TABLE 4. DESCRIPTIVE STATISTICS FOR IDEAL BSNOX (G/KW-HR) FOR
VARIOUS NTE SUBSETS
Minimum
Maximum
Mean
Median
Standard Deviation
0.0249
5.5261
2.8983
2.6782
1.3671
0.0249
6.6867
3.0068
2.6289
1.5175
0.0249
7.1927
3.0071
2.6033
1.4807
SwRI Report 03.12859.06
12 of 83
-------
4.0 CE-CERT MEASUREMENTS
The CE-CERT on-road measurements collected during the original PEMS study included
delta BS emissions for all three emissions and three calculation methods. As part of this study to
investigate all possible reasons why BSNOX did not validate for Methods 2 and 3, the 100 NTE
events from the CE-CERT data were examined for correctness by EPA. Since the CE-CERT
data had not been drift corrected in the original PEMS study, EPA performed drift corrections on
all three emissions for the 100 CE-CERT NTE events. As a result, several CE-CERT NTE
events did not pass the drift check criteria and, therefore, were not included in the simulation
performed for this study. In addition, time alignment problems were found in a few of the 100
CE-CERT NTE events and these were also excluded from the simulation. After deleting the
NTE events due to drift check or time alignment problems, the on-road delta BS emissions were
calculated from 81 NTE events for BSNOX and 87 NTE events for BSNMHC and BSCO. These
remaining CE-CERT NTE events were all drift corrected. In contrast, the original PEMS
program did not use drift correction in the CE-CERT on-road NTE events.
SwRI Report 03.12859.06 13 of 83
-------
5.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE
EVENTS USING MOD 1, MOD 2 AND MOD 3
The Monte Carlo simulations performed using the 23 reference NTE events listed in
Table 3 included modifications for the error surfaces listed in Table 2. All the error surfaces
from the original MC simulations were used except those listed as revised in Table 2. Each of
the 23 reference NTE events were run using the error surfaces revised for Mod 1 for either
10,000 or 30,000 trials. If the ideal BSNOX for an NTE event was less than 2.68204 g/kW-hr
then the simulation was run for 10,000 trials. Otherwise, the NTE was simulated using 30,000
trials. Once the MC simulations were completed using the Mod 1 revised error surfaces, the
same 23 NTE events were simulated a second time with the error surface modifications for Mod
2. Lastly, the 23 reference NTE events were simulated using the error surface modifications for
Mod 3.
In Mod 1, Mod 2, and Mod 3 simulation runs, all three emissions converged within 1% of
the emissions threshold value by all three calculation methods. Table 5 lists the convergence
criteria for all three brake-specific emissions. Thus, all 23 reference NTE events met the
convergence criteria.
TABLE 5. CONVERGENCE CRITERIA BY EMISSION
BSNOx
BSNMHC
BSCO
0.02682
0.00282
0.26015
From the MC simulations, the validation 5th and 95th percentile delta BS emissions were
extracted from the output files for each of the 23 reference NTE events. These delta emissions
were then plotted as empirical distribution functions (EDF) to form a validation interval for the
on-road data. Also plotted was the EDF computed from the on-road CE-CERT NTE events.
Figure 9 through Figure 11 represent the validation plots for the BSNOX for calculation methods
1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th percentile EDFs for the
Mod 1, Mod 2, and Mod 3 runs. Figure 12 through Figure 17 depict the validation plots for the
BSNMHC and BSCO simulation runs, respectively. The validation criteria set by the Steering
Committee for the original study was used in the modified program. It included the following
criteria:
• At least 90% of the CE-CERT emissions deltas must be within the 5th and 95th
percentiles of the MC validation cumulative emissions deltas.
• No more than 10% of the CE-CERT emissions deltas may fall less than the 5th
percentile or greater than the 95th percentile. This may indicate that the model is biased
low or high.
• Validation must be shown for all three calculation methods.
SwRI Report 03.12859.06 14 of 83
-------
A summary of the BSNOX validation conclusions based on the 23 reference NTE events
is provided in Table 6. All BSNOX, BSNMHC and BSCO emissions validated for all three
calculation methods and all three Mod runs.
TABLE 6. BSNOx VALIDATION RESULTS BASED ON 23 REFERENCE NTE
EVENTS
Mod 1
Mod 2
Mod3
• All CE-CERT data >
5th percentile
• All CE-CERT data <
95th percentile
•VALID
• All CE-CERT data >
5th percentile
• All CE-CERT data <
95th percentile
•VALID
• All CE-CERT data >
5th percentile
• All CE-CERT data <
95th percentile
•VALID
• All CE-CERT data >
5th percentile
• 1% of CE-CERT data
> 95th percentile
•VALID
• All CE-CERT data >
5th percentile
• 1% of CE-CERT data
> 95th percentile
•VALID
• All CE-CERT data >
5th percentile
• All CE-CERT data <
95th percentile
•VALID
• All CE-CERT data >
5th percentile
• All CE-CERT data <
95th percentile
•VALID
• All CE-CERT data >
5th percentile
• 1% of CE-CERT data
> 95th percentile
•VALID
• All CE-CERT data >
5th percentile
• All CE-CERT data <
95th percentile
•VALID
SwRI Report 03.12859.06
15 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
Error Models Mod 1, Mod 2 and Mod 3
NCK (g/kW—hr) Method 1
100
en
a.
rt
D
Ł
D
O
80-
60
40-
20-
0
—0.60 -O.45 -0,30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90
Delta NCK
FIGURE 9. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
NOX METHOD 1 MODS 1, 2 AND 3
SwRI Report 03.12859.06
16 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
Error Models Mod 1, Mod 2 and Mod 3
NCK (s^kW-hr) Method 2
100
80-
en
a.
O
N 23
^^^^^^^^^^^^^^m
^^^^^^^^^^^^^^m
95ti % Mod 2 Delta
-0.60 -Q.45 -0,30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90
FIGURE 10. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
NOX METHOD 2 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
17 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected SI CE—CERT Deltas
Error Models Mod 1, Mod Ł and Mod 3
NOx (g/kW-hr) Method 3
E
o
100-
80-
60-
51h % Mod 1 De 1a
5th W Mod 2 Delta
5th % Mod 3 Delta
951h M Mod 1 Delta
N 23
^^^^^^^^^^^^
^^^^^^^^^^^^^^H
95th W Mod 2 Delta
N 23
^^^^^^^^^^^^^^M
^^^^^^^^^^^^^^H
95th % Mod 3 Delta
-0,60 -0.45 -0,30 -0.15 0.00 0.15 0.30 0,45 0.60 0,75 0.90
Delta NCK g/kW—hr
4C"
20 <
FIGURE 11. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
NOX METHOD 3 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
18 of 83
-------
en
a.
rt
D
Ł
D
O
100
80-
60
40-
20-
0
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Models Mod 1, Mod 2 and Mod 3
NMHC (g/kW—hr) Method 1
|51h % Mod 1 Delta
—0.04 -O.03 -0,02 -0,01 0.00 0.01 0.02 0,03 0,04 0,05 0,06
Delta NMHC g/kW-hr
FIGURE 12. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
NMHC METHOD 1 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
19 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Models Mod 1, Mod 2 and Mod 3
NMHC (g/kW-hr) Method 2
100
en
a.
80-
60
rt
u
E 40-
o
20-
—0.03
-0,01 0.00 0.01
Delta NMHC g/kW-hr
0,02
0.03
0.04
FIGURE 13. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
NMHC METHOD 2 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
20 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Models Mod 1, Mod 2 and Mod 3
NMHC (g/kW-hr) Method 3
100
en
a.
80-
60
rt
u
E 40-
o
20-
—0.03
-0,01 0.00 0.01
Delta NMHC g/kW-hr
0,02
0.03
0.04
FIGURE 14. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
NMHC METHOD 3 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
21 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Models Mod 1, Mod 2 and Mod 3
CO (gykW-hr) Method 1
100
80-
en
a.
O
9E>th % Mod 1 DeHa
23
^m
^m
95th % Mod 2 Delta
-0,60 —0,46 —0,30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1,05
Delta CO a/kW-hr
FIGURE 15. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
CO METHOD 1 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
22 of 83
-------
en
a.
rt
D
Ł
D
O
100
80-
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Models Mod 1, Mod 2 and Mod 3
CO (g/RW—hr) Method 2
5th % Mod 1 Delta
9E>th % Mod 1 De ta
23
^a
^H
95th % Mod 2 Delta
-0,60 —0,46 —0,30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1,05
Delta CO a/kW-hr
FIGURE 16. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
CO METHOD 2 MODS 1, 2 AND 3
SwRI Report 03.12859.06
23 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Models Mod 1, Mod 2 and Mod 3
CO (g/RW—hr) Method 3
100
80-
en
a.
O
9E>th % Mod 1 DeHa
23
^a
^H
95th % Mod 2 Delta
-0,60 —0,46 —0,30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1,05
Delta CO a/kW-hr
FIGURE 17. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
CO METHOD 3 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
24 of 83
-------
6.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE
EVENTS USING MOD B
Since all three methods validated using the 23 reference NTE events by all three
modification runs for BSNOX, BSNMHC and BSCO, the question arose as to which of the
revised error surfaces had the most influence in the validation. Three possible explanations
included (1) the steady-state CC>2 bias had been eliminated, (2) the change in the steady-state
NOx error values, and (3) the bias in the steady-state exhaust flow rate had been eliminated. To
study these possible explanations, none of the revised SSNOX error surfaces or the SSCO error
surface was used in the simulations. These error surfaces were set to their original format in the
original PEMS study. Therefore, only the steady-state exhaust flow rate and the steady-state
CC>2 error surfaces were revised and the other remaining validation error surfaces were set to
their original definitions for running the 23 reference NTE events for the Mod B MC simulation.
This set of simulations is referred to as the 'Mod B' runs. Again, the simulations were
performed at 10,000 trials for NTE events with ideal BSNOx values less than 2.68204 g/kW-hr
and at 30,00 trials otherwise. All 23 reference NTE events met the convergence criteria listed in
Table 5.
From the Mod B MC simulations, the validation 5th and 95th percentile delta BS
emissions were extracted from the output files for each of the 23 reference NTE events. These
delta emissions were then plotted as empirical distribution functions (EDF) to form a validation
interval for the on-road data. Also plotted was the EDF computed from the on-road CE-CERT
NTE events. Figure 18 through Figure 20 represent the validation plots for the BSNOX for
calculation methods 1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th
percentile EDFs for the Mod B runs and the on-road CE-CERT data. Figure 21 through Figure
26 depict the validation plots for the BSNMHC and BSCO simulation runs, respectively. All
BSNOx, BSNMHC and BSCO emissions validated for all three calculation methods for the Mod
B runs.
Conclusions made from the results of the Mod B analysis were that the steady-state NOx
error surface changes did not have as much of an effect on the validation as the bias elimination
in the steady-state CO2 and steady-state exhaust flow rate error surfaces.
SwRI Report 03.12859.06 25 of 83
-------
c
u
m
u
I
a
t
v
e
P
e
r
c
e
n
t
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
Error Model Mod B
NCK (g/kW-hr) Method 1
100
5th % Mod B Delta
-0.60 -0,45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90
Delta NQx o/kW-hr
FIGURE 18. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
NOX METHOD 1 MOD B
SwRI Report 03.12859.06
26 of 83
-------
c
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected SI CE—CERT Deltas
Error Model Mod B
NQx (g/kW-hr) Method 2
100
-0,60 -0.45 -0.30 -0.15 0.00
FIGURE 19. VALIDATION NTE EVENTS USING 23 REFERENCE NTE EVENTS FOR
NOX METHOD 2 MOD B
SwRI Report 03.12859.06
27 of 83
-------
c
u
m
u
I
a
t
v
e
p
e
r
c
e
n
t
100'
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected SI CE—CERT Deltas
Error Model Mod B
NOK (g/kW—hr) Method 3
5th W Mod B Delta
5th % Mod B Ddta
-0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90
Delta NOx a/kW-hr
FIGURE 20. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
NOX METHOD 3 MOD B
SwRI Report 03.12859.06
28 of 83
-------
c
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
Error Model Mod B
NMHC (a/kW^nr) Method 1
100
60
40'
20-
0
-0.04 -0.03 -0.02 -0.01 0.00 0.01 0.02 0.03 0.04 0.05 0.06
Delta NMHC g/kW-hr
FIGURE 21. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
NMHC METHOD 1 MOD B
SwRI Report 03.12859.06
29 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
Error Model Mod B
NMHC (g/kW-hr) Method 2
100
C
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
60
40'
20-
-0.03
-0,02
-0,01 0.00 0.01
Delta NMHC g/kW-hr
0.02
0,03
0.04
FIGURE 22. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
NMHC METHOD 2 MOD B
SwRI Report 03.12859.06
30 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
Error Model Mod B
NMHC (g/kW-hr) Method 3
100
C
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
60
40'
20-
-0.03
-0,02
-0,01 0.00 0.01
Delta NMHC g/kW-hr
0.02
0,03
0.04
FIGURE 23. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
NMHC METHOD 3 MOD B
SwRI Report 03.12859.06
31 of 83
-------
c
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
100
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Model Mod B
CO (g/kW-hr) Method 1
5th % Mod B Delta
95th »i Mod B De ta
-0,60 -0,45 -0.30 -0.15 0,00 0.15 0.30 0,45 0.60 0,75 0,90 1.05 1.20
Delta CO g/kW-hr
FIGURE 24. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
CO METHOD 1 MOD B
SwRI Report 03.12859.06
32 of 83
-------
c
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Model Mod B
CO (g/RW^nr) Method 2
100
5th % Mod B Delta
95th »i Mod B De ta
-0,60 -0,45 -0.30 -0.15 0,00 0.15 0.30 0,45 0.60 0,75 0,90 1.05 1.20
Delta CO g/kW-hr
FIGURE 25. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
CO METHOD 2 MOD B
SwRI Report 03.12859.06
33 of 83
-------
c
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
100
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Model Mod B
CO (g/RW^nr) Method 3
5th % Mod B Delta
95th »i Mod B De ta
-0,60 -0,45 -0,30 -0.15 0,00 0.15 0.30 0,45 0.60 0,75 0,90 1.05 1.20
Delta CO g/kW-hr
FIGURE 26. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
CO METHOD 3 MOD B
SwRI Report 03.12859.06
34 of 83
-------
7.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE
EVENTS USING MOD 1
Based on the information provided in the validation plots for the Mod 1, Mod 2, Mod 3
and Mod B runs, changes made to the various steady-state error surfaces resulted in the
validation of the MC model for all three emissions and all three calculation methods for each of
the four Mod runs using the 23 reference NTE events. EPA chose to continue the simulation
runs using only the Mod 1 and Mod 2 revised error surfaces by running an additional 27
reference NTE events (total = 50 reference NTE events) through the MC model. These two
Mods were chosen because they both represented steady-state NOx error surfaces that were level
independent (as compared to the Mod 3 steady-state error surface that represented a combination
of level dependent and level independent NOx errors). The results from these 50 simulations
were used to calculate the measurement allowances provided in Section 9.0 of this report. This
section details the results of the validation based on the 50 reference NTE events run with Mod
1.
The Mod 1 simulations were performed at 10,000 trials for NTE events with ideal
BSNOx values less than 2.68204 g/kW-hr and at 30,00 trials otherwise. All 50 reference NTE
events met the convergence criteria listed in Table 5.
From the Mod 1 MC simulations, the validation 5th and 95th percentile delta BS emissions
were extracted from the output files for each of the 50 reference NTE events. These percentiles
were then plotted as empirical distribution functions (EDF) to form a validation interval for the
on-road data. Also plotted was the EDF computed from the on-road CE-CERT NTE events.
Figure 27 through Figure 29 represent the validation plots for the BSNOX for calculation methods
1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th percentile EDFs for the
Mod 1 runs and the on-road CE-CERT data. Figure 30 through Figure 35 depict the validation
plots for the BSNMHC and BSCO simulation runs, respectively. All BSNOx, BSNMHC and
BSCO emissions validated for all three calculation methods for the Mod 1 runs with the 50
reference NTE events.
SwRIReport03.12859.06 35 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
Error Model Mod 1
NCK (g/kW—hr) Method 1
100'
80
60 <
40'
20
C
u
m
u
I
a
t
v
e
p
e
r
c
e
n
t
0
-0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90
Delta NQx s/kW-hr
FIGURE 27. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
NOX METHOD 1 MOD 1
SwRI Report 03.12859.06
36 of 83
-------
c
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected SI CE—CERT Deltas
Error Model Mod 1
NQx (g/kW-hr) Method 2
100
5th % Mod B Delta
-0,60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90
Delta NQx g/kW—hr
FIGURE 28. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
NOX METHOD 2 MOD 1
SwRI Report 03.12859.06
37 of 83
-------
c
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
Error Model Mod 1
NQx (g/kW-hr) Method 3
100
5th % Mod B Delta
-0,60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90
Delta NQx g/kW—hr
FIGURE 29. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
NOX METHOD 3 MOD 1
SwRI Report 03.12859.06
38 of 83
-------
c
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
Error Model Mod 1
NMHC (a/kW^nr) Method 1
100
60
40'
20-
0
-0.04 -0.03 -0.02 -0.01 0.00 0.01 0.02 0.03 0.04 0.05 0.06
Delta NMHC g/kW-hr
FIGURE 30. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
NMHC METHOD 1 MOD 1
SwRI Report 03.12859.06
39 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Model Mod 1
NMHC (g/kW-hr) Method 2
100
C
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
60
40'
20-
15th M Mud B Delta I
-0.03
-0,02
-0,01 0.00 0.01
Delta NMHC g/kW-hr
0.02
0,03
0.04
FIGURE 31. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
NMHC METHOD 2 MOD 1
SwRI Report 03.12859.06
40 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Model Mod 1
NMHC (g/kW-hr) Method 3
100
C
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
60
40'
20-
-0.03
-0,02
-0,01 0.00 0.01
Delta NMHC g/kW-hr
0.02
0,03
0.04
FIGURE 32. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
NMHC METHOD 3 MOD 1
SwRI Report 03.12859.06
41 of 83
-------
c
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
100
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
Error Model Mod 1
CO (g/kW-hr) Method 1
5th % Mod B Delta
95th % Mod 1 De 1
-0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1.05
Delta CO Q/kW-hr
FIGURE 33. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
CO METHOD 1 MOD 1
SwRI Report 03.12859.06
42 of 83
-------
c
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
Error Model Mod 1
CO (g/RW^nr) Method 2
100
5th % Mod B Delta
95th % Mod 1 De 1
-0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1.05
Delta CO Q/kW-hr
FIGURE 34. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
CO METHOD 2 MOD 1
SwRI Report 03.12859.06
43 of 83
-------
c
u
m
u
I
a
t
i
v
e
P
e
r
c
e
n
t
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
Error Model Mod 1
CO (g/RW^nr) Method 3
100
5th % Mod B Delta
95th % Mod 1 De 1
-0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1.05
Delta CO Q/kW-hr
FIGURE 35. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
CO METHOD 3 MOD 1
SwRI Report 03.12859.06
44 of 83
-------
8.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE
EVENTS USING MOD 2
This section details the results of the validation based on the 50 reference NTE events run
with Mod 2. EPA chose to continue simulations using the Mod 1 and Mod 2 revised error
surfaces by running an additional 27 reference NTE events (total = 50 reference NTE events)
through the MC model. These two Mods were chosen because they both represented steady-
state NOx error surfaces that were level independent (as compared to the Mod 3 steady-state
error surface that represented a combination of level dependent and level independent NOX
errors).
The Mod 2 simulations were performed at 10,000 trials for NTE events with ideal
BSNOx values less than 2.68204 g/kW-hr and at 30,00 trials otherwise. All 50 reference NTE
events met the convergence criteria listed in Table 5.
From the Mod 2 MC simulations, the validation 5th and 95th percentile delta BS emissions
were extracted from the output files for each of the 50 reference NTE events. These percentile
delta emissions were then plotted as empirical distribution functions (EDF) to form a validation
interval for the on-road data. Also plotted was the EDF computed from the on-road CE-CERT
NTE events. Figure 36 through Figure 38 represent the validation plots for the BSNOX for
calculation methods 1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th
percentile EDFs for the Mod 2 runs and the on-road CE-CERT data. Figure 39 through Figure
44 depict the validation plots for the BSNMHC and BSCO simulation runs, respectively. All
BSNOx, BSNMHC and BSCO emissions validated for all three calculation methods for the Mod
2 runs with the 50 reference NTE events.
SwRI Report 03.12859.06 45 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
Error Model Mod 2
NCK (g/kW—hr) Method 1
100'
80
60 <
40'
20
C
u
m
u
I
a
t
v
e
p
e
r
c
e
n
t
95ih % Mod 2 Delta I
0
-0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90
Delta NQx s/kW-hr
FIGURE 36. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
NOX METHOD 1 MOD 2
SwRI Report 03.12859.06
46 of 83
-------
c
u
m
u
I
a
t
v
e
p
e
r
c
e
n
t
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
Error Model Mod 2
NOK (g/kW—hr) Method 2
100'
5th W Mod 2 Delta
99th % Mod 2 Delta
-0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90
Delta NOx a/kW-hr
FIGURE 37. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
NOX METHOD 2 MOD 2
SwRI Report 03.12859.06
47 of 83
-------
c
u
m
u
I
a
t
v
e
p
e
r
c
e
n
t
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
Error Model Mod 2
NOK (g/kW—hr) Method 3
100'
5th W Mod 2 Delta
99th % Mod 2 Delta
-0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90
Delta NOx a/kW-hr
FIGURE 38. VALIDATION EDF PLOTS FOR 50 REFERENCE NTE EVENTS FOR
NOX METHOD 3 MOD 2
SwRI Report 03.12859.06
48 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Model Mod 2
NMHC (g/KW-hr) Method 1
100'
80
60 <
40'
20
C
u
m
u
I
a
t
v
e
p
e
r
c
e
n
t
0
-0.04 -0.03 -0.02 -0.01 0.00 0.01 0.02 0.03 0.04 0.05 0.06
Delta NMHC gykW-hr
FIGURE 39. VALIDATION EDF PLOTS FOR 50 REFERENCE NTE EVENTS FOR
NMHC METHOD 1 MOD 2
SwRI Report 03.12859.06
49 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Model Mod 2
NMHC (g/kW-hr) Method 2
100'
80
60 <
40'
20
C
u
m
u
I
a
t
v
e
p
e
r
c
e
n
t
0
-0.04 -0.03 -0.02 -0.01 0.00 0.01 0.02 0.03 0.04 0.05 0.06
Delta NMHC gykW-hr
FIGURE 40. VALIDATION EDF PLOTS FOR 50 REFERENCE NTE EVENTS FOR
NMHC METHOD 2 MOD 2
SwRI Report 03.12859.06
50 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Model Mod 2
NMHC (g/kW-hr) Method 3
100'
80
60 <
40'
20
C
u
m
u
I
a
t
v
e
p
e
r
c
e
n
t
15th M Mod 2 Delta I
-0.03
-0,02
-0,01 0.00 0.01
Delta NMHC gykW-hr
0.02
0.03
0,04
FIGURE 41. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
NMHC METHOD 3 MOD 2
SwRI Report 03.12859.06
51 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Model Mod 2
CO (g/kW-hr) Method 1
C
u
m
u
I
a
t
v
e
p
e
r
c
e
n
t
100'
80
60
5th W Mod 2 Delta
-0.60 -0.45 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1.05
Delta CO g/kW-hr
FIGURE 42. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
CO METHOD 1 MOD 2
SwRI Report 03.12859.06
52 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Model Mod 2
CO (g/RW—hr) Method 2
c
u
m
u
I
a
t
v
e
p
e
r
c
e
n
t
100'
80
60
5th W Mod 2 Delta
-0.60 -Ci.45 -O.30 -0.15 0,00 0.15 0,30 0,45 0,60 0.75 0.90 1,05
Delta CO a/kW-hr
FIGURE 43. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
CO METHOD 2 MOD 2
SwRI Report 03.12859.06
53 of 83
-------
Validation Analysis 5th and 95th Percentile Deltas
Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
Error Model Mod 2
CO (g/RW—hr) Method 3
c
u
m
u
I
a
t
v
e
p
e
r
c
e
n
t
100'
80
60
5th W Mod 2 Delta
-0.60 -Ci.45 -O.30 -0.15 0,00 0.15 0,30 0,45 0,60 0.75 0.90 1,05
Delta CO a/kW-hr
FIGURE 44. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
CO METHOD 3 MOD 2
Table 7 provides a summary of all the validation results for the various MC simulations
as described in the above sections of this report. All emissions and all calculation methods
validated using the Mod 1 and Mod 2 runs with 50 reference NTE events while the Mod 3 and
Mod B runs validated using 23 reference NTE events.
SwRI Report 03.12859.06
54 of 83
-------
TABLE 7. SUMMARY OF VALIDATION RESULTS
BSNOx
Method 1
BSNOx
Method 2
BSNOx
Method 3
BSNMHC
Method 1
BSNMHC
Method 2
BSNMHC
Method 3
BSCO
Method 1
BSCO
Method 2
BSCO
Method 3
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
SwRI Report 03.12859.06
55 of 83
-------
9.0 MEASUREMENT ALLOWANCE CALCULATIONS
As detailed in the original PEMS study, the measurement error allowances were
computed using both a regression method and a median method to determine the measurement
allowance. The procedure was applied to the simulation data for the 50 reference NTE events
from the Mod 1 and Mod 2 runs for each of the three emissions and all three calculation
methods.
Figure 45 contains a regression plot of the 95th percentile delta BSNOX values (using
Method 1 Mod 1) versus the ideal BSNOX values for the 50 reference NTE events. Included
within the plot is the equation for the fitted regression line and the R-square (R2) value. Table 8
includes a comparison of the results of the regression method based on Figure 45 and the median
method (described in Ref 1). Under the heading "Regression Method" in the table, it is shown
that the R-square criterion is not met by the data (R-square must be > 0.90). Thus, the median
method must be used. Under the heading "Median Method" in the table, the measurement error
at the BSNOX threshold, based on using the median of the fifty 95th percentile delta BSNOX
values, is 11.5932% when expressed as a percent of the threshold value of 2.68204 g/kW-hr.
Similar regression plots and measurement error tables are provided in the remaining part
of this section for the Mod 1 and Mod 2 results based on the 50 reference NTE events. Figure 45
through Figure 47 and Table 8 through Table 10 provide results for BSNOX Mod 1. Figure 48
through Figure 50 and Table 11 through Table 13 provide results for BSNMHC Mod 1, and
Figure 51 through Figure 53 and Table 14 through Table 16 provide results for BSCO Mod 1.
Mod 2 results for BSNOX can be found in Figure 54 through Figure 56 and Table 17 through
Table 19, BSNMHC results are shown in Figure 57 through Figure 59 and Table 20 through
Table 22 and BSCO results are shown in Figure 60 through Figure 62 and Table 23 through
Table 25.
SwRI Report 03.12859.06 56 of 83
-------
NOx g/kW-hr Method 1 Mod 1
0.90 -,
0.80
0.70
.c
| 0.60
x „ -,.
O 0.50
z
TO
o 0.40
Q
^ 0.30
u>
OT 0.20 -
0.10
0.00
c
50 Ref NTE Events With Time Alignment Adjustment
* ^
» ,S
* *^ 4
** ^^ *
*
-------
NOx g/kW-hr Method 2 Mod 1
50 Ref NTE Events With Time Alignment Adjustment
0.50
0.45
0.40
0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00
y = 0.0558x + 0.0893
R2 = 0.9058
345
Ideal NOx g/kW-hr
FIGURE 46. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS
IDEAL BSNOx FOR METHOD 2 MOD 1
TABLE 9. MEASUREMENT ERROR AT THRESHOLD FOR BSNOx USING
REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 1
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=2.68204
0.9058
0.0276
0.1314
0.2390
Met
Criteria
Met
Criteria
Median 95th% Delta
Measurement Error @
Threshold=2.68204
0.2270
8.4641%
SwRI Report 03.12859.06
58 of 83
-------
NOx g/kW-hr Method 3 Mod 1
50 Ref NTE Events With Time Alignment Adjustment
0.70
0.60
y = 0.0869X + 0.0533
R2 = 0.8714
0.00
2345
Ideal NOx g/kW-hr
FIGURE 47. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS
IDEAL BSNOx FOR METHOD 3 MOD 1
TABLE 10. MEASUREMENT ERROR AT THRESHOLD FOR BSNOx USING
REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 1
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=2.68204
0.8714
0.0512
0.1314
0.3000
11.1845%
Did Not
Meet
Criteria
Met
Criteria
Median 95th% Delta
Measurement Error @
Threshold=2.68204
0.2572
SwRI Report 03.12859.06
59 of 83
-------
n ny
n DR
.c n nc;
1
3)
O o 04
ro
7ii n m
Q
55 0 02 1
" !
0 01
0 00
0.
NMHC g/kW-hr Method 1 Mod 1
50 Ref NTE Events
•
^^
^^
^^ y = 0.0955X + 0.01 74
\» R2 = 0.8109
V^' '
k •'
00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.
Ideal NMHCg/kW-hr
45
FIGURE 48. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC
VERSUS IDEAL BSNMHC FOR METHOD 1 MOD 1
TABLE 11. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING
REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 1
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=0.28161
0.8109
0.0037
0.0002
0.0443
15.7288%
Did Not
Meet
Criteria
Did Not
Meet
Criteria
Median 95th% Delta
Measurement Error @
Threshold=0.28161
0.0197
SwRI Report 03.12859.06
60 of 83
-------
k.
.C
i
B>
o
2
z
ro
±i
01
Q
S?
Ł
55
O>
NMHC g/kW-hr Method 2 Mod 1
50 Ref NTE Events
0 045
n D4n
n mR
n mn
n n9ci
0.020 j
0.015 I
n mn
n rin^
n nnn
^^^^
^^^^
| * «r y = 0.0382X + 0.01 74
| ' ""* 4 R2 = 0.4768
Ł*— "**^~ *
» *s*
Sr
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45
Ideal NMHCg/kW-hr
FIGURE 49. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC
VERSUS IDEAL BSNMHC FOR METHOD 2 MOD 1
TABLE 12. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING
REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 1
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=0.28161
0.4768
0.0032
0.0002
0.0282
9.9988%
Did Not
Meet
Criteria
Did Not
Meet
Criteria
Median 95th% Delta
Measurement Error @
Threshold=0.28161
0.0187
SwRI Report 03.12859.06
61 of 83
-------
0.050
o
s
z
ra
1
0.045 -
0.040 -
0.035 -
0.030 -
0.025
0.020
Ł 0.015
0.010
0.005 -
0.000
NMHC g/kW-hr Method 3 Mod 1
50 Ref NTE Events
y =0.0545x +0.0171
R2 = 0.6202
0.00 0.05 0.10 0.15 0.20 0.25 0.30
Ideal NMHCg/kW-hr
0.35
0.40
0.45
FIGURE 50. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC
VERSUS IDEAL BSNMHC FOR METHOD 3 MOD 1
TABLE 13. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING
REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 1
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=0.28161
0.6202
0.0034
0.0002
0.0324
11.5222%
Did Not
Meet
Criteria
Did Not
Meet
Criteria
Median 95th% Delta
Measurement Error @
Threshold=0.28161
0.0188
SwRI Report 03.12859.06
62 of 83
-------
CO g/kW-hr Method 1 Mod 1
50 Ref NTE Events With Time Alignment Adjustment
1.20
y=0.1112x+0.3148
R2 = 0.6518
0.00
3 4
Ideal CO g/kW-hr
FIGURE 51. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS
IDEAL BSCO FOR METHOD 1 MOD 1
TABLE 14. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING
REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 1
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=26.015
0.6518
0.0846
0.0249
0.9814
3.7726%
Did Not
Meet
Criteria
Did Not
Meet
Criteria
Median 95th% Delta
Measurement Error @
Threshold=26.015
0.3600
SwRI Report 03.12859.06
63 of 83
-------
CO g/kW-hr Method 2 Mod 1
50 Ref NTE Events With Time Alignment Adjustment
0.90
y = 0.0543X + 0.3211
R2 = 0.3471
3 4
Ideal CO g/kW-hr
FIGURE 52. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS
IDEAL BSCO FOR METHOD 2 MOD 1
TABLE 15. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING
REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 1
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=26.015
0.3471
0.0775
0.0249
0.6466
2.4856%
Did Not
Meet
Criteria
Did Not
Meet
Criteria
Median 95th% Delta
Measurement Error @
Threshold=26.015
0.3451
SwRI Report 03.12859.06
64 of 83
-------
1 10 -i
1 00
0 90 -
*r n fin
^
B>
O 0 70 -
o
ro
±i
a) n fin
S °-60 ;
s?
Ł 0 50 -
55 U'DU
o>
<
n 4n
n ^n
n 9n
C
CO g/kW-hr Method 3 Mod 1
50 Ref NTE Events With Time Alignment Adjustment
* .s^
^^
y = 0.0988X + 0.31 1
R2 = 0.5987
» ^r
** * tx^^^
"v ix<; *
X *
^*i%*
4r**' »
) 1 234567
Ideal CO g/kW-hr
7
FIGURE 53. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS
IDEAL BSCO FOR METHOD 3 MOD 1
TABLE 16. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING
REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 1
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=26.015
0.5987
0.0846
0.0249
0.9033
3.4722%
Did Not
Meet
Criteria
Did Not
Meet
Criteria
Median 95th% Delta
Measurement Error @
Threshold=26.015
0.3621
SwRI Report 03.12859.06
65 of 83
-------
NOx g/kW-hr Method 1 Mod 2
50 Ref NTE Events With Time Alignment Adjustment
0.90
0.80 -
0.70 -
0.60 -
o) 0.50
x
O
^ 0.40 -]
0.3
>o
a>
0.10 -
0.00
-0.10
*4
y =0.1 272x- 0.0086
R2 = 0.8638
Ideal NOx g/kW-hr
FIGURE 54. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS
IDEAL BSNOx FOR METHOD 1 MOD 2
TABLE 17. MEASUREMENT ERROR AT THRESHOLD FOR BSNOX USING
REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 2
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=2.68204
0.8638
0.0774
0.1314
0.3326
12.3993%
Did Not
Meet
Criteria
Met
Criteria
Median 95th% Delta
Measurement Error @
Threshold=2.68204
0.2834
SwRI Report 03.12859.06
66 of 83
-------
0.50
0.45
0.40 -
0.35
O)
X
O
0.30
0.25
0.05 -
0.00
NOx g/kW-hr Method 2 Mod 2
50 Ref NTE Events With Time Alignment Adjustment
y =0.0614x+0.0375
R2 = 0.9543
345
Ideal NOx g/kW-hr
FIGURE 55. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS
IDEAL BSNOx FOR METHOD 2 MOD 2
TABLE 18. MEASUREMENT ERROR AT THRESHOLD FOR BSNOx USING
REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 2
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=2.68204
0.9543
0.0206
0.1314
0.2022
Met
Criteria
Met
Criteria
Median 95th% Delta
Measurement Error @
Threshold=2.68204
0.1935
7.2141%
SwRI Report 03.12859.06
67 of 83
-------
0.70
0.60
0.50 -
0.40 -
0.30
0.20 -
0.10 -
0.00
NOx g/kW-hr Method 3 Mod 2
50 Ref NTE Events With Time Alignment Adjustment
y = 0.0932X + 0.0056
R2 = 0.8998
345
Ideal NOx g/kW-hr
FIGURE 56. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS
IDEAL BSNOx FOR METHOD 3 MOD 2
TABLE 19. MEASUREMENT ERROR AT THRESHOLD FOR BSNOx USING
REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 2
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=2.68204
0.8998
0.0477
0.1314
0.2556
9.5288%
Did Not
Meet
Criteria
Met
Criteria
Median 95th% Delta
Measurement Error @
Threshold=2.68204
0.2295
SwRI Report 03.12859.06
68 of 83
-------
n ny
0 06
.c n nc;
1
3)
O 0 04 -
ro
? 0 03
Q
55 0 02
0 01
0 00 -
0.
NMHC g/kW-hr Method 1 Mod 2
50 ref NTE Events
•
^^
^^
+ ^ y =0.0951x+0.0173
^ R2 = 0.816
k!>^r
"^^ . 4»
00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.
Ideal NMHCg/kW-hr
45
FIGURE 57. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC
VERSUS IDEAL BSNMHC FOR METHOD 1 MOD 2
TABLE 20. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING
REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 2
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=0.28161
0.8160
0.0036
0.0002
0.0441
15.6533%
Did Not
Meet
Criteria
Did Not
Meet
Criteria
Median 95th% Delta 0.0194
Measurement Error @
Threshold=0.28161
SwRI Report 03.12859.06
69 of 83
-------
k.
.C
i
B>
o
2
z
ro
±i
01
Q
S?
Ł
55
O>
NMHC g/kW-hr Method 2 Mod 2
50 Ref NTE Events
0 045
n n/in
n rn^
n n^n
n ro*>
0.020
n ni^ !
0.010 -
n nn^
n nnn
^^^
^^^^
+ + ^ "* y =0.0386x+0.0172
i * J R2 = 0.479
1**~~^^
*f^ 4 •
X. * *
^
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45
Ideal NMHCg/kW-hr
FIGURE 58. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC
VERSUS IDEAL BSNMHC FOR METHOD 2 MOD 2
TABLE 21. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING
REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 2
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=0.28161
0.4790
0.0032
0.0002
0.0281
12.3993%
Did Not
Meet
Criteria
Did Not
Meet
Criteria
Median 95th% Delta
Measurement Error @
Threshold=0.28161
0.0182
SwRI Report 03.12859.06
70 of 83
-------
0.040
0.035 -
0.030 -
o
0.025 -
0.020
0.015
0.010
NMHC g/kW-hr Method 3 Mod 2
50 Ref NTE Events
y = 0.054x+0.017
R2 = 0.6195
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45
Ideal NMHCg/kW-hr
FIGURE 59. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC
VERSUS IDEAL BSNMHC FOR METHOD 3 MOD 2
TABLE 22. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING
REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 2
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=0.28161
0.6195
0.0034
0.0002
0.0322
11.4367%
Did Not
Meet
Criteria
Did Not
Meet
Criteria
Median 95th% Delta 0.0182
Measurement Error @
Threshold=0.28161
SwRI Report 03.12859.06
71 of 83
-------
CO g/kW-hr Method 1 Mod 2
50 Ref NTE Events With Time Alignment Adjustment
1.20
y =0.1156x+0.3087
R2 = 0.6865
0.00
3 4
Ideal CO g/kW-hr
FIGURE 60. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS
IDEAL BSCO FOR METHOD 1 MOD 2
TABLE 23. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING
REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 2
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=26.015
0.6865
0.0813
0.0249
1.0017
3.8505%
Did Not
Meet
Criteria
Did Not
Meet
Criteria
Median 95th% Delta
Measurement Error @
Threshold=26.015
0.3576
SwRI Report 03.12859.06
72 of 83
-------
0.90
0.80 -
0.70 -
O
o
ra
±i
01
O
0.60 -
Ł 0.50
0.40
0.30
CO g/kW-hr Method 2 Mod 2
50 Ref NTE Events With Time Alignment Adjustment
34
Ideal CO g/kW-hr
FIGURE 61. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS
IDEAL BSCO FOR METHOD 2 MOD 2
TABLE 24. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING
REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 2
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=26.015
0.3785
0.0756
0.0249
0.6561
2.5221%
Did Not
Meet
Criteria
Did Not
Meet
Criteria
Median 95th% Delta
Measurement Error @
Threshold=26.015
0.3361
SwRI Report 03.12859.06
73 of 83
-------
CO g/kW-hr Method 3 Mod 2
50 Ref NTE Events With Time Alignment Adjustment
1.10
0.:
34
Ideal CO g/kW-hr
FIGURE 62. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS
IDEAL BSCO FOR METHOD 3 MOD 2
TABLE 25. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING
REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 2
R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=26.015
0.6195
0.0827
0.0249
0.9114
3.5033%
Did Not
Meet
Criteria
Did Not
Meet
Criteria
Median 95th% Delta
Measurement Error @
Threshold=26.015
0.3513
SwRI Report 03.12859.06
74 of 83
-------
Although only 23 reference NTE events were simulated under the Mod 3 and Mod B
runs, the measurement errors were computed for all three emissions and calculation methods
using the same regression and median methods. Table 26 lists a summary of the measurement
errors computed from simulation results in this study along with the results from the original
PEMS study. Note that BSNOX for methods 2 and 3 and BSCO for all three methods did not
validate in the original study.
TABLE 26. SUMMARY OF MEASUREMENT ERRORS AT RESPECTIVE
THRESHOLD (%)
BSNOx
BSNMHC
BSCO
195
50
50
23
23
195
50
50
23
23
195
50
50
23
23
Original
Mod 1
Mod 2
Mod 3
ModB
Original
Mod 1
Mod 2
Mod 3
ModB
Original
Mod 1
Mod 2
Mod 3
ModB
22.30
11.59
10.56
11.78
11.93
10.08
7.00
6.88
7.28
7.25
2.58*
1.38
1.37
1.52
2.19
4.45*
8.91
7.54
8.01
8.26
8.03
6.63
6.45
6.90
6.82
1.99*
1.33
1.29
1.41
1.99
6.61*
9.59
8.56
10.78
10.33
8.44
6.68
6.48
6.87
6.78
2.11*
1.39
1.35
1.48
2.16
Methods did not validate
The final BS measurement allowances were computed by multiplying each of the three
measurement errors (in percent) times their corresponding threshold values. Table 27 lists the
measurement allowances for all simulation Mod runs and the original PEMS study. The
emissions thresholds (g/hp-hr) are provided in the first column of Table 27.
SwRI Report 03.12859.06
75 of 83
-------
TABLE 27. SUMMARY OF MEASUREMENT ALLOWANCE, G/HP-HR
BSNOx
Threshold=2.0
BSNMHC
Threshold=0.21
BSCO
Threshold=19.40
195
50
50
23
23
195
50
50
23
23
195
50
50
23
23
Original
Mod 1
Mod 2
Mod 3
ModB
Original
Mod 1
Mod 2
Mod 3
ModB
Original
Mod 1
Mod 2
Mod 3
ModB
0.446
0.232
0.211
0.236
0.239
0.021
0.015
0.014
0.015
0.015
0.501*
0.268
0.266
0.295
0.425
0.089*
0.178
0.151
0.160
0.165
0.017
0.014
0.014
0.014
0.014
0.386*
0.258
0.250
0.274
0.386
0.132*
0.192
0.171
0.216
0.207
0.018
0.014
0.014
0.014
0.014
0.409*
0.270
0.262
0.287
0.419
Methods did not validate
SwRI Report 03.12859.06
76 of 83
-------
10.0 VALIDATION SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS
Simulation results for Mod 1, Mod 2, Mod 3 and Mod B from the 13 reference NTE
events chosen in the original study produced sensitivity values from the validation runs due to
variance and bias for all three 95th percentile delta emissions by all three calculation methods.
The tables below summarize the error surfaces in which either the contribution-to-variance
normalized sensitivity average value or the 'on/off bias check for the error surface was at least
5% in absolute magnitude compared to all the other error surfaces. If the label in the error
surface contains the words 'OnOff then it represents a check for bias; otherwise, the error
surface indicates a check for variance. If at least 5 of the 13 reference NTE events resulted in a
sensitivity average value > 5% or < -5% then the average contribution to variance is included in
the tables.
Table 28 lists the BSNOx sensitivity error surfaces associated with the original PEMS
study for Mod B, Mod 1, Mod 2 and the Mod 3 Monte Carlo validation simulations performed
during this project. Note in the output results that the bias due to steady-state exhaust flow rate
in Method 1 in the original study has been eliminated in the four modification simulations. In
addition, the steady-state CC>2 bias has been eliminated, but a sensitivity component due to
variance has been added for Methods 2 and 3.
Table 29 lists the sensitivities due to BSNMHC. Note that no changes have occurred for
the four modifications relative to the original study. This is due to the fact that the revised error
surfaces did not effect the NMHC calculations. Table 30 summarizes the results of the BSCO
sensitivities wherein the steady-state CO bias found in the original modeling has been eliminated
in Mod 1, Mod 2 and Mod 3 and replaced by an increased sensitivity due to variance.
SwRI Report 03.12859.06 77 of 83
-------
TABLE 28. SENSITIVITY RESULTS COMPARING 13 REFERENCE NTE EVENTS
ACROSS 5 MONTE CARLO VALIDATION SIMULATIONS FOR BSNOX
1
1
20
51
SSNOx
SS Exhaust Flow
NOx Time
Alignment
Pulse Flow OnOff
SS Exhaust
Flow OnOff
Original
29.0
9.6
9.0
25.8
20.4
33.2
19.2
10.7
30.5
42.9
16.6
9.1
26.3
32.0
20.4
10.9
33.1
36.3
18.8
10.6
29.1
2
1
2
45
51
SSNOx
TRNOx
SSCO2
NOx Time
Alignment
SSCO2 OnOff
37.0
-54.3
46.9
5.5
-40.1
6.7
57.8
-33.7
7.5
44.6
6.6
-43.6
7.6
52.5
-37.7
6.2
3
1
45
51
SSNOx
SSCO2
NOx Time
Alignment
SSCO2 OnOff
35.6
10.4
-50.9
44.0
-37.0
13.9
54.9
-31.3
11.5
41.6
-39.6
15.8
49.4
-34.8
12.9
DEFINITIONS
Original April 2007 Study: All original error surfaces
Mod B Revised SS Exhaust Flow and SS CO2 error surfaces
, , , , Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error
Mod 1 ,,
surfaces
Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error
surfaces
, , , _ Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error
Mod 3 „
surfaces
SwRI Report 03.12859.06
78 of 83
-------
TABLE 29. SENSITIVITY RESULTS COMPARING 13 REFERENCE NTE EVENTS
ACROSS 5 MONTE CARLO VALIDATION SIMULATIONS FOR BSNMHC
1
13
19
SSNMHC
NMHC Ambient
SSNMHC OnOff
Original
7.1
60.9
14.5
7.1
62.8
14.3
7.0
63.2
13.5
6.9
62.5
12.9
6.6
63.2
14.2
2
13
19
SSNMHC
NMHC Ambient
SSNMHC OnOff
6.7
64.5
14.3
6.8
66.6
12.9
6.8
66.7
13.5
6.9
66.3
13.2
6.4
66.9
12.8
3
13
19
SSNMHC
NMHC Ambient
SSNMHC OnOff
6.7
64.4
14.4
6.9
66.5
13.0
6.8
66.6
13.6
7.0
66.1
12.5
6.4
66.8
13.0
DEFINITIONS
Original April 2007 Study: All original error surfaces
Mod B Revised SS Exhaust Flow and SS CO2 error surfaces
, , , , Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error
Mod 1 ,,
surfaces
Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error
surfaces
Mod 3 Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error
surfaces
SwRI Report 03.12859.06
79 of 83
-------
TABLE 30. SENSITIVITY RESULTS COMPARING 13 REFERENCE NTE EVENTS
ACROSS 5 MONTE CARLO VALIDATION SIMULATIONS FOR BSCO
1
7
52
SSCO
CO Time
Alignment
SSCO OnOff
Original
6.2
10.3
82.0
6.5
82.2
79.3
11.0
79.1
10.1
79.3
10.7
2
7
52
SSCO
CO Time
Alignment
SSCO OnOff
6.4
83.6
6.7
84.6
82.9
10.5
83.0
11.8
83.0
10.4
3
7
52
SSCO
CO Time
Alignment
SSCO OnOff
6.2
11.8
80.8
6.3
10.1
81.0
77.9
19.2
78.0
17.9
77.9
19.0
DEFINITIONS
Original April 2007 Study: All original error surfaces
Mod B Revised SS Exhaust Flow and SS CO2 error surfaces
, , , , Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error
Mod I _c
surfaces
, , , . Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error
Mod 2 ,,
surfaces
, , , _ Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error
Mod 3 „
surfaces
SwRI Report 03.12859.06
80 of 83
-------
11.0 FULL MODEL SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS
This section contains a summary of the error surfaces that contributed the most to bias
and variance of the BSNOx emissions based on the full MC model. Simulation results from the
13 reference NTE events produced sensitivity values for all three 95th percentile delta emissions
by all three calculation methods. Table 31 lists the error surfaces in which either the
contribution-to-variance normalized sensitivity average value or the 'on/off bias check for the
error surface was at least 5% in absolute magnitude compared to all the other error surfaces for
BSNOx. If at least 5 of the 13 reference NTE events resulted in a sensitivity average value > 5%
or < -5% then the average contribution to variance is included in the table. The sensitivity results
from the original study are included for comparison.
The results from the BSNOx Method 1 sensitivities show that the steady-state exhaust
flow rate bias from the original study has been eliminated in Mod B, Mod 1, Mod 2, and Mod 3.
There is also a slight increase (from the original study) in the average sensitivity due to variation
in the steady-state exhaust flow rate with all the modification runs. Slight increases in the
average sensitivity for the steady-state NOx error surface are seen in all modification runs except
Mod 2.
Method 2 results for BSNOx show that the steady-state CO2 bias effect and the NOx time
alignment variation effect were eliminated in all the modification runs. However, new error
surfaces that showed sensitivities in the modification runs were the steady-state CO2 variation
effect and the BSFC DOE Test bias effect. There were also slight increases in the average
sensitivity due to variation for the steady-state NOx error surface in all the modification runs.
The steady-state CO2 bias effect for Method 3 was eliminated in all the modification runs
and a variation effect due to steady-state CO2 was included. Also, there was a slight increase in
the average sensitivity due to variation for the steady-state NOx error surface in all the
modification runs.
SwRI Report 03.12859.06 81 of 83
-------
TABLE 31. SENSITIVITY RESULTS COMPARING 13 REFERENCE NTE EVENTS
ACROSS 5 MONTE CARLO FULL MODEL SIMULATIONS FOR BSNOX
1
1
20
31
35
51
SSNOx
SS Exhaust Flow
Warm-up torque
Engine Manuf
Torque
NOx Time
Alignment
Pulse Flow OnOff
SS Exhaust
Flow OnOff
Original
21.0
6.3
-23.2
-9.8
6.1
14.7
14.6
22.3
10.2
-23.8
-9.9
5.5
15.1
31.2
9.5
-20.7
-8.7
5.0
13.8
20.8
11.0
-23.7
-9.9
5.5
16.4
25.4
10.4
-22.6
-9.4
5.6
14.8
2
1
38
42
45
51
SSNOx
Warm-up BSFC
Engine Manuf BSFC
SSCO2
NOx Time
Alignment
BSFC DOE
Test OnOff
SSCO2 OnOff
29.5
6.3
13.6
6.1
-34.9
33.3
5.9
14.1
-23.2
-5.9
43.9
5.2
12.3
-20.0
-5.3
30.4
6.2
15.8
-24.1
-6.5
37.1
5.6
14.0
-21.9
-6.1
3
1
31
35
45
51
SSNOx
Warm-up Torque
Engine Manuf
Torque
SSCO2
NOx Time
Alignment
SSCO2 OnOff
23.6
-27.8
-11.7
6.0
-25.2
25.6
-28.5
-11.9
-16.3
5.4
35.1
-25.0
-10.4
-14.8
4.8
34.2
-29.4
-12.6
-16.8
5.8
29.2
-27.0
-11.8
-15.6
5.3
DEFINITIONS
Original April 2007 Study: All original error surfaces
Mod B Revised SS Exhaust Flow and SS CO2 error surfaces
Mod 1 Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error surfaces
Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error surfaces
Mod 3 Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error surfaces
SwRI Report 03.12859.06
82 of 83
-------
12.0 REFERENCES
Feist, M.D., Sharp, C. A., Mason, R.L., and Buckingham, J.P., "Determination of PEMS
Measurement Allowances for Gaseous Emissions Regulated Under the Heavy-Duty
Diesel Engine In-Use Testing Program," SwRI Project 03-12024, U.S. Environmental
Protection Agency, California Air Resources Board and Engine Manufacturers
Association, April 20.
SwRI Report 03.12859.06 83 of 83
------- |