Additional Analyses of the Monte Carlo
   Model Developed for the Determination
   of PEMS Measurement Allowances for
   Gaseous Emissions Regulated Under
   the Heavy-Duty Diesel Engine In-Use
   Testing Program
United States
Environmental Protection
Agency

-------
               Additional Analyses of the Monte Carlo
                Model Developed for the Determination
                of PEMS  Measurement Allowances for
                 Gaseous Emissions Regulated Under
               the Heavy-Duty Diesel Engine In-Use
                              Testing Program
                              Assessment and Standards Division
                             Office of Transportation and Air Quality
                             U.S. Environmental Protection Agency
                                   Prepared for EPA by
                                Southwest Research Institute
                                EPA Contract No. EP-C-05-018
                                 Work Assignment No. 2-6
v>EPA
                NOTICE

                This technical report does not necessarily represent final EPA decisions or
                positions. It is intended to present technical analysis of issues using data
                generated in the associated test program. The purpose in the release of such
                reports is to facilitate the exchange of technical information and to inform the
                public of technical developments which may form the basis for a final EPA
                decision, position, or regulatory action.
United States                                     EPA420-R-07-010
Environmental Protection                               .   ^ „„.,
Agency                                        August 2007

-------
                              EXECUTIVE SUMMARY
       This report documents the program conducted by Southwest Research Institute® (SwRI),
on behalf of the U.S. Environmental Protection Agency (EPA), the objective of which was to
perform additional  analyses on the Portable Emissions Measurement Systems (PEMS) Monte
Carlo simulation models in order to  determine validation of the three  emissions  (BSNOx,
BSNMHC and BSCO) using three calculation methods.

       Several steady-state error surfaces were modified based on recommendations from the
Heavy-Duty In-Use Testing (HDIUT) Steering Committee.  These included modifications to the
steady-state NOx, exhaust flow rate, CO and CO2 error surfaces. Several reference NTE events
were run which produced validation results from all three emissions and all three calculation
methods.

       Measurement allowances were computed for two of the simulation strategies based  on
using 50 reference NTE events. The Mod 1 strategy involved changing the steady-state CO, CO2
and exhaust flow rate error surfaces to  eliminate bias while changing the steady-state NOx to a
level independent error  surface including all the test data. The Mod 2 strategy was the same as
Mod 1  except the steady-state NOx error surface was also changed to a level independent error
surface  but excluded several questionable low NOx values from one of the test engines. The
measurement  allowance values by calculation method determined at the conclusion of these
analyses are summarized in  Table 1.   This report details the process  used  to determine the
measurement values reported in Table 1.
        TABLE 1. MEASUREMENT ALLOWANCES FOR MOD 1 AND MOD 2
Pollutant
NOX
NMHC
CO
Calculation
Method
1
2
3
1
2
3
1
2
3
Mod 1 Measurement
Allowance, g/hp-hr
0.232
0.178
0.192
0.015
0.014
0.014
0.268
0.258
0.270
Mod 2 Measurement
Allowance, g/hp-hr
0.211
0.151
0.171
0.014
0.014
0.014
0.266
0.250
0.262
SwRI Report 03.12859.06

-------
                             LIST OF ACRONYMS

Brake-Specific                        BS
California Air Resources Board          CARB
Center for Environmental Research
& Technology                         CE-CERT
Code of Federal Regulations             CFR
Electronic Flow Meter                  EFM
Empirical Distribution Function          EDF
Engine Control Module                 ECM
Engine Manufacturer's Association       EMA
Environmental Protection Agency         EPA
Heavy Duty In-Use Testing              HDIUT
Heavy Heavy Duty                    HHD
Light Heavy Duty                     LHD
Median Absolute Deviation              MAD
Medium Heavy Duty                   MHD
Memorandum of Agreement             MO A
Mobile Emissions Laboratory            MEL
Not To Exceed                        NTE
Portable Emissions Measurement System   PEMS
Root Mean Square                     RMS
SEMTECH-DS  SN G05-SDS04          PEMS 1
SEMTECH-DS  SN G05-SDS02          PEMS 2
SEMTECH-DS  SN G05-SDS03          PEMS 3
SEMTECH-DS  SN G05-SDS01          PEMS 4
SEMTECH-DS  SN D06-SDS01          PEMS 5
SEMTECH-DS  SN D06-SDS06          PEMS 6
SEMTECH-DS  SN F06-SDS02          PEMS 7
Southwest Research Institute             SwRI
Standard Deviation                    SD
SwRI Report 03.12859.06
11

-------
                         TABLE OF CONTENTS
EXECUTIVE SUMMARY	i
LIST OF ACRONYMS	ii
LIST OF FIGURES	iv
LIST OF TABLES	vi
1.0   INTRODUCTION	1
2.0   ERROR SURFACE MODIFICATIONS	2
2.1   Steady-State Exhaust Flow Rate Error Surface	2
2.2   Steady-State CO2 Error Surface	3
2.3   Steady-State CO Error Surface	4
2.4   Steady-State NOX Error Surface	5
3.0   REFERENCE NTE EVENTS	9
4.0   CE-CERT MEASUREMENTS	13
5.0   MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE EVENTS
     USING MOD 1, MOD 2 AND MOD 3	14
6.0   MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE EVENTS
     USING MODE	25
7.0   MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE EVENTS
     USING MOD 1	35
8.0   MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE EVENTS
     USING MOD 2	45
9.0   MEASUREMENT ALLOWANCE CALCULATIONS	56
10.0  VALIDATION SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS.... 77
11.0  FULL MODEL SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS ... 81
12.0  REFERENCES	83
SwRI Report 03.12859.06                   iii

-------
                                 LIST OF FIGURES
1            Revised Error Surface for Steady-State Exhaust Flow Rate	3
2            Revised Error Surface for Steady-State CC>2	4
3            Revised Error Surface for Steady-State CO	5
4            Revised Error Surface Mod 1 for Steady-State NOX	6
5            Revised Error Surface Mod 2 for Steady-State NOX	7
6            Revised Error Surface Mod 3 for Steady-State NOX	8
7            Distribution of Ideal NOx (g/kW-hr) Method  1 for 23 Selected Ref NTE
             Events and 195 Ref NTE Events	11
8            Distribution of Ideal NOx (g/kW-hr) Method  1 for 50 Selected Ref NTE
             Events and 195 Ref NTE Events	12
9            Validation EOF Plots Using 23 Reference NTE Events for NOX Method 1
             Mods 1,2 and 3	16
10           Validation EOF Plots Using 23 Reference NTE Events for NOX Method 2 for
             Mods 1,2 and 3	17
11           Validation EOF Plots Using 23 Reference NTE Events for NOX Method 3 for
             Mods 1,2 and 3	18
12           Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 1
             for Mods 1,2 and 3	19
13           Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 2
             for Mods 1,2 and 3	20
14           Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 3
             for Mods 1,2 and 3	21
15           Validation EOF Plots Using 23 Reference NTE Events for CO Method 1 for
             Mods 1,2 and 3	22
16           Validation EOF Plots Using 23 Reference NTE Events for CO Method 2 Mods
             1,2 and 3	23
17           Validation EOF Plots Using 23 Reference NTE Events for CO Method 3 for
             Mods 1,2 and 3	24
18           Validation EOF Plots Using 23 Reference NTE Events for NOX Method 1 Mod B	26
19           Validation NTE Events Using 23 Reference NTE Events for NOX Method 2 Mod B. .27
20           Validation EOF Plots Using 23 Reference NTE Events for NOX Method 3 Mod B	28
21           Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 1 Mod B .. 29
22           Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 2 Mod B .. 3 0
23           Validation EOF Plots Using 23 Reference NTE Events for NMHC Method 3 Mod B .. 31
24           Validation EOF Plots Using 23 Reference NTE Events for CO Method 1 Mod B	32
25           Validation EOF Plots Using 23 Reference NTE Events for CO Method 2 Mod B	33
26           Validation EOF Plots Using 23 Reference NTE Events for CO Method 3 Mod B	34
27           Validation EOF Plots Using 50 Reference NTE Events for NOX Method 1 Mod 1	36
28           Validation EOF Plots Using 50 Reference NTE Events for NOX Method 2 Mod 1	37
29           Validation EOF Plots Using 50 Reference NTE Events for NOX Method 3 Mod 1	38
30           Validation EOF Plots Using 50 Reference NTE Events for NMHC Method 1 Mod 1...39
31           Validation EOF Plots Using 50 Reference NTE Events for NMHC Method 2 Mod 1.. .40
32           Validation EOF Plots Using 50 Reference NTE Events for NMHC Method 3 Mod 1.. .41
33           Validation EOF Plots Using 50 Reference NTE Events for CO Method 1 Mod 1	42
34           Validation EOF Plots Using 50 Reference NTE Events for CO Method 2 Mod 1	43
SwRI Report 03.12859.06                      iv

-------
3 5           Validation EOF Plots Using 50 Reference NTE Events for CO Method 3 Mod 1	44
36           Validation EOF Plots Using 50 Reference NTE Events for NOX Method 1 Mod 2	46
37           Validation EOF Plots Using 50 Reference NTE Events for NOX Method 2 Mod 2	47
38           Validation EOF Plots for 50 Reference NTE Events for NOX Method 3 Mod 2.. .48
39           Validation EOF Plots for 50 Reference NTE Events for NMHC Method 1 Mod 2	49
40           Validation EOF Plots for 50 Reference NTE Events for NMHC Method 2 Mod 2	50
41           Validation EOF Plots Using 50 Reference NTE Events for NMHC Method 3 Mod 2... 51
42           Validation EOF Plots Using 50 Reference NTE Events for CO Method 1 Mod2	52
43           Validation EOF Plots Using 50 reference NTE Events for CO Method 2 Mod 2.53
44           Validation EOF Plots Using 50 Reference NTE Events for CO Method 3 Mod 2	54
45           Regression Plot of 95th Percentile Delta BSNOX Versus Ideal BSNOX for Method 1
             Modi	57
46           Regression Plot of 95th Percentile Delta BSNOX Versus Ideal BSNOX for Method 2
             Modi	58
47           Regression Plot of 95th percentile Delta B SNOX Versus Ideal B SNOX for Method 3
             Modi	59
48           Regression Plot of 95th percentile Delta BSNMHC Versus Ideal BSNMHC for
             Method 1 Modi	60
49           Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for
             Method2Modl	61
5 0           Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for
             Method 3 Modi	62
51           Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 1
             Modi	63
52           Regression Plot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 2
             Modi	64
5 3           Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 3
             Modi	65
54           Regression Plot of 95th Percentile Delta BSNOx Versus Ideal BSNOx for Method 1
             Mod2	66
55           Regression Plot of 95th Percentile Delta BSNOX Versus Ideal BSNOX for Method 2
             Mod2	67
56           Regression Plot of 95th Percentile Delta BSNOX Versus Ideal BSNOX for Method 3
             Mod2	68
5 7           Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for
             Method !Mod2	69
5 8           Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for
             Method2Mod2	70
5 9           Regression Plot of 95th Percentile Delta B SNMHC Versus Ideal B SNMHC for
             Method 3 Mod 2	71
60           Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 1
             Mod 2	72
61           Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 2
             Mod 2	73
62           Regression Plot of 95th Percentile Delta B SCO Versus Ideal B SCO for Method 3
             Mod 2	74
SwRI Report 03.12859.06

-------
                                 LIST OF TABLES

Table                                                                         Page

1            Measurement Allowances For MOD 1 and Mod 2	i
2            Revised Error Surfaces Used in Monte Carlo Simulations	8
3            Reference NTE Events Used in Monte Carlo Simulations	10
4            Descriptive Statistics for Ideal BSNOX (g/kW-hr) for Various NTE Subsets	12
5            Convergence criteria by Emission	14
6            BSNOX Validation Results Based on 23 Reference NTE Events	15
7            Summary of Validation Results	55
8            Measurement Error at Threshold for BSNOX Using Regression and Median
             Methods for Method 1 Mod 1	57
9            Measurement Error at Threshold for BSNOX Using Regression and Median
             Methods for Method 2 Mod 1	58
10           Measurement Error at Threshold for BSNOX Using Regression and Median
             Methods for Method 3 Mod 1	59
11           Measurement Error at Threshold for BSNMHC Using Regression and Median
             Methods for Method 1 Mod 1	60
12           Measurement Error at Threshold for BSNMHC Using Regression and Median
             Methods for Method 2 Mod 1	61
13           Measurement Error at Threshold for BSNMHC Using Regression and Median
             Methods for Method 3 Mod 1	62
14           Measurement Error at Threshold for B SCO Using Regression and Median
             Methods for Method 1 Mod 1	63
15           Measurement Error at Threshold for B SCO Using Regression and Median
             Methods for Method 2 Mod 1	64
16           Measurement Error at Threshold for B SCO Using Regression and Median
             Methods for Method 3 Mod 1	65
17           Measurement Error at Threshold for BSNOX Using regression and Median
             methods  for Method 1 Mod 2	66
18           Measurement Error at Threshold for BSNOX Using Regression and Median
             methods  for Method 2 Mod 2	67
19           Measurement Error at Threshold for BSNOX Using Regression and Median
             Methods for Method 3 Mod 2	68
20           Measurement Error at Threshold for BSNMHC Using Regression and Median
             Methods for Method 1 Mod 2	69
21           Measurement Error at Threshold for BSNMHC Using Regression and Median
             Methods for Method 2 Mod 2	70
22           Measurement Error at Threshold for BSNMHC Using Regression and Median
             Methods for Method 3 Mod 2	71
23           Measurement Error at Threshold for B SCO Using Regression and Median
             Methods for Method 1 Mod 2	72
24           Measurement Error at Threshold for B SCO Using Regression and Median
             Methods for Method 2 Mod 2	73
SwRI Report 03.12859.06                      vi

-------
25           Measurement Error at Threshold for B SCO Using Regression and Median
             Methods for Method 3 Mod 2	74
26           Summary of Measurement Errors at Respective Threshold (%)	75
27           Summary of Measurement Allowance, g/hp-hr	76
28           Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte
             Carlo validation Simulations for BSNOX	78
29           Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte
             Carlo validation Simulations for BSNMHC	79
30           Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte
             Carlo validation Simulations for BSCO	80
31           Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte
             Carlo full model Simulations for BSNOX	82
SwRI Report 03.12859.06                      vii

-------
                               1.0    INTRODUCTION

       Southwest Research Institute  completed a Portable Emissions  Measurement System
(PEMS) program on behalf of the U.S. Environmental Protection Agency (EPA) in early April
2007.  The  purpose  of this project was to determine the brake-specific (BS)  measurement
allowances for the gaseous pollutants regulated under the Heavy-Duty In-Use Testing (HDIUT)
program [Ref 1]. The study was performed under cooperation between the EPA, the California
Air Resources Board  (CARB), and the Engine  Manufacturer's Association (EMA).  All efforts
during this  program  were conducted under  the  direction of  a joint  body,  the  HDIUT
Measurement Allowance Steering Committee,  referred to in this report simply as the Steering
Committee.

       The program consisted of modeling various PEMS measurement errors using a statistical
Monte Carlo modeling simulation.  The simulation results were used to  generate the brake-
specific measurement allowances based on three calculation methods for BS emissions of NOX,
NMHC and CO.  To confirm the results of the simulation, a SEMTECH-DS PEMS was operated
in-use with the CE-CERT Mobile Emission Laboratory (MEL) over several routes  in California.
The differences between the PEMS and MEL gaseous emission measurements were used to
validate the SwRI Monte Carlo modeling simulation results.

       Calculation  Methods 2 (BSFC based) and 3  (ECM Fuel Specific)  did not validate for
BSNOX based on the CE-CERT MEL emissions and the Monte Carlo PEMS simulations. Given
this result, EPA contracted with SwRI in April 2007 to conduct additional analyses to identify
possible causes as to why the BSNOX did not validate. The results of these analyses are included
in this report.
SwRI Report 03.12859.06                     1 of 83

-------
                     2.0    ERROR SURFACE MODIFICATIONS

       Since calculation Methods 2 and 3 did not validate for BSNOX after the Monte Carlo
model simulation runs during the original PEMS study, the associated Steering Committee chose
to modify key error surfaces in order to determine if such alterations would effect the validation
of Methods 2 and 3.  Over the course of several meetings, the Steering Committee brainstormed
to generate  alternate  error surface processing methods.  The Committee decided to modify
Steady-State CO, CO2, Exhaust Flow Rate, and NOX, as these surfaces had significant influence
on the Model results based on the sensitivity analyses.

       In reprocessing  the  error surfaces, the Steering Committee agreed  that it would be
necessary to remove any biases that were recorded in the SwRI laboratory. It was assumed that
these bias errors were due to the limited number of PEMS units and comparative observations,
and that  if more PEMS and Sensors Inc. exhaust flow meters were tested the biases may have
been eliminated.  Therefore, to remove  the biases the original steady-state error surfaces were
transformed to be a symmetric error surface with the 50th percentile errors set to zero and the 5th
and 95th  percentile errors modified to be a mirror image of one another.  Two analysis methods
were used to generate the revised error surfaces.  If the recorded error data showed an emissions
level  dependency, an  envelope was generated to encompass the extreme 5th  or  95th percentile
error data. The envelope segments with the largest absolute error were then mirrored to generate
a symmetric error surface.  A more  rigorous analysis was used to reprocess level independent
error surfaces.

2.1     Steady-State Exhaust Flow Rate Error Surface

       Figure 1 shows the  reprocessed, level-dependent  steady-state exhaust flow rate error
surface as well as the exhaust flow error surface data that was used in the original MC model
runs.  The largest errors recorded during laboratory testing were positive deltas measured during
Engine 3 testing with the 3-inch EFMs.  The 3-inch EFM errors defined the 95th percentile error
contours for the original error surface as well as the  reprocessed error surface.  The difference
between  the two 95l  percentile contours was  due to a correction of the 3-inch EFM calibrations
when the exhaust flow rate  error  surface  was reprocessed.   SwRI and  Sensors  Inc. had
recalibrated  the 3-inch flow meters  using data  generated  at  SwRI.  The  SwRI calibration
increased the slope multiplier by  approximately 4 % for each 3-inch flow meter.  Later in the
program, Sensors Inc.  discovered the 3-inch EFMs were likely not operating correctly on the
SwRI flow stand due to EFM pressures below ambient levels.  Therefore, each 3-inch flow meter
calibration was corrected to the original coefficients generated by Sensors.  The corrected 3-inch
flow meter data generated the 95th  percentile  contour  shown in Figure  1.   Because the 95th
percentile errors were larger than the 5th percentile errors, the 95th percentile data was mirrored to
generate  the 5th percentile contour in the reprocessed error surface.
SwRI Report 03.12859.06                     2 of 83

-------
                    -95th percentile
                    • Previous 95th
-50th percentile (median)
• Previous 50th
-5th percentile
• Previous 5th
                               Lab Reference Mean Exhaust Flow Rate [% of Max]

  FIGURE 1. REVISED ERROR SURFACE FOR STEADY-STATE EXHAUST FLOW
                                          RATE
2.2    Steady-State CO2 Error Surface
                                                                               th
       Figure 2 shows the reprocessed, level-dependent CC>2 error surface. The 95  percentile
                                                                          th
contour represents the envelope that was created to encompass the original 95  percentile error
data.   Line segments were used  to  connect the most  extreme points from the original  95
                                               th
                     -th
                                                                     th
percentile data. The 95  percentile contour was mirrored to produce the 5  percentile error data,
with the 50th percentile deltas set to zero. The reprocessed error surface was meant to encompass
all possible PEMS CC>2 measurement errors, not just those measured at SwRI.
SwRI Report 03.12859.06
3 of 83

-------
                     -95th percentile
                     • Previous 95th
                                       -50th percentile (median)
                                       • Previous 50th
-5th percentile
• Previous 5th
     0.8
     0.6
   " 0.4
     0.2
     0.0
   c
   o
   o
   CM
   O
   O
   40

-0.2
     -0.4
     -0.6
     -0.8
                4.5
                        5.0
                                5.5
                                         6.0
                                                 6.5
                                                         7.0
                                                                  7.5
                                                                          8.0
                                                                                  8.5
                                                                                          90
                                  Lab Reference Mean CO2 Concentration [%]

          FIGURE 2. REVISED ERROR SURFACE FOR STEADY-STATE CO2

2.3    Steady-State CO Error Surface

       Figure 3 shows the revised steady-state error surface for CO.  Because the CO error data
showed no level dependence, a  statistical calculation process,  developed by Bill Martin from
Cummins  Inc.,  was used to generate the  reprocessed CO  error surface.   The  total standard
deviation of the pooled CO error data was calculated by combining the variance of the PEMS
mean delta standard deviation about zero and the PEMS pooled estimate of repeatability standard
deviation.  The calculation of the total standard deviation is shown below.
                         ^U total   \
                                           repeatability  ~~  -  mean, zero
                                    repeatability
                                               IV S7)       2
                                                / j ^-^indiv,perns
                                                      n
  SDindiV,pems = standard deviation of each PEMS calculated over 20 repeats of each steady-state
                                            point

                n = 10 steady-state points per engine * 3 engines * 3 PEMS = 90
SwRI Report 03.12859.06
                                      4 of 83

-------
                              SD,
                                                    n-l
 PEMSindiv,mean = mean delta of each PEMS calculated over 20 repeats of each steady-state point

       Shown below, the  multiplier  for  a  90%  confidence  interval  assuming  a  normal
distribution for the errors was applied to the total standard deviation (48.84 ppm = 0.004882%)
to generate the constant 5th and 95th percentile error values.

                              5* % = -1.645 * SDtotal = - 0.008%
                              95*% = 1.645 * SDtotal = + 0.008%
                    •95th percentile
                    • Previous 95th
                                      •50th percentile (median)
                                      • Previous 50th
-5th percentile
• Previous 5th
     0.012
     0.010
     0.008  -     - 4M»-
     0.006
     0.004
   5. 0.002
   LU
   Q.
                        •4*4-
                                              «  «»
0.000
   o.opoo
-0.002
-0.004

-0.006

-0.008 \      - - |

-0.010
                        0.0005
                                        0.0010
                                                       0.0015
                                                                       0.0020
                                                                                       0.0 )25
                                   Lab Reference Mean CO Concentration [%]
          FIGURE 3. REVISED ERROR SURFACE FOR STEADY-STATE CO
2.4    Steady-State NOX Error Surface

       For  reprocessing  the  original steady-state NOX, three different methods were used  to
generate three NOX  concentration error surfaces.  For each of the three modified NOX error
surfaces, the original set of Engine 2 steady-state data was added to the original pooled data set.
The Engine 2 steady-state data was collected in a repeat run in the original study due to PEMS
failures that caused a large time gap in the original data.  The time gap was associated with a
NOX concentration shift.  Since the NOX shift would have caused problems with the steady-state
SwRI Report 03.12859.06
                                      5 of 83

-------
variance correction of the transient data the repeat run data was used.   However,  the original
Engine 2 delta data was later determined to be valid and thus was included in the modified data
set to increase the PEMS delta observations.
       The first NOX error surface modification (Mod  1) followed a procedure  similar to the
      -state CO error surface. Using the complete NOX delta data set, the 5th and 95th percentile
error values were  calculated with the total standard deviation  (15.08 ppm). The results of the
first NOX erro
±24.80 ppm.
steady-state CO error surface. Using the complete NOX delta data set, the 5th and 95th percentile
                                                                >8l   ,
first NOX error surface modification are shown in Figure 4 with the 5l  and 95l  percentiles set to
         40
         20 -
      Q.
      Q.
      I -20 H
      re

      m
      § -40
      O
      5
      LU
      0.
        -80
        -100
                      •95th percentile
                      • Previous 95th
                                          -50th percentile (median)
                                          • Previous 50th
•5th percentile
• Previous 5th
                     ,- .2.00
                                   300
                                               400
                                                           500
                                                                       600
                                 Lab Reference Mean NOx Concentration [ppm]
     FIGURE 4. REVISED ERROR SURFACE MOD 1 FOR STEADY-STATE NOX

       The second NOX error surface modification (Mod 2) was similar to the first modification
except six outlying low delta values recorded during Engine 3 testing were removed from the
pooled data set.  During Engine 3 steady-state testing, PEMS 1 and 6 showed extremely low
deltas at high NOX  concentration levels.  PEMS 4 also showed low biases at high NOX levels;
however, the biases were not as large as those measured for the other PEMS.  Therefore, PEMS
1 and 6 delta measurements at the three highest NOX levels measured during engine testing were
removed from the data set.  This resulted in a total standard deviation equal to 9.39 ppm and the
-th
         -th
5  and 95  percentile deltas equal to ±15.45 ppm.  The results of this analysis are shown in
Figure 5.
SwRI Report 03.12859.06
                                          6 of 83

-------
                      -95th percentile
                      • Previous 95th
-50th percentile (median)
• Prefious 50th
-5th percentile
• Previous 5th
        -100
                                 Lab Reference Mean NOx Concentration [ppm]
     FIGURE 5. REVISED ERROR SURFACE MOD 2 FOR STEADY-STATE NOX

       Shown in Figure 6, the third NOX error surface modification (Mod 3) combined the level-
dependent and level-independent analysis methods. Below 311 ppm, the delta data was assumed
to be level independent.  Therefore, a statistical method similar to that used for CO was used for
the NOX delta data below 311 ppm.  The total standard deviation for the level-independent data
was 9.41 ppm and the resulting 5th and 95th percentiles were ±  15.48 ppm.  Above 311 ppm, the
original 5l  percentile profile was used and mirrored  to  the  95th percentile to generate a
symmetric error surface.
SwRI Report 03.12859.06
7 of 83

-------
                      -95th percentile
                      • Previous 95th
-50th percentile (median)
• Previous 50th
-5th percentile
• Previous 5th
        100

         80

         60

         40

         20

          0

        -20

        -40

        -60

        -80

        -100
                                 Lab Reference Mean NOx Concentration [ppm]
     FIGURE 6. REVISED ERROR SURFACE MOD 3 FOR STEADY-STATE NOX

       In addition to running the MC simulations with the three different changes to the steady-
state NOx error surface, it was useful to run a simulation with the original steady-state NOx error
surface and only include the revised steady-state exhaust flow rate and steady-state CO2 error
surfaces.  This set of simulations is called the Mod B runs.  Table 2 provides a summary of the
four MC simulation runs made in this study with the revised error surfaces that were included in
each run.  All the other error surfaces that are not listed in Table 2 are the same as those run in
the original PEMS study.
 TABLE 2. REVISED ERROR SURFACES USED IN MONTE CARLO SIMULATIONS

Steady-State Exhaust Flow
Rate
Steady-State CO2
Steady-State CO
Steady- State NOX Mod 1
Steady- State NOX Mod 2
Steady-State NOX Mod 3

X
X





X
X
X
X



X
X
X

X


X
X
X


X
SwRI Report 03.12859.06
8 of 83

-------
                          3.0    REFERENCE NTE EVENTS

       The Monte Carlo simulation results from the original study included a reference data set
consisting of 195 NTE events gathered from a number of sources.  These included five engine
manufacturers,  SwRI  transient  lab  tests  and pre-pilot CE-CERT  data.  The investigations
performed in this modified program included only a subset of the original 195 reference NTE
events. Simulations from three different subsets of the original NTE events were run to study the
validation of the emissions by the three calculation methods using the revised error surfaces.

       Table 3 lists the NTE events along with their ideal BSNOX values  selected for each
simulation. The various NTE subsets are described as follows:

   •   13 Reference Events - These are the same  13  NTE events chosen during the original
       study to investigate the error surface sensitivities due to bias and variance.  These events
       were selected to bound the BSNOX threshold of 2.6820 g/kW-hr.  These 13 NTE events
       were used to examine the error surface sensitivities for both the validation and full model
       simulation runs.
   •   23 Reference Events - Ten additional NTE events were added to the original 13  NTE
       events described above in order to increase the sample size for generating the  validation
       EDF plots using the drift corrected CE-CERT data.
   •   50 Reference Events - After the model validation was confirmed for the three  emissions
       across the three calculation methods, Mods  1  and 2 were chosen for continued analysis.
       Twenty-seven additional reference NTE events were simulated with the  Monte  Carlo
       model in order to estimate the measurement allowances for each emission.  The selection
       of the additional 27 reference NTE events was based on maintaining a similar distribution
       of ideal BSNOX values for Method 1 established from the original study containing 195
       reference NTE events.

       Figure  7 shows  comparison histograms  of the Method  1  BSNOX values for the 23
reference NTE events  (lower histogram) and the 195 reference NTE events (upper histogram)
while Figure 8 compares the  Method 1 BSNOX histograms for the 50 reference NTE events
(lower histogram) and the 195 reference NTE events (upper histogram). Descriptive statistics of
BSNOX for these NTE subsets are detailed in Table 4.
SwRI Report 03.12859.06                     9 of 83

-------
  TABLE 3. REFERENCE NTE EVENTS USED IN MONTE CARLO SIMULATIONS

1
3
4
7
11
16
20
22
23
25
29
37
38
40
43
44
46
51
57
63
65
66
67
69
71
82
86
87
89
92
96
99
103
115
125
127
136
139
146
148
157

4.0713
3.0668
3.5832
5.2516
1.8583
2.3511
3.7245
4.7916
3.1483
5.4061
5.5261
5.4511
0.0250
4.1675
1.1473
1.0730
2.6958
2.8299
4.3382
2.6670
2.7437
3.9378
5.8600
3.0257
6.6867
2.4569
1.7132
1.5207
2.2566
1.6041
1.6224
1.8147
1.9186
1.3854
2.3214
3.3005
2.6782
2.4018
2.3053
1.9985
3.4666

X








X


X


X
X
X

X



X

X

X











X
X

X
X

X


X


X
X

X


X
X
X
X
X



X

X

X
X

X





X
X

X
X

X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
SwRI Report 03.12859.06
10 of 83

-------
160
162
163
168
176
177
179
191
193
2.0773
2.1405
2.5908
1.5859
1.9671
1.7507
2.2023
6.0815
5.7521


X








X

X




X
X
X
X
X
X
X
X
X
          40
          30
          10
       sr
           0
          10
          20
               Ideal NOx Method  1  195RefNTE
              02468
               Ideal NOx Method 1 23 Ref NTE
   FIGURE 7. DISTRIBUTION OF IDEAL NOX (G/KW-HR) METHOD 1 FOR 23
         SELECTED REF NTE EVENTS AND 195 REF NTE EVENTS
SwRI Report 03.12859.06
11 of 83

-------
       Ideal NOx Method 1 195 Ref NTE Events
           40

           30

        Ł 20
        0)
        =3 10
        D"
        Ł  0
        LL
           10

           20
              02468
        Ideal  NOx Method 1 50 Ref NTE Events

   FIGURE 8. DISTRIBUTION OF IDEAL NOX (G/KW-HR) METHOD 1 FOR 50
         SELECTED REF NTE EVENTS AND 195 REF NTE EVENTS
   TABLE 4. DESCRIPTIVE STATISTICS FOR IDEAL BSNOX (G/KW-HR) FOR
                   VARIOUS NTE SUBSETS

Minimum
Maximum
Mean
Median
Standard Deviation

0.0249
5.5261
2.8983
2.6782
1.3671

0.0249
6.6867
3.0068
2.6289
1.5175

0.0249
7.1927
3.0071
2.6033
1.4807
SwRI Report 03.12859.06
12 of 83

-------
                         4.0    CE-CERT MEASUREMENTS

       The CE-CERT on-road measurements collected during the original PEMS study included
delta BS emissions for all three emissions and three calculation methods. As part of this study to
investigate all possible reasons why BSNOX did not validate for Methods 2 and 3,  the 100 NTE
events  from the CE-CERT data were examined for correctness by EPA.  Since the CE-CERT
data had not been drift corrected in the original PEMS study, EPA performed drift corrections on
all three emissions  for the 100 CE-CERT NTE events.  As a result, several CE-CERT NTE
events  did not pass the drift check criteria and, therefore, were not  included in the simulation
performed for this study.  In addition, time alignment problems were found in a few of the 100
CE-CERT NTE events and these were also  excluded from the simulation.  After deleting the
NTE events due to drift check or time alignment problems, the on-road delta BS emissions were
calculated from 81 NTE events for BSNOX and  87 NTE events for BSNMHC and BSCO.  These
remaining CE-CERT NTE events  were all  drift corrected.   In  contrast, the original PEMS
program did not use drift correction in the CE-CERT on-road NTE events.
SwRI Report 03.12859.06                    13 of 83

-------
    5.0    MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE
                       EVENTS USING MOD 1, MOD 2 AND MOD 3

       The Monte Carlo simulations performed using the 23 reference NTE events listed in
Table 3 included modifications for the error surfaces listed in Table 2.  All the error surfaces
from the original MC simulations were used except those listed as revised in Table 2. Each of
the 23 reference NTE events were run using the  error surfaces revised  for Mod 1 for either
10,000 or 30,000 trials.  If the ideal BSNOX for an NTE event was less than 2.68204 g/kW-hr
then the simulation was run for 10,000 trials. Otherwise, the NTE was simulated using  30,000
trials.  Once the MC simulations were completed  using the Mod 1 revised error surfaces, the
same 23 NTE events were simulated a second time with the error surface modifications for Mod
2.  Lastly, the 23 reference NTE events were simulated using the error surface modifications for
Mod 3.

       In Mod 1, Mod 2, and Mod 3 simulation runs, all three emissions converged within 1% of
the emissions threshold value by all three calculation methods. Table 5  lists the convergence
criteria  for all three brake-specific  emissions.  Thus, all  23 reference  NTE events met the
convergence criteria.

                TABLE 5.  CONVERGENCE CRITERIA BY EMISSION

BSNOx
BSNMHC
BSCO

0.02682
0.00282
0.26015
       From the MC simulations, the validation 5th and 95th percentile delta BS emissions were
extracted from the output files for each of the 23 reference NTE events.  These delta emissions
were then plotted as empirical distribution functions (EDF) to form a validation interval for the
on-road data.  Also plotted was the EDF computed from the on-road CE-CERT NTE events.
Figure 9 through Figure 11 represent the validation plots for the BSNOX for calculation methods
1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th percentile EDFs for the
Mod 1, Mod 2, and Mod 3 runs. Figure 12 through Figure 17 depict the validation plots for the
BSNMHC and BSCO simulation runs, respectively. The validation criteria set by the  Steering
Committee for the original study was used in the modified program.  It included the following
criteria:

    •   At least 90% of the  CE-CERT emissions deltas  must be within the  5th  and 95th
        percentiles of the MC validation cumulative emissions deltas.
    •   No  more than  10% of the CE-CERT emissions deltas  may  fall  less  than  the 5th
        percentile or greater than the 95th percentile. This may indicate that the model is biased
        low or high.
    •   Validation must be shown for all three calculation methods.
SwRI Report 03.12859.06                     14 of 83

-------
      A summary of the BSNOX validation conclusions based on the 23 reference NTE events
is provided in Table 6.  All BSNOX, BSNMHC and BSCO emissions validated for all three
calculation methods and all three Mod runs.
    TABLE 6. BSNOx VALIDATION RESULTS BASED ON 23 REFERENCE NTE
                                   EVENTS

Mod 1
Mod 2
Mod3


• All CE-CERT data >
5th percentile
• All CE-CERT data <
95th percentile
•VALID
• All CE-CERT data >
5th percentile
• All CE-CERT data <
95th percentile
•VALID
• All CE-CERT data >
5th percentile
• All CE-CERT data <
95th percentile
•VALID

• All CE-CERT data >
5th percentile
• 1% of CE-CERT data
> 95th percentile
•VALID
• All CE-CERT data >
5th percentile
• 1% of CE-CERT data
> 95th percentile
•VALID
• All CE-CERT data >
5th percentile
• All CE-CERT data <
95th percentile
•VALID

• All CE-CERT data >
5th percentile
• All CE-CERT data <
95th percentile
•VALID
• All CE-CERT data >
5th percentile
• 1% of CE-CERT data
> 95th percentile
•VALID
• All CE-CERT data >
5th percentile
• All CE-CERT data <
95th percentile
•VALID
SwRI Report 03.12859.06
15 of 83

-------
                           Validation Analysis 5th and 95th Percentile Deltas
                    Compared  23 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
                               Error Models Mod 1, Mod 2 and Mod 3
                                     NCK (g/kW—hr) Method 1
    100
 en
 a.
 rt
 D
 Ł
 D
 O
     80-
     60
40-
     20-
      0
      —0.60    -O.45    -0,30    -0.15     0.00      0.15      0.30     0.45     0.60     0.75     0.90

                                          Delta NCK
  FIGURE 9.  VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                            NOX METHOD 1 MODS 1, 2 AND 3
SwRI Report 03.12859.06
                                       16 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
                                Error Models Mod 1, Mod  2 and Mod 3
                                      NCK (s^kW-hr) Method 2
    100
     80-
 en
 a.
 O
                                                                               N	23
                                                                               ^^^^^^^^^^^^^^m
                                                                               ^^^^^^^^^^^^^^m
                                                                               95ti % Mod 2 Delta
      -0.60    -Q.45    -0,30    -0.15     0.00      0.15     0.30      0.45     0.60     0.75      0.90
 FIGURE 10.  VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                          NOX METHOD 2 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
17 of 83

-------
                             Validation Analysis 5th and 95th Percentile Deltas
                     Compared 23 Ref NTE Events to Drift Corrected SI CE—CERT Deltas
                                 Error Models Mod 1, Mod Ł and  Mod 3
                                       NOx (g/kW-hr) Method 3
 E
 o
    100-
     80-
     60-
          51h % Mod 1 De 1a
          5th W Mod 2 Delta
          5th % Mod 3 Delta
                                                                                 951h M Mod 1 Delta
                                   N	23
                                   ^^^^^^^^^^^^
                                   ^^^^^^^^^^^^^^H
                                   95th W Mod 2 Delta
                                                                                 N	23
                                                                                 ^^^^^^^^^^^^^^M
                                                                                 ^^^^^^^^^^^^^^H
                                                                                 95th % Mod 3 Delta
      -0,60    -0.45    -0,30    -0.15     0.00     0.15      0.30      0,45     0.60     0,75     0.90
                                             Delta NCK g/kW—hr
4C"
     20 <
 FIGURE 11. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                           NOX METHOD 3 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
18 of 83

-------
 en
 a.
 rt
 D
 Ł
 D
 O
    100
     80-
     60
40-
     20-
      0
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                Error Models Mod 1, Mod 2 and Mod 3
                                     NMHC (g/kW—hr) Method 1
         |51h % Mod 1 Delta
      —0.04    -O.03    -0,02    -0,01     0.00     0.01     0.02     0,03     0,04     0,05     0,06

                                          Delta NMHC g/kW-hr


 FIGURE 12.  VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                        NMHC METHOD 1 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
                                        19 of 83

-------
                           Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                               Error Models Mod 1, Mod 2 and Mod 3
                                    NMHC (g/kW-hr) Method 2
    100
 en
 a.
     80-
     60
 rt
 u
 E   40-
 o
     20-
      —0.03
                              -0,01        0.00         0.01

                                         Delta NMHC g/kW-hr
                                                                  0,02
                                                                              0.03
                                                                                          0.04
 FIGURE 13. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                        NMHC METHOD 2 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
20 of 83

-------
                           Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                               Error Models Mod 1, Mod 2 and Mod 3
                                    NMHC (g/kW-hr) Method 3
    100
 en
 a.
     80-
     60
 rt
 u
 E   40-
 o
     20-
      —0.03
                              -0,01        0.00         0.01

                                         Delta NMHC g/kW-hr
                                                                  0,02
                                                                              0.03
                                                                                          0.04
 FIGURE 14. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                        NMHC METHOD 3 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
21 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                Error Models Mod 1, Mod 2 and Mod 3
                                      CO (gykW-hr) Method 1
    100
     80-
 en
 a.
 O
                                                                               9E>th % Mod 1 DeHa
                                                                                          23
                                                                                          ^m
                                                                                          ^m
                                                                              95th % Mod 2 Delta
      -0,60   —0,46    —0,30    -0.15    0.00     0.15    0.30    0.45     0.60    0.75    0.90     1,05
                                            Delta CO a/kW-hr
 FIGURE 15.  VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                           CO METHOD 1 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
22 of 83

-------
 en
 a.
 rt
 D
 Ł
 D
 O
    100
     80-
                             Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                 Error Models Mod 1, Mod 2 and Mod 3
                                       CO (g/RW—hr) Method 2
          5th % Mod 1 Delta
                                                                                9E>th % Mod 1 De ta
                                              23
                                              ^a
                                              ^H
                                  95th % Mod 2 Delta
      -0,60   —0,46   —0,30   -0.15    0.00     0.15     0.30    0.45     0.60     0.75     0.90     1,05
                                            Delta CO a/kW-hr
 FIGURE 16. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                              CO METHOD 2 MODS 1, 2 AND 3
SwRI Report 03.12859.06
23 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                Error Models Mod 1, Mod 2 and Mod 3
                                      CO (g/RW—hr) Method 3
    100
     80-
 en
 a.
 O
                                                                               9E>th % Mod 1 DeHa
                                                                                          23
                                                                                          ^a
                                                                                          ^H
                                                                               95th % Mod 2 Delta
      -0,60   —0,46   —0,30    -0.15    0.00     0.15    0.30    0.45     0.60     0.75     0.90    1,05
                                            Delta CO a/kW-hr
 FIGURE 17.  VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                           CO METHOD 3 FOR MODS 1, 2 AND 3
SwRI Report 03.12859.06
24 of 83

-------
    6.0    MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE
                                 EVENTS USING MOD B

       Since all three methods validated using the  23  reference NTE events by all  three
modification runs for BSNOX, BSNMHC and BSCO, the question arose  as to which of the
revised error surfaces had the most influence in the validation.   Three possible explanations
included (1) the steady-state CC>2 bias had been eliminated,  (2) the change in the steady-state
NOx error values, and (3) the bias in the steady-state exhaust flow rate had been eliminated.  To
study these possible explanations, none of the revised  SSNOX error surfaces or the SSCO error
surface was used in the simulations.  These error surfaces were set to their original format in the
original PEMS study.  Therefore, only the  steady-state exhaust flow rate and the steady-state
CC>2 error surfaces were revised and the  other remaining validation error surfaces were set to
their original definitions for running the 23 reference NTE events for the Mod B MC simulation.
This set of simulations is referred to as the 'Mod  B'  runs.  Again,  the simulations were
performed at 10,000 trials for NTE events with ideal BSNOx values less  than 2.68204 g/kW-hr
and at 30,00 trials otherwise. All 23 reference NTE events met the convergence criteria listed in
Table 5.

       From the  Mod B  MC  simulations,  the validation  5th and 95th percentile  delta  BS
emissions were extracted from the output files for each of the 23 reference NTE events.  These
delta emissions were then plotted as empirical distribution functions (EDF) to form a validation
interval for the on-road data.  Also plotted was the EDF computed from the on-road CE-CERT
NTE  events.  Figure  18 through  Figure  20  represent the validation plots  for the BSNOX for
calculation methods 1, 2 and 3,  respectively.  Each of the validation plots includes 5th and 95th
percentile EDFs for the Mod B runs and the on-road CE-CERT data. Figure 21 through Figure
26 depict the validation plots for the BSNMHC and BSCO simulation runs, respectively.  All
BSNOx, BSNMHC and BSCO emissions validated for all three calculation methods for the Mod
B runs.

       Conclusions made from the results of the Mod B analysis were that the steady-state NOx
error surface changes did not have as much of an effect on the validation as  the bias elimination
in the steady-state CO2 and steady-state exhaust flow rate error surfaces.
SwRI Report 03.12859.06                    25 of 83

-------
 c
 u
 m
 u
 I
 a
 t

 v
 e

 P
 e
 r
 c
 e
 n
 t
                           Validation Analysis 5th and 95th Percentile Deltas
                    Compared  23 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
                                       Error Model Mod B
                                     NCK (g/kW-hr) Method 1
    100
         5th % Mod B Delta
      -0.60    -0,45     -0.30    -0.15     0.00     0.15      0.30     0.45      0.60     0.75     0.90
                                          Delta NQx o/kW-hr
 FIGURE 18.  VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                                  NOX METHOD 1 MOD B
SwRI Report 03.12859.06
26 of 83

-------
 c
 u
 m
 u
 I
 a
 t
 i
 v
 e

 P
 e
 r
 c
 e
 n
 t
                           Validation Analysis 5th and 95th Percentile Deltas
                   Compared 23 Ref NTE Events to Drift Corrected SI CE—CERT Deltas
                                       Error Model Mod B
                                    NQx (g/kW-hr) Method 2
    100
      -0,60    -0.45    -0.30    -0.15     0.00
FIGURE 19. VALIDATION NTE EVENTS USING 23 REFERENCE NTE EVENTS FOR
                                 NOX METHOD 2 MOD B
SwRI Report 03.12859.06
27 of 83

-------
 c
 u
 m
 u
 I
 a
 t

 v
 e

 p
 e
 r
 c
 e
 n
 t
    100'
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected SI CE—CERT Deltas
                                        Error Model Mod B
                                      NOK (g/kW—hr) Method 3
          5th W Mod B Delta
                                  5th % Mod B Ddta
      -0.60    -0.45    -0.30    -0.15    0.00     0.15     0.30     0.45      0.60     0.75      0.90
                                           Delta NOx a/kW-hr
 FIGURE 20.  VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                                  NOX METHOD 3 MOD B
SwRI Report 03.12859.06
28 of 83

-------
 c
 u
 m
 u
 I
 a
 t
 i
 v
 e

 P
 e
 r
 c
 e
 n
 t
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
                                        Error Model Mod B
                                     NMHC (a/kW^nr) Method 1
    100
60
40'
    20-
     0
      -0.04    -0.03    -0.02    -0.01     0.00      0.01     0.02      0.03     0.04     0.05     0.06

                                          Delta NMHC g/kW-hr



 FIGURE 21. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                                 NMHC METHOD 1 MOD B
SwRI Report 03.12859.06
                                        29 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
                                        Error Model Mod B
                                    NMHC (g/kW-hr) Method 2
    100
C
u
m
u
I
a
t
i
v
e

P
e
r
c
e
n
t
    60
    40'
    20-
      -0.03
                  -0,02
                              -0,01        0.00         0.01

                                         Delta NMHC g/kW-hr
                                                                  0.02
                                                                              0,03
                                                                                          0.04
 FIGURE 22. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                                 NMHC METHOD 2 MOD B
SwRI Report 03.12859.06
                                           30 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
                                        Error Model Mod B
                                    NMHC (g/kW-hr) Method 3
    100
C
u
m
u
I
a
t
i
v
e

P
e
r
c
e
n
t
    60
    40'
    20-
      -0.03
                  -0,02
                              -0,01        0.00         0.01

                                         Delta NMHC g/kW-hr
                                                                  0.02
                                                                              0,03
                                                                                          0.04
 FIGURE 23. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                                 NMHC METHOD 3 MOD B
SwRI Report 03.12859.06
                                           31 of 83

-------
 c
 u
 m
 u
 I
 a
 t
 i
 v
 e

 P
 e
 r
 c
 e
 n
 t
    100
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                         Error Model Mod B
                                       CO (g/kW-hr) Method 1
          5th % Mod B Delta
                                  95th »i Mod B De ta
      -0,60   -0,45  -0.30   -0.15   0,00    0.15    0.30    0,45    0.60    0,75    0,90    1.05    1.20
                                            Delta CO g/kW-hr
 FIGURE 24.  VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                                    CO METHOD 1 MOD B
SwRI Report 03.12859.06
32 of 83

-------
 c
 u
 m
 u
 I
 a
 t
 i
 v
 e

 P
 e
 r
 c
 e
 n
 t
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                         Error Model Mod B
                                      CO (g/RW^nr) Method 2
    100
          5th % Mod B Delta
                                  95th »i Mod B De ta
      -0,60   -0,45  -0.30   -0.15   0,00    0.15    0.30    0,45    0.60    0,75    0,90    1.05    1.20
                                            Delta CO g/kW-hr
 FIGURE 25.  VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                                    CO METHOD 2 MOD B
SwRI Report 03.12859.06
33 of 83

-------
 c
 u
 m
 u
 I
 a
 t
 i
 v
 e

 P
 e
 r
 c
 e
 n
 t
    100
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                         Error Model Mod B
                                      CO (g/RW^nr) Method 3
          5th % Mod B Delta
                                  95th »i Mod B De ta
      -0,60   -0,45  -0,30   -0.15   0,00    0.15    0.30    0,45    0.60    0,75    0,90    1.05    1.20
                                            Delta CO g/kW-hr
 FIGURE 26.  VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR
                                    CO METHOD 3 MOD B
SwRI Report 03.12859.06
34 of 83

-------
    7.0    MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE
                                 EVENTS USING MOD 1

       Based on the information provided in the validation plots for the Mod 1, Mod 2, Mod 3
and Mod B  runs, changes made to the various steady-state error surfaces resulted in  the
validation of the MC model for all three emissions and all three calculation methods for each of
the four Mod runs using the 23 reference NTE events.  EPA chose to continue the simulation
runs  using only the  Mod 1 and Mod 2 revised error  surfaces  by running an additional 27
reference NTE  events (total =  50 reference NTE events)  through the MC model.  These two
Mods were chosen because they both represented steady-state NOx error surfaces that were level
independent (as compared to the Mod 3 steady-state error surface that represented a combination
of level dependent and level independent NOx  errors).  The results from these 50 simulations
were used to calculate the measurement allowances provided in Section 9.0 of this report. This
section details the results of the validation based on the 50 reference NTE events run with Mod
1.

       The Mod 1  simulations were performed at  10,000 trials for NTE events with ideal
BSNOx values less than 2.68204 g/kW-hr and at 30,00 trials otherwise.  All 50 reference NTE
events met the convergence criteria listed  in Table 5.

       From the Mod 1 MC simulations, the validation 5th and 95th percentile delta BS emissions
were extracted from the output  files for each of the 50 reference NTE events.  These percentiles
were then plotted as empirical distribution functions (EDF) to form a validation interval for the
on-road data.   Also plotted was the EDF computed  from the on-road CE-CERT NTE events.
Figure 27 through Figure 29 represent the validation plots for the BSNOX for calculation methods
1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th percentile EDFs for the
Mod 1 runs and the on-road CE-CERT data.  Figure 30 through Figure 35 depict the validation
plots for the BSNMHC and BSCO simulation runs, respectively.  All BSNOx, BSNMHC and
BSCO emissions validated  for all three  calculation methods  for the Mod 1  runs with the 50
reference NTE events.
SwRIReport03.12859.06                     35 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
                                        Error Model Mod 1
                                      NCK (g/kW—hr) Method 1
    100'
    80
    60 <
    40'
    20
C
u
m
u
I
a
t

v
e

p
e
r
c
e
n
t
     0
      -0.60    -0.45     -0.30    -0.15     0.00      0.15      0.30     0.45     0.60     0.75      0.90

                                          Delta NQx s/kW-hr


 FIGURE 27. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                  NOX METHOD 1 MOD 1
SwRI Report 03.12859.06
                                            36 of 83

-------
 c
 u
 m
 u
 I
 a
 t
 i
 v
 e

 P
 e
 r
 c
 e
 n
 t
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected SI CE—CERT Deltas
                                        Error Model Mod 1
                                      NQx (g/kW-hr) Method 2
    100
          5th % Mod B Delta
      -0,60    -0.45    -0.30     -0.15    0.00     0.15     0.30     0.45      0.60     0.75      0.90
                                           Delta NQx g/kW—hr
 FIGURE 28. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                   NOX METHOD 2 MOD 1
SwRI Report 03.12859.06
37 of 83

-------
 c
 u
 m
 u
 I
 a
 t
 i
 v
 e

 P
 e
 r
 c
 e
 n
 t
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
                                        Error Model Mod 1
                                      NQx (g/kW-hr) Method 3
    100
          5th % Mod B Delta
      -0,60    -0.45    -0.30     -0.15    0.00     0.15     0.30     0.45      0.60     0.75      0.90
                                           Delta NQx g/kW—hr
 FIGURE 29. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                   NOX METHOD 3 MOD 1
SwRI Report 03.12859.06
38 of 83

-------
 c
 u
 m
 u
 I
 a
 t
 i
 v
 e

 P
 e
 r
 c
 e
 n
 t
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
                                        Error Model Mod 1
                                     NMHC (a/kW^nr) Method 1
    100
60
40'
    20-
     0
      -0.04    -0.03    -0.02    -0.01     0.00      0.01     0.02      0.03     0.04     0.05     0.06

                                          Delta NMHC g/kW-hr



 FIGURE 30. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                 NMHC METHOD 1 MOD 1
SwRI Report 03.12859.06
                                        39 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                        Error Model Mod 1
                                    NMHC (g/kW-hr) Method 2
    100
C
u
m
u
I
a
t
i
v
e

P
e
r
c
e
n
t
    60
    40'
    20-
         15th M Mud B Delta I
      -0.03
                  -0,02
                              -0,01         0.00         0.01

                                          Delta NMHC g/kW-hr
                                                                   0.02
                                                                               0,03
                                                                                           0.04
 FIGURE 31. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                 NMHC METHOD 2 MOD 1
SwRI Report 03.12859.06
                                            40 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                        Error Model Mod 1
                                    NMHC (g/kW-hr) Method 3
    100
C
u
m
u
I
a
t
i
v
e

P
e
r
c
e
n
t
    60
    40'
    20-
      -0.03
                  -0,02
                              -0,01        0.00         0.01

                                         Delta NMHC g/kW-hr
                                                                  0.02
                                                                              0,03
                                                                                          0.04
 FIGURE 32. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                 NMHC METHOD 3 MOD 1
SwRI Report 03.12859.06
                                           41 of 83

-------
 c
 u
 m
 u
 I
 a
 t
 i
 v
 e

 P
 e
 r
 c
 e
 n
 t
    100
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
                                         Error Model Mod 1
                                       CO (g/kW-hr) Method 1
          5th % Mod B Delta
                                  95th % Mod 1 De 1
      -0.60    -0.45    -0.30    -0.15    0.00     0.15     0.30    0.45    0.60    0.75    0.90     1.05
                                            Delta CO Q/kW-hr
 FIGURE 33.  VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                    CO METHOD 1 MOD 1
SwRI Report 03.12859.06
42 of 83

-------
 c
 u
 m
 u
 I
 a
 t
 i
 v
 e

 P
 e
 r
 c
 e
 n
 t
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
                                         Error Model Mod 1
                                       CO (g/RW^nr) Method 2
    100
          5th % Mod B Delta
                                  95th % Mod 1 De 1
      -0.60    -0.45    -0.30    -0.15    0.00     0.15     0.30     0.45    0.60    0.75    0.90     1.05
                                            Delta CO Q/kW-hr
 FIGURE 34.  VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                    CO METHOD 2 MOD 1
SwRI Report 03.12859.06
43 of 83

-------
 c
 u
 m
 u
 I
 a
 t
 i
 v
 e

 P
 e
 r
 c
 e
 n
 t
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 87 CE—CERT Deltas
                                         Error Model Mod 1
                                       CO (g/RW^nr) Method 3
    100
          5th % Mod B Delta
                                  95th % Mod 1 De 1
      -0.60    -0.45    -0.30    -0.15    0.00     0.15     0.30     0.45    0.60    0.75    0.90     1.05
                                            Delta CO Q/kW-hr
 FIGURE 35.  VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                    CO METHOD 3 MOD 1
SwRI Report 03.12859.06
44 of 83

-------
    8.0    MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE
                                 EVENTS USING MOD 2

       This section details the results of the validation based on the 50 reference NTE events run
with Mod 2.   EPA  chose to continue simulations using the Mod  1  and Mod 2 revised error
surfaces by running an additional 27  reference NTE events (total = 50 reference NTE events)
through the MC model.   These two Mods were  chosen because  they both represented steady-
state NOx error surfaces  that were level independent (as compared to the Mod 3 steady-state
error surface  that represented a combination of level  dependent and level independent NOX
errors).

       The Mod 2  simulations were performed at 10,000 trials for NTE events with ideal
BSNOx values less than 2.68204 g/kW-hr and at 30,00 trials otherwise. All 50 reference NTE
events met the convergence criteria listed in Table 5.

       From the Mod 2 MC simulations, the validation 5th and 95th percentile delta BS emissions
were extracted from the output files for each of the 50 reference NTE events.  These percentile
delta emissions were then plotted as empirical distribution functions (EDF) to form a validation
interval for the on-road data. Also plotted was the EDF computed from the on-road CE-CERT
NTE events.   Figure 36  through Figure 38 represent the validation plots for the BSNOX for
calculation  methods 1, 2 and 3, respectively.  Each of the validation plots includes 5th and 95th
percentile EDFs for  the Mod 2 runs and the on-road CE-CERT data.  Figure 39 through Figure
44 depict the validation plots for the  BSNMHC  and BSCO simulation runs,  respectively. All
BSNOx, BSNMHC and BSCO emissions validated for all three calculation methods for the Mod
2 runs with the 50 reference NTE events.
SwRI Report 03.12859.06                    45 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
                                        Error Model Mod 2
                                      NCK (g/kW—hr) Method 1
    100'
    80
    60 <
    40'
    20
C
u
m
u
I
a
t

v
e

p
e
r
c
e
n
t
                                                                              95ih % Mod 2 Delta I
     0
      -0.60    -0.45    -0.30    -0.15     0.00      0.15     0.30     0.45     0.60     0.75     0.90

                                           Delta NQx s/kW-hr


 FIGURE 36. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                   NOX METHOD 1 MOD 2
SwRI Report 03.12859.06
                                            46 of 83

-------
 c
 u
 m
 u
 I
 a
 t

 v
 e

 p
 e
 r
 c
 e
 n
 t
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
                                        Error Model Mod 2
                                      NOK (g/kW—hr) Method 2
    100'
          5th W Mod 2 Delta
                                  99th % Mod 2 Delta
      -0.60    -0.45    -0.30     -0.15    0.00     0.15      0.30     0.45      0.60     0.75     0.90
                                           Delta NOx a/kW-hr
 FIGURE 37.  VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                   NOX METHOD 2 MOD 2
SwRI Report 03.12859.06
47 of 83

-------
 c
 u
 m
 u
 I
 a
 t

 v
 e

 p
 e
 r
 c
 e
 n
 t
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 81 CE—CERT Deltas
                                        Error Model Mod 2
                                      NOK (g/kW—hr) Method 3
    100'
          5th W Mod 2 Delta
                                  99th % Mod 2 Delta
      -0.60    -0.45    -0.30     -0.15     0.00     0.15      0.30     0.45     0.60      0.75     0.90
                                           Delta NOx a/kW-hr
  FIGURE 38. VALIDATION EDF PLOTS FOR 50 REFERENCE NTE EVENTS FOR
                                   NOX METHOD 3 MOD 2
SwRI Report 03.12859.06
48 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                        Error Model Mod 2
                                    NMHC (g/KW-hr) Method 1
    100'
    80
    60 <
    40'
    20
C
u
m
u
I
a
t

v
e

p
e
r
c
e
n
t
     0
      -0.04    -0.03   -0.02    -0.01     0.00     0.01      0.02     0.03     0.04     0.05     0.06

                                         Delta NMHC gykW-hr


  FIGURE 39.  VALIDATION EDF PLOTS FOR 50 REFERENCE NTE EVENTS FOR
                                 NMHC METHOD 1 MOD 2
SwRI Report 03.12859.06
                                            49 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                        Error Model Mod 2
                                    NMHC (g/kW-hr) Method 2
    100'
    80
    60 <
    40'
    20
C
u
m
u
I
a
t

v
e

p
e
r
c
e
n
t
     0
      -0.04    -0.03    -0.02    -0.01     0.00      0.01      0.02     0.03      0.04     0.05      0.06

                                         Delta NMHC gykW-hr


  FIGURE 40.  VALIDATION EDF PLOTS FOR 50 REFERENCE NTE EVENTS FOR
                                 NMHC METHOD 2 MOD 2
SwRI Report 03.12859.06
                                            50 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 50 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                        Error Model Mod 2
                                    NMHC (g/kW-hr) Method 3
    100'
    80
    60 <
    40'
    20
C
u
m
u
I
a
t

v
e

p
e
r
c
e
n
t
         15th M Mod 2 Delta I
      -0.03
                  -0,02
                              -0,01        0.00         0.01

                                         Delta NMHC gykW-hr
                                                                  0.02
                                                                              0.03
                                                                                          0,04
 FIGURE 41. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                 NMHC METHOD 3 MOD 2
SwRI Report 03.12859.06
                                            51 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                        Error Model Mod 2
                                      CO (g/kW-hr) Method 1
 C
 u
 m
 u
 I
 a
 t

 v
 e

 p
 e
 r
 c
 e
 n
 t
    100'
    80
60
          5th W Mod 2 Delta
      -0.60   -0.45   -0.30   -0.15    0.00     0.15     0.30     0.45    0.60     0.75     0.90     1.05
                                           Delta CO g/kW-hr
 FIGURE 42.  VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                   CO METHOD 1 MOD 2
SwRI Report 03.12859.06
                                        52 of 83

-------
                            Validation Analysis 5th and 95th Percentile Deltas
                    Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                        Error Model Mod 2
                                      CO (g/RW—hr) Method 2
 c
 u
 m
 u
 I
 a
 t

 v
 e

 p
 e
 r
 c
 e
 n
 t
    100'
    80
60
          5th W Mod 2 Delta
      -0.60    -Ci.45   -O.30   -0.15   0,00    0.15     0,30     0,45     0,60    0.75    0.90    1,05
                                           Delta CO a/kW-hr
 FIGURE 43.  VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                    CO METHOD 2 MOD 2
SwRI Report 03.12859.06
                                        53 of 83

-------
                           Validation Analysis 5th and 95th Percentile Deltas
                   Compared 23 Ref NTE Events to Drift Corrected 87 CE-CERT Deltas
                                      Error Model Mod 2
                                    CO (g/RW—hr) Method 3
 c
 u
 m
 u
 I
 a
 t

 v
 e

 p
 e
 r
 c
 e
 n
 t
    100'
    80
60
         5th W Mod 2 Delta
      -0.60   -Ci.45   -O.30    -0.15   0,00     0.15    0,30    0,45     0,60    0.75    0.90     1,05
                                         Delta CO a/kW-hr
 FIGURE 44. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR
                                  CO METHOD 3 MOD 2
       Table 7 provides a summary of all the validation results for the various MC simulations
as described in the above sections of this report.   All  emissions and all  calculation methods
validated using the Mod 1 and Mod 2 runs with 50 reference NTE events while the Mod 3  and
Mod B runs validated using 23 reference NTE events.
SwRI Report 03.12859.06
                                      54 of 83

-------
               TABLE 7.  SUMMARY OF VALIDATION RESULTS

BSNOx
Method 1
BSNOx
Method 2
BSNOx
Method 3
BSNMHC
Method 1
BSNMHC
Method 2
BSNMHC
Method 3
BSCO
Method 1
BSCO
Method 2
BSCO
Method 3

Validated


Validated
Validated
Validated




Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated

Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated

Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated

Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
Validated
SwRI Report 03.12859.06
55 of 83

-------
              9.0    MEASUREMENT ALLOWANCE CALCULATIONS

       As detailed  in  the  original  PEMS  study,  the  measurement  error allowances were
computed using both a regression method and a median method to determine the measurement
allowance.  The procedure was applied to the simulation data for the 50 reference NTE events
from the Mod 1  and Mod  2 runs for each of the three emissions and all three calculation
methods.

       Figure 45  contains a regression plot of the 95th percentile delta BSNOX values (using
Method 1 Mod 1) versus the ideal BSNOX values for the 50 reference NTE events.  Included
within the plot is the equation for the fitted regression line and the R-square (R2) value.  Table 8
includes a comparison of the results of the regression method based on Figure 45 and the median
method (described in Ref 1). Under the heading "Regression Method"  in the table, it is shown
that the R-square criterion is not met by the data (R-square must be > 0.90). Thus, the median
method must be used.  Under the heading "Median Method" in the table, the measurement error
at the BSNOX threshold,  based on using the median of the fifty 95th percentile delta BSNOX
values, is 11.5932% when expressed as a percent of the threshold value of 2.68204 g/kW-hr.

       Similar regression plots and measurement error tables are provided in the remaining part
of this section for the Mod 1  and Mod 2 results based on the 50 reference NTE events. Figure 45
through Figure 47 and Table 8 through Table 10 provide results for BSNOX Mod 1.  Figure 48
through Figure 50 and  Table 11 through Table 13 provide results for BSNMHC Mod 1,  and
Figure  51 through Figure 53 and Table 14 through Table 16 provide results for BSCO Mod 1.
Mod 2 results for BSNOX can be found in Figure 54 through Figure 56 and Table 17 through
Table 19, BSNMHC results are shown in Figure 57 through Figure 59 and Table 20 through
Table 22 and BSCO results are shown in Figure 60 through Figure 62 and Table 23 through
Table 25.
SwRI Report 03.12859.06                    56 of 83

-------
NOx g/kW-hr Method 1 Mod 1
0.90 -,
0.80
0.70
.c
| 0.60
x „ -,.
O 0.50
z
TO
o 0.40
Q
^ 0.30
u>
OT 0.20 -
0.10
0.00
c
50 Ref NTE Events With Time Alignment Adjustment
* ^
» ,S
* *^ 4
** ^^ *
* 
-------
                           NOx g/kW-hr Method 2  Mod 1
                  50 Ref NTE Events With Time Alignment Adjustment
           0.50

           0.45

           0.40

           0.35

           0.30

           0.25

           0.20

           0.15

           0.10

           0.05

           0.00
                   y = 0.0558x + 0.0893
                      R2 = 0.9058
                                   345

                                    Ideal NOx g/kW-hr
 FIGURE 46. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS
                     IDEAL BSNOx FOR METHOD 2 MOD 1
     TABLE 9.  MEASUREMENT ERROR AT THRESHOLD FOR BSNOx USING
         REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 1

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=2.68204
0.9058
0.0276
0.1314
0.2390

Met
Criteria
Met
Criteria





Median 95th% Delta
Measurement Error @
Threshold=2.68204
0.2270
8.4641%
SwRI Report 03.12859.06
58 of 83

-------
                           NOx g/kW-hr Method 3 Mod 1
                  50 Ref NTE Events With Time Alignment Adjustment
           0.70
           0.60
                                                  y = 0.0869X + 0.0533
                                                     R2 = 0.8714
           0.00
                           2345
                                   Ideal NOx g/kW-hr
 FIGURE 47.  REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS
                    IDEAL BSNOx FOR METHOD 3 MOD 1
    TABLE 10.  MEASUREMENT ERROR AT THRESHOLD FOR BSNOx USING
         REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 1

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=2.68204
0.8714
0.0512
0.1314
0.3000
11.1845%
Did Not
Meet
Criteria
Met
Criteria





Median 95th% Delta
Measurement Error @
Threshold=2.68204
0.2572

SwRI Report 03.12859.06
59 of 83

-------
n ny
n DR
.c n nc;
1
3)
O o 04
ro
7ii n m
Q
55 0 02 1
" !
0 01
0 00
0.
NMHC g/kW-hr Method 1 Mod 1
50 Ref NTE Events
•
^^
^^
^^ 	 y = 0.0955X + 0.01 74
\» 	 R2 = 0.8109
V^' '
k •'

00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.
Ideal NMHCg/kW-hr








45
    FIGURE 48. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC
              VERSUS IDEAL BSNMHC FOR METHOD 1 MOD 1
   TABLE 11. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING
        REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 1

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=0.28161
0.8109
0.0037
0.0002
0.0443
15.7288%
Did Not
Meet
Criteria
Did Not
Meet
Criteria





Median 95th% Delta
Measurement Error @
Threshold=0.28161
0.0197

SwRI Report 03.12859.06
60 of 83

-------
k.
.C
i
B>
o
2
z
ro
±i
01
Q
S?
Ł
55
O>
NMHC g/kW-hr Method 2 Mod 1
50 Ref NTE Events
0 045
n D4n
n mR
n mn
n n9ci
0.020 j
0.015 I
n mn
n rin^
n nnn


^^^^
^^^^
| * 	 «r 	 y = 0.0382X + 0.01 74
| 	 ' 	 ""* 4 R2 = 0.4768
Ł*— "**^~ *
» *s*
Sr



0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45
Ideal NMHCg/kW-hr
    FIGURE 49. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC
              VERSUS IDEAL BSNMHC FOR METHOD 2 MOD 1
   TABLE 12. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING
        REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 1

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=0.28161
0.4768
0.0032
0.0002
0.0282
9.9988%
Did Not
Meet
Criteria
Did Not
Meet
Criteria





Median 95th% Delta
Measurement Error @
Threshold=0.28161
0.0187

SwRI Report 03.12859.06
61 of 83

-------
           0.050
o
s
z
ra
1
           0.045 -
           0.040 -
           0.035 -
           0.030 -
           0.025
           0.020
         Ł 0.015
           0.010
           0.005 -
           0.000
                            NMHC g/kW-hr Method 3 Mod 1
                                  50 Ref NTE Events
                                                        y =0.0545x +0.0171
                                                           R2 = 0.6202
              0.00    0.05    0.10   0.15    0.20    0.25    0.30
                                     Ideal NMHCg/kW-hr
                                                  0.35
0.40
0.45
    FIGURE 50.  REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC
                 VERSUS IDEAL BSNMHC FOR METHOD 3 MOD 1
   TABLE 13. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING
         REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 1

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=0.28161
0.6202
0.0034
0.0002
0.0324
11.5222%
Did Not
Meet
Criteria
Did Not
Meet
Criteria





Median 95th% Delta
Measurement Error @
Threshold=0.28161
0.0188

SwRI Report 03.12859.06
                             62 of 83

-------
                           CO g/kW-hr Method 1 Mod 1
                 50 Ref NTE Events  With Time Alignment Adjustment
           1.20
                                                     y=0.1112x+0.3148
                                                        R2 = 0.6518
           0.00
                                     3       4

                                   Ideal CO g/kW-hr
  FIGURE 51. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS
                     IDEAL BSCO FOR METHOD 1 MOD 1
     TABLE 14. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING
         REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 1

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=26.015
0.6518
0.0846
0.0249
0.9814
3.7726%
Did Not
Meet
Criteria
Did Not
Meet
Criteria





Median 95th% Delta
Measurement Error @
Threshold=26.015
0.3600

SwRI Report 03.12859.06
63 of 83

-------
                           CO g/kW-hr Method 2 Mod 1
                  50 Ref NTE Events  With Time Alignment Adjustment
          0.90
                                                       y = 0.0543X + 0.3211
                                                          R2 = 0.3471
                                     3       4
                                   Ideal CO g/kW-hr
  FIGURE 52. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS
                     IDEAL BSCO FOR METHOD 2 MOD 1
     TABLE 15. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING
         REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 1

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=26.015
0.3471
0.0775
0.0249
0.6466
2.4856%
Did Not
Meet
Criteria
Did Not
Meet
Criteria





Median 95th% Delta
Measurement Error @
Threshold=26.015
0.3451

SwRI Report 03.12859.06
64 of 83

-------
1 10 -i
1 00
0 90 -
*r n fin
^
B>
O 0 70 -
o
ro
±i
a) n fin
S °-60 ;
s?
Ł 0 50 -
55 U'DU
o>
<
n 4n
n ^n
n 9n
C
CO g/kW-hr Method 3 Mod 1
50 Ref NTE Events With Time Alignment Adjustment


* .s^
^^
	 y = 0.0988X + 0.31 1
	 R2 = 0.5987
» ^r
** * tx^^^
"v ix<; *
X *
^*i%*
4r**' »
) 1 234567
Ideal CO g/kW-hr










7
 FIGURE 53. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS
                   IDEAL BSCO FOR METHOD 3 MOD 1
    TABLE 16. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING
        REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 1

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=26.015
0.5987
0.0846
0.0249
0.9033
3.4722%
Did Not
Meet
Criteria
Did Not
Meet
Criteria





Median 95th% Delta
Measurement Error @
Threshold=26.015
0.3621

SwRI Report 03.12859.06
65 of 83

-------
                           NOx g/kW-hr Method 1  Mod 2
                  50 Ref NTE Events  With Time Alignment Adjustment
           0.90
           0.80 -

           0.70 -

           0.60 -
o)  0.50
x
O
^  0.40 -]
           0.3
         >o
         a>
           0.10 -
           0.00
           -0.10
                                 *4
                                                y =0.1 272x- 0.0086
                                                   R2 = 0.8638
                                    Ideal NOx g/kW-hr
 FIGURE 54. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS
                     IDEAL BSNOx FOR METHOD 1 MOD 2
    TABLE 17. MEASUREMENT ERROR AT THRESHOLD FOR BSNOX USING
         REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 2

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=2.68204
0.8638
0.0774
0.1314
0.3326
12.3993%
Did Not
Meet
Criteria
Met
Criteria





Median 95th% Delta
Measurement Error @
Threshold=2.68204
0.2834

SwRI Report 03.12859.06
                           66 of 83

-------
           0.50
           0.45
           0.40 -
           0.35
         O)
         X
         O
           0.30
           0.25
           0.05 -
           0.00
                           NOx g/kW-hr Method 2 Mod 2
                  50 Ref NTE Events  With Time Alignment Adjustment
                y =0.0614x+0.0375
                   R2 = 0.9543
                                   345

                                    Ideal NOx g/kW-hr
 FIGURE 55. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS
                     IDEAL BSNOx FOR METHOD 2 MOD 2
    TABLE 18. MEASUREMENT ERROR AT THRESHOLD FOR BSNOx USING
         REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 2

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=2.68204
0.9543
0.0206
0.1314
0.2022

Met
Criteria
Met
Criteria





Median 95th% Delta
Measurement Error @
Threshold=2.68204
0.1935
7.2141%
SwRI Report 03.12859.06
67 of 83

-------
           0.70
           0.60
           0.50 -
           0.40 -
           0.30
           0.20 -
           0.10 -
           0.00
                           NOx g/kW-hr Method 3 Mod 2
                  50 Ref NTE Events  With Time Alignment Adjustment
                                                    y = 0.0932X + 0.0056
                                                       R2 = 0.8998
                                   345

                                    Ideal NOx g/kW-hr
 FIGURE 56. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS
                     IDEAL BSNOx FOR METHOD 3 MOD 2
    TABLE 19. MEASUREMENT ERROR AT THRESHOLD FOR BSNOx USING
         REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 2

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=2.68204
0.8998
0.0477
0.1314
0.2556
9.5288%
Did Not
Meet
Criteria
Met
Criteria





Median 95th% Delta
Measurement Error @
Threshold=2.68204
0.2295

SwRI Report 03.12859.06
68 of 83

-------
n ny
0 06
.c n nc;
1
3)
O 0 04 -
ro
? 0 03
Q
55 0 02
0 01
0 00 -
0.
NMHC g/kW-hr Method 1 Mod 2
50 ref NTE Events
•
^^
^^
+ ^ y =0.0951x+0.0173
^ R2 = 0.816
k!>^r
"^^ . 4»

00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.
Ideal NMHCg/kW-hr








45
    FIGURE 57. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC
              VERSUS IDEAL BSNMHC FOR METHOD 1 MOD 2
   TABLE 20. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING
        REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 2

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=0.28161
0.8160
0.0036
0.0002
0.0441
15.6533%
Did Not
Meet
Criteria
Did Not
Meet
Criteria





Median 95th% Delta 0.0194
Measurement Error @
Threshold=0.28161
SwRI Report 03.12859.06
69 of 83

-------
k.
.C
i
B>
o
2
z
ro
±i
01
Q
S?
Ł
55
O>
NMHC g/kW-hr Method 2 Mod 2
50 Ref NTE Events
0 045
n n/in
n rn^
n n^n
n ro*>
0.020
n ni^ !
0.010 -
n nn^
n nnn


^^^
^^^^
+ + ^ 	 "* y =0.0386x+0.0172
i * 	 J R2 = 0.479
1**~~^^
*f^ 4 •
X. * *
^



0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45
Ideal NMHCg/kW-hr
    FIGURE 58. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC
              VERSUS IDEAL BSNMHC FOR METHOD 2 MOD 2
   TABLE 21. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING
        REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 2

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=0.28161
0.4790
0.0032
0.0002
0.0281
12.3993%
Did Not
Meet
Criteria
Did Not
Meet
Criteria





Median 95th% Delta
Measurement Error @
Threshold=0.28161
0.0182

SwRI Report 03.12859.06
70 of 83

-------
           0.040
           0.035 -
           0.030 -
         o
           0.025 -
           0.020
           0.015
           0.010
                            NMHC g/kW-hr Method 3 Mod 2
                                 50 Ref NTE Events
               y = 0.054x+0.017
                 R2 = 0.6195
              0.00    0.05    0.10    0.15    0.20   0.25   0.30   0.35   0.40    0.45

                                     Ideal NMHCg/kW-hr
    FIGURE 59. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC
                VERSUS IDEAL BSNMHC FOR METHOD 3 MOD 2
   TABLE 22. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING
         REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 2

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=0.28161
0.6195
0.0034
0.0002
0.0322
11.4367%
Did Not
Meet
Criteria
Did Not
Meet
Criteria





Median 95th% Delta 0.0182
Measurement Error @
Threshold=0.28161
SwRI Report 03.12859.06
71 of 83

-------
                           CO g/kW-hr Method 1  Mod 2
                  50 Ref NTE Events With Time Alignment Adjustment
           1.20
                                                y =0.1156x+0.3087
                                                   R2 = 0.6865
           0.00
                                     3       4

                                    Ideal CO g/kW-hr
  FIGURE 60. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS
                     IDEAL BSCO FOR METHOD 1 MOD 2
     TABLE 23. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING
         REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 2

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=26.015
0.6865
0.0813
0.0249
1.0017
3.8505%
Did Not
Meet
Criteria
Did Not
Meet
Criteria





Median 95th% Delta
Measurement Error @
Threshold=26.015
0.3576

SwRI Report 03.12859.06
72 of 83

-------
           0.90
           0.80 -
           0.70 -
         O
         o
         ra
         ±i
         01
         O
0.60 -
         Ł 0.50
           0.40
           0.30
                           CO g/kW-hr Method 2 Mod 2

                  50 Ref NTE Events With Time Alignment Adjustment
                                     34


                                    Ideal CO g/kW-hr
  FIGURE 61. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS

                     IDEAL BSCO FOR METHOD 2 MOD 2
     TABLE 24. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING

         REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 2

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=26.015
0.3785
0.0756
0.0249
0.6561
2.5221%
Did Not
Meet
Criteria
Did Not
Meet
Criteria





Median 95th% Delta
Measurement Error @
Threshold=26.015
0.3361

SwRI Report 03.12859.06
                        73 of 83

-------
                          CO g/kW-hr Method 3 Mod 2
                 50 Ref NTE Events With Time Alignment Adjustment
          1.10
          0.:
                                    34

                                   Ideal CO g/kW-hr
 FIGURE 62. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS
                    IDEAL BSCO FOR METHOD 3 MOD 2
    TABLE 25. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING
        REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 2

R2
RMSE(SEE)
5% Median Ideal
Predicted 95th% Delta at
Threshold
Measurement Error @
Threshold=26.015
0.6195
0.0827
0.0249
0.9114
3.5033%
Did Not
Meet
Criteria
Did Not
Meet
Criteria





Median 95th% Delta
Measurement Error @
Threshold=26.015
0.3513

SwRI Report 03.12859.06
74 of 83

-------
       Although only 23 reference NTE events were simulated under the Mod 3 and Mod B
runs, the measurement errors were computed for all three emissions and calculation methods
using the same regression and median methods.  Table 26 lists a summary of the measurement
errors computed from simulation results in this study along with the results from the original
PEMS  study.  Note that BSNOX for methods 2  and 3  and BSCO for all three methods did not
validate in the original study.

      TABLE 26. SUMMARY OF MEASUREMENT ERRORS AT RESPECTIVE
                                  THRESHOLD (%)

BSNOx
BSNMHC
BSCO

195
50
50
23
23
195
50
50
23
23
195
50
50
23
23

Original
Mod 1
Mod 2
Mod 3
ModB
Original
Mod 1
Mod 2
Mod 3
ModB
Original
Mod 1
Mod 2
Mod 3
ModB


22.30
11.59
10.56
11.78
11.93
10.08
7.00
6.88
7.28
7.25
2.58*
1.38
1.37
1.52
2.19

4.45*
8.91
7.54
8.01
8.26
8.03
6.63
6.45
6.90
6.82
1.99*
1.33
1.29
1.41
1.99

6.61*
9.59
8.56
10.78
10.33
8.44
6.68
6.48
6.87
6.78
2.11*
1.39
1.35
1.48
2.16
 Methods did not validate

       The final BS measurement allowances were computed by multiplying each of the three
measurement  errors (in percent) times their corresponding threshold values. Table 27 lists the
measurement  allowances for all simulation Mod runs  and the original  PEMS study.   The
emissions thresholds (g/hp-hr) are provided in the first column of Table 27.
SwRI Report 03.12859.06
75 of 83

-------
       TABLE 27.  SUMMARY OF MEASUREMENT ALLOWANCE, G/HP-HR

BSNOx
Threshold=2.0
BSNMHC
Threshold=0.21
BSCO
Threshold=19.40

195
50
50
23
23
195
50
50
23
23
195
50
50
23
23

Original
Mod 1
Mod 2
Mod 3
ModB
Original
Mod 1
Mod 2
Mod 3
ModB
Original
Mod 1
Mod 2
Mod 3
ModB


0.446
0.232
0.211
0.236
0.239
0.021
0.015
0.014
0.015
0.015
0.501*
0.268
0.266
0.295
0.425

0.089*
0.178
0.151
0.160
0.165
0.017
0.014
0.014
0.014
0.014
0.386*
0.258
0.250
0.274
0.386

0.132*
0.192
0.171
0.216
0.207
0.018
0.014
0.014
0.014
0.014
0.409*
0.270
0.262
0.287
0.419
 Methods did not validate
SwRI Report 03.12859.06
76 of 83

-------
 10.0   VALIDATION SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS

       Simulation results  for Mod 1, Mod 2, Mod 3 and Mod B from the 13 reference NTE
events chosen in the original study produced sensitivity values from the validation runs due to
variance  and bias for all three 95th percentile delta emissions by all three calculation methods.
The tables below summarize the  error surfaces in  which either the  contribution-to-variance
normalized sensitivity average value  or the 'on/off bias check for the error surface was at least
5% in absolute magnitude compared to all the  other error surfaces.  If the label in the error
surface contains the words 'OnOff  then it represents a  check for bias; otherwise, the error
surface indicates a check for variance. If at least 5 of the 13 reference NTE events resulted in a
sensitivity average value > 5% or < -5% then the average contribution to variance is included in
the tables.

       Table 28 lists the BSNOx sensitivity error surfaces associated with the original PEMS
study  for Mod B, Mod 1, Mod 2 and the Mod 3 Monte Carlo validation simulations performed
during this project.  Note in the output results that the bias due to  steady-state exhaust flow rate
in Method  1  in the original study has been eliminated in the four modification simulations.  In
addition, the steady-state  CC>2 bias  has been eliminated,  but a sensitivity component  due to
variance  has been added for Methods  2 and 3.

       Table 29 lists the sensitivities due to BSNMHC.  Note that no changes have occurred for
the four modifications relative to the  original study.  This is due to the fact that the revised error
surfaces  did not effect the NMHC  calculations.  Table  30  summarizes the results  of the BSCO
sensitivities wherein the steady-state CO bias found in the original modeling has been eliminated
in Mod 1, Mod 2 and Mod 3 and replaced by an increased sensitivity due to variance.
SwRI Report 03.12859.06                     77 of 83

-------
 TABLE 28. SENSITIVITY RESULTS COMPARING 13 REFERENCE NTE EVENTS
      ACROSS 5 MONTE CARLO VALIDATION SIMULATIONS FOR BSNOX

1

1
20
51



SSNOx
SS Exhaust Flow
NOx Time
Alignment
Pulse Flow OnOff
SS Exhaust
Flow OnOff

Original
29.0
9.6
9.0
25.8
20.4

33.2
19.2
10.7
30.5


42.9
16.6
9.1
26.3


32.0
20.4
10.9
33.1


36.3
18.8
10.6
29.1


2
1
2
45
51

SSNOx
TRNOx
SSCO2
NOx Time
Alignment
SSCO2 OnOff
37.0



-54.3
46.9
5.5
-40.1
6.7

57.8

-33.7
7.5

44.6
6.6
-43.6
7.6

52.5

-37.7
6.2


3
1
45
51

SSNOx
SSCO2
NOx Time
Alignment
SSCO2 OnOff
35.6

10.4
-50.9
44.0
-37.0
13.9

54.9
-31.3
11.5

41.6
-39.6
15.8

49.4
-34.8
12.9


DEFINITIONS
Original April 2007 Study: All original error surfaces
Mod B Revised SS Exhaust Flow and SS CO2 error surfaces
, , , , Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error
Mod 1 ,,
surfaces
Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error
surfaces
, , , _ Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error
Mod 3 „
surfaces
SwRI Report 03.12859.06
78 of 83

-------
 TABLE 29. SENSITIVITY RESULTS COMPARING 13 REFERENCE NTE EVENTS
    ACROSS 5 MONTE CARLO VALIDATION SIMULATIONS FOR BSNMHC

1

13
19


SSNMHC
NMHC Ambient
SSNMHC OnOff

Original
7.1
60.9
14.5

7.1
62.8
14.3

7.0
63.2
13.5

6.9
62.5
12.9

6.6
63.2
14.2

2
13
19

SSNMHC
NMHC Ambient
SSNMHC OnOff
6.7
64.5
14.3
6.8
66.6
12.9
6.8
66.7
13.5
6.9
66.3
13.2
6.4
66.9
12.8

3
13
19

SSNMHC
NMHC Ambient
SSNMHC OnOff
6.7
64.4
14.4
6.9
66.5
13.0
6.8
66.6
13.6
7.0
66.1
12.5
6.4
66.8
13.0

DEFINITIONS
Original April 2007 Study: All original error surfaces
Mod B Revised SS Exhaust Flow and SS CO2 error surfaces
, , , , Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error
Mod 1 ,,
surfaces
Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error
surfaces
Mod 3 Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error
surfaces
SwRI Report 03.12859.06
79 of 83

-------
  TABLE 30. SENSITIVITY RESULTS COMPARING 13 REFERENCE NTE EVENTS
      ACROSS 5 MONTE CARLO VALIDATION SIMULATIONS FOR BSCO

1

7
52


SSCO
CO Time
Alignment
SSCO OnOff

Original
6.2
10.3
82.0

6.5

82.2

79.3
11.0


79.1
10.1


79.3
10.7


2
7
52

SSCO
CO Time
Alignment
SSCO OnOff
6.4

83.6
6.7

84.6
82.9
10.5

83.0
11.8

83.0
10.4


3
7
52

SSCO
CO Time
Alignment
SSCO OnOff
6.2
11.8
80.8
6.3
10.1
81.0
77.9
19.2

78.0
17.9

77.9
19.0


DEFINITIONS
Original April 2007 Study: All original error surfaces
Mod B Revised SS Exhaust Flow and SS CO2 error surfaces
, , , , Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error
Mod I _c
surfaces
, , , . Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error
Mod 2 ,,
surfaces
, , , _ Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error
Mod 3 „
surfaces
SwRI Report 03.12859.06
80 of 83

-------
11.0   FULL MODEL SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS

       This section contains a summary of the error surfaces that contributed the most to bias
and variance of the BSNOx emissions based on the full MC model.  Simulation results from the
13 reference NTE events produced sensitivity values for all three 95th percentile delta emissions
by  all three calculation methods.   Table  31  lists  the  error  surfaces in  which either  the
contribution-to-variance normalized sensitivity average value or  the 'on/off bias check for the
error surface was at least 5% in absolute magnitude compared to all the other error surfaces for
BSNOx.  If at least 5 of the 13 reference NTE events resulted in a sensitivity average value > 5%
or < -5% then the average contribution to variance is included in the table.  The sensitivity results
from the  original study are included for comparison.

        The results from the BSNOx Method 1  sensitivities show that the steady-state exhaust
flow rate bias from the original study has been eliminated in Mod B, Mod 1, Mod 2, and Mod 3.
There is also a slight increase (from the original study) in the average sensitivity due to variation
in the  steady-state exhaust flow rate with all the modification  runs.  Slight increases in the
average sensitivity for the steady-state NOx error surface are seen in all modification runs except
Mod 2.

       Method 2 results for BSNOx show that the steady-state CO2 bias effect and the NOx time
alignment variation effect were eliminated in all the  modification runs.   However, new error
surfaces  that showed sensitivities in  the modification  runs were  the steady-state CO2  variation
effect and  the BSFC DOE Test bias effect.   There were also slight increases in the average
sensitivity due to variation for the steady-state NOx error surface in all the modification runs.

       The steady-state CO2 bias effect for Method 3 was eliminated in all the modification runs
and a variation effect due to steady-state CO2 was included. Also, there was a slight increase in
the average sensitivity due to variation  for the  steady-state NOx  error surface in all  the
modification runs.
SwRI Report 03.12859.06                     81 of 83

-------
 TABLE 31. SENSITIVITY RESULTS COMPARING 13 REFERENCE NTE EVENTS
     ACROSS 5 MONTE CARLO FULL MODEL SIMULATIONS FOR BSNOX

1

1
20
31
35
51



SSNOx
SS Exhaust Flow
Warm-up torque
Engine Manuf
Torque
NOx Time
Alignment
Pulse Flow OnOff
SS Exhaust
Flow OnOff

Original
21.0
6.3
-23.2
-9.8
6.1
14.7
14.6

22.3
10.2
-23.8
-9.9
5.5
15.1


31.2
9.5
-20.7
-8.7
5.0
13.8


20.8
11.0
-23.7
-9.9
5.5
16.4


25.4
10.4
-22.6
-9.4
5.6
14.8


2
1
38
42
45
51


SSNOx
Warm-up BSFC
Engine Manuf BSFC
SSCO2
NOx Time
Alignment
BSFC DOE
Test OnOff
SSCO2 OnOff
29.5
6.3
13.6

6.1

-34.9
33.3
5.9
14.1
-23.2

-5.9

43.9
5.2
12.3
-20.0

-5.3

30.4
6.2
15.8
-24.1

-6.5

37.1
5.6
14.0
-21.9

-6.1


3
1
31
35
45
51

SSNOx
Warm-up Torque
Engine Manuf
Torque
SSCO2
NOx Time
Alignment
SSCO2 OnOff
23.6
-27.8
-11.7

6.0
-25.2
25.6
-28.5
-11.9
-16.3
5.4

35.1
-25.0
-10.4
-14.8
4.8

34.2
-29.4
-12.6
-16.8
5.8

29.2
-27.0
-11.8
-15.6
5.3


DEFINITIONS
Original April 2007 Study: All original error surfaces
Mod B Revised SS Exhaust Flow and SS CO2 error surfaces
Mod 1 Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error surfaces
Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error surfaces
Mod 3 Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error surfaces
SwRI Report 03.12859.06
82 of 83

-------
                                12.0   REFERENCES
      Feist, M.D., Sharp, C. A., Mason, R.L., and Buckingham, J.P., "Determination of PEMS
      Measurement Allowances for Gaseous Emissions Regulated Under the Heavy-Duty
      Diesel Engine In-Use Testing Program,"  SwRI Project 03-12024, U.S. Environmental
      Protection  Agency,  California   Air  Resources  Board  and  Engine  Manufacturers
      Association, April 20.
SwRI Report 03.12859.06                   83 of 83

-------