United States Environmental Protection Agency Office Of TransP°rtation EPA420-P-05-002 and Air Quality February 2005 MOVES2004 Validation Results Draft Report ------- EPA420-P-05-002 February 2005 MOVES2004 Validation Results Draft Report John Koupal Sujan Srivastava Assessment and Standards Division Office of Transportation and Air Quality U.S. Environmental Protection Agency NOTICE This Technical Report does not necessarily represent final EPA decisions or positions. It is intended to present technical analysis of issues using data that are currently available. The purpose in the release of such reports is to facilitate an exchange of technical information and to inform the public of technical developments. ------- Table of Contents 1. Introduction 2 2. Total Fuel Consumption Validation 3 2.1 FHWA Fuel Consumption Estimates 3 2.2 MOVES2004 Estimates 5 2.2.1 Aggregation Level 5 2.2.2 Calculation of Fuel Consumption 6 2.3 Results 7 2.3.1 National Results 7 2.3.3 State-by-State Results 8 3. CH4 and N2O Inventory Comparison 12 3.1 U.S. Inventory Estimates 12 3.2 MOVES Estimates 12 3.3 Results 12 4. Fuel Economy Comparison 13 4.1 Fleet MPG vs. FHWA Estimates 14 4.2 Model Year MPG vs. EPA Fuel Economy Trends 16 5. Conclusions 19 Appendix A: Peer Review Comments 20 References 26 ------- 1. Introduction MOVES2004 is the first iteration of EPA's new generation mobile source modeling framework. The model estimates energy consumption (total, petroleum-based, fossil-based) and emissions of methane (CH/t) and nitrous oxide (N2O) for all on-road sources, for the U.S (by state or county if desired), for calendar years 1999 through 2050. Ultimately, the model will include emissions of "criteria" pollutants including ozone precursors (HC and NOx), CO, PM and air toxics from all on- and off-road mobile sources, and will replace the current EPA models MOBILE6 and NONROAD. Additional background information and detail on the MOVES design and technical inputs are contained in several other reports, which are best navigated using the overview document "A Roadmap to MOVES2004". The primary reason the first version of MOVES estimates energy consumption is to validate model performance against top-down estimates of fuel consumption, compiled from fuel sales tax records. In the report "Modeling Mobile Source Emissions" published in 2000, the National Research Council stressed the need for EPA's models to undergo more systematic validation and sensitivity analysis,1 and from the beginning validation has been a top priority in the design and development of MOVES. Validation of MOVES2004 results is important not only to gauge the accuracy of MOVES energy consumption estimates, but also because many aspects of energy and emission estimation methodology used in MOVES2004 will form the basis for criteria pollutant emission estimation in later versions. Positive validation results provide assurance that the underlying MOVES methodology is fundamentally sound. Validation efforts have been conducted in the past on the MOBILE series of models - however, as there is no true "top-down" measure of criteria pollutant emissions, validation efforts are limited to a variety of methods meant to give a snapshot of overall model performance. These validation methods include: tunnel studies, comparison to independent emission data (e.g. chassis dynamometer or remote sensing studies), or using ambient monitoring data to construct pollutant ratios (e.g. VOC:NOx) for comparison to model predictions of these ratios. The results of such validations are often difficult to draw conclusions from - for example the results of a recent MOBILE6 validation effort sponsored by the Coordinating Research Council differed greatly depending on the method used.2 Validation of fuel consumption is considered more reliable than criteria pollutants because an estimate of top-down fuel consumption is available through fuel tax records. Recent validation studies of EPA's NONROAD and the California Air Resources Board's EMFAC models have employed top-down fuel sales, in the latter case as a step in generating criteria pollutants validation based on fuel-specific emission rates.3'4 Top- down fuel sales and energy consumption estimates are compiled by the federal government - the Department of Transportation's Federal Highway Administration (FHWA) and Department of Energy's Energy Information Agency (EIA) - and reported annually, after some adjustment to attempt to account for uncertainties in end use and fuel losses. Fuel tax receipts collected by individual states are used as the basis for ------- compiling these estimates. The validation performed for this analysis compares the bottom-up estimates of fuel consumption (via energy consumption) generated by MOVES to these independent top-down consumption estimates for the entire U.S., and by state. In addition to this validation comparison, this report presents a comparison of MOVES2004 methane (CH4) and nitrous oxide (N2O) estimates to the "official" U.S. Inventory prepared by EPA's Office of Air and Radiation, and a comparison of MOVES fuel economy estimates to those produced by FHWA and EPA's Fuel Economy Trends Report. These comparisons are not considered to be independent validations of MOVES because either they are derivative of total fuel consumption estimates, or they employ methods of estimation which overlap with the "bottom-up" methods employed in MOVES; however, we are including them in this report to give a sense for how MOVES compares to current state-of-the-practice estimates of CH4 and N2O emissions and fuel economy, and why differences occur. The comparisons between MOVES and fuel consumption, CH4 and N2O emissions are only made in the calendar year range from 1999 through 2002, since 1999 is the "base" year for MOVES2004 and the earliest for which the model produces estimates, and 2002 is the latest year for which top-down fuel sales and CH4 and N2O emission estimates are available. The comparisons can therefore provide a sense for model performance during this short span of years, but it does not allow a check on the validity of longer-term growth assumptions. A pre-publication version of this report underwent formal peer review by Professor Robert Harley of UC Berkeley; the resulting comments and our responses to these comments are contained in Appendix A. 2. Total Fuel Consumption Validation 2.1 FHWA Fuel Consumption Estimates Fuel consumption records are compiled by each state and submitted to FHWA, who publishes the estimates in the Highway Statistics annual report series.5 Both raw and adjusted results are presented in Highway Statistics, broken out by gasoline and "special fuel". According to the Highway Statistics website, special fuels "include diesel fuel and, to the extent they can be quantified, liquefied petroleum gases [LPG] such as propane". According to MOVES, LPG accounts for less than 0.05 percent of special fuel volume, so the "special fuel" category is essentially all diesel. For this analysis, we used results reported in Table MF-21 from Highway Statistics 1999 through 2002. For gasoline, we used total gasoline for highway use. This includes gasohol (E10) and reformulated gasoline (RFG) sold as motor fuel. For diesel, we used special fuel for private and commercial highway use. ------- The following information on the FHWA fuel consumption estimates is for the most part copied from the Highway Statistics website. The source of the motor fuel data are state tax records, which are submitted to FHWA as raw consumption values. FHWA adjusts the results by subtracting estimated non-highway use from the total use reported by the states. According to the Highway Statistics website, "Over the last several years, there have been numerous changes in State fuel tax laws and procedures that have resulted in improved fuel tax compliance, especially for diesel fuel. The improved compliance has resulted in increased fuel volumes being reported by the States to FHWA." The FHWA reported values do not include data on fuel purchased by the federal government for military use or fuel exported from the United States. The gasoline consumption levels include estimates of public use (separated into federal civilian and state, county and municipal government), but the special fuel levels do not. This is of note because this would mean that relatively large publicly-owned fleets of diesel vehicles - e.g. garbage trucks, transit buses, school buses - would not be included in the FHWA estimates. The sources included in the FHWA and MOVES fuel consumption estimates are therefore not completely aligned, which contributes to some uncertainty in the comparison between the two. FHWA made additional adjustments to allow for losses from destruction, evaporation, spillage, etc, and reports some variability among states in how this is quantified: "Some States make a flat percentage allowance for losses in storage and handling, and others allow for actual losses not to exceed a specified percentage. Still others permit distributors to claim stock losses in reconciliations of inventories, thus exempting the lost volume from taxation. Losses by destruction, where reported separately, are also included in this column. The maximum allowance used in the analysis to cover losses in storage and handling was one percent. Because of accounting methods, losses can be reported as a net gain." Adjustments are made in the annual data to exclude percentage losses in excess of 1 percent and to reflect usage rather than tax collections. The FHWA estimates used for this analysis are shown in Table 2-1. ------- Table 2-1: U.S. Annual Highway Fuel Consumption Estimates from FHWA (billion gallons) Year 1999 2000 2001 2002 Gasoline 128.7 128.9 129.7 133.0 Special Fuel 31.9 33.4 33.4 34.8 50 State plus District of Columbia 2.2 MOVES2004 Estimates 2.2.1 Aggregation Level MOVES2004 calculates energy consumption rather than fuel consumption. To generate fuel consumption estimates, total energy consumption estimates were first generated and converted to fuel consumption as a post-processing step. There are several ways to produce national, annual totals of energy consumption in MOVES2004 based on the choices the user has for pre-aggregation of geographic and temporal resolution. At the most disaggregate level, the model could be run for every county in the nation, by hour of the day, across all days of the week and months of the year. At the most aggregate level, specifying a pre-aggregation level of "nation" and "year" in the model will result in a pre-aggregation routine creating average inputs for the entire nation as a single "county" and the entire year as a single "hour". There are several options in between these two bounds: for example, geography could be pre- aggregated up to the state level from county or time span could be pre-aggregated up to the day or month level. The key point with regard to aggregation is that results will be different depending on the level of aggregation chosen. This is because some effects in the model (including temperature, air conditioning, and the distribution of operating modes) are not linear, and a single model run using an arithmetic average of several input data points will not yield the same results as separate model runs at each data point. For example, a run at 40 degrees will not equal the average of the results from runs at 30 and 50 degrees, because the equation for temperature effects (documented in the report "Energy and Emission Inputs for MOVES2004", or Energy and Emission Report) is not linear. In MOVES, temperature inputs vary by each county in the nation, by month and by hour. Aggregating these inputs to the nation / year level produces a single average temperature of 61 degrees. At this temperature, air conditioning effects are suppressed, but start energy consumption is increased by about 37 percent due to temperature effect based on Equation 9-5 in the Energy and Emission Report. ------- Operating mode distributions are derived from the mapping of driving schedules to average speed ranges, as discussed in the report "MOVES2004 Software Design Reference Manual". In the pre-aggregation routine, average speeds are aggregated before mapping to driving schedules, so the mix of drive schedules for the aggregated case would not be the same as the mix for the disaggregate case. Average speed varies only by hour of the day in MOVES2004 (not geographic location), so aggregation to the day level (and higher) results in a mix of speeds that differ from the hourly levels. The benefit of using the aggregation options in MOVES is model runtime performance. At this stage the time required to do a run at the most disaggregate level would be prohibitively long in single-computer configuration. The most aggregate case (nation / year) is the quickest method for producing national / annual results, and would be a likely choice for doing national-level runs. To assess the magnitude of difference among pre-aggregation options, we performed a sensitivity analysis on total energy results by generating MOVES national results for 2002 at four different levels of pre- aggregation: nation/year, state/year, nation/month, and state/month. The results of each run are shown in Table 2-2. Table 2-2: Sensitivity Analysis of 2002 U.S. Annual Energy Consumption Estimates at Different Levels of Pre-Aggregation (Petajoules) Pre-Aggregation Level Nation / Year State / Year Nation / Month State / Month Gasoline 15,751 15,853 16,089 16,142 Diesel 4,521 4,546 4,609 4,621 The trend shown here is that MOVES energy consumption results increase slightly as geographic and temporal resolution is increased. When geography is varied between nation and state and the time span is varied from year to month, results vary by about 2-3 percent. Additional work is necessary to determine the sensitivity of MOVES results to fine levels of dissaggregation, i.e. to the county level for geography, and day or hour for time. 2.2.2 Calculation of Fuel Consumption Total energy results from MOVES2004 for the state/month aggregation level were generated for all MOVES source types (i.e. vehicle classes) for gasoline (including E10 and RFG) and diesel, for each year from 1999 through 2002. Although MOVES can generate estimate of petroleum-based energy, total energy is the appropriate metric of comparison for this analysis since the FHWA estimates include E10 and RFG, and total energy consumption estimates in MOVES accounts for the oxygenate used in these fuels. ------- The next step was to convert the MOVES total energy results to fuel consumption, which required estimates of heating value for each fuel, shown in Table 2- 3. There are many estimates of heating value in the literature, as they depend on fuel density and quality; for this analysis we used Lower Heating Values (LHVs) from Heywood6 to stay consistent with the methodology used in the development of MOVES2004 energy rates (as documented in the Energy and Emission Input report). For the validation analysis we developed an average gasoline energy content by weighting together conventional gasoline, RFG and E10 energy contents according to the national/annual volume shares of these fuels estimated by MOVES (67 percent Conventional Gasoline, 21 percent RFG and 12 percent E10). Table 2-3: Energy Content by Fuel Type Fuel Subtype Conventional Gasoline Reformulated Gasoline E10a National Average Gasolineb Conventional Diesel LPG E100C Lower Heating Value (KJ/gram) 44.0 42.9 - - 43.2 46.4 26.9 Density (Kg/gallon) 2.8 2.8 - - 3.2 1.9 3.0 Energy Content (MJ/gallon) 124 121 120 123 137 89 80 a Volume-based weighted average of Conventional Gas (90%) and E100 (10%) b VMT-based weighted average of Conventional Gas (67%), RFG (21%) and E10 (12%) c Reference only 2.3 Results 2.3.1 National Results The fuel consumption calculated from the MOVES total energy results are shown in Table 2-4. The "special fuel" category for MOVES reflects diesel fuel only; LPG is estimated to represent only about 0.05 percent of special fuel volume. ------- Table 2-4: U.S. Annual Highway Fuel Consumption Estimates from FHWA and MOVES2004 (billion gallons) Year 1999 2000 2001 2002 Gasoline FHWA 128.7 128.9 129.7 133.0 MOVES 126.6 127.9 129.0 131.5 % Diff -2% -1% -1% -1% Special Fuel FHWA 31.9 33.4 33.4 34.8 MOVES 30.8 32.0 32.7 33.8 % Diff -3% -4% -2% -3% As shown in Table 2-4, the MOVES results compare well with the top-down estimates. Gasoline results from MOVES are 1-2 percent lower than FHWA, depending on the year. Special fuel results are 2-4 percent lower. For both fuels, MOVES tracks the increase in consumption reported by FHWA across the years analyzed. We would expect off-road use of motor gasoline and differences between states in reporting methods and accounting for spillage and losses would contribute to the overall uncertainty of the FHWA estimates. FHWA doesn't quantify this uncertainty, so it is not known whether MOVES estimates would fall within the uncertainty bounds of the top-down estimates. One source of uncertainty in this comparison is the inclusion of some vehicles in the MOVES estimates which are not included in the FHWA estimates. The FHWA gasoline totals exclude military vehicles and the special fuel totals exclude all publicly- owned vehicles. Military vehicle travel is not accounted for explicitly in MOVES, but would be accounted for to the extent the FHWA VMT estimates (the basis of MOVES activity estimates) include their travel. The MOVES source types do include some vehicle categories likely dominated by publicly-owned vehicles, such as refuse trucks, transit buses and school buses. In 2002, these three source types comprise about 1.3 percent of MOVES special fuel estimates. Perhaps a more equitable comparison would be to remove these vehicles from the special fuel totals shown in Table 2-4, although removing these vehicles wouldn't fully address this issue since a) some of these categories do include privately-owned vehicles (e.g. commercial refuse haulers), and b) there are publicly-owned vehicles in other MOVES categories (e.g. passenger fleet vehicles) that cannot be removed from the MOVES estimates. 2.3.3 State-by-State Results In response to peer review comments (Appendix A), we also conducted a comparison of calendar year 2002 fuel consumption results between MOVES and FHWA on a state-by-state basis. State level estimates are shown in Table 2-5, and maps of the contiguous 48 states color-coded by the absolute difference in fuel consumption between MOVES and FHWA are shown in Figures 2-1 and 2-2. Differences between MOVES estimates and FHWA estimates vary quite a bit state-to-state, although overall agreement ------- is good in many states, particularly for gasoline; the difference is 5 percent or better for 19 states, and 10 percent or better for 35 states. As illustrated in Figures 2-1 and 2-2, the spread and magnitude of difference between MOVES and FHWA is larger for special fuel than for gasoline. The maximum difference among states is 23 percent for gasoline (Oklahoma), and 146 percent for special fuel (Hawaii). Two factors contributing to larger differences in state-by-state comparisons relative to the comparison of national totals are a) cross-border travel and b) differences between fleet composition and activity patterns state-to-state, relative to the national defaults used in MOVES. It is difficult to discern which factor contributes more in a given state, although the cross-border travel issue is more relevant for smaller states, particularly those with a lot of travel (e.g. commuter traffic) to nearby states. For example, the MOVES gasoline results for New Jersey and New Hampshire are 16 and 19 percent lower than the FHWA estimates, respectively. Many vehicles based in these states commute to nearby states; for example, 2000 U.S. Census data estimates that 13 percent of New Hampshire residents work in Massachusetts.7 The difference between MOVES and FHWA fuel consumption estimates for states with high commuter outflow likely reflects that a substantial amount of the fuel purchased in these states is used in other states. The cross-border travel issue is accounted for to some degree with freight trucks in the FHWA estimates through the International Fuel Tax Agreement (IFTA), whose intent is to reallocate fuel taxes to states where fuel is used rather than sold. Cross-border travel for light-duty vehicles are not accounted for explicitly by FHWA. The second factor driving larger variability in state-by-state comparisons is how well the national default assumptions characterize a particular state. The less representative the national defaults are for a given state, the larger deviation from top- down fuel sales we would expect. There are numerous inputs this would apply to, including: the age of the vehicle fleet, congestion levels, meteorology, or terrain. For example, MOVES would tend to overpredict fuel consumption for a state with a vehicle fleet that is younger than the national average. For best results in state-level analysis, we would recommend customizing MOVES to include state-specific inputs such as VMT, age distribution and average speed distribution where available. ------- Table 2-5: Comparison of Year 2002 Fuel Consumption Estimates by State (x 100,000 gallons) ALABAMA ALASKA ARIZONA ARKANSAS CALIFORNIA COLORADO CONNECTICUT DELAWARE DISTRICT OF COLUMBIA FLORIDA GEORGIA HAWAII IDAHO ILLINOIS INDIANA IOWA KANSAS KENTUCKY LOUISIANA MAINE MARYLAND MASSACHUSETTS MICHIGAN MINNESOTA MISSISSIPPI MISSOURI MONTANA NEBRASKA NEVADA NEW HAMPSHIRE NEW JERSEY NEW MEXICO NEW YORK NORTH CAROLINA NORTH DAKOTA OHIO OKLAHOMA OREGON PENNSYLVANIA RHODE ISLAND SOUTH CAROLINA SOUTH DAKOTA TENNESSEE TEXAS UTAH VERMONT VIRGINIA WASHINGTON WEST VIRGINIA WISCONSIN WYOMING FHWA 2,541 237 2,541 1,383 15,386 2,023 1,520 409 141 7,720 4,827 433 625 5,061 3,077 1,519 1,161 2,063 2,219 699 2,505 2,786 4,997 2,587 1,537 3,008 476 830 974 688 4,003 905 5,617 4,140 336 5,094 1,708 1,515 5,114 390 2,270 419 2,989 11,116 993 336 3,792 2,660 799 2,474 313 Gasoline MOVES 2,609 225 2,325 1,329 13,633 1,950 1,470 428 188 7,262 4,812 409 674 5,068 3,403 1,373 1,329 2,266 1,896 635 2,349 2,641 4,695 2,513 1,648 3,342 452 841 848 554 3,360 1,046 6,449 4,407 342 5,279 2,105 1,642 4,933 415 1,981 370 3,076 10,470 1,079 336 3,523 2,556 832 2,847 357 %Diff 3% -5% -8% -4% -11% -4% -3% 4% 34% -6% 0% -6% 8% 0% 11% -10% 14% 10% -15% -9% -6% -5% -6% -3% 7% 11% -5% 1% -13% -19% -16% 16% 15% 6% 2% 4% 23% 8% -4% 6% -13% -12% 3% -6% 9% 0% -7% -4% 4% 15% 14% FHWA 673 110 709 573 2,733 550 229 62 27 1,355 1,459 38 224 1,373 1,360 507 407 914 588 168 504 399 944 653 540 951 202 375 288 105 834 420 1,168 981 150 1,505 807 460 1,375 55 623 159 905 3,125 355 64 936 572 270 684 305 Special Fuel MOVES 715 62 633 426 3,014 549 355 99 30 1,687 1,302 94 199 1,274 944 412 375 664 576 196 598 578 1,138 653 496 897 154 251 218 159 689 341 1,394 1,114 106 1,347 557 452 1,290 89 654 130 873 2,626 297 98 977 630 282 734 130 %Diff 6% -44% -11% -26% 10% 0% 55% 60% 12% 24% -11% 146% -11% -7% -31% -19% -8% -27% -2% 16% 19% 45% 21% 0% -8% -6% -24% -33% -24% 51% -17% -19% 19% 14% -29% -10% -31% -2% -6% 61% 5% -18% -4% -16% -16% 52% 4% 10% 4% 7% -57% 10 ------- Figure 2-1: Absolute Percent Difference between MOVES and FHWA Gasoline Consumption Estimates by State (Continental U.S.) Figure 2-2: Absolute Percent Difference between MOVES and FHWA Special Fuel Consumption Estimates by State (Continental U.S.) 11 ------- 3. CH4 and N2O Inventory Comparison 3.1 U.S. Inventory Estimates In accordance with International Panel on Climate Change (IPCC) guidelines, the U.S. EPA Office of Air and Radiation compiles and reports retrospective greenhouse gas emission inventories for all sectors each year in the "Inventory of U.S. Greenhouse Gas Emissions and Sinks". The latest version of this was published in April 2004, reporting results for 1990 though 2002.8 The methods for generating CH4 and N2O in the Emissions & Sinks report is a standard "bottom-up" methodology, combining gram per mile emission rates for these pollutants with estimates of vehicle miles traveled (VMT) and fleet mix to generate total mass emission estimates. CH4 and N2O estimates from the Emissions & Sinks report were derived from Tables 3-22 and 3-23, "CH4 [N2O] Emissions from Mobile Combustion". We also summed these pollutants by calendar year, for gasoline and diesel. The summed results are shown in Table 3-1 and 3-2. 3.2 MOVES Estimates For CFLt and N2O, MOVES is essentially an update of the process used for Emissions & Sinks, with similar estimates for VMT and fleet mix but more recent data for emission rates. CH4 and N2O emission rates were updated in MOVES2004 to incorporate recent vehicle testing conducted primarily by EPA (for CFLt) and the California Air Resources Board (for N2O). The details of this analysis are documented in a separate report prepared by ICF Consulting.9 Since the emission rates have changed significantly from those used in the Emissions & Sinks report, we do not expect MOVES CFLt and N2O inventory results to track those from the Emissions & Sinks report. 3.3 Results We compared aggregate on-road totals of CFLt and N2O generated by MOVES to the 2004 Emissions & Sinks report. MOVES was run in national/annual pre-aggregation mode to generate these results (unlike energy consumption, the level of pre-aggregation will not affect CFLt and N2O results because temperature, A/C or speed effects are not applied to these pollutants). CFLt and N2O results were summed across source use types for gasoline and diesel. The results are shown in Tables 3-1 and 3-2. 12 ------- Table 3-1: U.S. Annual CH4 emissions from Emissions & Sinks Report and MOVES2004 (gigagrams = 109 grams) Year 1999 2000 2001 2002 Gasoline E&S 174 169 164 159 MOVES 94 85 77 72 % Diff -46% -50% -53% -55% Diesel E&S 14 14 14 14 MOVES 0.4 0.5 0.5 0.5 % Diff -96% -96% -96% -96% Table 3-2: U.S. Annual N2O emissions from Emissions & Sinks Report and MOVES2004 (gigagrams = 109 grams) Year 1999 2000 2001 2002 Gasoline E&S 169 164 157 150 MOVES 115 111 106 102 % Diff -32% -32% -33% -32% Diesel E&S 10 10 10 10 MOVES 0.5 0.6 0.6 0.6 % Diff -95% -94% -94% -94% The large differences between MOVES and Emissions & Sinks for CH^ and N2O can be attributed directly to the updated emission rates. The new rates are significantly lower than the rates used in Emissions & Sinks, particularly for diesel. 4. Fuel Economy Comparison A useful comparison to make is between fuel economy (MPG) results derived from MOVES2004 output and alternative estimates - on a fleetwide basis as estimated by FHWA, and by model year as estimated from EPA label values. However, these comparisons are not considered to be independent validations of MOVES results because either they share some of the same data used in MOVES (in the case of FHWA), or because they are simply alternate methods of generating fuel economy estimates (in the case of the EPA label values). 13 ------- 4.1 Fleet MPG vs. FHWA Estimates MPG estimates for the entire on-road fleet are published by FHWA in the annual Highway Statistics series, broken down by vehicle categories: passenger car, other 2-axle 4-tire vehicle (i.e. light trucks), bus, single-unit truck, combination truck and motorcycles. The source of the "gallon" estimates are the top-down fuel consumption estimates compiled from fuel tax records presented in Section 2. According to the Highway Statistics Table VM-1, FHWA allocates total fuel consumption into each vehicle category based on "miles per gallon for both diesel and gasoline powered vehicles using state-supplied data, the 1997 VIUS, and other sources as a baseline"; hence, the FHWA MPG estimates by vehicle category are based on an estimated allocation of total fuel consumption, and are not a true top-down measure. The "miles" estimates are based on the estimates of total vehicle miles traveled (VMT) compiled through FHWA's Highway Performance Monitoring System (HPMS) and reported in the Highway Statistics reports (Tables VM-1 & 2); these VMT estimates are also used in MOVES. MOVES MPG estimates were calculated from VMT estimates produced by the model divided by fuel consumption as derived from total energy results presented in Section 2. To match the vehicle categories reported by FHWA, a post-processing step was necessary to combine MOVES source types (which were subsets of the HPMS vehicle categories) into the HPMS categories according to the breakdown shown in Table 4-1. 14 ------- Table 4-1: HPMS Vehicle Classes & MOVES Source Use Type HPMS Class Passenger Cars Other 2-axle / 4-tire Vehicles Single Unit Trucks Buses Combination Trucks Motorcycles MOVES Use Type Passenger Car Passenger Truck Light Commercial Truck Refuse Truck Single-Unit Short-Haul Truck Single-Unit Long-Haul Truck Motorhome Intercity Bus Transit Bus School Bus Combination Short-Haul Truck Combination Long-Haul Truck Motorcycle Because MOVES2004 uses the VMT data compiled by FHWA directly in the model, differences between MOVES MPG and FHWA MPG on a vehicle category basis would be traced to the differences in total fuel consumption estimates discussed in Section 2, and different methods for deriving fuel consumption by vehicle category. The fleet fuel economy estimates for calendar year 2002 by HPMS category are shown in Table 4-2. The passenger car, light truck and combination truck categories (which dominate both gasoline and diesel consumption) differ by five percent or lower, while the relative differences for buses, single unit trucks and motorcycles differ on the order of 30 to 40 percent. 15 ------- Table 4-2: 2002 U.S. Fleetwide Fuel Economy from FHWA and MOVES2004 (MPG) Vehicle Class Passenger Car Light Truck Bus Single Unit Truck Combination Truck Motorcycles FHWA 22.0 17.4 6.8 7.4 5.2 50.0 MOVES 22.8 16.6 9.7 9.8 5.3 30.5 % diff 4% -5% 43% 33% 1% -39% 4.2 Model Year MPG vs. EPA Fuel Economy Trends Another way to gauge MOVES fuel economy results is to compare by-model year MPG from MOVES with the sales-weighted model year averages reported in the Fuel Economy Trends report published yearly by EPA.10 The Trends report analyzes official fuel economy data generated for the process of determining compliance with Corporate Average Fuel Economy (CAFE) requirements and the EPA fuel economy labeling program. The report includes an estimate of harmonically averaged fuel economy for each model year from 1975 through 2004, weighted by sales of each vehicle line. For this analysis, we used the combined city/highway values reported in Table 1 of the Trends report. We used the "adjusted" values from this table, which reflect the downward adjustment (roughly 15 percent) applied to the raw measured values in order to better estimate "real world" fuel economy. To generate by-model year MPG estimates for MOVES, we did a national / annual run for calendar year 2004 in which we specified output reporting at the model year level (as noted in Section 2, more disaggregate runs will result in fuel economy results which are a few percent lower). This reports total energy and distance (VMT) for each model year in the 30 year window prior to and including the analysis year. MOVES does not account for deterioration in energy consumption due to vehicle age, hence it is assumed that MPG does not change with age. We then converted energy consumption to fuel consumption using the methods discussed in Section 2, and calculated MPG as VMT divided by fuel consumption for each model year. Since the Trends report only addresses light-duty cars and trucks, we limited the MOVES estimates to the passenger car and passenger truck use types. The passenger car use types should map directly to the light-duty vehicle class used in the Trends report. This is not the case for trucks, however. Trucks included in the Trends report have an upper weight cutoff of 8,500 Ibs gross vehicle weight (GVW), which is the cutoff for the CAFE regulations, whereas MOVES includes trucks heavier than 8,500 Ibs GVW in the 16 ------- passenger truck category. Vehicle weight classes in MOVES are based on loaded weight, not GVW, and the distribution of loaded vehicle weight is based on information from the U.S. Census Bureau's Vehicle Inventory and Use Survey (VIUS 1997).11 Based on VIUS, MOVES estimates that about 6 percent of passenger trucks have a loaded weight above 6000 Ibs (up to 14,000 Ibs), many of which would have a GVW rating higher than 8,500 Ibs. This means that passenger trucks exceeding 8500 Ibs GVW, such as the Ford Expedition or the GM Hummer, are included in the MOVES estimates but not the Trends estimates. The by-model year comparisons for model years 1975 through 2004 are shown for cars and trucks in Figures 4-1 and 4-2. Figure 4-1: Passenger Car Fuel Economy by Model Year for Fuel Economy Trends Report and MOVES2004 (MPG) 30.0 25.0 20.0 . ™ •• 7~ Si '7Z -i R n & E • 0 ° -in n 1 U.U . ' / £^ — MOVES --•-•FE Trends 0.0 1975 1980 1985 1990 Model Year 1995 2000 2005 17 ------- Figure 4-2: Light Truck Fuel Economy by Model Year for Fuel Economy Trends Report and MOVES2004 (MPG) 25.0 20.0 5.0 0.0 MOVES FE Trends 1975 1980 1985 1990 Model Year 1995 2000 2005 For passenger cars, the MOVES estimates track changes in the Trends results well but are generally lower, with larger differences in earlier model years and closer agreement in later years. The lower MOVES estimates highlight possible differences between the real-world methodologies employed in MOVES (e.g. driving patterns derived from in-use driving surveys, cold temperature effects for starts) and the methodology used to estimate real-world fuel economy in the Trends estimate, i.e. applying a downward adjustment to raw results from the Fuel Economy Test Procedure. Changes to this procedure are currently being considered by EPA in order to better reflect real world results. The MOVES passenger truck results are generally also lower than the Trends light- truck estimates, except for a stretch of years in the early 1990's where the estimates are higher by a few percent. Large swings in MOVES results in the early 1980's can be traced to anomalies in truck weight data as derived from VIUS and Oak Ridge National Lab datasets. The large drop in MOVES fuel economy from 1996 to 1997 can be traced to a jump in average truck weight, which appears to be brought on by the introduction of heavier trucks on the market in the 1997 model year - most notably the Ford Expedition (the reader should consult the report "MOVES2004 Highway Population and Activity Data" for more detail on the default weight distributions used in the model). The cause of the higher MOVES estimates relative to Trends in the early 1990's merits further investigation. 18 ------- 5. Conclusions Our conclusions from this analysis are as follows: • National fuel consumption estimates derived from MOVES2004 total energy results show good agreement with top-down estimates from fuel tax records compiled by FHWA. In 2002, MOVES estimates for gasoline consumption for a state/month aggregation case were 1 percent lower than FHWA estimates, and diesel consumption estimates were 3-4 percent lower depending on the treatment of publicly-owned vehicles. • State-by-state comparisons of MOVES and FHWA fuel consumption are generally favorable, particularly for gasoline, with larger variability in diesel results. Increased variability in state-by-state results is likely a function of a) cross-border travel, and b) how applicable national defaults are to a particular state. • A sensitivity analysis of MOVES energy consumption results versus the level of geographic and temporal resolution showed a 2-3 percent difference between the highest level of pre-aggregation (nation/year) and the state/month level, with intermediate levels (state/year and nation/month) falling in between. • MOVES CH4 and N2O emission inventory estimates are significantly lower than inventory estimates compiled by EPA's Office of Air and Radiation: roughly 30 to 60 percent lower for gasoline, over 90 percent for diesel. This difference is almost entirely due to new emission factors developed for MOVES which incorporate recent test data. • A comparison was made to fuel economy estimates for the entire on-road fleet generated by FHWA, and by-model year estimates from EPA's Fuel Economy Trends Report. MOVES MPG estimates agree well with the FHWA estimates for the vehicle categories which dominate fuel consumption (passenger car, light truck, and combination truck), which is expected based on the total fuel consumption results. The MOVES by-model year results are generally lower than the Fuel Economy Trends Report estimates, although agreement is closest in the most recent model years. Overall, the comparisons presented in this report are encouraging, particularly the good agreement between fuel consumption estimates derived from MOVES and the top-down fuel sales data compiled by FHWA. We believe the analyses presented here are a responsive first step towards the charge given to EPA by the National Research Council and other parties to employ more systematic model validation. At the same time, we recognize that model validation and evaluation is an ongoing process which must expand into a fuller assessment of sensitivities and uncertainties, and incorporate new data as it becomes available. 19 ------- Appendix A: Peer Review Comments Professor Robert Harley of the Department of Civil and Environmental Engineering of the University of California at Berkeley was contracted to provide formal peer review on a pre-publication version of this document. His comments are included verbatim in this Appendix. Responses to substantive (i.e. non-editorial) comments have been added following each comment, in italics to differentiate it from the original comments. It is important to keep in mind when reading these comments that MOVES2004 was revised in the time between the pre-publication and published versions of this report, to correct errors and update default inputs; as the peer review comments were made based on draft results, some of the specific comments apply to results which are no longer in this report. 20 ------- Review of "MOVES2004 Validation Results' by Robert Harley Department of Civil and Environmental Engineering University of California at Berkeley October 2004 MAJOR COMMENTS Ml. The title of the report should be changed to give a more detailed picture of the issues that are being addressed. For example, based on the title alone, I would have assumed that the report provided an assessment of CO, NOX, VOC, and PM emission estimates. I recommend the following revised report title: "Assessment of MOVES Model Estimates of Fuel Consumption, Fuel Economy, and Greenhouse Gas Emissions". M2a. Rates of growth for gasoline and diesel differ: on-road diesel fuel use is growing much faster than gasoline. Therefore, I recommend comparisons of MOVES with FFEWA national on-road fuel consumption estimates for earlier years such as 1990 as well as circa 2000 comparisons that are already included in the report. While useful and informative, the comparisons around the year 2000 don't give a clear picture of whether the differing long-term rates of growth of gasoline and diesel use are accurately represented in the MOVES model. Capturing differences in activity growth rates by sector will be very important for making future year emission projections. Some older transportation models focus on total traffic and incorrectly assume diesel is a small fixed fraction of the total, growing at the same rate as the total vehicle miles of travel (VMT), which is dominated by gasoline vehicles. M2b. The differences in national fuel consumption shown on page 5 (see Table 2-3) for 1999-2003 are small and all about the same for gasoline, with larger and more variable differences for diesel (special) fuel. This narrow window of years does not help to assess whether long-term growth rates are represented accurately within MOVES. EPA Response to 2a & 2b: The earliest calendar year MOVES2004 can currently provide estimates for is 1999, so the suggested comparison isn 't currently possible. M2c. The text on page 2 notes improved fuel tax compliance for diesel fuel resulting in states reporting higher diesel fuel sales to FFEWA over time. This may contribute to a faster growth rate for diesel fuel compared to gasoline. Other relevant factors could include differences in fuel economy trends between light-duty gasoline vehicles and heavy-duty diesel trucks, different rates of growth in the numbers of light-duty gasoline vs. heavy-duty diesel vehicles on the road, and different rates of growth in the amount of 21 ------- driving per vehicle per year. EPA Response: Differences in fuel economy trends between light-duty gasoline vehicles and heavy-duty diesel truck are addressed in MOVES to the extend they are reflected in the data used to generate total energy rates. Different rates of growth in the numbers of light-duty gasoline vs. heavy-duty diesel vehicles on the road are derived from El A 's National Energy Modeling System (NEMS). Different rates of growth in the amount of driving per vehicle per year are accounted for in MOVES only indirectly; total VMT and sales grows at different rates, which implies differences in per-vehicle VMT growth. M3. Another evaluation of MOVES that should be done is to compare state-level annual on-road fuel consumption between FHWA and MOVES for the year 2000 (aggregrates by EPA Region, PAD District, etc. could also be examined). A common misconception is that state-level fuel comparisons cannot be done for diesel fuel as heavy-duty trucks can drive -1000 miles between refuelings and therefore often cross state lines. Note however that current law requires inter-state truckers to file quarterly international fuel tax agreement (IFTA) returns in their home state only, with fuel excise taxes then remitted by the home state to other jurisdictions in proportion to where fuel is used rather than where it is purchased. According to FHWA, all Canadian provinces and all states except Alaska and Hawaii were participating in the IFTA program as of October 1996. EPA Response: In response to this comment we 've added a state-by-state comparison of fuel consumption levels estimated by FHWA and MOVES, in Section 2.3.3. M4. For diesel fuel, where off-road fuel use is a significant fraction of the total, how will EPA check for balance between refinery distillate fuel production supplied to the U.S. market and total on-road + off-road engine activity? Though an accurate separation of on-road from off-road fuel use may be difficult to achieve, the combined total may still be well-defined. The validation efforts reported here encompass only on-road fuel use, but I thought that MOVES includes off-road engine activity and emission estimates as well as on-road. EPA Response: MOVES will eventually include off-road, but not at this time. When off- road has been integrated into MOVES we can go a broader comparison to refinery distillate production. For now, we are relying on FHWA 's process for separating highway and non-highway fuel use. M5. There is a pervasive issue of tabular data being presented with unjustified precision in the draft report. For example, in the comparison of greenhouse gas emission estimates (section 3), the cited EPA GHG report presented CH4 and N2O to the nearest Gg, whereas Tables 3-2 and 3-3 append ".0" to each of those numbers, implying a sudden order of magnitude improvement in the precision of the estimates. The stated uncertainties in the EPA greenhouse gas emission inventory report are 7, 9, and 18% respectively for CO2, CH4, and N2O. The uncertainty corresponds to -80 Tg of CO2 from gasoline engines, so these numbers should not be reported in Table 3-1 as 1096.3 etc. For CO2 emissions from gasoline engines, all that can be concluded is that the two emission estimates agree within 22 ------- their associated ranges of uncertainty. I couldn't reproduce the diesel CC>2 numbers from E&S report due to use in the MOVES2004 report of unpublished data from a contractor, but uncertainties in excluding off-road diesel fuel use imply at minimum the diesel CC>2 numbers should be rounded to the nearest 1 Tg, not 0.1 Tg. As the comparison of GHG emissions is central to the present report, more supporting information should be included on the reasons and data sources for revisions to CH4 and N2O emission factors. M6. The tone of the concluding paragraph is upbeat in promoting the MOVES model and its validity. More balanced wording should be used here. For example, the first sentence of this paragraph could be dropped without detracting from the conclusions. Agreement between FHWA and MOVES fuel consumption should be characterized as "close", not "very close". The diesel fuel estimates disagree by more than 10%. I recommend additional rewording as follows: "... NRC's review of mobile source emissions models recommended increased attention by EPA to model validation efforts, and the results reported here are responsive to that charge for overall levels of on-road engine activity and greenhouse gas emissions." EPA Response: We have modified these sections in response to this comment, although the magnitude of difference in the comparison changed since these comments were made as a results of changes to the model. 23 ------- DETAILED COMMENTS Dl. On page 1, an introductory paragraph should be added that describes the MOVES model and its intended uses, to provide context for the reader. D2. On page 1, citations to recent validation studies of NONROAD and EMFAC models are missing (note 3). D3. At the bottom of page 4, there is a reference to Heywood. This needs to be footnoted and added to the reference list. D4. Table 2.2 needs numerous revisions. Precision in the final column of the table (energy content) is to 6 figures, which is excessive. Restate in MJ/L and/or MJ/gallon units. Units for the second column should be kJ not KJ. I recommend stating fuel density in specific gravity or kg/L units. Excessive precision of 4 figures is currently specified for fuel densities. The density difference between conventional and reformulated gasoline indicated in the table may be too small: reducing aromatic content is expected to reduce fuel density. In California, we reported that the density of gasoline fell from 0.76 to 0.74 kg/L with the 1996 introduction of Phase 2 RFG (Kean et al., SAE technical paper no 2002-01-1713, see Table 1), though note this fuel reformulation was more severe than required in Federal RFG areas outside of California. Response: the RFG fuel density is based on GREET estimates for Federal RFG. GREET does account for California RFG separately, but uses the same density. D5. On page 5 (section 2.3), it would be helpful to state what fraction of total special fuel is attributed to LPG by the MOVES model. I expect it to be a negligible fraction on the national scale. This information could be derived from the EPA GHG report, which includes inventories of mobile source CO2 emissions by fuel type including LPG. Response: the LPG contribution has been added to the report (approximately 0.05 percent of "specialfuel" volume). D6. On page 6, the meaning of the last sentence of the first full paragraph is unclear, and the text is garbled: "but we would the uncertainty in off-road use..." D7. The stated units of measure in Tables 3-2 and 3-3 are incorrect. Emission estimates for CFL; and N2O are in Gigagrams (Gg), not Teragrams (Tg). It may help to include notes that Tera = 1012 and Giga = 109. Percentage differences should be rounded to the nearest whole value in all tables appearing on this page. D8. On page 10, Table 4-1, spaces are missing in some of the entries in the 2nd column. In Table 4-2, the percentage differences should be rounded to the nearest whole number. The text at the top of page 11 should not given reasons for a 4% difference as those numbers agree within the range of uncertainty in the estimates. 24 ------- D9. On page 11, a citation for the Vehicle Inventory and Use Survey (VIUS) is missing from the reference list. The text on page 11 should read "Vehicle Inventory and Use" rather than "Vehicle In Use". D10. On page 13, the Figure is incorrectly numbered 4-1; it should be 4-2. For both figures, the y-axis tick mark labels should be rounded to the nearest 1 mpg rather than 0.1 mpg. DILI recommend caution in making statements such as that appearing at the bottom of page 13 ("MOVES results suggest that this adjustment is no longer adequate to reflect true on-road fuel economy"). MOVES provides estimates of in-use vehicle fuel economy, not ground-truth data. The 15% downward adjustment applied to fuel economy measured in the FTP may indeed be inadequate. However, I would want to see direct measurements of in-use vehicle fuel economy and comparisons to FTP values for the same vehicles before making strong statements about the need for revisions to in-use fuel economy adjustments. Response: We have modified the report in light of these comments. D12. The first bullet in the conclusions section traces the higher fuel consumption estimated by MOVES in part to the exclusion of publicly-owned vehicles from the FHWA special fuel consumption estimates. But when MOVES estimates were adjusted to exclude refuse trucks and buses (see Table 2-3), the comparison with FHWA still indicated an 8-14% offset vs. a 10-15% offset before adjustment. Publicly-owned vehicles therefore do not appear to be a major contributor to the offset, at least assuming that most of the relevant fuel use is by buses. Other factors may be at work here, and/or the differences may not be significant relative to uncertainties in the estimates. Response: The magnitude and direction of this difference has changed since these comments were made. In general the contribution of publicly owned vehicles can not be quantified at this time, and further work will be required to understand this. D13. In the reference list, web links to cited reports should be included where available. Many of the cited references will otherwise be difficult to locate. 25 ------- References 1 National Research Council. 2000. Modeling Mobile Source Emissions. National Academy Press; Washington, D.C. URL with link to this book: http://books.nap.edu/catalog/9857.html 2 Environ International Corp., Evaluation of the U.S. EPA MOBILE6 Highway Vehicle Emission Factor Model, Coordinating Research Council Project E-64 Final Report, March 2004. URL with link to report: http://www.crcao.com/reports/recentstudies2004/CRC_E-64_Final_032004.pdf 3 Kean, A.J.; Sawyer, R.F.; Harley, R.A. (2000) A Fuel-Based Assessment of Off-Road Engine Diesel Engine Emissions Journal of the Air & Waste Management Association 50, 1929-1939. URL with a link to article: http://www.ce.berkeley.edu/~harley/research_3.php 4 Singer, B.C.; Harley, R.A. (2000) A Fuel-Based Inventory of Motor Vehicle Exhaust Emissions in the Los Angeles Area During Summer 1991 Atmospheric Environment 34, 1783-1795. URL with a link to article: http://www.ce.berkeley.edu/~harley/research_3.php 5 FHWA Office of Highway Policy Information, Highway Statistics 2002, 2004. URL with link to report: http://www.fhwa.dot.gov/policy/ohim/hs02/index.htm 6 Heywood, J., Internal Combustion Engine Fundamentals (p 915), McGraw-Hill, 1988 7 U.S. Census 2000 County-To-County Worker Flow Files, http://www.census.gov/population/www/cen2000/mcdworkerflow.html 8 U.S. EPA Office of Atmospheric Programs, Inventory of U.S. Greenhouse Gas Emissions and Sinks: 1990 - 2002, EPA Report No. EPA430-R-04-003, April 2004. URL with link to report: http://yosemite.epa.gov/oar/globalwarming.nsf/content/ResourceCenterPublicationsGHGEmissionsUSEmis sionslnventory2004.html 9 ICF, Update of Methane and Nitrous Oxide Emission Factors for On-Highway Vehicles, 2004 10 U.S. EPA Office of Transportation & Air Quality, Light-Duty Automotive Technology and Fuel Economy Trends: 1975 through 2004, EPA420-R-04-001, April 2004 . URL with link to report: http://www.epa.gov/otaq/fetrends.htm 11 U.S. Census Bureau, 1997 Economic Census: Vehicle Inventory and Use Survey, October 1999 URL Link to report: http://www.census.gov/svsd/www/97vehinv.html 26 ------- |