Strengths and Weaknesses of Models Used in Past Benefit Assessments from an Economist's Perspective William Wheeler Office of Research and Development, US EPA Presented at the Workshop on Water Quality Modeling for National-Scale Economic Benefit Assessment, Washington DC, February 9-10, 2005 ------- DISCLAIMER These proceedings have been prepared by Alpha-Gamma Technologies, Inc. under Contract No. 68-W-01-055 by United States Environmental Protection Agency Office of Water. These proceedings have been funded by the United States Environmental Protection Agency. The contents of this document may not necessarily reflect the views of the Agency and no official endorsement should be inferred. ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9, 2005 Strengths and Weaknesses of Models Used in Past Benefit Assessments from an Economist's Perspective Speaker: Will Wheeler Office of Research and Development, US EPA Slide #1 "Strengths and Weaknesses of Models Used in Past Benefit Assessments from an Economist's Perspective," Tape 2A, Tape counter starts at 540 Thank you, Martin, and good morning to everyone. Before I start, I want to acknowledge Martin and Chris and especially Charles for talking with me about my presentation and giving me some really helpful comments and trying to coordinate with the ones you just saw. I also want to emphasize that this is an economist's perspective, so it's my perspective and I'm probably reflecting ignorance or misunderstanding. Also, they "punted" a few things to me, and I'm worried now that expectations for this talk are too high, (laughter) I'm not going to be able to touch on everything. Slide #2 "Introduction," Tape counter starts at 579 I was in the Office of Water for five years, working primarily as an economist on effluent guidelines, focusing on benefits assessments. I now work in the Office of Research and Development in the STAR Program, but I do still focus on benefits. I hope I remember everything correctly from back then, but I may not. I am supposed to talk about a lot of issues, and some people asked me to talk about a few things, but I'm really going to emphasize constraints to our analysis and sort of why we do what we do, which maybe isn't the ideal. Slide #3 "Context: Tight Deadline and Limited Resources," Tape counter starts at 623 The big point, which I'm afraid I'm hammering too much in this talk, is the tight deadlines and limited resources we have for these sorts of assessments, and Charles and Chris talked about this. In OW, just as an overall average, we have 473 days from the start of a rule- making to a proposal, and then 903 more days from the proposal to the final rule promulgation. I think some of those averages might be a little low, based on including some simple procedural rules in there, but that's not a lot of time considering that that includes 90 days for the proposal, 90 days for final—for OMB to review the rule, management briefing— which takes many months because you have to start at the bottom of the chain and go all the way up the chain and have benefits estimates at the beginning to have every manager aware of 1 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9, 2005 what the benefits are, any other tasks you have to do—including information collection requests, which are pretty time-consuming. Then after proposal there's a comment period, and there may be a notice where you talk about what you learned in the comment period. The reason I highlight that is because it's really hard to get any work done during the comment period—it's kind of dead time—because the point is you're supposed to wait and see what people say you did wrong and how stupid you were and then fix it. So, you can't really make improvements to your models during that period because then someone will say something that you didn't anticipate. Slide #4 "Other Hurdles, " Tape counter starts at 712 A couple of other hurdles that kind of limit the analysis we choose to do are OMB and management acceptance of what we've done in the past. There is an inertia, I would say. If you have a type of analysis and you've used a model—you've explained it to OMB—you've explained it to management—they understand it and they accept it—you're not going to change it every rulemaking; no one really has time for that. As an example, we had numerous side meetings with OMB about using NWPCAM in benefits assessments and how to apply the water quality ladder valuation. Once we did that, there's an implicit agreement that that's the model we're going to use and that's how we're going to apply that model. So, a lot of people are reluctant, I think, to change how you do the analysis. Another constraint is that it's hard for us to use models off the shelf, given our need to add in the policy contacts, the sources of pollutants, and the benefit endpoints—I'll talk about that in a little bit. Slide #5 "Implications," Tape counter starts at 782 Implications of these constraints are: We really can only rely on contractors or what we can do ourselves. However, in any given rulemaking there are far more contractor hours used than EPA staff hours. Any given EPA staff member working on a rulemaking is supervising numerous contractors, and any number of staff at the contractors, in doing the work. So, really, it's the contractors who end up doing the work. Frankly, most people wouldn't put up with what we ask of them if they weren't being paid in that manner, (laughter) Contractors are the only people to whom you can say, "Here are the inputs to your analysis—we need it tomorrow," and they'll do it, but a government employee might not. (laughter) So, "What is the best model for this application?" is not a question we usually ask or answer—it's not something we have time to do, but it is obviously something we should be doing. It's easier for me to say that now, obviously, not that I'm in ORD. (laughter) 2 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9, 2005 Slide #6 "Example—CAFOs, " Tape counter starts at 844 Let me just give you an example: CAFOs, which was a very high-profile rule, included among other things a lot of collaboration and consultation with USD A. I don't have files on this anymore, but we started work on that around early 1998; we proposed in December 2000; a comment period lasted until July 2001; we published a notice of data availability, talking about what we learned in the comments; another comment period until January 2002— another notice—and another comment period and the final rule in February 2003. Slide #7 "Example—CAFOs (cont), " Tape counter starts at 885 We did use NWPCAM for the benefits assessment for that. These are just examples, not really reflective of what we decide for every rule, but we considered using SPARROW for CAFOs. We thought that given the match of the pollutants in SPARROW and the pollutants of concern in CAFOs there would be a really good match. At the time we didn't have access to the most recent version and there was no version available that we could get someone to run internally, so we really couldn't use that. We also thought that maybe it would be really good to do a case study of the Chesapeake Bay. That's an area that's obviously impacted a lot by CAFOs, and it would be really interesting to policy makers, but again we didn't have a mechanism to get at that model to use in the timeframes we were interested in. Slide #8 "Steps in Benefit Analysis (CAFOs), " Tape counter starts at 937 I don't really need to talk about this much—Chris went over this really well. The reasoning is that there's just all these steps in the benefits analysis: engineers have to provide the edge-of- field loadings—they have to get that to a reach, then run the NWPCAM model, which takes kind of a long turn-around if you're running the full model (the mainframe version—at least it did—it might not anymore). Then those inputs go to the benefits assessments—we go to other models sometimes—so there's just always a time crunch. One of the issues is: the loadings from the engineers come late, and they're probably not right the first time, so as soon as you get them you have to turn stuff around right away. Slide #9 "What Models Have We Used?" Tape counter starts at 981 I apologize that my presentation doesn't flow—I'm supposed to talk about a few different things. I just want to really set the context of why we choose the models we choose, based on some tight constraints of time and resources. 3 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9, 2005 There are, I think, five different things that we've used that you could call a water quality model or attempt at water quality modeling. This reflects what I was able to find for the last 7 or 8 years. I doubt that it's a comprehensive list, but I hope it's pretty close. I put these in the order I found them, but the dilution models and NWPCAM are probably the two things we've used the most, and that's why they're highlighted in blue. I don't have a count of "here are all the rules and here are the models we've used," but I'm fairly confident saying those are the things that are most prominent in our analyses. Slide #10 "Proportional Reductions, " Tape counter starts at 1036 I have "proportional reductions" up there as a model, which I guess it's not really when you just say that water quality improves in direct proportion to the reduction in loadings. I found a couple instances where we've done that—in a couple of water quality standards, the California Toxics Rule, and the Great Lakes Water Quality Standards. They're not national— they're regional or state-level rules, where we probably had to do something quickly. To answer one of the questions from the previous session on the water quality standards, the last I heard the relative importance of water quality standards and TMDLs versus effluent guidelines—in terms of being written into a permit—is right now about half and half. However, it's going towards more permits being written to meet water quality standards and TMDLs and less so for effluent guidelines as time goes on. Half and half is the last number I remember; I hope it's still right. Slide #11 "DilutionModels," Tape counter starts at 1091 Dilution models are just flow of pollutant divided by flow of the reach. We've tended to use those most for the effluent guidelines where there are a large number of toxic pollutants— metal products and machinery, iron and steel, pulp and paper. There's a lot of information about that in one of the appendices in the background paper about how many pollutants of concern there were for each rule. For a lot of these rules, we had more than a hundred pollutants we wanted to try to model. I don't know that dilution models are really all that accurate, but they're cheap and easy to use, and we could really get the job done with them. Slide #12 "NWPCAM," Tape counter starts at 1128 Chris talked a lot about NWPCAM. We've used it for CAFOs, Storm Water Phase II, Meat and Poultry Products, Construction and Development (which may not be a complete list). I doubt that you're familiar with all of those rules, but they're all high-profile rules with lots of prep work. Except for Meat and Poultry Products, they all dealt with non-point sources, so 4 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9, 2005 we had to put a lot of effort into identifying where those sources were and linking them up to this model. They also have an emphasis on conventionals, nutrients, and lots of sources, so there was a decent fit with NWPCAM, which is a national model and can accommodate including all of these sources. Now, note that this model is expensive and time-consuming, but it's a lot more complex than the dilution models. It also has the benefit of being accepted by management and OMB because it was created primarily with EPA funds, and it is run by a contractor so we do have access to use it. Right now, however, as far as I know, it's only run at one contractor. Slide #13 "QUAL2E," Tape counter starts at 1183 I don't know a whole lot about this—it was used after I left the division, but Chris talked about it. As far as I know, it was only used in Aquatic Animal Production. It is an in-house model and focuses on conventionals and nutrients, so it was a very good fit for that rule. Slide #14 "Dioxin Reassessment, " Tape counter starts at 1200 The last one I think we've used is the dioxin reassessment model. The dioxin reassessment evaluation model is what we called it in the Effluent Guidelines Program. I Googled that phrase and only "effluent guidelines" came up, so I'm not sure if it has a more-formal name that you'd be familiar with. We used it in effluent guidelines, especially Pulp and Paper, where dioxins and furans were really the pollutants of concern. EPA has been reassessing its treatment of dioxin—it's an ORD project that's not finished yet. So, this was an in-house model we had access to, and it focused on the major pollutants of concern. So, if there's a message I'm giving you, it's that the models we've chosen are models we have an ability to run easily, and that we do try to match them up to our pollutants. Slide #15 "Other Things I Was Asked to Talk About, " Tape counter starts at 1247 There are a lot of topics in the background paper, a number of things I was asked to talk about. Frankly, I don't have a whole lot to say. We usually try to match spatial scales. I don't think it's that much of a big deal—I think economists can generally handle most spatial scales from water quality models. NWPCAM and Mitchell-Carson, for example, are both national and they match up really well. I can't say that temporal scales match at all. It's not something that we worry about, given that most models predict an average summer flow or whatever, but benefits vary highly across different pollutant concentrations as pollutants change over the course of the season. It matters a lot, and summer pollutants probably matter more than anything else. 5 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9, 2005 Martin asked me about the precision of the models and how that affects benefits estimation. Until he asked me last week, I had never ever thought about that (laughter), so I don't have anything to say, but I thought it was important to note that we've never thought about that, (more laughter) As a couple people mentioned, there are more-stringent requirements to deal with uncertainty in our estimates, and I think the precision of the models is really going to matter as we go to a more-complicated uncertainty analysis. Slide #16 "Case Study Approach, " Tape counter starts at 1314 Chris mentioned that they used the case study approach in Aquatic Animal Production. We've often used case studies to verify economic estimates of benefits, but didn't use a more complicated model. Aquatic Animal Production is an example where we extrapolated from what we thought was a representative sample of benefit estimates to the national scale. They also used it in an NPNM to supplement the national estimates, where they used a more detailed water quality model. I think this is a very promising approach, if there's a tradeoff between developing a national model and developing accurate regional models, if we can solve the sampling problem of finding a representative set of case studies of benefits estimates and using more accurate regional- or local-scale models to do that. Hopefully, that's something to think about in the next two days. Slide #17 "Conclusions" Tape counter starts at 1366 I just wanted to plug something: The Ecological Benefits Assessment Strategic Plan is a project a large number of us in EPA have been working on as a way to figure out how to do a better job overall of calculating the ecological benefits of rules, not just water rules but all rules. One of our big, big recommendations is that different disciplines need to communicate better and need to start working together earlier in the process, because what we've seen is that disciplines don't work together—they work parallel—and then kind of get together at the end. Really, I'm optimistic. This is a great start, in this area, of doing that sort of thing, and talking together about what our needs are and what modelers can provide and how to get everything to mesh. So, that's pretty much it. I just think we should develop more capability and more models that can be used off the shelf. 6 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9 2005 Strengths and Weaknesses of Models Used in Past Benefit Assessments from an Economist's Perspective Introduction OW Economist for five years ¦ Worked primarily on benefit assessments for Effluent Guidelines Now in ORD ¦ Still focus on benefits I'm supposed to talk about a lot of issues ¦ Emphasize constraints to analysis 7 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9 2005 RESEARCH £ DEVELOPMENT Building a scientific foundation for .sound envtfpnmentaI decisions Context: Tight Deadline and Limited Resources OW averages: ¦ 473 days from start to proposal ¦ 903 days from proposal This includes: ¦ 90 days each for OMB review ¦ Management briefing ¦ All other tasks (Information Collection) ¦ Comments and Notice Other Hu OMB arid Management acceptance ¦ Several side meetings with OMB on use of NWPCAM in benefits assessment Can't use models off the shelf ¦ Need to add sources, benefit endpoints 8 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9 2005 RESEARCH £ DEVELOPMENT Building a scientific foundation for .sound envtfpnmentaI decisions Implications Reliance on contractor or internal modeling But EPA is heavily tilted toward contractors So contractors end up doing most of the work "What is the best model for this application?" is not a question we ask or answer Example-CAFOs Selected around January 1998 Proposal in December 2000 ¦ Comment period closed July 2001 NODA published November 2001 ¦ Comment period closed January 2002 NODA published July 2002 ¦ Comment period closed August 2002 Final rule published February 2003 9 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9 2005 Example-CAFOs Used NWPCAM for benefits assessment Considered SPARROW ¦ At the time, we didn't have access to most recent version Considered Chesapeake Bay case study ¦ Didn't have a mechanism Bttildinga scientific foundation for sowid environmental decisions Steps in B Analysis (C Model farm edge-of-field loadings Transport to nearest RF3 reach Run full model Benefit Assessment ¦ Water Quality Index ¦ NWPCAM feeds into other analyses There is ALWAYS a time crunch 10 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9 2005 RESEARCH £ DEVELOPMENT Building a scientific foundation for .sound envtfpnmentaI decisions What Models Used? Proportional Reductions (Not modeling) Dilution Models NWPCAM QUAL2E Dioxin Reassessment Evaluation Proportional California Toxics Rule, Great Lakes Water Quality Standards ¦ Water Quality Standards ¦ Smaller Scale 10 11 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9 2005 RESEARCH £ DEVELOPMENT Building a scientific foundation for .sound envtfpnmentaI decisions Dilution Used for most Effluent Guidelines with a large number of toxic pollutants ¦ MP&M, Iron and Steel, Pulp and Paper Not accurate, but cheap and easy to use NWPCAM Used for CAFOs, Stormwater Phase II, Meat and Poultry Products, Construction and Development ¦ High profile rules, lots of prep work ¦ Emphasis on conventionals, nutrients, lots of sources Expensive and time-consuming, but more complex Accepted by management, OMB Run by a (single) contractor 12 12 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9. 2005 QUAL2E Used only in Aquatic Animal Production In-house (EPA) Model Focus on conventionals, nutrients Dioxin Reasses Used in certain effluent guidelines ¦ Pulp and Paper Result of EPA's reassessment In-house model Focus on major pollutant(s) of concern 14 13 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9 2005 Building a scientific foundation for .sound envtfpnmentaI decisions Other Things Asked to Talk About Spatial Scales ¦ Often match (NWPCAM and Mitchell- Carson), probably not a big deal Temporal Scales ¦ Don't really match at all Precision ¦ Uncertainty (new OMB requirements) 15 Bitlldiiiga scientific: foundation for soxmd environmental decisions Case Study • Often used to verify economic estimates of benefits, but not modeling (e.g. Pulp and Paper) • Used in Aquatic Animal Production • Used to supplement national estimates (e.g., Metal Products and Machinery) ¦ Dilution model for national estimates ¦ Simple fate and transport for Ohio • Need to solve sampling problem 14 ------- Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments February 9 2005 Conclusions Improved Communication: Ecological Benefits Assessment Strategic Plan ¦ Improved communication ¦ Disciplines working together earlier in the process Develop in-house (EPA or contractor) capability More models that can be used off the shelf 15 ------- |