Strengths and Weaknesses of Models
Used in Past Benefit Assessments from an
Economist's Perspective
William Wheeler
Office of Research and Development, US EPA
Presented at the Workshop on Water Quality Modeling for
National-Scale Economic Benefit Assessment,
Washington DC, February 9-10, 2005

-------
DISCLAIMER
These proceedings have been prepared by Alpha-Gamma Technologies, Inc. under
Contract No. 68-W-01-055 by United States Environmental Protection Agency Office of
Water. These proceedings have been funded by the United States Environmental
Protection Agency. The contents of this document may not necessarily reflect the views
of the Agency and no official endorsement should be inferred.

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
February 9, 2005
Strengths and Weaknesses of Models Used in Past Benefit Assessments from an
Economist's Perspective
Speaker: Will Wheeler
Office of Research and Development, US EPA
Slide #1 "Strengths and Weaknesses of Models Used in Past Benefit Assessments from an
Economist's Perspective,"
Tape 2A, Tape counter starts at 540
Thank you, Martin, and good morning to everyone. Before I start, I want to acknowledge
Martin and Chris and especially Charles for talking with me about my presentation and giving
me some really helpful comments and trying to coordinate with the ones you just saw. I also
want to emphasize that this is an economist's perspective, so it's my perspective and I'm
probably reflecting ignorance or misunderstanding. Also, they "punted" a few things to me,
and I'm worried now that expectations for this talk are too high, (laughter) I'm not going to
be able to touch on everything.
Slide #2 "Introduction,"
Tape counter starts at 579
I was in the Office of Water for five years, working primarily as an economist on effluent
guidelines, focusing on benefits assessments. I now work in the Office of Research and
Development in the STAR Program, but I do still focus on benefits. I hope I remember
everything correctly from back then, but I may not. I am supposed to talk about a lot of
issues, and some people asked me to talk about a few things, but I'm really going to
emphasize constraints to our analysis and sort of why we do what we do, which maybe isn't
the ideal.
Slide #3 "Context: Tight Deadline and Limited Resources,"
Tape counter starts at 623
The big point, which I'm afraid I'm hammering too much in this talk, is the tight deadlines
and limited resources we have for these sorts of assessments, and Charles and Chris talked
about this. In OW, just as an overall average, we have 473 days from the start of a rule-
making to a proposal, and then 903 more days from the proposal to the final rule
promulgation. I think some of those averages might be a little low, based on including some
simple procedural rules in there, but that's not a lot of time considering that that includes 90
days for the proposal, 90 days for final—for OMB to review the rule, management briefing—
which takes many months because you have to start at the bottom of the chain and go all the
way up the chain and have benefits estimates at the beginning to have every manager aware of
1

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
February 9, 2005
what the benefits are, any other tasks you have to do—including information collection
requests, which are pretty time-consuming. Then after proposal there's a comment period,
and there may be a notice where you talk about what you learned in the comment period. The
reason I highlight that is because it's really hard to get any work done during the comment
period—it's kind of dead time—because the point is you're supposed to wait and see what
people say you did wrong and how stupid you were and then fix it. So, you can't really make
improvements to your models during that period because then someone will say something
that you didn't anticipate.
Slide #4 "Other Hurdles, "
Tape counter starts at 712
A couple of other hurdles that kind of limit the analysis we choose to do are OMB and
management acceptance of what we've done in the past. There is an inertia, I would say. If
you have a type of analysis and you've used a model—you've explained it to OMB—you've
explained it to management—they understand it and they accept it—you're not going to
change it every rulemaking; no one really has time for that. As an example, we had numerous
side meetings with OMB about using NWPCAM in benefits assessments and how to apply
the water quality ladder valuation. Once we did that, there's an implicit agreement that that's
the model we're going to use and that's how we're going to apply that model. So, a lot of
people are reluctant, I think, to change how you do the analysis.
Another constraint is that it's hard for us to use models off the shelf, given our need to add in
the policy contacts, the sources of pollutants, and the benefit endpoints—I'll talk about that in
a little bit.
Slide #5 "Implications,"
Tape counter starts at 782
Implications of these constraints are: We really can only rely on contractors or what we can
do ourselves. However, in any given rulemaking there are far more contractor hours used
than EPA staff hours. Any given EPA staff member working on a rulemaking is supervising
numerous contractors, and any number of staff at the contractors, in doing the work. So,
really, it's the contractors who end up doing the work. Frankly, most people wouldn't put up
with what we ask of them if they weren't being paid in that manner, (laughter) Contractors
are the only people to whom you can say, "Here are the inputs to your analysis—we need it
tomorrow," and they'll do it, but a government employee might not. (laughter) So, "What is
the best model for this application?" is not a question we usually ask or answer—it's not
something we have time to do, but it is obviously something we should be doing. It's easier
for me to say that now, obviously, not that I'm in ORD. (laughter)
2

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
February 9, 2005
Slide #6 "Example—CAFOs, "
Tape counter starts at 844
Let me just give you an example: CAFOs, which was a very high-profile rule, included
among other things a lot of collaboration and consultation with USD A. I don't have files on
this anymore, but we started work on that around early 1998; we proposed in December 2000;
a comment period lasted until July 2001; we published a notice of data availability, talking
about what we learned in the comments; another comment period until January 2002—
another notice—and another comment period and the final rule in February 2003.
Slide #7 "Example—CAFOs (cont), "
Tape counter starts at 885
We did use NWPCAM for the benefits assessment for that. These are just examples, not
really reflective of what we decide for every rule, but we considered using SPARROW for
CAFOs. We thought that given the match of the pollutants in SPARROW and the pollutants
of concern in CAFOs there would be a really good match. At the time we didn't have access
to the most recent version and there was no version available that we could get someone to
run internally, so we really couldn't use that.
We also thought that maybe it would be really good to do a case study of the Chesapeake Bay.
That's an area that's obviously impacted a lot by CAFOs, and it would be really interesting to
policy makers, but again we didn't have a mechanism to get at that model to use in the
timeframes we were interested in.
Slide #8 "Steps in Benefit Analysis (CAFOs), "
Tape counter starts at 937
I don't really need to talk about this much—Chris went over this really well. The reasoning is
that there's just all these steps in the benefits analysis: engineers have to provide the edge-of-
field loadings—they have to get that to a reach, then run the NWPCAM model, which takes
kind of a long turn-around if you're running the full model (the mainframe version—at least it
did—it might not anymore). Then those inputs go to the benefits assessments—we go to
other models sometimes—so there's just always a time crunch. One of the issues is: the
loadings from the engineers come late, and they're probably not right the first time, so as soon
as you get them you have to turn stuff around right away.
Slide #9 "What Models Have We Used?"
Tape counter starts at 981
I apologize that my presentation doesn't flow—I'm supposed to talk about a few different
things. I just want to really set the context of why we choose the models we choose, based on
some tight constraints of time and resources.
3

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
February 9, 2005
There are, I think, five different things that we've used that you could call a water quality
model or attempt at water quality modeling. This reflects what I was able to find for the last 7
or 8 years. I doubt that it's a comprehensive list, but I hope it's pretty close. I put these in the
order I found them, but the dilution models and NWPCAM are probably the two things we've
used the most, and that's why they're highlighted in blue. I don't have a count of "here are all
the rules and here are the models we've used," but I'm fairly confident saying those are the
things that are most prominent in our analyses.
Slide #10 "Proportional Reductions, "
Tape counter starts at 1036
I have "proportional reductions" up there as a model, which I guess it's not really when you
just say that water quality improves in direct proportion to the reduction in loadings. I found
a couple instances where we've done that—in a couple of water quality standards, the
California Toxics Rule, and the Great Lakes Water Quality Standards. They're not national—
they're regional or state-level rules, where we probably had to do something quickly.
To answer one of the questions from the previous session on the water quality standards, the
last I heard the relative importance of water quality standards and TMDLs versus effluent
guidelines—in terms of being written into a permit—is right now about half and half.
However, it's going towards more permits being written to meet water quality standards and
TMDLs and less so for effluent guidelines as time goes on. Half and half is the last number I
remember; I hope it's still right.
Slide #11 "DilutionModels,"
Tape counter starts at 1091
Dilution models are just flow of pollutant divided by flow of the reach. We've tended to use
those most for the effluent guidelines where there are a large number of toxic pollutants—
metal products and machinery, iron and steel, pulp and paper. There's a lot of information
about that in one of the appendices in the background paper about how many pollutants of
concern there were for each rule. For a lot of these rules, we had more than a hundred
pollutants we wanted to try to model. I don't know that dilution models are really all that
accurate, but they're cheap and easy to use, and we could really get the job done with them.
Slide #12 "NWPCAM,"
Tape counter starts at 1128
Chris talked a lot about NWPCAM. We've used it for CAFOs, Storm Water Phase II, Meat
and Poultry Products, Construction and Development (which may not be a complete list). I
doubt that you're familiar with all of those rules, but they're all high-profile rules with lots of
prep work. Except for Meat and Poultry Products, they all dealt with non-point sources, so
4

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
February 9, 2005
we had to put a lot of effort into identifying where those sources were and linking them up to
this model. They also have an emphasis on conventionals, nutrients, and lots of sources, so
there was a decent fit with NWPCAM, which is a national model and can accommodate
including all of these sources. Now, note that this model is expensive and time-consuming,
but it's a lot more complex than the dilution models. It also has the benefit of being accepted
by management and OMB because it was created primarily with EPA funds, and it is run by a
contractor so we do have access to use it. Right now, however, as far as I know, it's only run
at one contractor.
Slide #13 "QUAL2E,"
Tape counter starts at 1183
I don't know a whole lot about this—it was used after I left the division, but Chris talked
about it. As far as I know, it was only used in Aquatic Animal Production. It is an in-house
model and focuses on conventionals and nutrients, so it was a very good fit for that rule.
Slide #14 "Dioxin Reassessment, "
Tape counter starts at 1200
The last one I think we've used is the dioxin reassessment model. The dioxin reassessment
evaluation model is what we called it in the Effluent Guidelines Program. I Googled that
phrase and only "effluent guidelines" came up, so I'm not sure if it has a more-formal name
that you'd be familiar with. We used it in effluent guidelines, especially Pulp and Paper,
where dioxins and furans were really the pollutants of concern. EPA has been reassessing its
treatment of dioxin—it's an ORD project that's not finished yet. So, this was an in-house
model we had access to, and it focused on the major pollutants of concern.
So, if there's a message I'm giving you, it's that the models we've chosen are models we have
an ability to run easily, and that we do try to match them up to our pollutants.
Slide #15 "Other Things I Was Asked to Talk About, "
Tape counter starts at 1247
There are a lot of topics in the background paper, a number of things I was asked to talk
about. Frankly, I don't have a whole lot to say. We usually try to match spatial scales. I
don't think it's that much of a big deal—I think economists can generally handle most spatial
scales from water quality models. NWPCAM and Mitchell-Carson, for example, are both
national and they match up really well.
I can't say that temporal scales match at all. It's not something that we worry about, given
that most models predict an average summer flow or whatever, but benefits vary highly across
different pollutant concentrations as pollutants change over the course of the season. It
matters a lot, and summer pollutants probably matter more than anything else.
5

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
February 9, 2005
Martin asked me about the precision of the models and how that affects benefits estimation.
Until he asked me last week, I had never ever thought about that (laughter), so I don't have
anything to say, but I thought it was important to note that we've never thought about that,
(more laughter) As a couple people mentioned, there are more-stringent requirements to deal
with uncertainty in our estimates, and I think the precision of the models is really going to
matter as we go to a more-complicated uncertainty analysis.
Slide #16 "Case Study Approach, "
Tape counter starts at 1314
Chris mentioned that they used the case study approach in Aquatic Animal Production. We've
often used case studies to verify economic estimates of benefits, but didn't use a more
complicated model. Aquatic Animal Production is an example where we extrapolated from
what we thought was a representative sample of benefit estimates to the national scale. They
also used it in an NPNM to supplement the national estimates, where they used a more
detailed water quality model. I think this is a very promising approach, if there's a tradeoff
between developing a national model and developing accurate regional models, if we can
solve the sampling problem of finding a representative set of case studies of benefits estimates
and using more accurate regional- or local-scale models to do that. Hopefully, that's
something to think about in the next two days.
Slide #17 "Conclusions"
Tape counter starts at 1366
I just wanted to plug something: The Ecological Benefits Assessment Strategic Plan is a
project a large number of us in EPA have been working on as a way to figure out how to do a
better job overall of calculating the ecological benefits of rules, not just water rules but all
rules. One of our big, big recommendations is that different disciplines need to communicate
better and need to start working together earlier in the process, because what we've seen is
that disciplines don't work together—they work parallel—and then kind of get together at the
end.
Really, I'm optimistic. This is a great start, in this area, of doing that sort of thing, and talking
together about what our needs are and what modelers can provide and how to get everything
to mesh. So, that's pretty much it. I just think we should develop more capability and more
models that can be used off the shelf.
6

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
	February 9 2005
Strengths and Weaknesses
of Models Used in Past
Benefit Assessments from
an Economist's Perspective
Introduction
OW Economist for five years
¦	Worked primarily on benefit
assessments for Effluent Guidelines
Now in ORD
¦	Still focus on benefits
I'm supposed to talk about a lot of
issues
¦	Emphasize constraints to analysis
7

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
	February 9 2005
RESEARCH £
DEVELOPMENT
Building a
scientific
foundation
for .sound
envtfpnmentaI
decisions
Context: Tight Deadline
and Limited Resources
OW averages:
¦	473 days from start to proposal
¦	903 days from proposal
This includes:
¦	90 days each for OMB review
¦	Management briefing
¦	All other tasks (Information Collection)
¦	Comments and Notice
Other Hu
OMB arid Management
acceptance
¦	Several side meetings with OMB on
use of NWPCAM in benefits
assessment
Can't use models off the shelf
¦	Need to add sources, benefit
endpoints
8

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
	February 9 2005
RESEARCH £
DEVELOPMENT
Building a
scientific
foundation
for .sound
envtfpnmentaI
decisions
Implications
Reliance on contractor or internal
modeling
But EPA is heavily tilted toward
contractors
So contractors end up doing most of
the work
"What is the best model for this
application?" is not a question we ask
or answer
Example-CAFOs
Selected around January 1998
Proposal in December 2000
¦	Comment period closed July 2001
NODA published November 2001
¦	Comment period closed January 2002
NODA published July 2002
¦	Comment period closed August 2002
Final rule published February 2003
9

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
	February 9 2005
Example-CAFOs
Used NWPCAM for benefits
assessment
Considered SPARROW
¦	At the time, we didn't have access to
most recent version
Considered Chesapeake Bay
case study
¦	Didn't have a mechanism

Bttildinga
scientific
foundation
for sowid
environmental
decisions
Steps in B
Analysis	(C
Model farm edge-of-field loadings
Transport to nearest RF3 reach
Run full model
Benefit Assessment
¦	Water Quality Index
¦	NWPCAM feeds into other analyses
There is ALWAYS a time crunch
10

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
	February 9 2005
RESEARCH £
DEVELOPMENT
Building a
scientific
foundation
for .sound
envtfpnmentaI
decisions
What Models
Used?
Proportional Reductions (Not
modeling)
Dilution Models
NWPCAM
QUAL2E
Dioxin Reassessment Evaluation
Proportional
California Toxics Rule, Great
Lakes Water Quality Standards
¦	Water Quality Standards
¦	Smaller Scale
10
11

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
	February 9 2005
RESEARCH £
DEVELOPMENT
Building a
scientific
foundation
for .sound
envtfpnmentaI
decisions
Dilution
Used for most Effluent Guidelines
with a large number of toxic
pollutants
¦ MP&M, Iron and Steel, Pulp and
Paper
Not accurate, but cheap and easy
to use
NWPCAM
Used for CAFOs, Stormwater Phase II,
Meat and Poultry Products,
Construction and Development
¦	High profile rules, lots of prep work
¦	Emphasis on conventionals, nutrients, lots
of sources
Expensive and time-consuming, but
more complex
Accepted by management, OMB
Run by a (single) contractor
12
12

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
February 9. 2005
QUAL2E
Used only in Aquatic Animal
Production
In-house (EPA) Model
Focus on conventionals, nutrients
Dioxin	Reasses
Used in certain effluent guidelines
¦ Pulp and Paper
Result of EPA's reassessment
In-house model
Focus on major pollutant(s) of
concern
14
13

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
	February 9 2005
Building a
scientific
foundation
for .sound
envtfpnmentaI
decisions
Other Things
Asked to Talk About
Spatial Scales
¦	Often match (NWPCAM and Mitchell-
Carson), probably not a big deal
Temporal Scales
¦	Don't really match at all
Precision
¦	Uncertainty (new OMB requirements)
15
Bitlldiiiga
scientific:
foundation
for soxmd
environmental
decisions
Case Study
•	Often used to verify economic
estimates of benefits, but not modeling
(e.g. Pulp and Paper)
•	Used in Aquatic Animal Production
•	Used to supplement national estimates
(e.g., Metal Products and Machinery)
¦	Dilution model for national estimates
¦	Simple fate and transport for Ohio
•	Need to solve sampling problem
14

-------
Wheeler: Strengths and Weaknesses of Models Used in Past Benefit Assessments
	February 9 2005
Conclusions
Improved Communication: Ecological
Benefits Assessment Strategic Plan
¦	Improved communication
¦	Disciplines working together earlier in the
process
Develop in-house (EPA or contractor)
capability
More models that can be used off the
shelf
15

-------