EPA Response to Peer Review of BenMAP
BenMAP is the result of years of research and development, and reflects methods based on the
peer-reviewed health and benefits analysis literature. BenMAP is first and foremost a tool for
estimating health and welfare benefits associated with air pollution regulations proposed or
finalized by the U.S. Environmental Protection Agency. Secondary objectives in developing
BenMAP were to provide a well documented, flexible, and relatively easy to use program that
could be shared with the public to enhance the transparency of our complex benefits analyses.
The basis of BenMAP is a damage function approach to estimating changes in the incidence of
health effects and valuing those changes. This approach is consistent with many recently
published health impact and benefit assessments (see for example Banzhaf et al., 2002; Levy et
al., 2001; Kunzli et al., 2000; Levy et al., 1999; and Ostro and Chestnut, 1998). The specific
theoretical approach and underlying inputs to the models (concentration-response functions,
spatial interpolation methods, etc.) have for the most part been reviewed and accepted
individually by both the EPA Science Advisory Board and by the National Academy of Sciences
Committee on Estimating the Health-Risk-Reduction Benefits of Proposed Air Pollution
Regulations. The purpose of the peer review of BenMAP was to provide a critical review of the
specific tool developed to combine the individual data and methodological elements in the
generation and reporting of benefits estimates for national scale benefits analyses. To that end,
EPA requested the peer reviewers to address the following questions:
1. General:
a) Does BenMAP provide a useful and sensible structure for addressing policy
analysis needs?
b) Does it provide adequate flexibility to users for addressing important policy
questions?
c) Are the different components of the model appropriately defined and linked to one
another?
2. Input Databases: Keeping in mind that some of the input databases are fixed in the model and
some
can be adapted or supplemented...
a) Are the fixed input databases appropriately selected and defined using
• Census and projected demographic data?
• Modeled air quality data?
• Baseline incidence data?
b) Are the adaptable input databases appropriately selected and defined for
• Monitored air quality data?
• Concentration-response functions?
• Valuation functions?
3. Exposure Estimation Algorithms: To spatially and temporally align population, air quality, and
incidence data as inputs to the concentration-response functions, BenMAP uses several
estimation and
interpolation algorithms.
a) Are these methods scientifically sound and appropriate for
-------
• interpolating and projecting population estimates for non-Census years?
• estimating population subgroups (e.g., by age, gender, ethnicity, etc. )?
• estimating the spatial distribution of populations and linking them to air quality
grids?
• spatial and temporal interpolation of pollutant monitoring data?
b) Does BenMAP offer an appropriate menu of interpolation options for estimating
exposures?
c) Do these methods provide appropriate inputs for the class of
concentration-response functions allowable in the model?
4. Aggregation and Pooling Methods: BenMAP offers alternative approaches for spatially
aggregating health effects estimates and for pooling separate estimates of health effects for
valuation.
a) Does BenMAP offer an appropriate menu of aggregation and pooling options?
b) Do these methods provide appropriate inputs for the class of valuation functions
allowable in the model?
5. Uncertainty Analysis Methods: BenMap offers options for including and evaluating how
uncertainty regarding (1) C-R relationships and (2) valuation functions affect model outputs?
a) Does BenMAP allow the user to adequately and appropriately specify uncertainty
for these two areas?
b) Are the uncertainty routines properly specified and incorporated in the model?
c) Without greatly complicating the structure of the model, are there additional areas
or types of uncertainty that could or should be incorporated in BenMAP?
6. Report and Mapping Results: BenMAP offers several options for reporting and mapping air
quality, population, incidence, and valuation data and results.
a) Do these options provide an adequate and appropriate framework for displaying
results?
b) Are the results displays appropriately specified and configured to address the
intended uses and analytical needs of BenMAP (as defined above and in the
model documentation)?
7. User Interface and User Guide: BenMAP is a menu driven interactive software tool with
multiple
options and features, as described above.
a) Is the user interface appropriately organized, easy to use, and easy to follow?
Through the user interface, are the options and features well described and easy to
navigate?
b) Is the user guide appropriately organized, easy to use, and easy to follow?
c) Does the user guide appropriately complement and correspond with the user
interface?
d) Does the user guide provide the necessary explanation and background for
installing and running the model, selecting options, and displaying results?
e) Does the user guide adequately explain the model's objectives and the model's
underlying structure, assumptions, data, methods, and routines?
The peer review document is provided on this CD-ROM (BenMAP Peer Review.pdf), along with
EPA's response to the peer review. Two of the reviewers comments were generally favorable,
-------
providing mainly suggestions for improvements to the existing model structure. One reviewer
had more serious criticisms of the basic tenets of the analytical framework underlying the model.
While we appreciate the reviewers detailed critique, we note that the health impact and benefit
analysis framework that underlies BenMAP is based on a substantial body of peer-reviewed
literature. As such, we focus mainly on addressing the comments that pertain to improvements in
the BenMAP software, rather than on those that suggest changes to the underlying analytical
structure of EPA benefits analyses.
EPA is working to revise BenMAP to address many of the suggestions provided by the peer
reviewers. We expect to release a revised version of BenMAP in Fall 2004.
We wish to express thanks to Dr. Alan Krupnick of Resources for the Future, Dr. Nino Kiinzli of
the University of Southern California, and Dr. George Christakos of the University of North
Carolina for their thoughtful and constructive comments and suggestions.
References:
Banzhaf, H. S. D. Burtraw and K. Palmer, Efficient emission fees in the US electricity sector,
Resource and Energy Economics, Volume 26, Issue 3, September 2004, Pages 317-341.
Kunzli, N., R. Kaiser, S. Medina, M. Studnicka, O. Chanel, P. Filliger, M. Herry, F. Horak Jr., V.
Puybonnieux-Texier, P. Quenel, J. Schneider, R. Seethaler, J-C Vergnaud, and H.
Sommer. 2000. Public-health Impact of Outdoor and Traffic-related Air Pollution: A
European Assessment. The Lancet, 356: 795-801.
Levy, J.I., J.K. Hammitt, Y. Yanagisawa, and J.D. Spengler. 1999. Development of a New
Damage Function Model for Power Plants: Methodology and Applications.
Environmental Science and Technology, 33: 4364-4372.
Levy, J.I., T.J. Carrothers, J.T. Tuomisto, J.K. Hammitt, and J.S. Evans. 2001. Assessing the
Public Health Benefits of Reduced Ozone Concentrations. Environmental Health
Perspectives. 109: 1215-1226.
Ostro, B. and L. Chestnut. 1998. Assessing the Health Benefits of Reducing Particulate Matter
Air Pollution in the United States. Environmental Research, Section A, 76: 94-106.
-------
EPA Responses to Peer Review Comments for BenMAP
Acknowledgment
EPA would like to thank the three peer reviewers (Dr. Krupnick, Dr. Kunzli, and Dr. Christakos)
for providing a thorough and rigorous assessment of BenMAP and the accompanying manual.
We found the comments to be very helpful in identifying ways in which both the model and
manual can be improved in subsequent versions to improve both ease-of-use and the ability of
BenMAP to support environmental analysis and decision making. We have already incorporated
a number of suggestions made by reviewers into our plans for model enhancement in the near-
term and fully expect to incorporate additional recommendations in later stages of model
development.
1.0 Introduction
This reports presents the EPA's response to the BenMAP (Benefits Mapping and Analysis
Program) Peer Review completed in March, 2004. BenMAP is an economic benefits analysis
program designed to estimate monetized benefits associated with changes in ambient air quality.
The model was developed by EPA to support regulatory development within EPA and as a tool
for broader applications both within and outside of Agency including a variety of environmental
planning and analysis applications. As currently configured, the model can generate both health
effects incidence estimates and monetized benefits for changes in ambient ozone and particulate
matter. However, the model may be expanded in the future to support a wider spectrum of
benefit analysis beyond criteria air pollutants.
Currently, EPA is funding the development of the next version of BenMAP (BenMAP Version
2.0) with a estimated fall/winter 2004 release date. The next version will have a number of
enhancements, many of which will directly address comments raised in this peer review.
Enhancements for BenMAP Version 2.0 include:
• Enhanced flexibility for the user to model custom study areas (e.g., specific urban centers,
regions or nations) and input the GIS coverages and datasets required to model these
study areas.
• Introduction of the concept of a "user-defined setup" within BenMAP that allows the user
to save all of the configuration settings associated with a particular study area including
specifications of key input datasets (e.g., the demographic data, health
incidence/prevalence data, air quality data, GIS coverages defining the study area). The
user can then easily access setups associated with a given study area or analysis in order
to analyze, QA, or rerun those analyses.
• Modifications to BenMAP's graphic user interface (GUI) to make the GUI more intuitive
and guide the user in specifying the "setup" mentioned above (i.e., inputting custom
datasets to support modeling of user-defined study areas).
-------
• Use of a relational database for storing the variety of data required by BenMAP (e.g., air
quality, demographic, health incidence/prevalence, health impact functions, valuation
functions). Use of a single relational database with multiple tables for individual data
elements, enhances the robustness of the data handling within BenMAP and reduces the
potential for introducing error.
• Expanding BenMAP to support multi-pollutant health impact assessment and benefits
analysis.
EPA will also be developing a new manual for BenMAP Version 2.0, which will expand on the
original manual in a number of ways: (a) new set of chapters discussing key modeling steps
within BenMAP, including key caveats and limitations associated with the model, (c) inclusion
of case study examples designed to illustrate specific types of analyses and related technical
issues, and (d) inclusion of an expanded set of appendices to the manual designed to provide
improved coverage of detailed technical issues related to BenMAP (e.g., development of effects
estimates from epidemiology studies, predicting population growth over the next 1-2 decades).
EPA is also planning to develop case study datasets for distribution with BenMAP Version 2.0
that will parallel the case studies included in the manual. In this way, users can interact with
BenMAP directly as they review the manual. This should significantly improve the accessibility
and effectiveness of the manual as a training/instructional tool.
This document is organized as follows. Section 2.0 provides a brief summary of comments
received from each of the three reviewers. Then, sections 3.0 through 5.0, discuss comments
from the standpoint of those that are (a) likely to be addressed by the next version of BenMAP
(Section 3.0), (b) potentially addressed in later versions of BenMAP (Section 4.0) and (c)
unlikely to be addressed in the foreseeable future (Section 5.0). In addition, detailed responses to
key comments from individual reviewers are included as Attachment A.
2.0 Brief Summary of Comments Received from Each Reviewer
This section presents a brief overview of comments received from each reviewer and EPA's
response to those comments. The reader is referred to Sections 3.0 through 5.0 and Attachment
A for detailed discussions of EPA's responses to comments provided by the reviewers.
Krupnick
Dr. Krupnick called BenMAP "...astonishing in its comprehensiveness and flexibility," but also
noted that he does "...have a number of concerns about it and suggestions for making it better."
Many of his concerns and suggestions centered around ease of use issues - wanting "...to make
the manual as little needed as possible", and also wanting the text of the manual to be more
extensive on certain topics. Dr. Krupnick also wanted the ability to more easily generate
aggregated benefits distributions, and to test the impact of alternative assumptions on these
benefits.
-------
EPA agrees with many of Dr. Krupnick's suggestions and criticisms of BenMAP and the manual.
Many of his points will be addressed either in near-term enhancements to BenMAP and the
manual or possibly in later versions of the model.
Kunzli
Dr. Kunzli called BenMAP "...very ambitious and both technically and scientifically on a very
high standard". Like Dr. Krupnick, many of Dr. Kunzli's suggestions centered on ease of use
issues - especially Graphic User Interface and User Manual issues. Dr. Kunzli was also quite
concerned that users understand the issues surrounding uncertainty, and the impact of the various
assumptions that are incorporated into the analysis.
Again, EPA agrees with many of the suggestions and criticisms provided by Dr. Kunzli and is
planning to address them either in near-term or possibly longer-term efforts to enhance BenMAP.
Christakos
Dr. Christakos found that "...the BenMAP approach suffers from a number of conceptual,
methodological and practical drawbacks and, thus, it needs a major revision and redesign".
Much of his criticism seems to be based on the assumption that BenMAP is intended as a
"integrated human exposure assessment (IHEA) model" and that therefore, it should utilize
highly refined micro-environmental and compartmental exposure and dose modeling in
estimating exposure and subsequent health impacts. His recommendations include the
estimation of body burdens and their subsequent use in toxicokinetic modeling of pollutant
impacts for individuals comprising modeled populations.
EPA believes that many of the criticisms provided by Dr. Christakos reflect an incorrect
interpretation of the intended scope and purpose of BenMAP. Because the model is intended to
support economic benefits estimation, it needs to generate health effects (disease) incidence
within modeled populations. The type of detailed mechanistic exposure modeling recommended
by Dr. Christakos, while valuable in allowing relative comparison of exposure and risk levels
across segments of study populations, is hampered significantly in generating actual disease
incidence estimates, by a lack of available data in key areas (e.g., concentration-response data
linking internally-modeled target tissue doses to actual probabilities of disease for modeled
individuals and aggregations of modeled individuals - i.e., modeled populations). While
theoretically, incorporating this type of highly resolved mechanistic exposure modeling into
BenMAP could be valuable in enhancing exposure assessment in support of epidemiological
studies, a lack of data prevents these high refined exposure models from being used in generating
disease incidence as part of benefits estimation. Therefore, EPA considers the majority of the
comments submitted by Dr. Christakos regarding the incorporation of IHEA methods and
techniques into BenMAP to have little direct relevance to the current or future model
development effort. However, EPA will continue to track developments in toxicology and
epidemiology and will revise its position on incorporating this type of modeling framework into
BenMAP should the required data become available.
-------
3.0 Peer Review Comments Addressed in the next Version of BenMAP and
Accompanying Manual
As mentioned in Section 1.0, a number of the peer review comments will be addressed by
enhancements and modifications planned for the next version of BenMAP and through the
development of an improved manual. This section identifies specific enhancements to BenMAP
and to the manual that will address key comments made by reviewers.
Several reviewers noted confusion in identifying and understanding user access to datasets used
within BenMAP (e.g., demographic, health incidence/prevalence). The new database structure
and accompanying GUI interface for inputting data will provide the user with ready access to all
datasets used in BenMAP. In addition to having the GUI expanded to support data input by the
user, existing GUIs associated with key modeling steps (e.g., creation of air quality grids,
analysis/pooling and valuation) will be enhanced to improve usability. This should address
reviewers' concerns regarding the "quirkiness" of the model interface such as the automatic
resetting of all configuration choices when the user returned to screens accessed earlier to make
changes. GUI enhancements should also prevent confusion regarding capabilities within
BenMAP (e.g., reviewers overlooking the ability of BenMAP to model monitor rollback
scenarios due to difficulty in finding the appropriate GUI window).
Plans to incorporate multi-pollutant modeling functionality into BenMAP Version 2.1 will
directly address comments from the reviewers. This functionality will allow effects estimates
based on consideration for ambient concentrations of two or more pollutants to be employed.
Many of the reviewer comments addressed either directly, or indirectly, the content and
organization of the BenMAP manual. The new version of the manual (developed to accompany
BenMAP Version 2.1) will be developed in stages with later versions of the manual
incorporating additional case study (tutorial) examples and expanded treatment of technical
issues in the appendices. These additional case study tutorials and technical appendices will be
developed as EPA continues to support outreach and collaborative initiatives involving
BenMAP.
One of the first chapters in the manual will provide an expanded discussion of the intended scope
of the model, including the types of decisions/analyses it is intended to support, the types of
health incidence and valuation metrics it can produce and the conceptual and theoretical
framework underlying the model. Clearly laying out this information will provide users with an
understanding of the appropriate applications and limitations of the model and should avoid
misunderstandings regarding the intended purpose of the model (some of the comments received
during the peer review clearly reflected a misperception regarding the intended scope and
application of the model, as well as the conceptual framework underlying the model - See
Section 4.0).
-------
Reviewers noted that the audience for the manual was not clearly defined, which reduced its
overall effectiveness as a training tool. The new manual will have different sections targeted at
specific user groups. For example, the main set of chapters introducing key modeling steps will
be designed to provide general users with the ability to run BenMAP with an awareness of key
caveats and limitations associated with the model's use. The case study tutorials will be intended
for the general user group, as well as more specialized users, since they will provided an
expanded treatment of specific types of analytical scenarios. Finally, the technical appendices
will be developed to provided a detailed overview of specific topics related to the model and as
such, will be applicable to those either customizing the model, or using it for research-oriented
applications. The manual will also include an expanded discussion of the framework used by
EPA to guide the selection of the set of health impact and valuation functions included with the
model, as well as an expanded treatment of key modeling steps such as analysis and pooling of
incidence and valuation estimates. The treatment of caveats and limitations will be given special
attention throughout the manual to insure that users are aware of contexts in which the model can
generate estimates with significantly increased levels of uncertainty.
Several reviewers requested an expanded discussion of uncertainty and variability, as modeled in
BenMAP. Chapters providing an overview of key modeling steps will include an expanded
discussion of uncertainty/variability characterization in BenMAP including (a) which elements of
both uncertainty and variability are "covered" in the model and which elements are not covered
and (b) how the model generates estimates characterizing the impact of these two phenomena
from both a theoretical and technical standpoint.
Reviewers commented on the complexity associated with configuring (or setting up) BenMAP.
The case study tutorials which will be developed for the new manual (when combined with the
parallel case study datasets) should significantly improve ease of learning for this aspect of the
model. In addition, reviewers noted difficulty in accessing the range of mapping functions in
BenMAP. This functionality will likely be covered in greater detail in both the case study
tutorials and the set of chapters covering modeling steps (a chapter could be dedicated to the
issue of mapping intermediate and final results).
4.0 Peer Review Comments Potentially Addressed in Later (Version 2.0++) Versions of
BenMAP
A number of the peer review comments, although not likely to be addressed in the next version
of BenMAP, would be considered by EPA for later versions of the model (Versions 2.0++) or for
related resource development (e.g., creation of a BenMAP website that could serve as a clearing
house for setups, configurations and function sets developed by users for specific applications).
Key examples of these comments are discussed below.
-------
Several of the reviewers recommended that default configurations and datasets be included with
BenMAP. This would allow users applying the model to evaluate more "standard" scenarios to
do so with considerably less effort. This is especially true for endpoints requiring complex
aggregation and pooling where the potential for introducing errors in significantly increased.
Reviewers noted a lack of coverage for non-EPA selected health impact and valuation functions.
In addition, reviewers recommended that EPA develop a strategy for updating input data,
including functions, as new information becomes available and allowing BenMAP users to have
ready access to those data. EPA will consider the development of a dedicated BenMAP website
which could address both concerns. This would allow sets of non-EPA functions developed by
different users to be posted to the website for access by other users. The website would also
allow EPA to alert users when new "EPA-sanctioned" datasets were available for BenMAP users
and provide a means for users to access those datasets. The website would also provide a means
for posting technical updates related to BenMAP (e.g., additional technical guidance, model
enhancements etc).
For future versions of BenMAP, EPA has considered a number of enhancements to BenMAP that
could significantly improve user interaction and speed training. Key among these is the
development of tool tips (i.e., small windows activated when the mouse is moved over key
portions of the GUI). Several reviewers referenced tool-tips or similar concepts as a means for
conveying important clarifying information to the user. Tool tips could address comments by
several reviewers that they had difficulty when they strayed away from the manual. Specifically,
by incorporating concise information into GUI windows they could provide the user with
immediate access to clarifying information. The development of the tool tips could be based on
feedback from users regarding areas of the GUI that could benefit from additional explanation or
detail.
One reviewer suggested that the treatment of uncertainty and variability within both the model
and manual should be expanded and improved. Specifically, the model should be expanded to
cover additional sources of uncertainty and the relationship between uncertainty and variability
within the modeling framework should be clearly explained. The reviewer also suggested that the
model consider alternate methods and techniques for conducting specific modeling steps (e.g.,
the use of applied mathematical demography methods in projecting population change for future
modeling periods). EPA is currently in the process of developing a rigorous, scientifically-
defensible framework for conducting uncertainty and variability analysis of benefits estimates
generated using BenMAP. This framework will begin with an influence/sensitivity analysis
designed to identify key sources of uncertainty in the overall model. Once these sources of
uncertainty are identified, a probabilistic multi-stage framework will likely be employed to
characterize the impact of those sources of uncertainty (along with variability) on benefits
estimates. This uncertainty/variability analysis framework will be subject to peer review.
Reviewers recommended that BenMAP be capable of generating validity scores for modeled
-------
results. A validity score would be based on an assessment of overall compatibility across
modeling elements used in generating a particular model output (e.g., whether the CR function
used matched with the air concentration metric), combined with an assessment of data quality for
each element. EPA will consider the future addition of data fields in the input parameter
database which could be used to track and evaluate overall data quality and inter-element
compatibility for specific model runs. It might also be possible to develop automated procedures
in BenMAP for combining these quality/compatibility data for a given run to generate a
compatibility score, although this is not an enhancement currently in the planning.
One recommendation involved the integration of micro-environmental exposure modeling into
BenMAP as part of health effects incidence estimation. EPA has began to look at options for
incorporating this type of more detailed exposure modeling into BenMAP (or more specifically,
results from this modeling in the form of spatially-differentiated exposure profiles for modeled
populations). If this were undertaken, then it might be possible to use these more refined
exposure estimates to support health effect incidence estimation, assuming appropriate CR
functions are available. It might also be possible to use micro-environmental exposure modeling
within BenMAP to augment epidemiological analyses of both mortality and morbidity effects for
specific populations (i.e., further differentiate generalized ambient air concentrations to produce
more spatially-resolved exposure profiles for use in epidemiological studies). EPA is currently
looking into these refinements involving micro-environmental modeling, although specific plans
for model enhancements along these lines have not been finalized.
5.0 Peer Review Comments Classified as Not Requiring BenMAP Modification in the
Near-T erm
This section summarizes two categories of comments received from peer reviewers that are not
expected to require any enhancement or modification of BenMAP or the manual: (a) comments
not directly relevant to BenMAP given its intended scope and purpose and the intended scope of
the manual (i.e., the comment is based on an incorrect interpretation of the purpose of the model
and/or the manual), (b) comments suggesting a theoretical or methodological approach not
supported by available data and (c) comments where the reviewer was not aware of key
functionality (i.e., the model can do what the reviewer suggested was missing). Note, that for
"b" above, these comments may actually identify areas where either the GUI, the manual, or both
could be clarified and improved.
One reviewer suggested that BenMAP lacks methodological support in the context of integrated
human exposure assessment (IHEA) since it does not use a holistic approach to health incidence
assessment involving consideration for the duration/frequency of exposure, intake and uptake
pathways and compartmental analysis. EPA acknowledges that, if sufficient data were available,
we might model health disease incidence using detailed mechanistic exposure and risk modeling
(e.g., using microenvironmental exposure modeling, compartmental target-level dose estimation),
although epidemiological-derived effects estimates have many advantages. However, given that
-------
the goal of BenMAP is to estimate disease cases within the population (and not to generate risk
levels that allow relative risk comparisons across chemicals or sources, but do not support true
incidence estimation), incorporation of detailed mechanistic compartmental exposure and risk
modeling into BenMAP at this point, would yield little utility in improving benefits assessment.
One reviewer suggested that the manual does not provide sufficient discussion of the link
between theory and application in relation to health incidence estimation. Specifically, the
manual does not provide sufficient detail on the relationship between formal (math theory and
techniques) and the interpretive (physical meaning of the math terms and justification of the
formal assumptions). They assert that this can prevent the user from gaining important insights
between formal methods and experience. EPA recognizes the importance of having users
(especially more intensive research-level users) develop this level of critical understanding of the
analytical framework reflected in BenMAP. However, the manual is intended to guide users in
learning how to run BenMAP, including key caveats and limitations and is not intended to
provide this level of in-depth background information. The user is referred to a broad body of
literature related to benefits analysis and risk assessment for ambient air quality to gain a deeper
understanding on theoretical and conceptual issues related to the modeling approach used in
BenMAP.
Several comments were received regarding the geostatistical (kriging) functionality within
BenMAP. EPA has decided not to pursue development of a dedicated kriging functionality
within BenMAP since there are a number of geostatistical programs already available. Users can
employ geostatistical packages to develop air quality grids outside of BenMAP and then import
those grids (as model direct grids) into BenMAP.
-------
Attachment A: DETAILED RESPONSES
This Attachment provides detailed responses to specific comments made by peer reviewers.
These responses are organized by commentor. Our responses can be divided generally into
several categories:
" Agreement with the reviewer that a particular enhancement would be beneficial. In these
cases, the possible modification to the model or manual is classified as either "shorter-term"
(likely to be implemented in the near-term through modifications in the next version of the
model) or "longer-term" (EPA supports the enhancement but does not have immediate plans for
its implementation).
" Clarification of model functionality based on a perceived misperception by the reviewer
(e.g., they noted an absence of functionality which is actually present). Note, in many of these
cases, misperception or confusion on the part of the reviewer is important in pointing out areas
where we can improve clarity and user-accessibility in the model or manual.
" Determination that we will not be addressing the comment in the form of model or
manual modification.
A. Responses to Krupnick
la) Does not support linking of air concentrations back to emissions sources: You could run
BenMAP for individual source categories (or even for air concentration coverages reflecting
individual sources) to estimate benefits associated with each source category/source.
Longer-term: allow multiple air concentration (delta) coverages, each representing a different
source, to be used simultaneously in BenMAP to track total and source-specific benefits.
la) Lack of multi-pollutant modeling: Shorter-term: augmentation to allow multi-pollutant
modeling (with separate air concentration coverages for each pollutant). BenMAP CR functions
could also be entered to work with air concentrations for specific chemicals in generating
cumulative (cross-chemical) incidence estimates. Of course, this would require epi or tox study
data that could support this level of detailed multi-chemical analysis.
lb) Lack of flexibility in selecting and de-selecting CR functions and seeing resulting impact on
overall results (i.e., conducting sensitivity analysis): The user does have the ability to run
multiple pooling/aggregation schemes combined with multiple studies for the same endpoint in
order to examine the sensitivity of overall results to CR functions. Once run in this fashion, the
user could export to full suite of results to a database such as Access where various comparison
of results based on different combinations of CR functions and aggregation/pooling
configurations could be rapidly assessed.
-------
lb) Lack of roll-back capability: The current model does have this capability, but the appropriate
window is not activated at the optimal time (i.e., it doesn't show up until you have activated a
region on the map for the rollback, which can be confusing). Shorter-term: modify the GUI to
have the roll-back window appear as soon as the "Monitor Rollback Screen" is activated (but be
grayed-out until a region is selected by the user).
2a) Population and incidence datasets could be accessible to users: These datasets are accessible
within the "data" subdirectory. However, EPA agrees that it would be useful to provide users
with access to these data just as they have access to the CR and valuation functions through a
more direct GUI-accessed avenue. This could be a shorter-term modification (adding these
options to the first GUI screen).
2b) Incomplete coverage for non-EPA valuation methods (VSLY): Shorter - to Longer-term: we
could consider stratifying the functions (valuation and incidence) into two distinct groupings.
First, we would have a set intended to support EPA-type cost-benefit analysis. This set would
not include types of functions explicitly excluded/removed from EPA protocol. Then there
would be a second set of functions ("research library" or some such) which would have a more
complete set. Ultimately, we could have a website where users can post functions that they
develop - this might represent a third (open source) grouping of functions.
2b) Structure of CR-function database tree (lack of ozone/PM differentiation and clutter
associated with age-group differentiation): Shorter-term: We could tighten up the structure of the
CR and valuation trees to include the key constituent (ozone/PM) attribute and organize the
functions accordingly. We could also provide a collapsed node for age group and any other
classifier that results in multiple related branchings (e.g., multi-pollutant versions of functions).
The user would then click on these collapsed nodes to access detailed specifications of the
functions. These latter modifications involving collapsing of multiple-branch hubs (e.g., age
ranges) might represent a longer-term modification.
2b) Need additional detail on how CR functions were selected for BenMAP. In addition, the
commentor asserts that the list of functions may be incomplete: Shorter-term: We could provide
a detailed step-wise description of the procedure used in selecting CR (and valuation) functions
including the specific criteria used.
2b) Request for examples of how incidence and prevalence are used: Longer-term: An
overarching response to many of the comments provided by the commentors is that we should
include some form of tooltips (i.e., comments imbedded in the GUI that are activated as the user
moves their cursor over key elements of the GUI). These tooltips could cover much of the
clarifying information currently provided in User Manual. In this specific context, incidence and
prevalence cells in the CR Function GUI could have tool tip links that pull up brief descriptions
and possibly even examples of how each type of data is used in incidence estimation.
-------
4a) Issue of potential double-counting of results through pooling - possible inclusion of a default
setting for pooling and inclusion of a example demonstrating how the pooling is implemented:
Shorter-term: We may want to consider developing one or more default EPA pooling and
aggregation schemes as part of an overall default EPA benefits estimation routine. These default
EPA routines could be based on a recent RIA strategy that has been publicly released. Not only
would this routine provide an example of pooling and aggregation, it would also show how
multiple studies were used to provide coverage for specific mortality and morbidity categories.
We could include dedicated text in the form of a cast study example that would describe key
elements of each of these default routines (this would allow us also to introduce and demonstrate
key BenMAP functionality).
5c) Generation of final distributions reflecting uncertainty in incidence and valuation functions
through use of ADD SUM button not clearly described in GUI: Longer-term: This issue could be
addressed through the use of the tool tip concept described earlier. Specifically, text explaining
the functionality of the ADD SUM button could be accessed through a pop-up tool tip. In
addition, we may want to consider using some form of standardized coding to identify elements
across the various GUI screens that are associated with uncertainty characterization (and other
categories of analysis for that matter). For example, a small "UA" (presented in a bolded colored
font) could appear as a superscript next for all GUI windows with functionality related to
uncertainty analysis (e.g., ADD SUMS, Latin Hypercube specification etc). This would help the
user identify modeling elements related to this important topic.
5c) Additional sources of uncertainty could be addressed (e.g., demographics, background
incidence): We are currently in the process of developing a comprehensive approach for
characterizing uncertainty in our benefits estimates generated using BenMAP. This methodology
will include an influence analysis which will consider specific sources of uncertainty in each
stage of the modeling process (e.g., emissions characterization, air quality modeling, incidence
estimation and valuation). Based on the results of the influence analysis (and associated
sensitivity analyses), a probabilistic framework will be developed for evaluating the impact of
key sources of uncertainty on model results. Potential sources of uncertainty referenced by the
commentor will be considered in developing this methodology.
6a) Difficulty accessing and displaying data on population and air quality: A longer-term model
refinement could involve a GUI screen that displays in greater detail the specific analytical steps
(including key data sets such as population and incidence) involved in benefits estimation. The
user could click on database icons associated with each analytical step to access the relevant
dataset. This GUI design would have the advantage of clearly identifying the analytical
framework and the position of specific datasets within that framework.
7a) The setup procedure is intimidating and not that helpful: Longer-term: inclusion of tool tips
would likely help with this. In addition, development of a GUI (as described in 6a above) that
provides a more detailed but comprehensive picture of the modeling framework and where
-------
specific analytical steps and datasets come into play, might help with the accessibility of the
model.
7b) Compliments on the manual, but he encountered problems when he tried to stray from the
tutorial and especially in completing various mapping steps: Shorter-term: The manual could be
expanded to include a number of tutorials (case studies) designed to illustrate specific functional
elements of BenMAP (including mapping). Tutorials could be designed around anticipated
application contexts (e.g., a regulator who wants to run a fairly standard benefits assessment,
versus a researcher who wants to conduct a range of different sensitivity analyses using the
model).
7c) The use guide is essential for learning to use the model, as compared to friendlier software
where the manual is more of a backup: The process of generating benefits estimates is
complicated and it would be difficult to design a benefits modeling program and GUI that did not
require the user to complete a tutorial to learn fundamental elements of the analytical/modeling
approach and framework. The process of gaining familiarity with the modeling framework might
be reduced somewhat if the more detailed step-wise description of the analytical framework was
incorporated into the GUI. This longer-term augmentation, would allow the user to see the
overall relationship of key sub-elements in the model. But it is likely that some form of tutorial
would still be required to give the user sufficient knowledge and perspective on the modeling
approach to use BenMAP.
7e) Not clear which audience(s) the manual is intended for. Once the intended audience(s) is
identified, it would be possible to tailor the manual accordingly: Shorter- to Longer-term: The
manual could be reorganized to include a "general approach" section and an "advanced methods"
section, each intended for a different audience (i.e., those using BenMAP in straight-forward
regulatory/decision support contexts and those wanting to tailor their analyses for specific
applications or in order to support benefits analysis research, accordingly). Then, case studies
and tutorials could be designed for each of these "sections" of the manual.
B. Responses to Kunzli
la) BenMAP is an ambitious collection of data sets and scientific issues. In that context, it is
important to be aware of and support different categories of users with the manual (e.g., those
using the more complex functionality for purposes of research versus those wanting to run
standard default-type analyses). Therefore, it might be useful to include a set of standardized
default scenarios for users conducting more routine analyses: A useful refinement for BenMAP
(both the manual and the program) would be to more explicitly define and separate modeling
elements and functionality related to these two broad categories of BenMAP application (i.e.,
standard-default regulatory analyses and more intensive research initiatives). This could take the
form of several augmentations including: (a) shorter-term modification: inclusion of case
study/tutorials for both the standard/default and intensive/technical BenMAP applications (these
-------
tutorials would each cover individual or logically grouped functional elements associated with
these two types of BenMAP applications), (b) shorter-term: separation of key datasets into two
distinct categories (standard EPA-default and expanded research-oriented) - this is certainly
applicable in the context of the incidence and valuation functions, and (c) in the longer-term,
consider re-designing the BenMAP GUI to explicitly separate out the more intensive/research
type functionality from steps required for standard default-type analyses. Note, some of these
points have already been made in response to Krupnick's comments.
2a and b) It would be useful to develop a strategy for updating BenMAP datasets as new data
become available: Shorter- to longer-term: a website could be set up that posts the most
up-to-date datasets for supporting default EPA-type BenMAP assessments. The user could then
check the website for updates on key input datasets used in BenMAP. A different portion of the
website could post additional "research" data sets that include a wider variety of valuation and
incidence functions and other types of data sets (e.g., krigged air concentration coverages
developed for special applications, different types of demographic and incidence projection data
sets etc). The default EPA content would be controlled by EPA, while the other "research" area
could either be left fairly open for posting, or controlled more loosely.
2a and b) The protocol for adding new studies and determining when older studies are "dropped"
from the database should be clearly defined (similar to Krupnick 2b): Shorter-term: we could
clearly outline our protocol for completing this process and include it more explicitly in the
manual. Note, a different protocol could be used for the EPA-standard versus Research versions
of the datasets, if we were to implement that stratification.
2a and b) It is advantageous to combine data inputs that are compatible in terms of date and other
key attributes. In this context, it is difficult to understand the "validity score" of all the input
options: All of the information necessary for evaluating the mix of studies and data used in
generating a particular benefits estimate is currently available within BenMAP. The user has
access to these data through the audit trail function. Development of a true validity score based
on the attributes of studies and data used in a particular benefits assessment requires a
combination of scientific analysis and professional judgment. Given the semi-subjective nature
of this process, at this stage, we would prefer to leave the process of developing and presenting a
validity score in the hands of the user and not include a specific framework for implementing this
within BenMAP. Note, however that a longer-term augmentation would be to provide a concise
summary of analysis attributes associated with each modeled endpoint - these could then be
readily used to construct a validity score, or to otherwise assess potential uncertainty introduced
through choices made in linking data sets and incidence/valuation functions.
3a) Insufficient caveats provided to user regarding potential combinations of modeling elements
within BenMAP (e.g., generating estimates at large scale based on epi studies conducted at small
spatial scale): Although inclusion of validity scores is not advisable at this stage (see above), it
would be possible to in the longer-term include some standard "caveats" or "warning messages"
-------
for estimates generated using non-optimal combinations of input parameters (e.g., mismatch in
scale of analysis versus scale of underlying studies providing incidence functions, extrapolation
of functions to cover ages groups or constituents that are different from those considered in the
original analysis). Regarding the specific issue of spatial scale in benefits assessment and the
uncertainty introduced through modeling using different spatial templates, EPA is currently
conducting research on this topic and will reflect findings in future versions of BenMAP
(including both the model and documentation, as appropriate).
4a) The process of pooling is complex especially for certain effects categories. A default pooling
structure (guidance) would be helpful: If we develop a set of standard default EPA configurations
(shorter-term augmentation), these would by definition include pooling structures for key
endpoints. In addition, the BenMAP manual could be expanded to include one or more tutorials
focusing specifically on pooling and aggregation. This tutorial could include, or be paired to a
discussion (brief) of technical issues related to pooling.
5a) The manual may give an insufficient perspective on uncertainties: EPA is currently working
to develop a framework for evaluating uncertainty associated with benefits estimates generated
using BenMAP (this is likely a shorter- to longer-term enhancement). Once that framework is
developed, the manual will be updated to reflect this new capability within BenMAP, including
both the technical background/overview of the uncertainty characterization framework, as well as
guidance on how to implement it to generate confidence intervals for incidence and valuation
estimates.
6b) It would be helpful to have ready access to all input data in mappable format (e.g., including
incidence estimates): As discussed earlier, a longer-term enhancement could include a more
detailed step-wise description of the analytical structure within the main (first) GUI screen. In
addition to having hot link buttons for accessing key data sets, it would also be possible to have
buttons for accessing mapping functions for specific datasets of interest.
7a) Manual gray boxes could be "pop-ups" in BenMAP: We could certainly incorporate this type
of information as pop-up boxes (or tool tips) depending on the amount of information that is to
be displayed (more pop-ups, less for tool tips). This would likely be a longer-term enhancement.
7a) Several limitations/quirks associated with setting up BenMAP configurations are noted by
the commentator (e.g., inability to step back through set-up steps without resetting entire portion
of the BenMAP setup sequence): Shorter-term modifications to the model could allow for more
user-friendly parameterization of the model (e.g., ability to revisit earlier screens without losing
options specified by the user in those screens).
7e) Could provide default options with discussion of their design and strengths/weaknesses.
Could also provide a few examples of different setups, illustrating the impact that these choices
-------
can have on results: Shorter-term augmentation would involve development of default
configuration files that would be included with BenMAP. Would include discussion of strengths
and weaknesses of these default configurations in the manual. Could also include several
examples of different configurations illustrating how results differ depending on design of
configuration files. Longer-term modification could include (a) redesigning BenMAP GUI to
support distinct tasks of applying default/standard configurations and designing and
implementing customized configurations and (b) inclusion of dynamic linking system that builds
caveats for specific benefits categories depending on the analytical framework used to derive
those estimates (e.g., caveats regarding age of the study, application of functions to non-study age
groups, application of functions to different category of pollutant etc.).
C. Responses to Christakos
Part 1: Summary Comments
1) Methodology: BenMAP lacks a methodological support in the context of integrated human
exposure assessment (IHEA). BenMAP does not demonstrate an understanding of the holistic
nature of IHEA, including the importance of the epistemic process in this type of model
development.
The Manual for BenMAP focuses on providing the user with an introduction to the functionality
of the model and places less emphasis on laying out the conceptual and methodological
foundations for BenMAP. However, EPA recognizes that presenting a description of the
conceptual framework (including both its conceptual and methodological foundations) would
help users engaged in more sophisticated research using BenMAP, gain a better understanding of
the overall model.
Having said that, it is important to point out that BenMAP uses an analytical framework for
generating incidence estimates that is the product of an extensive and structured development
process involving continuous refinement by both EPA scientists and outside peer-reviewers (e.g.,
SAB-HES). In developing this analytical framework, EPA scientists have been careful to match
the theoretical and conceptual basis for the model to the specific types of health assessments that
BenMAP will need to support. The current version of BenMAP is designed to evaluate health
impacts resulting from criteria pollutant exposure (specifically PM and ozone). Therefore, the
conceptual framework for the model reflects an emphasis on modeling morbidity and mortality
using concentration-response (CR) functions developed from both time-series and
chronic-duration epidemiology studies. This focus on epidemiologically-derived CR functions
explicitly determines the analytical structure for up-stream elements of the overall modeling
framework, including both the air modeling that is required, as well as the demographic and
baseline disease incidence and prevalence data that are used. Specifically, the spatial and
temporal scale underlying the epidemiological studies necessitates a matching degree of spatial
and temporal precision in the various input datasets required as inputs to those functions. In
-------
those cases where mismatches in spatial or temporal precision exist, uncertainty may be
introduced.
Although the analytical and conceptual framework utilized in BenMAP does not currently
emphasis probabilistic risk assessment, should BenMAP be expanded in the future to provide
cover for individual chemical-level population risk estimation, elements of this paradigm will be
incorporated in a structured manner. Depending on data availability (a key factor to consider in
translating a conceptual framework into an operationalized analytical framework), EPA will
certainly consider the use of advanced methods including (a) the "modeled individual approach"
towards exposure assessment which utilizes activity diaries and simulated mobility to develop
exposure profiles for sets of modeled individuals, and (c) multi-compartmental modeling of
internal tissue-level doses. However, it is important to reiterate that the applicability of these
methods in the context of estimating disease incidence for a specific chemical will be dictated by
available data. While it is certainly possible to describe an optimal analytical framework that
includes sophisticated and innovative techniques, if the information required to "drive" those
techniques is not available, then that exercise remains strictly academic. While EPA constantly
monitors cutting-edge research and, in the context of benefits estimation, is often at the forefront
of that research, we are also careful to match model enhancement the types of data that are
currently available, or likely to be in the near future.
EPA recognizes the importance of the epistemic process in model development. If a model or
conceptual framework is to be developed and refined in a structured and efficient manner, then
that process needs to be conducted in accord with epistemic principles (i.e., the scientific basis
and evidence for all key assumptions and modeling elements needs to be clearly documented and
evaluated). Note, however, that it is important in addressing the epistemological issue, to avoid
the pitfall of the infinite regress, which can paralyze model development. Although not
specifically referenced by EPA as reflecting an epistemic process, we are currently conducting a
comprehensive sensitivity and uncertainty analysis of the analytical framework used in BenMAP.
An important component of this process is the identification of critical inputs and assumptions
associated with each component of the analytical framework. Once all of the critical
inputs/assumptions are documented along with their associated rationales, then an influence
analysis will be conducted to identify key conceptual and mechanistic dependencies and
relationships between various modeling elements. Underlying this entire process will be a
comprehensive review of the conceptual basis underpinning the entire analytical framework.
This step is important to guarantee that the macro architecture of the model is matched to the
types of analyses being conducted, especially if BenMAP is enhanced to cover different
categories of chemical exposure-related risk.
2) Internal Consistency: BenMAP (assuming that the commentator is actually referring to the
manual) does not provide an adequate consideration for the internal consistency of between the
formal (mathematical theory, techniques) and interpretive (physical meaning of the mathematical
terms, justification of the formal assumptions). This prevents the user from gaining important
insights regarding the connection between of the formal methods and experience (including
-------
physically testable hypothesis).
As described above under #1, the BenMAP manual is intended to provide the user with an
introduction to the functionality and use of the model. As currently drafted, it does not provide a
comprehensive treatment of issues related to the conceptual underpinnings of the model. EPA
agrees that expansion of the manual to provide a formal overview of the conceptual framework
of the model, including the "internal consistency" issue discussed by the commentator would be
useful and would aid users engaging in more sophisticated research-oriented applications of the
model.
It is important to note that this issue of internal consistency and the relation between the
conceptual framework used in the model and the specific types of results that are generated has
received significant attention from the EPA development team. Often, data limitations (e.g.,
mismatches in the temporal and spatial scale of modeled air concentrations in relation to the
epidemiological studies providing the CR functions) necessitate that an optimized conceptual
framework be stretched slightly to allow the generation of quantitative incidence estimates. In
these cases, while internal consistency may have been achieved in the theoretical structure/design
of the modeling framework, data limitations have necessitated the use of non-optimal data that
does compromise this internal consistency to some extent. In these situations, the ability to
conduct quantitative sensitivity and uncertainty analyses to evaluate the degree of error
potentially introduced by these divergences from the optimal design, is critical in providing
decision makers with a complete picture of health impact for a study area. This is part of the
impetus for the current efforts to develop a quantitative uncertainty analysis methodology for
BenMAP.
3) Exposure estimation/interpolation: Range of criticism for our approach (and lack thereof)
regarding geostatistical interpolation (kriging). Includes criticism that we (a) do not address the
composite space-time nature of inhalation health impacts sufficiently, (b) do not consider more
detailed micro-environmental exposure, and (c) use a simplistic annual average generation
approach.
The EPA development team has decided to de-couple all geostatistical interpolation functionality
from BenMAP. Rather than providing kriging capability within BenMAP, users will be able to
use readily available software to derive their own krigged surfaces which they can then import
into BenMAP. This decision was made for two reasons. First, application of kriging to derive
air concentration surfaces requires expertise in both spatial statistics and air quality modeling, in
order to avoid introduction of significant error into analyses. Many users will not have this type
of expertise and inclusion of kriging software within BenMAP could encourage misuse of the
software and introduction of additional uncertainty into BenMAP output. Second, there are a
number of software packages that support geostatistical interpolation, including both dedicated
packages and GIS software with add-on capability. Because of the complexity associated with
developing code capable of supporting more advanced kriging methods (e.g., co-kriging,
-------
Bayesian maximum entropy), it was decided to fall back on existing software for this capability
and not expend significant resources duplicating this functionality within BenMAP.
Although kriging functionality will not be incorporated into BenMAP, the EPA development
team fully expects that geostatistical techniques will play an increasingly important role in
supporting benefits assessment conducted using BenMAP. Research at EPA to identify optimal
strategies for interpolating air quality surfaces is ongoing. This research is addressing a number
of technical issues including: (a) efficacy of different techniques (e.g., covariance and trend
extensions in kriging, incorporation of Bayesian methods) given spatial and temporal variability
in air quality data, (b) methods for combining empirical monitored data and modeled data (e.g.,
co-kriging, Bayesian maximum entropy) in conducting interpolation, and (c) methods for
evaluating kriging techniques (e.g., single and multiple variogram cross-validation, orthonormal
residuals) and their ability to handle such challenges as clustering of air monitors.
As methods and strategies for interpolating air quality surfaces are developed, the BenMAP
manual will be updated to reflect the current state of the science regarding this application of
geostatistical interpolation. However, as noted above, we do not anticipate developing new code
to support geostatistical interpolation functionality within BenMAP.
Regarding the use of refined exposure assessment methods that take into account such factors as
micro-environmental exposure and cross-correlations and dependencies between spatial and
temporal exposure processes, an important factor to consider in using more refined exposure
assessment modeling is the nature of the concentration-response (CR) functions used in
generating incidence estimates. It is important to match the level of precision and specificity of
exposure assessment with the corresponding level of resolution in the epidemiology studies
providing the CR functions. Generally, for both mortality and morbidity, CR functions are based
on ecological and/or cohort epidemiology studies that do not include highly refined assessment
of individual-level exposures (e.g., micro-environmental tracking or modeling of exposure
levels). Typically, these studies use generalized exposure estimates based on proximity to
monitoring stations. Therefore, incorporation of detailed micro-environmental exposure
modeling based on activity diary-based generation of exposure profiles for sets of modeled
individuals, could result in a mismatch between exposure assessment and concentration-response
assessment methods. This would introduce the potential for bias resulting from ecological
fallacy-related errors. It is important to note, however that the EPA is looking into the potential
for incorporating detailed modeled exposure profile data (e.g., activity diary-based
microenvironmental model based results) into BenMAP as a means for supporting enhanced
epidemiological analysis. In this context, detailed exposure profiles could be used to supplement
ambient monitor-based metrics as a means for assessing cohort exposure and deriving CR
functions. In addition, the EPA research team will continuing to monitor research in
epidemiology, toxicology, exposure assessment and risk assessment related to air quality-related
health impacts. Should new data supporting a more refined characterization of individual-level
(or small group-level) exposure become available, we will update our methods for benefits
assessment accordingly.
-------
4) Criticism of the uncertainty characterization methodology used in BenMAP including (a) the
need to consider uncertainty from an epistemic viewpoint and treat it as more than simply
variability between studies, (b) recognition that variability between studies can reflect a number
of factors including both uncertainty and variability, (c) the Latin Hypercube sampling technique
is simplistic and does not consider incidence changes at the grid cell and combined effects of
exposure parameters acting in synergy, (d) need to evaluate uncertainty in the analytical function
for expected health effects (reflecting exposure parameters through CR functions and
toxicokinetics).
As already noted, the BenMAP manual was not intended to provide a rigorous description of the
theoretical and methodological underpinnings of the analytical framework used to generate
benefits estimates. Instead, it was designed to provide guidance in using the model to generate
estimates. However, the manual could be enhanced in the future to provide a more rigorous
description of the analytical framework, which would be useful in supporting more sophisticated
research-related applications of BenMAP.
At present, BenMAP includes capability to handle aspects of uncertainty in two components of
the benefits analysis framework: (a) uncertainty resulting from the availability of multiple studies
characterizing exposure-response relationships for a given health endpoint and (b) uncertainty
resulting from multiple studies characterizing willingness to pay (WTP) to avoid the risk of
specific health effects including statistical mortality incidence.
With regard to incidence estimation, BenMAP allows the user to evaluate the impact of statistical
sampling error associated with the effects estimates from a specific study, as well as "study
variability" when more than one study exists for a given health endpoint. In both cases, BenMAP
is actually evaluating the aggregate impact of sources of uncertainty and variability on incidence
estimation. With the effects estimate, a study typically provides a confidence interval reflecting
the goodness of fit of the regression model to the available data. That confidence interval likely
reflects a combination of inter-group variability (to the extent that it can be discerned in the
ecological or cohort study) as well as uncertainty resulting from multiple factors including failure
to fully consider confounding and effects modification. While it may not be possible with many
of the CR functions to explicitly differentiate between specific sources of uncertainty and
variability reflected in the confidence intervals, the EPA research team is aware of the
importance of recognizing and tracking these different sources. Having a clear handle on
different sources of uncertainty and variability impacting CR functions can be important in
guiding and tracking ongoing research to improve our ability to model incidence.
With regard to "study variability", BenMAP allows the pooling of incidence estimates from
different studies using either fixed effects or random effects methods (user-specific weights can
also be used, but are of less interest in the context of this discussion). These pooling techniques
are based on the assumption that variance in the CR functions (i.e., the size of the confidence
interval) is associated with overall confidence (or conversely uncertainty) in the study findings.
-------
Therefore, incidence estimates with larger confidence intervals will be given less weight than
incidence estimates with tighter confidence intervals. While this approach certainly has
statistical merit and is generally defensible from a theoretical standpoint, we recognize that it
does not reflect a rigorous epistemological treatment of uncertainty. In other words, it does not
represent an effort to systematically reflect the combined impact of multiple "up stream" sources
of uncertainty and variability on incidence estimates. Therefore, EPA has initiated the
development of a comprehensive uncertainty characterization methodology for its benefits
estimation framework. This methodology will use an influence analysis to identify elements of
the analytical framework that contribute significant uncertainty and/or variability to benefits
estimates. A Monte Carlo-based probabilistic framework will then be developed to propagate the
aggregated impact of these sources of uncertainty and variability on both incidence and benefits
estimates. This framework will likely incorporate a nested structure to allow the separate (and
combined) impact of uncertain and variable sources to be evaluated. Depending on technical
limitations (i.e., run time and data volume issues), it maybe possible to implement this
comprehensive uncertainty characterization methodology at the cell-level (i.e., generating
distributions characterizing uncertainty and/or variability for each cell, prior to study area-level
aggregation of incidence and valuation estimates). However, practical considerations may
require that the probabilistic uncertainty and variability analysis be implemented at a level of
spatial aggregation above that of the individual cell. We believe that, once implemented, this
uncertainty characterization methodology will address many, if not all, of the points raised by the
reviewer.
With regard to uncertainty in the valuation (WTP) component of the benefits analysis framework,
most of the points made above in relation to incidence estimation also apply to the valuation
estimates. Similarly, the comprehensive uncertainty characterization framework discussed above
will include coverage for aspects of the model used in valuation.
5) Estimation of adverse health effects: The discussion of adverse health effects in the manual is
uneven. Specific criticisms include: (a) use of CR functions based on black box associations
between central-site exposure measurements and population-wide health endpoints, (b) no
discussion of physiology-based toxicokinetic models and inclusion of biomarker modeling in
BenMAP, (c) BenMAP includes a large set of CR functions without identifying a list of the
"most significant" functions including critical evaluation, their relative advantages and
disadvantages and insightful recommendations regarding their use, (d) scale effects in applying
the CR functions at different scales is not addressed, (e) additional health impact factors such as
duration/frequency of exposure, intake and uptake pathways and compartmental analysis are not
considered, (f) non-science based approach used for population projections (need to use applied
mathematical demography methods), and (g) need to consider susceptible subgroups (asthmatics,
age, disease status) and the fact that demographic mix of the entire population vis a vie these
subgroups will change over time.
Regarding criticism of our use of CR functions based on ecological/cohort epidemiology studies
(i.e., studies relating central-site monitors to population-wide health effects), we would like to
-------
address this issue separately in relation to chronic and acute health effects. First, we believe that,
in the case of chronic PM-related health effects, there is no viable alternative to ecological
epidemiological studies for generating CR functions at this time. While it is certainly true that
the individual species comprising PM produce adverse health effects through a series of toxicity
pathways that could be modeled provided we had the data (e.g., lung deposition rates, uptake
fractions, clearance rates etc. for the general population and special subgroups), in reality, a
number of factors act in concert to prevent this level of detailed internal dose modeling. First,
while it is possible to integrate a PBPK model into our analytical framework for benefits
estimation, at this point, we simply do not have the data necessary to drive that model (i.e.,
deposition rates, uptake fractions etc. for PM as a mixture or for key individual component
species). In addition, while PBPK modeling has been successfully conducted in support of
population-level risk assessment and incidence estimation for specific chronic toxicants (lead,
mercury), PM poses a particular challenge because of its complex and variable composition. The
aggregate toxicity of PM as a mixture is likely not simply the sum total of its individual
constituents due to synergistic and antagonistic interactions. Therefore, disaggregating PM into
individual components and modeling each separately could significantly misrepresent overall
toxicity and introduce uncertainty. In addition, the component species comprising PM, which can
vary significantly depending on source type and additional factors effecting environmental
fate/transport/transformation, can interact with each other to either enhance or reduce uptake of
specific species. At present, we do not have the data to support this level of inter-active
pharmacokinetic modeling. For the reasons stated above, we believe that at this time, it is not
feasible to use individual-level exposure modeling in generating benefits estimates for chronic
PM exposure. Note, however, that research is currently ongoing in a number of areas that might
allow a more refined treatment of chronic-level health effects estimation for PM (e.g., linkages
between health surveillance and air quality data at much more refined spatial and temporal scales
that would support future derivation of more refined CR functions). As these new methods
become available, we will update our benefits estimation approach accordingly.
The second point we would like to make in response to the commentor's criticism of our use of
ecological epidemiology study-based CR functions relates to the modeling of acute health effects
(e.g., asthma exacerbations). EPA has developed sophisticated activity profiles models which it
uses for evaluating acute exposure to criteria pollutants as part of the NAAQS analysis. These
models use a probabilistic Monte Carlo-based framework along with activity diary, demographic
and population mobility data to generate activity profiles for sets of modeled individuals. These
exposure estimates can then be integrated with chamber study-based response data to evaluate
the current status of population exposure regionally and nationally. However, for incidence
estimation, it is critical that we generate estimates for specific, quantifiable health endpoints
(e.g., asthma exacerbations, school loss days, minor restricted activity days etc.). At this point,
only ecological epidemiology study data provides CR functions at the population-level for these
specific types of health endpoints. While chamber studies can identify likely thresholds for
potential effects which can be used in setting health standards, due to their small sample size,
these chamber studies can not be used to generate population-level effects estimates. Therefore,
we have continued to rely on ecological studies for our CR functions. Note, however, that as
with chronic health effects, we are carefully monitoring ongoing research and will adjust our
methods and data sources as necessary to reflect the latest findings.
-------
The suggestion that we provide a critical assessment of the CR functions we have incorporated
into BenMAP, including their relative advantages and disadvantages, is in-line with similar
suggestions made by the other reviewers. We agree that it would be helpful to the user to have
access to information on the strengths and weaknesses of each CR function, since this would
allow them to design their analytical approaches for specific benefits analysis in a more
scientifically defensible manner and would help them in interpreting their results. In the
longer-term we are also considering including functionality in BenMAP to generate summary
statements for each incidence and/or benefits estimate that summarize key factors that should be
considered in evaluating the overall quality of each estimate including: (a) potential mis-match
between modeled and study populations (e.g., demographic attributes, location), (b) implicit or
explicit extrapolations beyond the model domain of the study (e.g., using exposure levels well
below levels used in deriving the CR function), (c) treatment of co-pollutant exposure, and (d)
potential mismatch between the study constituent mix and the likely constituent mix being
evaluated in the analysis (this is relevant mainly for PM).
Regarding the scale effects comment, EPA recognizes that care must be taken in applying CR
functions at spatial and temporal scales other than those used in the underlying epidemiology
study. In fact, this concern extends to any application of CR functions beyond the model domain
of the original study. Specifically with regard to spatial scale, the primary concern in applying
CR functions at scales larger (i.e., more refined spatial scale) than the original study is the
introduction of ecological bias. In the context of health impact assessment, ecological bias refers
to the application of ecological (group-based) effects estimates to individuals or smaller
sub-groups of individuals. While an effects estimate maybe representative for a specific area
and grouping of individuals evaluated in the original study, if that area is examined in greater
detail, distinct stratifications of individuals into subgroups with different risk profiles may be
found. In this example, applying more generalized effects estimates to these smaller stratified
sub-populations would be incorrect and could mis-represent their actual risk. Given that
BenMAP can be used to evaluate virtually any spatial scale specified by the user, it is important
for caveats and guidance to be provided to the user regarding application of CR functions at
spatial scales other than those specified in the underlying study. EPA will consider enhancing
the BenMAP manual to include a more complete treatment of this topic.
Regarding comments on the simplistic approach used to generate population projections, it is
important to note that the User's Manual as currently drafted does not provide a sufficiently
detailed description of the analytical approach to allow the method to be critically evaluated. As
mentioned earlier, this reflects our decision to have this version of the manual provide guidance
on using BenMAP, rather than a rigorous description of the technical approach used for each key
element of the benefit methodology. Having said that, we are providing a more detailed overview
of the method used for population projection for completeness (this description comes from the
technical methods section of a Regulatory Impact Analysis that was completed recently).
We use sophisticated population projections based on economic forecasting models developed by
Woods and Poole, Inc. The Woods and Poole (WP) database contains county-level projections
-------
of population by age, sex, and race out to 2025. Projections in each county are determined
simultaneously with every other county in the United States to take into account patterns of
economic growth and migration. The sum of growth in county-level populations is constrained
to equal a previously determined national population growth, based on Bureau of Census
estimates (Hollman, Mulder and Kalian, 2000). According to WP, linking county-level growth
projections together and constraining to a national-level total growth avoids potential errors
introduced by forecasting each county independently. County projections are developed in a
four-stage process. First, national-level variables such as income, employment, populations, etc.
are forecasted. Second, employment projections are made for 172 economic areas defined by the
Bureau of Economic Analysis, using an "export-based" approach, which relies on linking
industrial sector production of nonlocally consumed production items, such as outputs from
mining, agriculture, and manufacturing with the national economy. The export-based approach
requires estimation of demand equations or calculation of historical growth rates for output and
employment by sector. Third, population is projected for each economic area based on net
migration rates derived from employment opportunities and following a cohort-component
method based on fertility and mortality in each area. Fourth, employment and population
projections are repeated for counties, using the economic region totals as bounds. The age, sex,
and race distributions for each region or county are determined by aging the population by single
year of age by sex and race for each year through 2015 based on historical rates of mortality,
fertility, and migration.
Although we believe the WP approach to represent a scientifically sound and defensible
approach for conducting population projections, we will review the category of demographic
projections noted by the commentor. To determine if any improvements/enhancements in our
methods should be made.
Regarding the comment that we should consider special subgroups in modeling mortality and
morbidity incidence (i.e., asthmatic and other disease status individuals), it is important to note
that most of the time series and chronic epidemiology studies included in BenMAP apply to the
general population or to fairly broad subgroups of that population (e.g., demographic age ranges,
or basic ethnic groups). However, few if any of the studies provide effects estimates specific to a
particular subgroup (e.g., individuals with specific health status). While these studies may
comment that morbidity or mortality is likely centered on a particular group with specific
predispositions to adverse impact, the sample size often prevents the development of a
statistically significant effects estimate for these smaller subpopulations. Having said that, as
new studies are published identifying more specific effects estimates, those will certainly be
reviewed and incorporated as appropriate into BenMAP.
6) Presentation: specific presentation elements that need improvement include: (a) need a chapter
providing a rigorous discussion of objectives, assumptions, theories, and methods, (b) improve
the display capability in the area of space-time analysis and mapping, and (c) inclusion of an
index for the manual.
-------
EPA agrees with the commentor that the manual could be expanded (in the short-medium term)
to include an overview of the theoretical and conceptual framework underpinning the benefits
model as well as the intended use and scope of the model. We do want to point out however,
that the benefits analysis framework in BenMAP is complex and combines a number of distinct
modeling elements (e.g., emissions characterization, air quality modeling, population growth
projections, epidemiology derived CR functions etc), each of which is grounded in its own theory
and is comprised of a complex array of analytical procedures. Therefore, providing a
comprehensive overview of the entire analytical framework in the manual is probably not
feasible. Instead, we plan to incorporate a general conceptual overview of the entire model and
then reference the reader to alternate publicly-available documentation where they can find
additional details.
Regarding space-time analysis and mapping, we do include the ability to export shape-file
coverages which could then be incorporated into a ArcGIS for display and further analysis,
including more refined discrete time-step mapping and display. At this point, we do not have
plans to incorporate functionality to support this type of sophisticated GIS-level display, since
this would be resource-intensive and simply duplicate capabilities already available in other
programs.
An index can certainly be added to the manual.
7) Computer programs: (a) computer library is not user friendly, (b) user interface is well laid
out, but the manual does not match the interface and many errors were encountered in working
with the GUI, (c) the BenMAP program is specifically designed to work with annual averages
and therefore is not as flexible as advertised, and (d) key elements of the computer library are
missing (e.g., fitting the covariance function in kriging).
Shorter-term modifications to the BenMAP GUI could allow the user to have ready access to all
of the different data libraries used in BenMAP (e.g., demographic data, demographic projections,
baseline incidence and prevalence rates). Currently, they are only allowed direct access to
several of the libraries (e.g., CR and valuation functions).
We are considering the development of additional case study tutorials that could be linked to
dedicated training datasets contained in BenMAP. These tutorials would focus on specific
functional elements of BenMAP and be designed to train the user quickly in the nuances of the
analytical framework. In addition, we keep a running record of quirks in the model that interfere
with ease of learning and operation by the user. Updated versions of the model with these issues
addressed are released on a relatively frequent basis and will continue to be.
Finally, regarding the comments on important missing elements in the computer library (e.g.,
ability to fit variograms to data), most of the technical issues cited by the author refer to
-------
geospatial interpolation. Because of the decision to decouple kriging functionality from
BenMAP, most of the commentary addressing this issue of missing data library elements.
------- |