UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
° WASHINGTON D.C. 20460
OFFICE OF THE ADMINISTRATOR
SCIENCE ADVISORY BOARD
September 4, 2009
EPA-SAB-09-017
The Honorable Lisa P. Jackson
Administrator
U.S. Environmental Protection Agency
1200 Pennsylvania Avenue, N.W.
Washington, D.C. 20460
Subject: Consultation on Development of EPA' s Report on the Environment (2012)
Dear Administrator Jackson:
At the Request of EPA' s Office of Research and Development (ORD), in 2007, the
Science Advisory Board (SAB) reviewed a draft of the Agency' s Report on the Environment:
Science Report (ROE). The SAB provided recommendations to improve the ROE before it was
subsequently published in 2008. In addition, the SAB strongly supported continued development
of the ROE and provided recommendations to improve future versions of the report. The 2008
ROE presented a compilation of scientific indicators of the environmental condition of the
nation's air, water, and land, and status and trends regarding human health and ecological
systems. The indicator information in the 2008 ROE was intended to answer status and trend
questions of critical importance to EPA's mission.
ORD requested further advice from the SAB to continue developing the ROE, and a new
SAB Committee was formed to advise EPA regarding development of the next version of the
report to be published in 2012. On June 30 - July 1, 2009, the SAB Committee held an initial
consultation with EPA on a paper, Issues for Initial Consultation with the SAB Advisory
Committee on EPA 's Report on the Environment. The paper describes EPA's proposed
approaches for restructuring the ROE and incorporating conceptual models and supplemental
information. Committee members discussed the paper and submitted the enclosed written
comments. This letter highlights key points of the discussion.
EPA's ROE is an important document that can be of great value to the Agency as a
strategic planning and public information tool. EPA has made remarkable progress in
developing the ROE and has responded to many of the SAB comments on previous drafts of the
report. However, Committee members found that a clear mission statement for the ROE needs
to be articulated, and that the clearest statement could originate from EPA leadership. ORD has
proposed aligning the structure of the ROE with goals and objectives in the Agency's Strategic
Plan. This may be a worthwhile goal for the ROE, but the current structure and content of the
-------
report would require revision to be effective in supporting the strategic planning process. A clear
articulation of the purpose and objectives of the ROE is needed to guide plans for the next
iteration of the report. In addition, if the ROE is to be used as a strategic planning tool, some
information should be included to provide an historical perspective on the condition of the
environment and benchmarks or goals for the future. Committee members also found that the
audience of the report is not clearly defined; there seem to be multiple audiences that may need
somewhat different information. The proposal to align the ROE with goals and objectives in the
Agency's Strategic Plan suggests that the main audience of the report would be EPA leadership.
However, there are multiple potential audiences for the ROE and it may not be possible to write
the report in a way that will be optimal for all of them. Therefore, EPA must decide who the
most critical audience is or prepare different versions of the ROE for different audiences.
As in 2007, SAB Committee members underscore the need for an overarching and
unifying framework to link and integrate the scientific elements of the ROE. EPA has not yet
developed a proposed framework or model that provides much insight into how chapters,
questions, and indicators in the report are connected. Possible approaches to consider in
developing such an overarching framework are discussed in the enclosed comments from
Committee members. Finally, Committee members urge EPA to invest more resources in
continued development of the ROE. Because of the breadth and depth of the ROE, developers of
the report periodically require additional technical support to update environmental indicators in
the report.
Because this was a consultation, there will be no formal report from the SAB. Enclosed
are my summary of key points discussed by the Committee and the individual comments from
Committee members. We hope these comments are helpful to EPA and look forward to
providing additional advice to the Agency as it continues the important work of developing the
next edition of the Report on the Environment.
Sincerely,
/Signed/
Dr. James Sanders, Chair
SAB Advisory Committee on EPA's Report on
the Environment
Enclosures:
Enclosure 1: Committee Roster
Enclosure 2: Comments from Committee Members
Enclosure 3: EPA Charge Questions to the Committee
cc: Dr. Deborah Swackhamer
Chair, EPA Science Advisory Board
-------
Enclosure 1: Committee Roster
U.S. Environmental Protection Agency
Science Advisory Board
Advisory Committee on EPA's Report on the Environment
CHAIR
Dr. James Sanders, Director and Professor, Skidaway Institute of Oceanography, Savannah,
GA
MEMBERS
*Dr. David T. Allen, Professor, Department of Chemical Engineering, University of Texas,
Austin, TX
Dr. Henry Anderson, Chief Medical Officer, Division of Public Health, Wisconsin Division of
Public Health, Madison, WI
**Dr John Bailar, Scholar in Residence, The National Academies, Washington, DC
Dr. Timothy Buckley, Associate Professor and Chair, Division of Environmental Health
Sciences, College of Public Health, The Ohio State University, Columbus, OH
Dr. Ingrid Burke, Director, Haub School and Ruckelshaus Institute of Environment and Natural
Resources, University of Wyoming , Laramie, WY
Ms. Lauraine Chestnut, Managing Economist, Stratus Consulting Inc., Boulder , CO
Dr. Aaron Cohen, Principal Scientist, Health Effects Institute, Boston, MA
Dr. Loveday Conquest, Professor, School of Aquatic and Fishery Sciences, University of
Washington, Seattle, WA
Dr. Jeffrey Griffiths, Associate Professor, Public Health and Community Medicine, School of
Medicine, Tufts University, Boston, MA
Dr. Charles Hawkins, Professor and Director, Western Center for Monitoring and Assessment
of Freshwater Ecosystems, Department of Aquatic, Watershed, and Earth Resources, Utah State
University, Logan, UT
Dr. Lynda Knobeloch, Senior Toxicologist, Wisconsin Department of Health Services,
Madison, WI
-------
Dr. Allan Legge, President, Biosphere Solutions, Calgary, Alberta, Canada
Dr. Amanda Rodewald, Associate Professor of Wildlife Ecology, School of Environment and
Natural Resources, The Ohio State University, Columbus, OH
*Dr. Peter Scheff, Professor, Environmental and Occupational Health Sciences, School of
Public Health, University of Illinois at Chicago, Chicago, IL
Dr. Mark Schwartz, Professor, Department of Environmental Science and Policy, College of
Agriculture, University of California, Davis, CA
Dr. Alan Steinman, Director, Annis Water Resources Institute, Grand Valley State University,
Muskegon, MI
Dr. Helen Suh, Associate Professor, Environmental Health, School of Public Health, Harvard
University, Boston, MA
Dr. Due Vugia, Chief of Infectious Diseases Branch, Division of Communicable Disease
Control, California Department of Public Health, Richmond, CA
*Dr. Kathleen Weathers, Senior Scientist, Gary Institute of Ecosystem Studies, Millbrook, NY
Dr. Judith S. Weis, Professor, Department of Biological Sciences, Rutgers University, Newark,
NJ
Dr. Stephen Weisberg, Executive Director, Southern California Water Research Project
Authority, Costa Mesa, CA
SCIENCE ADVISORY BOARD STAFF
Dr. Thomas Armitage, Designated Federal Officer. U.S. Environmental Protection Agency,
Washington, DC
* Unable to attend the June 30 - July 1, 2009 meeting
** Unable to attend the June 30 - July 1, 2009 meeting but submitted written comments
-------
Enclosure 2: Comments from Committee Members
Page
SUMMARY OF KEY POINTS OF DISCUSSION BY COMMITTEE CHAIR,
DR. JAMES SANDERS 7
COMMENTS FROM DR. HENRY ANDERSON 10
COMMENTS FROM DR. JOHN BAILAR 12
COMMENTS FROM DR. TIMOTHY BUCKLEY 14
COMMENTS FROM DR. INGRID BURKE 16
COMMENTS FROM MS. LAURAINE CHESTNUT 17
COMMENTS FROM DR. AARON COHEN 19
COMMENTS FROM DR. LOVEDAY CONQUEST 21
COMMENTS FROM DR. JEFFREY GRIFFITHS 23
COMMENTS FROM DR. CHARLES HAWKINS 24
COMMENTS FROM DR. LYNDA KNOBELOCH 26
COMMENTS FROM DR. ALLAN LEGGE 28
COMMENTS FROM DR. AMANDA RODEWALD 29
COMMENTS FROM DR. MARK SCHWARTZ 35
COMMENTS FROM DR. ALAN STEINMAN 38
COMMENTS FROM DR. HELEN SUH 41
COMMENTS FROM DR. DUC VUGIA 43
-------
COMMENTS FROM DR. JUDITH WEIS 46
COMMENTS FROM DR. STEPHEN WEISBERG 47
ISSUES FOR CONSULTATION 48
-------
Summary of Key Points of Discussion by Committee Chair, Dr. James Sanders
EPA's Report on the Environment is an important document that can be of great value
to the Agency and the public. EPA has made remarkable progress in developing the Report
on the Environment and has responded to many SAB comments on previous drafts of the
report.
Additional improvements are needed to make the ROE a more effective strategic
planning and public information tool. The developers of the ROE need more resources
to incorporate these improvements. Because of the breadth of the ROE, developers of the
report periodically require additional technical support in biostatistics, surveys,
epidemiology, toxicology, and other relevant disciplines. It is important that EPA commit to
providing that technical assistance either by hiring additional full-time staffer making
experts elsewhere in ORD available to work on the report.
A clear mission statement for the ROE is needed. A clear mission statement is needed not
only to define the intended audiences of the report, but to clearly define its objectives in a
more general sense. If properly conceived, the mission statement could provide guidance
and direction on other key issues such as choice of an overarching and unifying conceptual
model. The audience of the ROE is not clearly defined. EPA has stated that it intends to
restructure the ROE to more directly align its chapters with EPA's Strategic Plan goals, and
to align policy questions in the ROE with objectives in the Agency's Strategic Plan. The
main audience would therefore appear to be EPA leadership. However, there are multiple
potential audiences. It may not be possible to write the ROE in a way that will be optimal for
all audiences, so EPA must decide who the most critical audience is or prepare different
versions of the ROE for different audiences. For example, it may be useful to develop a
"Citizen's Guide" to the ROE for the public. This version of the ROE could be prepared
using simpler language, colorful graphics, and a transparent approach to convey the key
messages of the document.
There is a need for an overarching and unifying conceptual model for the ROE. The
current ROE lacks an overarching conceptual framework that brings together the scientific
elements of the report and allows the natural placement of each indicator in the framework.
The conceptual models proposed by EPA for use in the ROE are media specific and provide
little insight into how chapters, questions, and indicators in the report are interconnected.
The ROE conceptual framework should provide the foundation for understanding what EPA
is interested in knowing and why. The conceptual framework should connect the questions
in the report, embrace important new questions, issues, and indicators, and also provide
guidance on what should and should not be included in the ROE. EPA may wish to consider
using the European Environmental Agency's Pressure-State-Response approach to develop
an overarching conceptual framework for the ROE. Other possible conceptual modeling
approaches are provided in the attached comments.
It may be useful to incorporate additional process-oriented models into the media
chapters of the ROE. Specific chapter models could address the underlying scientific
foundation of many of the ROE questions and indicators. Slightly more detailed models may
-------
also be developed for some of the individual questions. The SAB previously recommended
that such models be process-oriented. Although EPA has stated that it is not the intent of the
ROE to identify mechanisms and drivers of the indicators, some recognition of the
underlying processes is necessary to present the scientific foundation of questions in the
report and identify strategies to remedy and avoid problems. EPA's proposed conceptual
model examples for the drinking water and air questions show linkages from emissions to
health and environmental effects and this is helpful in putting individual indicators into a
context. However, not all of the linkages in these models are equally important and not all of
the linkages are under EPA jurisdiction. Furthermore, the proposed models do not account
for important "effect modifiers" such as meteorology or age.
Alignment of the ROE with EPA's strategic goals and objectives makes sense but
indicators should be linked to EPA program actions. If the ROE is to be effectively used
in the strategic planning process it needs to link the indicators to EPA program actions and
describe benchmarks or goals, where possible, so that progress relative to goals can be
assessed. The link between reductions in pollutants and improvements in environmental
quality should be made, with the goal of answering the question, "how much reduction in
emissions or environmental concentration is needed to produce environmental
improvements?" The overarching conceptual model for the ROE needs to include the
feedback loop of EPA regulation and policy as an action/response that affects the
environment. Consideration of both an ecological process model as well as an operational
process model that focuses on linking management actions with desired environmental
outcomes would be appropriate in this regard. In addition, some historical perspective
should be provided. The reader needs to know where we have come from, where we are
today, and where we need to be in the future.
It is important to retain rigorous criteria for the selection of ROE indicators, but
supplemental information can be useful in filling gaps where national data are
unavailable and addressing questions of limited geographic scope. This information
can help identify emerging vulnerabilities or threats and consolidate information that
may be useful to a broad set of users. EPA's proposal to include supplemental
information in the ROE has merit. However, if the supplemental information is quarantined
in a ROE incubator site or appendix as proposed, the information may be of limited use.
Therefore, at the very least, the ROE should include information about the indicators that are
listed in the incubator site. If additional resources are focused on meta-analysis of
supplemental data, it may be possible to combine results from different supplemental studies
and develop aggregate results that could be used in the ROE. It may also be possible to
integrate supplemental information into the ROE and provide additional metadata
information about the data source, rigor, geographic region of significance, and level of
confidence in the data.
It is important to provide a defined place in the ROE for discussions of emerging issues.
The document should be forward-looking. Emerging issues should be discussed in the
ROE, perhaps in the supplemental information section but also in an executive summary, to
show that EPA is aware of and beginning to monitor or consider important new issues. It
will be these emerging issues that are likely to encompass the EPA's efforts in coming
-------
decades, and in order for the ROE to help drive strategic planning, these issues must be
identified and prioritized.
Regional problems require regional treatment. Not all problems are national This is
particularly true of ecosystem problems. For example, ecologists have long recognized
increased tree mortality in the eastern U.S. driven by acid rain, and increased tree mortality in
some regions of the U.S. driven by invasive species (e.g., gypsy moth) and urban island
effects (high urban tree mortality). In developing the ROE, EPA should consider using
regional indicators that represent important status and trend information. EPA may see its
mandate as national in scope, but many readers of the ROE do not make distinctions between
national, regional, and state jurisdictions. In considering the use of regional indicators it is
important to ensure use of appropriate scales for analysis of information.
The ROE should contain an executive summary that distills and synthesizes the
indicator information to address the questions in the report. The 2008 ROE is a great
collection of indicator data, but alone it is not sufficient for planning purposes because it does
not draw conclusions about the state of the environment. Some synthesis or interpretation,
rather than just providing summary information and allowing users to interpret or draw
conclusions, would be useful. If connections cannot be made between indicators and
outcomes, questions in the report should probably be reworded. The executive summary
should also be published as a separate document because it is likely to be of interest to the
general public, law makers, and policy makers who may not read the entire ROE.
In developing the ROE, EPA should emphasize producing a powerful and interactive
web-based platform with links to pertinent websites containing additional data and
information. The 2012 ROE should be available as a printed report, but each chapter should
be published in an electronic format that is fully searchable. Additional resources will be
needed to accomplish this.
Where possible trends in environmental indicators should be linked to indicators for
ecological condition and human health. For example, EPA has estimated the health
impacts of air pollution in quantitative terms in other reports. These estimates provide direct
answers to the ROE question for air and would help fill gaps that exist in the 2008 ROE.
Several other key points are provided for EPA's Consideration. 1) It is important to
maintain the statistical validity and scientific rigor of the ROE. 2) An efficient way to use
staff, produce fresh products and keep policy-makers focused would be to produce a different
ROE chapter once per year on a rotating schedule. 3) Integration across agencies that
conduct environmental monitoring is critical. An interagency monitoring
effort/clearinghouse/data source would be very useful.
-------
Comments from Dr. Henry Anderson
Charge Question 1: Please comment on whether the EPA's proposed conceptual modeling
approaches are logical and useful for:
a) Aligning the ROE questions with the Agency's strategic goals and objectives.
There is a need for an overarching conceptual model in addition to the more focused models
proposed. This needs to tie everything together. It will be very useful to have the ROE grounded
in the agency strategic goals and objectives and the conceptual models proposed can easily do
that. The first ROE was an Administrator directed initiative and it would give the proposed ROE
greater import if the current Administrator would not only endorse the production of the ROE
but provide some direction as to how she intends to have it used within the agency. Aligning the
ROE with the strategic plan gives the ROE a core agency constituency and use. This is not the
only constituency but is an important one that will give the ROE ongoing utility. For instance
the Administrator could suggest that all internal research proposals and external RFAs should
reference what gap or ROE area it will address. This would get the ROE continually referenced
and read and provide some import to the gaps and needs identified in the ROE. It would become
a driver for programs to support the ROE development out of self-interest.
b) Communicating the intent and scope of questions in the ROE.
The proposed ROE comes at a time of transition in information communication. While a paper
report needs to be provided, greater emphasis should be placed upon the electronic version of the
report. The ROE needs to become a multimedia enterprise and not just a single report. By the
time the ROE is completed in 2012 we can anticipate additional information technology
advances and the ROE needs to be sufficiently nimble to be "state of the art" when it comes out.
Right now the ROE needs to be written about in Wikipedia and the other personal
communication modalities explored. Cross linkages with the CDC Tracking Portal and other
"indicator" sites need to be negotiated to get the current ROE in greater circulation. An
electronic format allows for interactive utilization of the data behind static graphs and charts
allowing individualized reports. The ROE needs to become an integral part of a
communication/information system. The on-line material can also link to other resources as they
become available. There are a growing number of state and federal agencies developing
indicators and the ROE needs to recognize them and not duplicate effort.
A critical current need is for an evaluation of the previous ROE and an assessment of impact.
For the next ROE an evaluation plan needs to be developed before it comes out.
Other comments:
The ROE indicators should be broadened to include one or more social science indicators. These
might include the use of the many surveys of public awareness, perceptions and beliefs, some of
which can even provide trends. If not actual indicators added to each question answer, this type
of information needs to be provided in the supplemental information sections. Supplemental
information will add perspective and relevance to the information provided with the indicators.
10
-------
The ROE needs to provide interpretation and strategies to achieve the goals, not just present the
data. The questions in the current ROE cry out for answers and the reader is disappointed when
no interpretation is provided.
It would help to have a "citizen's guide" to the ROE.
11
-------
Comments from Dr. John Bailar
Comments on "Issues for Initial Consultation with the SAB Advisory Committee on EPA's
Report on the Environment"
First it is a pleasure to commend the quality of this document. It is clear, well organized, easy to
read, highly informative, and to the point. These features make it much easier to comment
intelligently, and contribute to making a good product even better (as opposed to making a poor
product marginally acceptable). I cannot make specific recommendations because I have not
discussed this with committee members or others, and may miss some of the critical background.
Still, there are several points that the committee should think about as it considers what
recommendations would be appropriate. The following are in no particular order.
The whole process is geared to EPA's strategic plan, despite words here and there about other
uses. This internal use is important, but other readers will have other uses for the data, and too
tight a focus on the strategic plan could undercut broader uses of the ROE that might contribute
to EPA's mission in critical but less direct ways. A few of these are: informing the public about
matters that interest them (and the public support that would generate), greater service to non-
EPA governmental organizations, including state and local agencies, and informing various
public interest groups about emerging issues that EPA cannot deal with for reasons of limitations
in its mandate or resources. By all means be sure that the ROE is supportive of and consistent
with the strategic plan, but I urge you to give these other uses of ROE at least as much weight as
its internal uses.
It would be helpful, and might significantly advance EPA's overall mission, to include a short
section on international comparisons and trends. Such data could support new initiatives at EPA
by showing what others have already demonstrated to be possible, and would help to put much
of ROE in a broader context. Candidate items might include age-adjusted mortality rates for
specific causes (e.g. liver cancer, or selected pneumoconioses), PM2.5 levels in a few cities, or
some measure of water quality.
I would adjust the frequency of ROE to match that of the strategic plan, perhaps offset by a year
to allow for feedback from the plan to ROE and for impact of ROE on the plan as it is developed.
Among other points, it is not clear to me why ROE should match the structure (as opposed to
content) of the strategic plan. This is surely a convenience for EPA, but at what cost in
organization to other users? Again I strongly recommend that you give a lot more attention to
users outside the EPA community.
Some policy-relevant questions are beyond EPA's mission - e.g. some kinds of highly localized
pollution, de minimus violations that nevertheless cause substantial local concern, or
exceedances that are very large but do not meet the frequency criterion for some EPA-mandated
intervention. ROE should address this general issue where it explains the rationale behind its
choice of metrics.
12
-------
A general comment is that future editions of ROE will become more and more valuable as they
include longer and longer series of data points. This makes it critical that the data series chosen
for inclusion be continued. Before you decide to include some series, please give some explicit
thought to the likelihood that the series will not be continued indefinitely.
I do not know the internal structure that has led to the production of the present and future ROEs,
but it seems to me critical that there be a dedicated, full-time, permanent staff for this, including
competence in biostatistics, surveys, epidemiology, toxicology, and other relevant disciplines.
I would give some thought to limiting the supplemental data to items that display (or will
display) trends, not one-offsets of observations.
The ROE should include a meaty section, written in conjunction with others from all major
components of EPA, that is focused on future needs for data, both new items and new
information about present items - new problems that EPA should address, improved data quality,
more extensive tabulations, etc. Getting this into today's version of ROE, while the gaps and
problems are still fresh in memory, and where users can see and comment on them, will be better
than trying to reconstruct the problems and solve them with old records, new staff, etc.
Every item in the present (and I am sure future) version of ROE is affected by some uncertainty
- both sampling uncertainty and bias. The discussion should address matters of bias for each
table, and sampling uncertainty should be addressed quite consistently with confidence bounds
or SEs. I feel quite strongly about the need for users to understand the limitations on both
accuracy and precision in each data point presented. This why I recommend that each section,
and generally each table, include some discussion of uncertainty, including SEs or confidence
bounds as well as plausible biases. These matters are particularly important for measures of
change, where uncertainty may be substantially less than for individual levels because some
kinds of bias will cancel out, and variances of change should include subtraction of covariances.
Finally, I see an important role for some single, overall, summary index of "environmental
quality". Such an index will require considerable time and effort to develop, including a great
deal of outside stakeholder involvement, but could be of great value to EPA and others in
tracking the broadest changes over time.
13
-------
Comments from Dr. Timothy Buckley
Charge Question: Does the Committee have recommendations concerning other possible
approaches to conceptual model development that would be useful in identifying or highlighting
important ROE topics, indicators for consideration, research, or development?
Response: This important and complex document has continued to evolve and become stronger
and stronger. It is very well written and is coming closer to meeting its potential. The critical
comments described here should not in any way be interpreted as a lack of enthusiasm over the
concept of the ROE and its value to EPA and the public that it serves. Accordingly, the critique
provided here is given in the context of supporting the document development and making it as
strong as possible. There is still room for improvement including in the area "other possible
approaches to conceptual model development" addressed in the charge question above. The
Agency's interest in developing appropriate conceptual models to represent both the science and
application of the ROE are well placed as a strategy to effectively organize, present, and
communicate the ROE. Previous review panels have highlighted the need for an overarching
conceptual model and the current ROE still lacks such a model. Figure 1-1 of the new report is
more oriented toward the application or process of the ROE and therefore does not adequately
address the need of providing a scientific framework. Such a conceptual model will provide a
valuable contribution to the ROE in:
Depicting and introducing the main scientific elements of the ROE and how they relate to
one another.
Highlighting the highly integrative nature of the ROE
An example of a simple integrative conceptual model was provided by a previous ROE review.
In the context of this charge, I provide an additional rough draft of a alternative model that the
Agency might consider for further development.
14
-------
Other
Non-environmental
Influences
In its current form, the ROE is largely limited to emissions (sources), the physical environment,
and their impact on human health and ecology. This orientation ignores a critical upstream
dimension that forms an underlying driver of sources and stressors on the environment, that is
the social and behavioral sciences that underlie behaviors and individual decisions of a growing
population. These behavioral factors have been shown to have a substantial influence on the
environment, health, and ecology, and therefore provide strong potential indicators thus far not
being considered include such issues as:
energy use and conservation;
water consumption
consumer product use (e.g. pesticides)
urban sprawl
vehicle mile traveled
combustion source pollution including cooking, grilling, fireplaces, campfires,
candle/incense burning
Such indicators are especially valuable to include because they are amenable to intervention
although not through traditional EPA regulatory strategies. I think that the ROE could provide
an incredibly valuable educational tool in showing the link between such behaviors,
environmental quality, and human health and ecological status.
15
-------
Comments from Dr. Ingrid Burke
1. Charge Questions 1 and 2: proposed conceptual models approaches.
My interpretation of the SAB review of the ROE is very different than what we heard this
morning about conceptual frameworks/models. I think what that review states is that there is a
need for an overarching conceptual framework that provides the glue for the entire ROE. The
presentation this morning represented an elaboration of the questions within each section of the
report, that is, conceptual models at a scale that is very much more detailed, rather than
providing the overarching glue or philosophical basis for the ROE.
I will dive right in and say that I was disappointed with the approach presented today, in terms of
depth of thinking. The ROE needs a foundation for answering the question, broadly, "What are
We Interested in Knowing and Why?"; something that can allow us to understand what falls
under the umbrella, and what DOESN'T. The Conceptual Framework should represent the
forest, not the trees.
The Conceptual Model should connect the questions, should embrace important new
questions/issues/indicators, and additionally, provide guidance for what should NOT be
included.
More comments from Indy:
1. The connection of the ROE to a conceptual framework such as the Millenium
Assessment that explicitly includes ecosystem services (provisioning of clean air,
adequate quantity and quality of water, provision of fuel, etc.) will allow the ROE to have
strong relevance to policy. HOW do we increase quality of air? We make clean air worth
something, etc. (see recent EPA report on Valuing, etc). This will integrate the
chapters/indicators/questions really well.
2. The executive summary is critical.
3. An efficient way to utilize the staff, produce fresh products, and keep policy-makers
focused would be a rotating schedule of chapters, one per year. Each year, on a calendar
date, a synthesis of one question could be produced, with a high visibility. The staff
would be focused during that year in depth on one particular issue, and when it is out, it
will be very fresh.
4. We need to find a way to send the message forward to the Administrator that integration
across agencies conducting environmental monitoring is Critical. An interagency
effort/clearinghouse/data source would be a gigantic contribution.
16
-------
Comments from Ms. Lauraine Chestnut
Overview comments: three main points
1. Alignment with the strategic plan, and applying the indicators to support the policy and
planning process, makes a lot of sense and is consistent with the apparent original intention of
the effort to develop a report on the state of the environment. However, to be effectively used in
the planning process the report needs to link the indicators to EPA's programs and needs
to describe benchmarks or goals, where possible, so that progress relative to goals can be
assessed.
2 If the primary purpose of the report is for planning and assessment of EPA's progress,
the overarching conceptual model needs to include the feedback loop of EPA regulation
and policy as an action/response that affects the environment. This loop includes an
assessment of how are we doing relative to benchmarks or goals, which then influences planning
in a feedback process.
3. The title, "report on the environment," suggests a comprehensive view and raises expectations
of conclusions and assessment The current (2008) document is a great collection of
indicator data, but alone it is not sufficient for planning purposes because it does not draw
conclusions about the state of the environment. A more descriptive title of this document
would be, "report on environmental indicators."
Specific charge questions
la: Do the proposed conceptual modeling approaches help to align the ROE questions with the
Agency's strategic plan?
The modeling approaches, as currently described in the proposal, show the linkages from
emissions to health and environmental effects, which is helpful in putting individual indicators
into a context. However, the models are not sufficient to support alignment of the indicators
with the strategic plan, because the models do not include EPA's programs and actions that are
intended to affect change in environmental conditions. For alignment with the planning process,
a more relevant overarching model would be one that includes the assessment of environmental
problems and priorities that comes from interpretation of the indicators, and the feedback of this
into the EPA's actions. These in turn influence emissions that ripple through to the human
health and environmental effects, as reflected in the indicator levels.
The proposal to align the ROE with the strategic plan makes a lot of sense and will help focus
the ROE on the most important indicators needed to support policy and planning decisions,
including identifying gaps in the available indicators that need to be addressed. It is a step
toward greater integration of the indicators work with the EPA's planning process, which is a
step in the right direction.
17
-------
An important caution or limitation in terms of alignment of the ROE with the strategic plan is
that some benefits of EPA's programs and regulations result from preventing problems. Not all
benefits come on the form of reducing existing problems. Environmental problems that have
been prevented cannot be measured with indicator trends (other than showing a flat trend
reflecting sustained quality of a resource); indicators based on observed data cannot show what
would have happened in the absence of the program or policy that prevented the problem from
happening.
Ib: Do the proposed conceptual modeling approaches help to communicate the intent and scope
of the ROE questions?
The proposed conceptual modeling approaches are very helpful in showing where individual
indicators fit into the chain from initial pollution emissions (or land use actions); to pollution
transport and transformation via air, water, land, and biota; and to ultimate human health and
environmental effects. Such models can help guide selection of appropriate indicators and show
how "upstream" indicators may be a proxy for "downstream" effects that may be more difficult
to measure with observational data. The selection issue is important because the ROE is in
danger of growing thicker and thicker without necessarily providing better answers to the
relevant questions.
However, the proposed conceptual models raise the same questions about linkages that the ROE
questions themselves raise, and that the 2008 report does not address. For example, the ROE
question, "What are the trends in outdoor air quality and their effects on human health and the
environment?" explicitly links air quality and human health, but the 2008 ROE puts the
indicators on air quality and on human health in separate chapters. It is left to the reader to draw
any conclusions about whether the trends in health indicators have anything to do with the trends
in air quality. Admittedly, it may not be possible to draw such conclusion given the multitude of
factors that influence many of the health indicators, but the ROE questions raise expectations
that an attempt to link them is going to be made.
One detailed comment on the draft model for outdoor air is that it seems to overlook human
welfare effects. Part of the problem is that it separates human and ecological exposure as if these
are independent tracks. It is hard to see how effects on visibility aesthetics, agricultural yields,
or manmade and cultural materials, and nonuse values for ecosystem preservation enter into the
chain. It also leaves unmentioned the link from ecological exposure to changes in ecological
services that in turn affect human health and welfare. It is important to keep the models
relatively simple, but at the same time general enough to encompass the detail. Another example
is with mercury where the effect on human health (fish consumption) and welfare (fishing
advisories and restrictions) travels through the ecological exposure.
18
-------
Comments from Dr. Aaron Cohen
Issues for discussion of questions 1 and 2
1. In general, EPA has responded to many of the comments in 2007 review, including the
request for conceptual models.
2. The proposed conceptual models for water and air are useful, but would suggest:
o Adding policy action as first stage of conceptual model for air (with appropriate
feedback loops): CAA has huge influence on air quality.
o Adding susceptibility factors at individual - and aggregate-levels: e.g., diet, health
care, socio-economic factors. These are relevant to both geographic scaling
issues and environmental justice concerns.
3. The current ROE does not provide answers to the ROE questions re. health
effects/impacts. Here are some suggestions:
o Authors should consider moving the relevant health information into the water and air
chapters. This would help draw the links necessary to answer ROE questions re.
health effects/impacts.
o EPA has estimated the health impacts of air pollution in quantitative terms, including
economic valuation, and these estimates have been extensively and independently
reviewed (e.g., by NAS). These methods and results are directly relevant to the ROE
question re. health effects of air pollution, and I have heard no good reason why they
should not be used. NB EPA's PM and ozone Criteria Documents (or ISA) and prior
(and future) staff papers are based on systematic review of the literature, as are EPA's
impact assessments. For this reason, they are not subject to criticisms re. "cherry-
picking." In contrast, citing only specific studies, no matter how compelling (e.g.,
Pope et al. 2009), opens the door for such criticism.
o The ROE should also note current efforts/results re. indicator development for health
impacts of air pollution. NB 2007 review suggested things such as the following:
CDC Environmental Public Health Tracking
EPA's BENMAP computer-based impact analysis program
Impact assessments by California Air Resources Board, EC and WHO,
including the Global Burden of Disease/Comparative Risk Assessments.
Top 3 recommendations
1. A clear mission statement for the ROE is needed. This should come from the highest
levels of the Agency, and should provide clear guidance on issues such as the relation to
EPA's strategic planning and the intended readership.
19
-------
2. A unified model/conceptual framework is needed in addition to specific conceptual
models, e.g. for water and air. The European Environmental Agency's Pressure, State,
Response model is an example of one possible approach (as noted at the June Panel
meeting).
3. EPA estimates of the health impacts of air pollution provide direct answers to the ROE
question for air and should be included in the ROE. These estimates use data and
indicators that have been extensively and independently reviewed, and have been used in
regulatory decision-making.
20
-------
Comments from Dr. Loveday Conquest
Three ROE Recommendations and Charge Question 3 and 4 comments.
Three recommendations for the ROE:
1. There needs to be a conceptual framework that brings together the different media and allows
the natural placement of each indicator in the framework. (The Example Conceptual Framework
Diagram in Appendix C-2, EPA-SAB-09-007 Report, is a very decent start, and it does
incorporate ecosystem processes and services.)
2. As more of the ROE ends up on the web, it will be important to devote an appropriate level of
resources (FTEs) to this effort so that sections are easy to access, the links are appropriate, etc.
A savvy Wikipedia entry is a good idea!
3. Continue to maintain the ROE's statistical validity and scientific rigor (particularly with
respect to use of Supplemental Informationsee comments below).
Charge Question 3. Comment on the logic and utility of EPA 's proposed use of supplemental
information to answer questions in the next version of the ROE.
Supplemental information (SI) can indeed be useful to fill gaps in the data and to identify
emerging issues, even while not meeting the all the original, stringent criteria that EPA has set
for its indicators. Presumably EPA has already investigated a variety of potential indicators,
many of which were not included in the ROE because they did not meet the rigorous criteria.
EPA should revisit the list and focus on those responses that might otherwise have been included
(e.g., they addressed an important issue, filled a data gap, addressed an emerging issue) but for
lack of one or more criteria. (It appears as though there were several indicators that "missed out"
due to not being collected on a national basis.) All candidate indicators should occupy a natural
place in EPA's conceptual framework.
It might be possible to use meta-analysis to statistically combine results from different studies
and come up with an aggregate result (realizing that there are still criteria of statistical
comparability to be met in order to carry this out). Even where meta-analysis is not possible,
data collected on a regional basis could still provide useful aggregate conclusions, using "weight
of the evidence" approaches.
Just as EPA developed rigorous, transparent criteria for national indicators, they should do the
same for regional indicators and for case studies.
Investigating all sources of data for potential supplemental information is a daunting task.
Therefore it is essential that EPA focus up front on key requirements (scientific, policy-driven,
and transparent) in seeking SI.
21
-------
The "incubator site" seems a natural place to put potential indicators that refer to emerging
issues. Canvass the scientific literature to find best papers to illustrate trends and to identify
emerging trends in the environment.
Charge Question 4. Recommendations for criteria to assure that supplemental information
included in the ROE is objective, free from bias, scientifically valid, and supports intended
purpose of the report.
Such data should in general meet the high standards of the ROE Indicator Definition and
Criteria. (Presumably datasets won't meet all of the criteria; if they had they would have been
included as indicators already.) The highest quality data will have been generated from a
probability sample. For example, EMAP and REMAP did a good job of getting regions, states,
and other public agencies started in terms of sampling using probability samples. EPA should
look at those indicators to ascertain which ones would provide useful SI.
Appendix F of the "Issues" paper also mentions potential usefulness of long-term data sets not
based on probability sampling approaches. These may still be incorporated into future ROEs,
but should be investigated carefully to ensure the rigorousness of their conclusions. Similarly,
case studies ("a sample size of one") may prove useful, for example, to illustrate a process. The
"Issues for Initial Consultation" (2009) paper (Appendix F) mentions use of the National Science
Foundation's Long-term Ecological Research (LTER) Program sites, USGS ground water basins,
state agencies, the Great Lakes, Lake Tahoe, and Lake Mendota. There are probably others.
It is important not to "cherry-pick" indicators for use as SI. Thus, only studies that have been
adequately peer-reviewed should be used. The appearance of cherry-picking can also be avoided
by using meta-analysis on a group of studies.
22
-------
Comments from Dr. Jeffrey Griffiths
The most recent Report on the Environment is an impressive and substantial work. The team that
has assembled the pieces of the Report, and the Report itself, deserve high praise.
Although my oral comments have been incorporated into the summary document, please allow
me to expand a bit on several points.
First, a number of the public health consequences of environmental pollutants - e.g., the effects
on the population - could be expanded. To say that mercury is toxic is accurate, however more
detail - perhaps even just a few sentences - may help to anchor the information in the Report
more concretely to protection of the public. In general, the background information on the issues
help to provide a foundation for judging the progress, or lack thereof, in addressing a public
health issue.
Along the same lines on the flip side of the equation, there is little interpretation of what the
improvements in the environment suggest or imply in terms of human health. For example, to
state that finished drinking water now has many fewer Cryptosporidium parasites in it due to
recent rules to enhance the efficiency of filtration is one thing; it would be more helpful to state
that this means that the population served by that system may have X or Y less cryptosporidiosis
because of this change.
Second, the report itself could be a living document on line... with individual areas or specific
topics frequently or continually updated as more information is acquired. We discussed this at
length during the meeting as the movement to the internet is socially well advanced and it may
be prudent to both (1) allow updates to occur as they are relevant, rather than on a pre-set
schedule; and (2) provide the Report in ways that will reach the most people.
Third, the Report will benefit from being linked to the EPA strategic plans which are currently
being developed. This will help to focus the document and provide a mechanism whereby the
public and the Agency can reflect upon the progress being made in the environment and judge its
success.
Lastly, one of the questions posed to us regarded the use of data that reflects regional rather than
national trends. Often no national data sets exists for an issue, which may be of regional but not
national importance. My recommendation is to use those regional data sets, when they are of
high quality, when they are illustrative of regional trends that can inform the public.
23
-------
Comments from Dr. Charles Hawkins
In general, the 2008 ROE presents information on a large number of environmental indicators
that will be of interest and use to many readers of the report including individual citizens through
policy makers. The report could be further strengthened by addressing the following issues:
1. The audience is not clearly defined. Given that the ROE staff tried to align the questions
included in the ROE with EPA's strategic goals and objectives as a way of better
informing Agency priorities, the main audience would appear to be EPA leadership.
However, from our discussions, it was rather clear that there are multiple potential
audiences. The ROE cannot be written in a way that will be optimal for all audiences, so
EPA must decide who the most critical audience is - or prepare different versions of the
ROE for difference audiences. In general, I expect that a format similar to the 2008 report
will be of most use to researchers and those in need of data. Such a compilation is
critically needed for a variety of reasons but it is probably not the best way to hold the
attention of most readers. The current Highlights of National Trends boils down some of
this information in a way that is suitable for the public, but another type of summary will
probably be most useful to EPA leadership - a summary that focuses on critical gaps and
spans/integrates the different chapters. Finally, a 1-2 page summary is needed for press
releases and Congressional staff.
2. Consider ways to present the indicator information in some type of common format.
Although tabular data will be useful to data miners, it is an ineffective way to convey
either status or trends. Also, consider ways to summarize all of the indicator information
into a central figure that provides an overall visualization of which parts of the
environment are in good shape and which ones are in poor shape. STAR diagrams are
often used effectively for these purposes, but the diversity of types of indicators and their
scales of measurement may make this approach difficult. Nonetheless, a standard
summary diagram that could be repeated in each report would go far toward describing
what the overall condition of the environment is. Assigning 'grades' to indicator values
may be one way to avoid this problem, but be cautious in how such grades are
implemented and what they imply (e.g., 25% species lost does not mean fair or a C in my
mind, a score that could be applied given the tyranny of how we often chunk up the 0-
100% scale into school grades).
3. The way the central questions are phrased ("What are the trends in and their effects
on human health and the environment?") sets the reader up to expect to see connections
between indicators and outcomes. But the ROE explicitly states elsewhere that no
connections will be made. This results in the reader becoming confused and/or frustrated
with the ROE's content. If connections cannot or should not be made, these questions
should probably be reworded. Also, note that you present only status information for
many indicators and trend data do not yet exist. Explicitly address/recognize this fact and
wordsmith accordingly.
4. I don't believe we (the Advisory Committee) provided very clear guidance regarding the
use of conceptual models and found our discussion of this issue somewhat vague and
24
-------
confusing. To the extent that you do explore incorporating a 'grand' conceptual model,
consider developing it in the sense of a 'conceptual framework' that helps the reader
better understand the general philosophy of indicator measurement and interpretation that
will be applied to all chapters. I suspect your choice of how to best do this will be
strongly influenced by who you decide you audience is. Also, consider if the amount of
detail you present in the specific conceptual models used in the report can affect either
choice or interpretation of indicators.
5. Retain the rigorous criteria for indicator selection when bringing in supplemental data of
regional importance.
25
-------
Comments from Dr. Lynda Knobeloch
Assignment: Does the committee have recommendations concerning other possible approaches
to conceptual model development that would be useful in identifying or highlighting important
ROE topics, indicators for consideration, research, or development?
Comments:
The conceptual model should be updated to reflect the most current publication formats and risk
assessment methodologies, as well as the rapidly evolving fields of environmental science and
synthetic chemistry.
1. Publication formats: The 2012 ROE should be available as a hard copy report. In addition,
each chapter should be published in an electronic format that is fully searchable. The ROE
website should include links to other relevant websites and publications and should be easy to
access using popular search engines.
2. Health Indicators: Indicators for human health should be updated to consider exposures
during prenatal development, early childhood, adolescence and old age. This change is
recommended because the biological response to many chemical exposures varies over the
course of the human lifespan. Developmental windows including prenatal growth, childhood
and puberty are particularly vulnerable to neurological toxicants and endocrine disrupters, for
example. The range of human health outcomes considered in the report should be expanded to
include endocrine disorders, such as thyroid disorders and diabetes, and chronic neurological
diseases, such as Parkinson's Disease and Amyotrophic Lateral Sclerosis.
3. Environmental Indicators: Several indicators used in the 2008 ROE chapters on air and water
focus on chemicals that are no longer produced or sold in the United States. While it may be
important to continue to track these substances in the environment, the report should provide
insights into the effects of newly-synthesized chemicals including chemicals used in industry,
nanomaterials, pharmaceuticals, and the new generation of agricultural pesticides.
4. Interpretation: When appropriate, environmental indicators should be linked to indicators for
ecological condition and human health. For example, temporal trends for air quality indicators,
such as ozone and particulate air pollution, should be linked to temporal trends for the incidence
of asthma, COPD, and cardiovascular disease. Similarly, temporal trends for mercury emissions
and mercury levels in fish tissue should be compared to trends in human biomonitoring for
mercury. The authors of the report should interpret the data for each indicator and provide
conclusions. Readers want to know whether things are getting better, staying the same, or
getting worse and whether there are regional differences within the U.S. Therefore, temporal
and regional trends should be described whenever possible.
Additional comments:
26
-------
For future editions of EPA's Report on the Environment to be useful to policy-makers and
others, it should provide information for chemicals that are currently sold and used in on
currently emitted substances. It should also address environmental concerns such as the
unexplained disappearance of honey bees, the finding of hermaphroditic fish and deformed frogs,
and the decline of wildlife populations such as grassland birds. Many experts have expressed
concern that newly designed pesticides, bioengineered crops, estrogenic pharmaceutical s or
industrial solvents may play a role in these disturbing trends. Previous editions of the ROE have
focused on a relatively small number of chemicals. Many of the chemicals, such as PCBs, DDT,
mirex and lindane are legacy pollutants that haven't been produced or sold in the U.S. for nearly
half a century. Continued focus on these chemicals to the exclusion of newer chemicals along
with the lack of linkage between ecologic or human health indicators to chemical pollutants
found in air and water limits the usefulness of the report.
Future ROEs should strive to find a more holistic way to assess the condition of our environment
that doesn't consider chemicals individually. Is there a way to consider, for example, the full
range of man-made chemicals that have been introduced in the past century to determine whether
our water and air are really cleaner today than they were 20 years ago? Or are we simply
replacing old chemicals that are closely monitored with newer chemicals that are not yet
recognized as threats to the environment?
27
-------
Comments from Dr. Allan Lesse
CHARGE QUESTIONS:
Charge Question 1. Please comment on whether EPA's proposed conceptual modeling
approaches are logical and useful for:
a) Aligning the ROE questions with the Agency's goals and objectives.
b) Communicating the intent and scope of questions in the ROE.
c) Presenting the underlying scientific foundation of questions in the ROE.
d) Providing a framework for selecting indicators and identifying associated gaps,
limitations, and useful supplemental information.
Charge Question 2. Does the Committee have recommendations concerning other possible
approaches to conceptual model development that would be useful in identifying or highlighting
ROE topics, indicators for consideration, research, or development?
Charge Question 3. Please comment on the logic and utility of EPA's proposed use of
supplemental information to answer questions in the next version of the ROE.
Charge Question 4. Does the Committee have recommendations for criteria to assure the
supplemental information included in the ROE is objective, free from bias, scientifically valid,
and supports the intended purpose of the report?
Comments on Charge Questions:
There is no doubt that a conceptual model is needed for the ROE. That being said, a simple
model is better than a complex model. My overriding concern, however, relates to the ROE as a
communications tool about the 'state of the environment'. The reader needs to know where we
have come from, where we are today and where we need to be in the future. The reader of the
ROE needs historical perspective. To know where you need to go you must know where you
have been. The linkages between and among environmental components need to be clearly
emphasized. The ROE must be more than simply a report card as to how we have responded to
environmental regulations/legislation. Further, it is not sufficient to simply provide data in the
ROE and leave it to the reader to interpret the data. The data/information in the ROE needs to be
interpreted for the reader because the writers/contributors to the ROE have the necessary
knowledge and understanding to provide the interpretations. One has a sense that the 'Agency'
has lost its' way and forgotten that it is called the Environmental Protection Agency with an
emphasis on 'protection'. Put another way, the Agency needs to go back to its' roots with a solid,
imaginative, proactive, forwarding-looking ' Strategic Plan'. First Nations Peoples place great
'value on the land' because they recognized long ago when you take care of the land, the land
will take care of you. It is not too late for us to learn. The ROE can help with this. It is essential
that adequate and timely support be provided to ROE staff and contributors to rethink and
reshape the ROE.
28
-------
Comments from Dr. Amanda Rodewald
Top three recommendations:
1) Clarify purpose and users of ROE, including supplemental information (see comments
below). In particular, clarifying the extent to which ROE will be used to evaluate
progress towards meeting strategic goals/objectives and/or identify/assess current and
emerging threats and vulnerabilities.
2) Construct an overarching, but simple, conceptual model that illustrates how ROE fits
within the agency's decision-making framework (e.g., the pressure, state, impact,
response model).
3) Craft an executive summary that distills and synthesizes (not summaries) the indicators to
address the questions. This synthesis requires some level of risk-taking to "take a stand",
so to say, rather than just summarize information to allow the user to interpret and draw
conclusions.
Charge Question 3. Logic and utility of EPA's proposed use of supplemental information
to answer questions in the next ROE.
Supplemental information could seem to be used in three primary ways:
(1) to fill gaps in indicators / strategic measures in order to better address the questions
articulated in ROE, with the ultimate aim to evaluate the effectiveness of policies to avoid,
reduce, or mitigate undesirable environmental and health outcomes.
(2) to identify emerging vulnerabilities or threats that require attention in the future (i.e.,
pressures/states/impacts that are not currently recognized as having high importance; giving us a
"heads up", so to say).
(3) to consolidate datasets or other information that may be useful to a broader set of users.
Relative to the first function, additional info is needed to develop strategic measures to track
progress of goals/objectives. To the greatest extent possible, indicators should parallel strategic
measures.
Relative to the second function, if part of the ROE's purpose is to identify emerging
problems/threats, then supplemental information is KEY because of the low likelihood that there
already are widely used/robust monitoring efforts of environmental or health attributes not yet
perceived to be threatened or vulnerable.
Relative to the third, it might suffice simply to include a list (with links) of datasets within an
Appendix. My supposition is that this third possible use has, at best, low priority.
29
-------
These two categories might be consistent with Dr. Anderson's suggestions of leading vs. trailing
indicators.
Currently, the EPA does not include either non-representative data sets (regional, case studies),
nor indicators that are not of national significance. The tension here is that many emerging
threats / future threats are not necessarily of national significance or widespread yet.
Some environmental / health problems or vulnerabilities that EPA should address are regional in
nature. The focus on U.S.-Mexico border, Pacific Island territories, Great Lakes all illustrate the
relevance of regional indicators.
Another benefit of considering regional data is that one could tract or identify issues or threats
that are obscured at national scales and only strongly affect certain regions or demographic
groups (e.g., urban vs. rural, economic status [environmental justice], ethnic groups).
Charge question 4. Recommendations for criteria to assure that supplemental information
included in ROE is objective, free from bias, scientifically valid and supports the intended
purpose of the report.
I like the concept of an indicator "incubator" site, where individuals could comment on
usefulness or shortcomings of various potential indicators. That said, the reality is likely to be
that few users/readers of ROE will access the incubator site. Consequently, these
future/potential indicators would be effectively invisible if not included in the report. At the
least, the report should include information about which indicators are listed in the incubator site.
Another possibility is that EPA could use an expert/Delphi process to select and/or evaluate
supplemental information.
One also could incorporate supplementary information with a simple explicit "coding" of the
reliability in the indicator (e.g., BBS uses color codes of blue, yellow, and red to indicate relative
reliability of tend data). This system provides a very quick, easy to read, approach to identifying
shortcomings.
OTHER CHARGE QUESTIONS
1. Usefulness and logic of proposed conceptual modeling approaches for:
a. Aligning ROE questions with Agency's strategic goals and objectives
I agree that this alignment is essential to make the ROE more useful and relevant for EPA and
evaluating progress towards achieving Strategic Plan objectives. The ROE team suggest that the
first three media chapters align directly with the first three goals (Clean Air & Global Climate
Change; Clean and Safe Water; Land Preservation and Restoration, but a new chapter for
Healthy Communities and Ecosystems; what about Compliance and Environmental
Stewardship?). Taking it further, organizing ROE into chapters where each represents a different
goal and the subsections the objectives (e.g., Healthier Outdoor Air, Protect the Ozone Layer)
with subheadings representing the Strategic Sub-objectives and/or Strategic Measures would be
30
-------
conceptually attractive. However, given the fact that the Strategic Plan may change
considerably, a complete reorganization of ROE may not be desirable now. At the least, ROE
could include a table at the end of each goal chapter that lists the Objectives/Subobjectives and
the results of the strategic measures.
Regarding cases where there are important indicators for which there is no corresponding
Strategic goal or objective, the EPA could (as they suggest) add a strategic goal/objective or
include a section within each chapter that includes additional indicators of environmental
condition.
b. Communicating the intent and scope of questions in ROE.
It seems that the last SAB Committee recommended that the models be process oriented, a
recommendation with which I agree. However, the EPA explicitly states that the intent of the
ROE is not to identify mechanisms / drivers of the indicators. I suggest that some recognition of
underlying processes is necessary in order to identify the best strategies to remedy/avoid
problems.
c. Presenting underlying scientific foundation of questions in ROE
Would seem to need some sort of process/mechanistic piece to do this.
d. Providing framework for selecting indicators and identifying associated gaps,
limitations, and useful supplemental information.
A closer alignment facilitates the identification of information gaps. For example, there were
many cases where the strategic measures identified in the paper for the next strategic plan (2009-
2009-2014) were not reflected in any of the current indicators.
2. Recommendations concerning other possible approaches to conceptual model
development that would be useful in identifying or highlighting important ROE topics,
indicators for consideration, research or development.
Perhaps using a Pressure-State-Response approach, as developed by Organization of Economic
Cooperation and Development (OECD).
The European Environmental Agency uses: Driving Force, Pressure-State-Impact-Response -
DF = underlying forces (e.g., basic sector trends, GDP)
P = human activities directly affecting environment
S = observable change of environment
I = Effect of the change in environment
R = response of society to solve the problem
In the case of the overarching conceptual model that integrates the ideas in the ROE and the tie
to policy, one could tie together the ROE components with the following fused model:
-------
Pressures
Responses
States
,i
Impacts
Pressures = human activities
(emissions, chemical
applications, etc.)
States = air/water/land
conditions (e.g., amt of
chemicals or pollutants in the air
or water)
Impacts on ecological condition
and human health
Responses by Agency that
influence pressures (e.g.,
policies, regulations).
Or in a more fleshed out example,
Human activities
(stressors, pressures)
Emissions, applications of
pesticides and fertilizers, land use
changes, etc.
Response of EPA
Policies, regulations,
adjustments to goals and
priorities.
States of air, water, & land
Concentrations of pollutants in air or
water, acidity of lakes and streams, high
and low flows, land cover, etc.
Impacts on human health
and ecological condition
Rates of illness and disease, cause-
specific mortality rates, biodiversity,
carbon sequestration, hypoxic zones, etc.
This model also provides a nice framework to see where the indicators fit in the iterative process,
something that helps the reader/user to conceptually "organize" the data and shows points where
the Agency is lacking information. For example, related to indicators, the detection time lag
increases through the model, with "impacts" being the most delayed. Therefore, if it became
32
-------
apparent that for some issues, all of the indicators fell on the impact end of things, that there
would be large time lags between the human actions and the ability of EPA to detect the
consequences. In such cases, EPA might expend more effect to develop indicators focused on
Pressures, given that they might provide more time for EPA to address the problem before the
unwanted environmental and/or health consequences are realized.
Other miscellaneous issues
1. Purpose:
The EPA develops policies to regulate particular stressors precisely because they are thought to
be drivers of, or contributors to, certain environmental and health outcomes. The ROE would
seem to be examining indicators of environmental and health outcomes in order to evaluate the
effectiveness of policies. Thus, the usefulness of the ROE rests upon an implicit, if not explicit,
link between the stressors and outcomes. This also touches on the previous ROE committee's
recommendations that the models be more process oriented, better articulating relationships
among stressors, media, health and environmental outcomes. Ultimately understanding the
processes that underlie the outcomes is needed to identify the most effective policies.
2. For ecological condition, they might want to more closely align both Strategic Plan and the
ROE with the EPEC's 2002 recommended framework for assessing and reporting on
ecological condition. EPEC uses the following PSR framework:
Goals/Objectives - Essential Ecological Attributes - Ecological Indicators (Endpoints) -
Measures (monitoring data)
The essential ecological attributes are:
1. landscape condition
2. biotic condition at ecosystem/community, species/population, and organism levels
3. chemical and physical characteristics (water, air, soil, sediment)
4. ecological processes
5. hydrology & geomorphology
6. natural disturbance regimes
The ROE uses only part of this framework to articulate questions in the ecological condition
chapter.
3. Including "scorecards" of sorts that overview the indicators. A simple table that has the
question followed by indicators (as in the table of contents) and the status (e.g., improving,
worsening, no change) would be very helpful. Inclusion of this overview would facilitate
use and interpretation of ROE.
4. While I think that the popular and widely-used ecosystem services framework is useful in
many contexts, I am concerned about structuring ROE around it, which would presumably
involve recasting the ROE questions to identify the levels/qualities of the supporting,
provisioning, regulating, and cultural services provided by the environment. Many of the
33
-------
specific questions and jurisdictional/regulatory mandates do not fit neatly within the services
framework because the indicators frequently assess stressors/pressures, states, and impacts.
Extending those metrics to services would involve quantifying the loss in services that follows
those impacts. I prefer aligning ROE with strategic goals/objectives.
Fig. 1
Ecosystem Services
Supporting
Carbon pool
storages
Carbon
sequestration
Nutrient
cycling
Habitat &
refugia
Primary
production
Biodiversity
Provisioning
food, fiber, fuel
water
Regulating
climate, air& water
quality, flood control,
disease & pest
control, fire control
Cultural
Recreational,
aesthetic, spiritual,
educational, existence
value, sense of place
Constituents of
Human Well-being
Security
Personal safety, secure
resource access,
security from disasters
Basic material for
good life
Livelihoods, nutrition,
shelter, goods
Good social relations
Social cohesion,
mutual respect,
ability to help others
Freedom of
choice and
action
Opportunity
to achieve
what an
individual
values doing
and being
Modified from Millennium Ecosystem Assessment
34
-------
Comments from Dr. Mark Schwartz
(Question #2).
A. Paragraph on question #2.
a. The conceptual model is incomplete as presented to us in the material for this
meeting. The conceptual model presented to us is also somewhat ungratifying
because it is very flat everything is connected to everything else at equal
intensity. In thinking about this, I see the need for thinking about this modeling
framework in three distinct ways. First, an overarching model (Dr. Rodewald's
model, the previous SAB panel suggestion, the Millennium Ecosystem
Assessment sorts of approaches). This overarching model acts as a roadmap that
can guide the reader throughout the document to understand where they are and
why what is being covered matters. The second sort of model would be an
organizational one for querying the report from different angles. I can imagine a
user wanting to query the eROE on how policies and enabling laws have worked
(has cap and trade worked); on strategic goals (how many are trending toward
achievement?; on particular issues (is haze and visibility getting better or worse
and where); or on outcomes (what are the causes of coastal water degradation and
what is the status of what we know about that). These all provide opportunities for
the ROE to allow a query to link to the indicators that address that issue. This
would alleviate comments such as whether media and outcome chapters should
remain separate or integrated, and why it is difficult to find some information.
This requires careful thought to a three dimensional linkage model for this and
how it gets programmed into a web access tool. The third kind of model is more
like the stressor to outcome model presented to us. This has to be a component of
the second model, but I distinguish it because it needs to make some arrows
darker than others: not all linkages are equally important; not all linkages are
under EPA jurisdiction. This kind of cause and effect model can help the reader
understand the structure of the report and why it contains the information it does
in the same way that the overarching model can.
B. Other thoughts on the remaining questions
a. (3). Supplemental indicators are critical for making this useful, particularly for
prospective problems. The proposed use of them is good. It could much further,
with additional resources.
b. (4). Meta-analysis; contracted peer reviewed reviews of major issues. Take a look
at the Annual Reviews of Ecology, Evolution and Systematics or Annual Reviews
of Environment and Resources. I could see the EPA sponsoring such reviews for
pathways in the overarching model that are not well captured by national
indicators and then using this as an unbiased report on status and trends on that
question.
35
-------
c. (la). Aligning the ROE questions with the strategic goals. This is an outstanding
idea. It is very important. This has to be accompanied with some rationale that the
reader can get to that explains the strategic goals. Although this is not the ROE
team purview per se, a goal is not strategic unless it is accompanied by a strategy.
As it is written, there are no obvious strategies linked to strategic goals. It would
be helpful to the reader to outline EPA's strategies for achieving these goals.
d. (Ib). Linking the report to strategic goals and a conceptual model that draws the
linkage from driving legislation through EPA policies all the way to human and
ecosystem health outcomes is a way to define the scope of this report. This is
critical because it is not intended to be a report on the entirety of the environment,
and that can make that point very clear. It needs to be very clear.
e. (labcd). There is room for interpretation of the data. I strongly urge the ROE team
to make the leap from simply reporting the data to interpreting the data for the
reader.
C. Other thoughts, in general.
a. Terrific work. This is coming much further, much faster than I had imagined
when I was on the panel in 2004. You have a huge task. You are definitely going
in the right direction.
b. I support the SAB to make a strong statement to the administrator on how this
represents a terrific opportunity for the agency to shine, but that it needs more
resources at its disposal to do the job justice. Probably at least double the staff and
a dedicated web design person, maybe more.
D. Three most important things
a. Get the model frameworks right for readability and accessibility. ROE needs an
overarching conceptual model that links the media (land, air, water) into an
integrated look at ecosystem and human health and well-being outcomes. But, an
electronic ROE also needs an operational model that allows one to approach the
information from a variety of angles and get to the information quickly and
efficiently. I describe this above in my thoughts on a modeling framework.
b. Simple access tools to increase impact. Use simple color (or other) codes for how
well the indicator actually captures problems such as: (a) the degree of uncertainly
in the indicator trend; (b) the indicator trend; or (c) the degree to which an
aggregate of trends indicate positive or negative change. Executive summaries
and briefing sheets are good. Staggered rollouts on specific issues would help
impact as well.
36
-------
Regional problems require regional treatment. Not all problems are national. This
is particularly true of ecosystem problems. Recent studies have emerged showing
an alarming increase in tree mortality in the west. This is likely climate change
driven. We have long recognized increased tree mortality in the east driven by
acid rain. We also have increased tree mortality through invasive species (e.g.,
gypsy moth) and urban island effects (high urban tree mortality). These impacts
have different drivers, and hence different monitoring programs and hence
different data sets. Together they represent something very important, but lacking
EMAP, there might not be a good way to capture this with an indicator. Regional
indicators would help alleviate this problem. The EPA may see its mandate as
national in scope, but most readers want to know about the state of the
environment as it relates to EPA and its state equivalents. They will not make the
fine distinction between national, regional and state jurisdictions. Hence the report
should make every effort not to distinguish either.
37
-------
Comments from Dr. Alan Steinman
Question 3. Please comment on the logic and utility of EPA 's proposed use of supplemental
information to answer questions in the next version of the ROE.
Supplemental information can play an important, complementary role in helping inform EPA
about the status of the environment in the US. I believe these data should be used in a
discriminating way, especially in cases where national data are unavailable.
One of the key questions will be how these data are used in the ROE. If the supplemental data
are quarantined in some appendix, they will have little value and in which case, it is hard to
recommend they be included given the limited resources available to produce the ROE. A much
better approach (in my opinion) would be to either 1) include hyperlinks in the main body of the
report to the relevant supplementary data set, which would render the physical location of these
data moot, at least in the electronic version; or 2) integrate them in the main body of the text,
clearly identified as supplementary, as they pertain to key questions.
One possible approach would be to attach a metadata file (or table) with each supplementary data
set that contains information on data source (i.e., agency, etc.), data rigor (i.e., have they been
peer-reviewed and/or published in the scientific literature?), geographic region of significance,
and EPA opinion on overall confidence level of data (low, moderate, high). We adopted a
similar approach when assessing the quality of economic data we used to generate valuations of
ecosystem services in west Michigan. See
http://invest.wri.gvsu.edu/croplandfoodproductionvalues.html for an example of our approach.
Question 4. Does the Committee have recommendations for criteria to assure that supplemental
information included in the ROE is objective, free from bias, scientifically valid, and supports
the intended purpose of the report?
The standard approaches for ensuring data quality should be applied first: have the data been
peer-reviewed and/or have they been published in the scientific literature? There should be a
secondary level of screening, as I suspect many potentially useful data sets have not been peer-
reviewed, which would include assessment of data collection methodology, such as the presence
or absence of a QAPP.
General Comments on Conceptual Models:
1) EPA may want to consider using conceptual models with similar architectures, especially
when at the same hierarchical level. There is an intuitive appeal in this type of parallel
construction, although I recognize there may be a need for different architectures for
human health vs. environmental condition models.
2) Two different general classes of conceptual models to consider include what I refer to as
a) ecological process models and b) operational process models. General examples of
each are given below:
38
-------
I. Ecological Process Model:
Land Use Change
Stormwater Runoff
Waterborne Pathogens
/ Attribute \
Beach Closings
II. Operational Process Model:
Improve Water Quality
Reduce stormwater runoff
Install rain gardens
IResearch\
\ Needs /
GISto identify best
locations
Three critical issues for ROE:
1) Greater integration across indicators and chapters.
39
-------
2) Addressing appropriate spatial scales for analysis (i.e. prior concern over use of EPA
regions as the spatial scale of choice).
3) Develop a "Citizen's Guide" to the ROE that uses simplistic language, colorful graphics,
and a transparent (but rigorous) approach to convey the key messages of the document.
Final General Comment:
This issue emerged at the very end of our meeting, but I raised it at the start of the meeting with
Dr. Teichman (funny how we remember our own questions best) about dedicated resource
allocation to the ROE effort. This was a concern we have expressed in the prior two ROE
committees, and Dr. Teichman's response was pithy (our staff is dedicated in multiple ways...)
but lacked substance in terms of whether or not the allocation was sufficient.
I believe it would be very instructive to have an analysis done on how many FTEs and resources
(since much of the work on the ROE is contracted out) have been allocated to this effort, per
fiscal year, since its inception. This should be relatively easy to track, and would provide us with
a useful "indicator".
40
-------
Comments from Dr. Helen Suh
MEMORANDUM
Charge Question: (1) Please comment on whether EPA 's proposed conceptual modeling
approaches are logical and useful for presenting the underlying scientific foundation of
questions in the ROE. (2) Provide a framework for selecting indicators and identifying
associated gaps, limitations, and useful supplemental information.
The proposed conceptual models are helpful and when finalized, should achieve their stated
goals to (1) better support for the Agency's planning, problem formulation, and decision making
and (2) make the conceptual underpinnings of the questions and indicators clearer to the reader.
It would be helpful to include an overall conceptual model that span all media and indicators and
media-specific models in the introduction to supplement media specific models in the relevant
chapters. The models could be improved further with the following:
A clear understanding and statement of the Report's mission, objectives and intended
audience;
Inclusion of boxes or arrows or some other indicators of "effect modifiers", such as
meteorology, time-activity patterns, or age that will influence movement from one
column to the next (e.g., emissions to concentrations or concentrations to exposures);
Explicit linkages between the conceptual model and the relevant indicators, their
limitations and gaps;
Illustration of how the indicator trends affect movement into the adjacent columns (e.g.,
contaminant source, concentration, exposure, etc.);
Inclusion of how climate change fits within the conceptual models;
An indication of priorities or most important aspects within each column of the
conceptual model and explanation of these priorities (e.g., drinking water from XXX
source is less important than that from a different source given the small percentage of
people who drink water from this source; inhalation not as important as ingestion for
some pollutants).
In addition, after the Report mission is decided, it will be important to determine where the
conceptual model fits into the organization of the report as soon as possible. For example, an
overall conceptual model that spans all media would conceivably fit into the ROE introduction;
however, it is not clear how the media-specific conceptual models would be integrated into the
document. Would the media-specific models go into the beginning of each media-specific
chapter? If so, would the health and exposure chapter refer back to the conceptual model in that
Chapter? If not, would the model be placed into a conclusion or summary of the document?
This quandary speaks to the need to reexamine the organization of the document and whether it
makes sense to incorporate the Health and Exposure and Ecology chapters within the media-
specific chapters, the possibility of moving to a one environment approach, and the division
between the print and web-based platforms.
Other Issues:
41
-------
In the preparation of the Report, the following issues or changes should be considered:
Greater linkage between indicators, trends and policy actions.
A defined place for discussions of emerging indicators and issues, perhaps in
supplemental information but also in an executive summary, to show that EPA is aware
and is beginning to monitor (or at least think about) these emerging or possibly important
issues.
Substantial emphasis on a powerful and interactive web-based platform, with links to
pertinent governmental websites; possible to de-emphasize the written document and
replace with greater presence of web-based platform; may need additional resources to
address this issue.
42
-------
Comments from Dr. Due Vugia
In the EPA's Office of Research and Development draft of June 2009 "Issues for Initial
Consultation with the SAB Advisory on EPA's ROE," the 2012 ROE is proposed to be
restructured to align more visibly with the Agency's strategic plan, to include conceptual models
for each ROE question to make the relevance of each question clearer and more useful to
readers, and to include "supplemental information" to provide additional insight into
environmental issues, particularly those with indicator gaps and limitations. Here are my
comments on whether EPA's proposed conceptual modeling approaches are logical and useful
for:
Ic. Presenting the underlying scientific foundation of questions in the ROE.
Two examples of possible conceptual models were presented, one on the 2008 ROE outdoor air
policy question and the other on the 2008 ROE drinking water question. I see these proposed
conceptual modeling approaches as very logical and very useful to both present and enhance the
understanding of the underlying scientific foundation of the ROE questions to readers. As
readers of the ROE will more likely be non-scientific people, the level of details of the models
should be easy to understand and to grasp, probably not much more detailed than the illustrated
examples given. And as several of the questions may overlap regarding a conceptual model, it
may be helpful to consider, for each medium such as air or water, starting with a more inclusive
conceptual model for that particular medium addressing several or as many questions under that
medium as possible, and then for some but not necessarily all questions, provide a slightly more
detailed conceptual model particular to that question. An overarching conceptual model
integrating all environmental media and the effects of their contamination on human, animal, and
ecological health would be very helpful at the beginning of the ROE, before breaking out into
each medium, to set the stage for readers on how components and concerns of the ROE fit into
this overall understanding of environment and health.
Id. Providing a framework for selecting indicators and identifying gaps, limitations, and useful
supplemental information.
The proposed conceptual models, such as the examples for the drinking water and outdoor air
questions, would provide a useful common framework for scientists to identify and discuss
information needs, gaps, and supplemental information. For non-scientists and policy makers,
the proposed conceptual models provide a common framework to understand why there are such
information needs and where the indicators and supplemental information fit within the
framework.
However, the strict criteria for selecting national indicators limit the number and variety of
useful measures that can be selected, thereby leaving wider information gaps than really need be
given the large amount of useful measures available in the current literature or in collected data.
If the criteria for national indicators can be relaxed slightly to include similarly rigorous and
scientifically reproducible data collected at some regional level, or at several but not all cities
and states, then several more useful measures can be considered to give a clearer understanding
43
-------
of the current status and trend of several environmental questions, and their informational gaps
would be decreased. If the national indicator criteria cannot be relaxed, then more supplemental
information is needed to provide better understanding of status and trend on the questions (more
comments on supplemental information below).
Regarding the three most critical points that EPA should consider for the 2012 ROE, my current
thoughts are:
1. Consider adding, into the beginning of the ROE before breaking out into the five EPA
areas of interest, an overarching conceptual model integrating all environmental media
and questions showing how elements of the environment affect human, animal, and
ecologic health.
2. Consider adding a moderately substantive and informative Supplemental Information
section to each of the five areas/media using defined criteria similar to but not as strict as
the criteria used for national indicators.
3. Consider expanding on Emerging Issues under each area/medium to acknowledge recent
or emerging environmental concerns and to provide some information of ongoing
research into these issues.
All these points will make the 2012 EPA ROE more useful and informative to readers of all
backgrounds, and it may help policy makers understand environmental issues better and be more
open to requests for funding for emerging environmental issues or where information gaps exist.
Substantive additions of Supplemental Information and Emerging Issues in the next ROE may
also help influence the direction of some research.
Regarding adding Supplemental Information, I fully support this as it will be an important part of
the next ROE to address information gaps or lack of suitable national indicators for several
questions. It shows readers that EPA is aware of and is willing to share or point out good data
and information that help shed more light on some questions, particularly since there are not
enough national indicators to show status and trend on all questions. Supplemental Information
to be included in the next ROE should be selective, and the criteria to consider for selection can
include many of the same criteria as for national indicators, particularly that the information is
useful (it answers a question in the ROE), objective (accurate, unbiased manner), transparent and
reproducible, characterized by sound collection methodologies, and comparable across time and
space. In contrast to criteria for national indicators, supplemental information do not have to be
available from all states across the nation, but can be from several states or cities or from
regional areas. Additionally, data from multiple sites and published in peer-reviewed journals
should also be considered; if meta-analyses have been done on such studies or multiple data sets,
then those comprehensive review studies should be even better as they likely standardized
criteria across studies to address an issue. Finally, potential national indicators may be identified
from Supplemental Information, which then serves as an incubator for future useful national
indicators.
44
-------
I look forward to the next conference call or meeting of this EPA SAB Advisory Committee on
the 2012 ROE.
45
-------
Comments from Dr. Judith Weis
Question 1 c, d. Conceptual modeling for presenting underlying scientific foundation of
questions and providing a framework for selecting indicators and identifying gaps, limitations
etc.
The conceptual models are a good framework for presenting the underlying science and thinking
about the questions. I think they could be placed initially into the larger picture of "How are we
doing?" Introductory material should discuss how unfettered pollution and environmental
degradation in the 1960s led to the creation of EPA and passage of regulatory laws. We have
been reducing emissions for -40 years and the ROE is a look at how well these reductions in
emissions has led to reductions in ambient concentrations of pollutants and to improved
environment and human health.
The links between reductions in releases/emissions of pollutants and the resulting improvements
in environmental quality or human health are not always clear or "provable," but, based on
decades of scientific study we know they are linked. Whenever there are data that can be used to
demonstrate this link, these should be emphasized (perhaps in supplemental information). A goal
within the report (that may not be realized, but should be stated as a major need) should be to be
able to answer the question "how much reduction in emissions or environmental concentration
is needed to produce how much environmental improvement?" For example, "how much do we
have to reduce nitrogen inputs to an estuary in order to reduce phytoplankton blooms and
hypoxia by x amount?" These questions may point the way toward needed regulatory changes -
it may be necessary to ratchet down the releases more than we have done thus far in order to
achieve the desired environmental improvements - and should lead to changes in Total
Maximum Daily Loads (TMDLs) or other standards (and perhaps, for the first time to regulation
of non-point source runoff).
46
-------
Comments from Dr. Stephen Weisbers
Question Ic. Are EPA's proposed conceptual modeling approaches logical and useful for
presenting the underlying scientific foundation for questions in the ROE?
Only to a limited degree. The models properly communicate a basic underlying scientific
foundation by connecting stressors with state and response. However, the models that were
presented are media specific, providing little insight into how priorities were assigned across
media or how the chapters are intertwined. There should be an overarching model that more
holistically integrates across media. Perhaps the SAB's recent endorsement of ecosystem
valuation could provide that framework.
It is also interesting that the chapter models are used differently in selecting the questions. For
instance, the air chapter includes numerous questions and indicators about effectiveness of
source control, but comparable questions/indicators are missing from the water chapter, even
though the same issues are important and apparent in the conceptual model for that chapter.
Question Id. Are EPA's proposed conceptual modeling approaches presenting a logical
framework for selecting indicators and identifying associated gaps, limitations and supplemental
information?
Again, only partially. EPA's conceptual model is based on ecologic processes, describing
drivers, stressors, state, and response. That may be appropriate for selecting the best possible
indicators from a scientific perspective, but that may not be the best model for meeting the needs
of this document's audience, for which scientists are secondary. The document seems targeted
toward providing feedback to EPA's management about their performance in protecting the
environment. This is emphasized in linking the ROE questions with EPA's strategic plan. In
that case, an operational process model that focuses on linking management actions with desired
environmental outcomes would be a more appropriate conceptual model. These two types of
models are not mutually exclusive, as the operational model is hopefully informed by the
ecological process model, but there are differences in emphasis which would lead the operational
model toward more relevant questions and indicator needs.
Top 3 priority suggestions:
1) Audience: Develop a clear mission statement that identifies your client(s) and the
decisions you hope to affect with the ROE.
2) Linkage: Move past presenting a series of independent status indicators, instead
presenting them in parallel with indicators that assess effectiveness of agency actions
intended to affect that status.
3) Communication: Prepare shorter summary documents that better communicate the core
messages.
47
-------
Enclosure 3: EPA Charge Questions to the Committee
Issues for Consultation
Background
Purpose and History of EPA 's Report on the Environment (ROE)
In 2001 EPA initiated work to assemble an extensive set of environmental indicators in order to
provide high quality information on the state of the environment. A goal of this effort was to
compile and present indicator status and trend information that would enable EPA and the public
to assess progress toward accomplishing EPA's goals of cleaner air, purer water and better
protected land. EPA presented these indicators in its 2003 Draft Report on the Environment
Technical Document and its summary Draft Report on the Environment Highlights Document.
Following Science Advisory Board (SAB) reviews of the 2003 draft ROE and a revised 2007
draft of the report, EPA published its 2008 Report on the Environment. In 2008 EPA also
published a shorter ROE document that highlighted national trends. EPA's 2008 ROE provided
indicator information to answer 23 "policy- relevant" questions that the Agency found to be
critically important to its mission. Thus, although the 2008 ROE provided information on a
broad range of indicators, the focus of the report shifted toward providing information that was
relevant to EPA's mission.
In September 2008, EPA also released an online electronic Report on the Environment, or eROE.
The eROE (www.epa.gov/roe) contains the current indicator data that are updated quarterly.
SAB Recommendations to Improve the draft 2007 ROE
In its peer review of the draft 2007 ROE, the SAB emphasized the value and importance of the
report. The SAB strongly supported continued development of the ROE and provided
recommendations to improve the report before its publication in 2008. The SAB also provided
long-term recommendations to improve future versions of the ROE. In general, the SAB found
that:
The ROE lacked a framework describing the scientific understanding of relationships
between indicators and the basis for including them in the report.
The ROE presented status information to establish baselines for reporting future trends, but
the lack of long-term trend information in the document precluded trend analysis for many
indicators.
48
-------
The rigid application of indicator selection criteria resulted in the exclusion of valuable and
relevant information that could be used to further analyze trends.
The ROE was limited because it contained little data interpretation and no conclusions
supported by statistical analysis.
The current SAB consultation focuses on issues regarding the first three bullets.
Scope of the ROE 2008
The 2008 ROE was organized around five main chapters, "Air," "Water," "Land," "Human
Exposure and Health," and "Ecological Condition." Each chapter was organized around a set of
critical "policy-relevant" questions that EPA wanted to answer with confidence in order to be
adequately informed about important environmental trends. However, EPA stated that these
questions could not necessarily be fully answered with indicators that met the Agency's indicator
definition and six indicator selection criteria in the report. In the ROE 2008 EPA defined an
indicator as a numerical value derived from actual measurements of a stressor, state, or ambient
condition, exposure, or human health or ecological condition over a specified geographic
domain, whose trends over time represent or draw attention to underlying trends in the condition
of the environment. The ROE 2008 did not include presentations of statistical confidence in the
status of and trends in the indicators. When indicator trends were reported they were interpreted
as the direction of change, and did not imply statistical significance. EPA recognized that
uncertainty is an important issue and stated that it planned to quantify uncertainty in future
versions of the ROE and its indicators.
Future Direction of the ROE
EPA intends to publish the next full edition of the ROE in 2012. Emphasis will be placed on
presenting the status of and trends in environmental and human health conditions of interest to
the EPA in order to inform the Agency's planning and decision making. EPA intends to
restructure the ROE to more directly align chapters of the report with EPA's Strategic Plan goals,
and to align the policy questions in the ROE with objectives in the Agency's Strategic Plan. Any
revision of the ROE in this regard must take into account the new Administration's long-term
perspectives on strategic planning. Although a draft of EPA's new strategic plan for 2009-2014
has not yet been released, the overall architecture of the Strategic Plan (with goals, objectives,
and sub-objectives) is not likely to change. As stated above, EPA also plans to enhance indicator
information in future versions of the ROE to include quantitative uncertainty information.
Issues for the SAB Consultation
Overarching Issues
In its review of the draft 2007 ROE, the SAB recommended including conceptual frameworks in
the report to illustrate scientific understanding of relationships between indicators and the basis
for including them in the report. The SAB also recommended that EPA relax restrictive
indicator selection criteria to enable the use of additional indicators that could inform the stated
49
-------
questions. EPA's Office of Research and Development (ORD) is seeking early consultation with
the SAB on conceptual models for restructuring and refining the next version of the ROE in
order to better support Agency planning, problem formulation, and decision making and make
the conceptual underpinnings of the questions and indicators clearer to the reader. ORD is also
seeking consultation with the SAB on the proposed use of regional and sub-regional indicators
and supplemental information to help answer ROE questions.
ORD has developed an issue paper for the consultation describing how EPA proposes to
restructure and refine the next version of ROE. The issue paper contains two examples of
conceptual models to illustrate the scope of the questions and to select indicators. One example is
a generalized conceptual model framing the 2008 ROE question, "What are the trends in the
quality of drinking water and their effects on human health?" The other example is a conceptual
model framing the 2008 ROE question, "What are the trends in outdoor air quality and their
effects on human health and the environment?" Section 4 of the issue paper discusses EPA's
proposed use of supplemental information in the next version of the ROE. Specifically, ORD
has requested consultation on the following issues.
Specific Issues for Consultation
1. Please comment on whether EPA's proposed conceptual modeling approaches are logical
and useful for:
Aligning the ROE questions with the Agency's strategic goals and objectives.
Communicating the intent and scope of questions in the ROE.
Presenting the underlying scientific foundation of questions in the ROE.
Providing a framework for selecting indicators and identifying associated gaps, limitations,
and useful supplemental information.
2. Does the Committee have recommendations concerning other possible approaches to
conceptual model development that would be useful in identifying or highlighting important
ROE topics, indicators for consideration, research, or development?
3. Please comment on the logic and utility of EPA's proposed use of supplemental information
to answer questions in the next version of the ROE.
4. Does the Committee have recommendations for criteria to assure that supplemental
information included in the ROE is objective, free from bias, scientifically valid, and
supports intended purpose of the report?
50
------- |