\ UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
\ ^SZZ * WASHINGTON, D.C. 20460
I t^MMK^ a
OFFICE OF
Gary S. Sayler, Ph.D. RESEARCH AND DEVELOPMENT
Chair, Board of Scientific Counselors
Center for Environmental Biotechnology
The University of Tennessee
676 Dabney Hall
Knoxville, Tennessee 37996
Dear Gary:
On April 25-26, 2007, the Science and Technology for Sustainability (STS) Program
Subcommittee of the Board of Scientific Counselors (BOSC) met in Cincinnati, Ohio to evaluate the
Office of Research and Development's (ORD) Sustainability Program. Following that review, the
Subcommittee presented a report of its findings and recommendations to the Executive Committee
of the BOSC on September 17, 2007, and the Executive Committee, in turn, provided a final BOSC
report to the ORD on April 22, 2008. With this letter, I am pleased to enclose the Agency's response
to the final BOSC report on its review of the STS Program.
The program benefited a great deal from the insight and advice offered by the Subcommittee,
and the recommendations were greatly appreciated. The attached narrative presents an overview of
the recommendations made by the BOSC and provides a brief comment in response that indicates
how the STS program has taken the findings into consideration. A table that summarizes each
recommendation, the action to be taken, and a schedule for completion of the action is also attached.
As you are aware, ORD conducts periodic evaluations of progress for each of its research
programs at intervals of four to five years. The purpose of these reviews is to determine progress
with regard to relevance, quality, performance, and scientific leadership. The reviews also focus on
identifying how the scientific community and programmatic clients utilize ORD's scientific outputs
to protect human health and the environment. In addition to these formal reviews, ORD evaluates
program progress midway through the review cycle. These mid-cycle reviews provide critical
feedback to the program concerning its progress since the last review and the extent to which
recommendations from that review are being met.
The date for the mid-cycle review of the STS program will be March 12, 2009, and we look
forward to demonstrating our progress at that time. In the meantime, if you or other members of the
BOSC have any questions about the enclosed response, please contact Alan Hecht, through Greg
Susanke the appropriate DFO of course. Thank you again.
Sincerely
KevinY: Teichman, Ph.D.
Deputy Assistant Administrator for Science
Enclosure
Internet Address (URL) • http://www.epa.gov
Recycled/Recyclable • Printed with Vegetable Oil Based Inks on 100% Postconsumer, Process Chlorine Free Recycled Paper
-------
Office of Research and Development's (ORD) Response to the
Board of Scientific Counselors (BOSC) Report on the Review of ORD's
Science and Technology for Sustainability (STS) Research Program
(final report received April 2008)
December 2008
BOSC Technology for Sustainability Subcommittee
Dr. John Giesy (Chair)
Dr. Martin Abraham
Dr. Earl Beaver
Dr. Conception Jimenez-Gonzalez
Dr. Wayne Landis
Dr. Ted Tomasi
Dr. Peter Corcoran
-------
December 2008 Office of Research and Development's (ORD) Response to the
Board of Scientific Counselors (BOSC) Report on the Review of ORD's Science and
Technology for Sustainability Research Program (final report received April 2008)
The following is a narrative response to the comments and recommendations of the
review by the Board of Scientific Counselors (BOSC) of ORD's Science and Technology
for Sustainability Research Program (STS), which took place in April 2007 in Cincinnati,
Ohio. We thank the members of the BOSC Science and Technology for Sustainability
Research Subcommittee for their comprehensive review of the Research Program's multi-
year plan (MYP). Based on the Subcommittee's comments and recommendations, ORD
has already initiated efforts to modify the STS Program and looks forward to the mid-
year review scheduled for March 2009 for further discussion and feedback on the
Program's evolution and achievements.
For the April 2007 review, which is considered to be part of a series of consultative
reviews, the BOSC Subcommittee focused on the STS's strategic goals; its program
relevance, quality, and structure; its scientific leadership, coordination, and
communication with stakeholders; and the degree to which research outputs are being
used by stakeholders, e.g., outcomes. The panel addressed a number of charge questions
intended to focus on each of these themes. It rated many parts of the STS Research
Program favorably and found that the Program was meeting expectations. The
Subcommittee recognized the high caliber of excellent researchers who are achieving
high-quality research with relatively limited resources. The Subcommittee recognized
that the current structure of the Program was well suited for the development of decision
support tools that promote environmental stewardship and sustainable management
practices. The BOSC also acknowledged that the Program as planned would be able to
develop, apply, and demonstrate innovative technologies that solve environmental
problems and provide sustainable outcomes. At the same time, the BOSC offered
numerous suggestions to improve the program. The insights provided by the BOSC
Subcommittee are helping ORD to better focus its planned STS research.
In preparing this narrative response, ORD grouped the Subcommittee's many detailed
comments into seven overarching strategic issues related to the charge questions. Under
each issue, relevant Subcommittee comments are printed in italics and ORD's response
follows in regular type. Attached to this document is a summary table that provides a
more comprehensive list of BOSC comments and ORD responses.
Strategic Issue #1: Better define, communicate, and coordinate metrics research
and its outputs and better explain their relationship with other components of the
STS.
The use of rigorous metrics is critically important in the development of decision-
making tools and also should drive research needs in both internal and external
programs. A key component of the development and testing of appropriate
metrics is a clear conceptual definition of what is to be measured with a
particular set of metrics. Thus, clear definitions of the Sustainability concepts
being addressed and the component elements of these concepts are required
-------
December 2008 Office of Research and Development's (ORD) Response to the
Board of Scientific Counselors (BOSC) Report on the Review of ORD's Science and
Technology for Sustainability Research Program (final report received April 2008)
before a specific metric can be assessed and its applicability in real-world
situations evaluated. Additional attention needs to be given in this element to the
process of development and evaluation of sustainability metrics. Testing of
specific metrics in real-world situations also is appropriate, but one needs to
propose and develop the metrics first. Then, the testing protocols should be
established to determine if the metrics are measuring the intended functions, if
they are consistent in their evaluation, if they are sufficiently independent, and if
they can be effectively used to determine if specific actions are driving society to
become more sustainable.
A clear definition of sustainability and a framework for its application is required
to fund appropriate extramural research programs and to determine the efficacy
of specific metric or decision tools.
The metrics developed under the P2NT Research Program have not pervaded
other programs.
Given their limited numbers, the team leaders are having an appropriate impact
on the development of scientifically based sustainability metrics. A team that was
better integrated throughout EPA could draw on additional resources that could
enhance its effectiveness.
The development of sustainability metrics is a critical component of the overall
effort, because these are the measures on which the success of all activities needs
to be evaluated. It is unclear, however, precisely how the metrics to be developed
within this element will be used in other LTGs, and it also is unclear how the
metrics to be developed will be informed by activities in the other LTGs.
While the P2NT Research Program was not specifically designed for or focused on
metrics development, ORD agrees with the BOSC perspective that development of
metrics is an important area and that we must work diligently to ensure they are widely
used within and outside the Agency to guide decisions on sustainable alternatives. The
STS is only one of many programs across the Agency that will address sustainability.
We concur that the Agency needs to further develop definitions of sustainability concepts
and define how sustainability can be made operational. However, resolution of this issue
transcends the STS and must be addressed across the Agency. Over the next year, ORD
will coordinate a series of workshops to facilitate a discussion on this issue. These
workshops will be designed to engage key Agency officials in Program Offices and
Regions to obtain their perspectives. Ultimately, the decisions regarding an Agency
sustainability framework, including appropriate definitions, will be the responsibility of
these officials. Any metrics resulting from these discussions will need to incorporate
clear conceptual definitions of what is to be measured. In general, development of
sustainability concepts and metrics will be produced to address specific problems of
national importance consistent with the following recommendations of the Environmental
Engineering Committee of EPA's external Science Advisory Board:
-------
December 2008 Office of Research and Development's (ORD) Response to the
Board of Scientific Counselors (BOSC) Report on the Review of ORD's Science and
Technology for Sustainability Research Program (final report received April 2008)
The Committee supports application of sustainability principles to address
and resolve specific, multi-faceted environmental problems.
The Agency should be prepared to undertake some "higher risk- higher
payoff projects, i.e., projects that because of complexity, data requirements,
methodological novelty, and interdisciplinary focus, may be challenging, but
would have a large impact if they are indeed developed successfully. The
project portfolio should also balance targeted Agency needs and geography.
In a few key areas, including urban development and biofuels, ORD is already working
on developing sustainability metrics that be used as the foundation for discussions at the
workshops mentioned above. ORD is using its new work on biofuels as a model of how
we might work across EPA to address such other key issues as materials management and
ecosystem protection. ORD is now working with other federal agencies to define a set of
criteria and indicators for sustainable biofuel production. The work requires clearly
identifying the critical elements of biofuel production and identifying relevant indicators
to measure progress toward sustainability. Information contained in the EPA Biofuels
Strategy can help to guide STS activities in the biofuels area. In addition to the biofuels
work, ORD is also developing a set of system metrics that represent the most
fundamental properties and processes that must be preserved to ensure the sustainability
of a particular geographical system or region. These metrics will be used to ascertain
whether the region is moving towards or away from sustainability over time and to
promote adaptive environmental management for sustainability.
We think the above examples address the BOSC concern about defining sustainability
and "testing of specific metrics in real-world situations also is appropriate." In the first
case, ORD is testing the model in the growing biofuel sector. In the second example,
ORD is developing and evaluating the application of metrics in specific geographic
region of importance. Also, the Collaborative Science and Technology Network for
Sustainability (CNS) has funded 23 projects across the country that are developing
coupled decision-making tools and metrics and applying them for real-world
sustainability decision-making in diverse settings. While we cannot adjust these projects,
ORD will sponsor a webinar series followed by a workshop to bring together internal and
external investigators to identify any lessons learned from the CNS program and
determine if there are any specific outputs from the ongoing projects that could be used to
enhance internal research efforts.
Also, ORD will expand the existing metrics team to include other parts of the Agency.
For example, an internal EPA team focused on sustainable biofuel production has been
organized and is already discussing issues related to metrics.
In the revised MYP, the general approach for the development of sustainability metrics
will be described including criteria to assess their utility. Finally, as part of the STS
revisions, ORD will add language to the MYP to ensure that appropriate linkages are
-------
December 2008 Office of Research and Development's (ORD) Response to the
Board of Scientific Counselors (BOSC) Report on the Review of ORD's Science and
Technology for Sustainability Research Program (final report received April 2008)
made between the metrics LTG and the balance of the Program. The revisions will factor
in the role of metrics and other driving forces.
Strategic Issue #2: Relevance of the Green Technology Program (GTP) is unclear in
light of activities in industry and other federal agencies.
The Green Technology Program is less relevant and there is a less clear
connection to Program goals. It was not clear what activities were underway that
could not be accomplished in the public or academic sectors in the absence of the
EPA program; thus, the public benefits were not evident.
The Green Technology Program as currently configured might be perceived to be
largely irrelevant. Consideration should be given to redirecting the Program or
replacing it with an extramural grants program.
The relevance and impact of the Green Technology Program is less apparent and
this program needs to be assessed internally to determine if it is serving a
function that is not being met already by the private sector andacademia.
Results derived from the Green Technology Program have not been effectively
communicated to larger industrial enterprises.
The original goal of the GTP was to perform research that could be broadly applied to
many processes in industry. The research results from the Program's many projects have
been the subject of numerous presentations at national professional meetings attended by
representatives of nearly all major chemical producers. In the ten year period from 1996
to 2006, the Green Chemistry program published 384 peer reviewed papers, 34% of
which are listed among the top 10% of all cited papers in their field. This work was
coupled with successful decision support tools such as PARIS (Program for Assisting the
Replacement of Industrial Solvents) and SAGE (Solvent Alternatives Guide) that have
assisted industry to identify cleaner products and processes. However, as the need for
metrics and decision support tools has increased, as well as the green technology efforts
of industry and others, ORD has eliminated in-house research to investigate new green
technologies and chemistries.
The extramural portion of the Program has continued its partnership with the National
Science Foundation (NSF) to identify research needs in areas such as green chemistry
and green engineering in rapidly growing (or changing) sectors (including energy and
nanotechnology); green building; advanced life cycle assessment, material flow analysis,
and related systems analyses; and environmental sensors and information systems. The
partnership is currently focusing on Green Building.
Strategic Issue #3: Use sustainability criteria to evaluate proposals (SBIR and P3).
-------
December 2008 Office of Research and Development's (ORD) Response to the
Board of Scientific Counselors (BOSC) Report on the Review of ORD's Science and
Technology for Sustainability Research Program (final report received April 2008)
The solicitation/judging criteria for the P3 Program should be improved to
require a clear statement by students as to effects articulated via sustainability
metrics or decision tools. More emphasis should be placed on measurement.
This will force students to more clearly articulate how their projects relate to
sustainability. The ETV Program should encourage an increased role in
supporting emerging markets in trades/mitigation/offsets, such as
mercury/greenhouse gases, etc. An analysis should be conducted to determine if
there are emerging markets in the trade/offset areas that have barriers to
verification. Then, research could focus on how to solve or minimize these
impediments to verification and subsequent use. The SBIR Program should
increase the use of sustainability metrics in the selection criteria. It also should
increase the linkages between Program outcomes and sustainability metrics.
The Program could benefit from a more systematic evaluation of the program
outcomes, such as tracking of careers of recipients to obtain information that can
be used to measure impact as outcome. Thus, a detailed analysis of the impacts
on the P3 Program on the student participants is desirable.
ORD understands the BOSC desire to ensure any proposal funded with ORD STS
resources include measurement of progress towards sustainability. Many of the ongoing
research projects already consider metrics and measurement in their design and
implementation. For example, the P3 Program requires student teams to quantitatively
and/or qualitatively articulate the benefits of their project in the social, environmental,
and economic dimensions, both at the proposal stage and at the final report stage. For the
Small Business Innovation Research (SBIR) program, the strongest opportunity to focus
on sustainability is in the identification of solicitation topics. Though SBIR is structured
to address environmental technology priorities beyond sustainability across the agency,
many of these priorities have been sustainability-related in recent years. SBIR does
release "success story" reports that quantify environmental benefits. EPA's scientific
expertise in this area, particularly related to metrics, can directly support future efforts to
more overtly consider sustainability criteria and concepts in the SBIR and P3 Programs.
Regarding the Environmental Technology Verification (ETV) Program, several of the
existing centers have or are currently verifying technologies in some emerging market
areas. For example, in the area of climate change, ETV has verified several combined
heat and power units designed to reduce CO2 and other pollutants.
Although the P3 Program is a relatively young program (it was started in 2003), NCER is
working with ORD/ORMA to assess the effectiveness of the program relative to its role
in (1) stimulating sustainability in academic institutions; (2) providing students with an
opportunity to work on a real-world problems and thereby learn the value of teamwork
and diversity; and (3) develop technologies, tools and processes that promote
sustainability.
-------
December 2008 Office of Research and Development's (ORD) Response to the
Board of Scientific Counselors (BOSC) Report on the Review of ORD's Science and
Technology for Sustainability Research Program (final report received April 2008)
Strategic Issue #4: Better program integration is needed as the P2NT transitions to
the STS.
In general, the Program structure appears to be adequate but it should be assured
that there is integration and continuity among the elements during the plan for
transition. The existing Program elements and the structure proposed for STS
research in the future are organized around the development of scientifically
based sustainability metrics. The current structure of the Program and the
proposed structure are well-suited for the development of decision support tools
that promote environmental stewardship and sustainable management practices.
There needs to be significant interaction between this LTG and, in particular,
LTG 2, which are intimately tied together.
Economics and other social dimensions should be incorporated as part of
feedback loops of process or output evaluated decision-making.
The Subcommittee recommends integrating an implementation plan as part of the
STSMYP. Some concepts in the APGs of the STSMYP need to be defined (e.g.,
'sustainable land use,' 'sustainable water use,' 'local level') to ensure clear
understanding by stakeholders and to ensure that all the aspects of sustainability
are incorporated. Strengthen and expand communication aspects of tools as part
of the MYP including: (1) guidance regarding scope (e.g., what LCA does and
does not do), outreach, and influence (how LTG 1, LTG 2, and LTG 3 tie together
in the path to sustainability); and (2) interrelations of different aspects of
sustainability. Ecological aspects should be incorporated into the tools for
decision analysis. Additional expertise might be needed to cover ecological
systems so it would be wise to strengthen collaborations with the ORD Ecology
Research Program. Geographic and landscape orientation should be
incorporated for local implementation.
The STS MYP was designed to describe an integrated program that explains how each of
the LTG outcomes/products link to provide a holistic perspective on sustainable
solutions. In order to clarify these linkages, ORD will incorporate changes that will
better explain how ORD and its partners will integrate the various research components
described in the STS MYP. While we agree that economic and social dimensions should
be integrated into the tools produced, these topics are primarily being addressed by others
in the Agency, e.g., the National Center for Environmental Economics. ORD will remain
abreast of what others in the Agency are doing and attempt, where feasible, to incorporate
the results.
An example of how this integration is already happening is in the biofuels area where
unlike other programs, research is starting with potential metrics as a way to influence
and inform research. ORD's Ecology Program, through the Future Midwest Study, is
also evaluating location-specific sustainability issues related to changes in land use. In
-------
December 2008 Office of Research and Development's (ORD) Response to the
Board of Scientific Counselors (BOSC) Report on the Review of ORD's Science and
Technology for Sustainability Research Program (final report received April 2008)
addition, as part of the revision of the STS MYP, ORD will also include any additional
appropriate language to better describe the planned research. However, the MYP will not
include specific implementation details on every research activity that ORD plans to
perform over the next five years. ORD Laboratories and Centers typically develop these
implementation details consistent with the MYP goals. ORD also recognizes that
sustainability will occur at various geographic scales.
Different sustainability metrics and indicator efforts that relate to LTG 1 in the STS MYP
are also related to each other. At the sub-national scale, ORD is developing place-based
local and regional scale metrics and indicators. For example, the STS Program is
deriving a suite of science-based sustainability metrics to assess and track system
condition, including dynamic order (Fisher information), ecological footprint, energy
content, and net regional product. Also, informed by collaborative processes, the CNS
Program is developing a series of case studies with sets of sustainability indicators and
desired outcomes to inform regional decision-making. Both the more theoretical metrics
and the more practical indicators rely on similar data sets. Over time, we will blend the
two efforts so that the scientific indicators become more accessible and more decision-
relevant, and the practical indicators become more scientifically sound. Sub-national
sustainability metrics based on resource use, principally focused on supporting the
decisions of industry, may relate to manufacturing and other activities at numerous
locations. For example, the ETV Program is supporting the development of sustainability
metrics to inform technology verification, and the CNS Program supports some industrial
metrics and indicators based on resource use.
The existing MYP attempts to describe the interaction and dependence of LTG 1
(metrics) and LTG 2 (decision support tools), which are meant to be coupled to provide
integrated solutions to complex environmental problems. In many cases, these metrics
couple with tools developed under Goal 2. The research supported in the STS MYP
provides tools and related underlying models [including life cycle assessment (LCA),
material flow analysis (MFA), Future Scenarios, and Geographic Information Systems
(GIS)] that support planning and decision-making for government (APG 2.1), companies
(APG 2.2), and communities (APG 2.3). The key programs include LCA, Environmental
Impact Assessment (EIA), and the Collaborative Science and Technology Network for
Sustainability (CNS).
Since it also transcends the STS MYP, the sustainability concept needs to be reflected in
other MYPs. ORD will work with the rest of the Agency to determine how sustainability
concepts can be integrated into the design and execution of other programs. Those
carrying out the STS MYP, particularly the ORD Sustainability Director, will continue to
suggest to ORD National Program Directors (NPDs) how they can use STS research
results and sustainability concepts to evaluate their proposed research agendas. However,
in order for sustainability concepts to become an Agency priority, senior Agency
officials, such as the members of the Science Policy Council, will need to promote the
concept.
-------
December 2008 Office of Research and Development's (ORD) Response to the
Board of Scientific Counselors (BOSC) Report on the Review of ORD's Science and
Technology for Sustainability Research Program (final report received April 2008)
Strategic Issue #5: ORD STS Program should be more strategic and focus on a
limited number of areas where it can make unique contributions and impacts.
Currently, much of the work being conducted by the STS Research Program is
eclipsed by the magnitude and pace of advancements of industrial and academic
communities. Thus, in developing the plan, the Program must make strategic
decisions on where it can make an impact on the overall field.
Some Program elements are small components and lack a critical mass of
personnel.
The potential impact of STS programs is limited by lack of a critical mass and
resources. In developing the STS Research Program, ORD must make better use
of capabilities across ORD.
The ETV Program should encourage an increased role in supporting emerging
markets in trades/mitigation/offsets, such as mercury/greenhouse gases, etc. An
analysis should be conducted to determine if there are emerging markets in this
trade/offset area that have a barrier surrounding verification issues.
Development of streamlined methods is needed as part of the expansion ofLCA
tools (e.g., make them user-friendly) as well as integration of material flow
analysis (e.g., industrial ecology concepts).
System-based methods are indispensable for moving towards sustainability.
These are integrated in the STSMYP, but need to be integrated into tools.
Carefully examine the rationale for the selection of target areas/technologies to
better address market failures and tie outcome measures to sustainable measures
and metrics.
The Program should incorporate additional decision-making tools, such as
probabilistic risk assessment, Bayesian networks, causal pathways, and Multi-
Criteria Decision Analysis (Igor Linkov and others) in the research program.
This recommendation is consistent with guidance from the June 2007 Science Advisory
Board (SAB) Advisory on ORD's Sustainability Research Strategy and the Science and
Technology for Sustainability Multi-year Plan), which "acknowledges that the judicious
selection of research projects within the Plan will help to facilitate the diffusion and
adoption of the sustainability paradigm both within and outside the Agency." In concert
with its internal and external partners, ORD will work to focus the STS Program on a
select number of issues in which its expertise and resources can provide unique outputs
that will meet client needs and advance sustainability solutions. Smaller and less relevant
components of the Green Technology Program will be eliminated and the associated
resources will be redirected to address higher-priority research. Based on BOSC
-------
December 2008 Office of Research and Development's (ORD) Response to the
Board of Scientific Counselors (BOSC) Report on the Review of ORD's Science and
Technology for Sustainability Research Program (final report received April 2008)
recommendations, ORD has already made some strategic adjustments by starting to focus
on issue of the Sustainability of various biofuel production options. This emphasis on
national environmental problems will ensure the Program will have a significant impact
now and into the future.
With respect to the Environmental Technology Verification (ETV) Program, while we
agree with this recommendation, we note that ORD has already engaged in market
analysis research to identify where verification will be most useful. For example, in the
area of climate change, ETV has verified several combined heat and power units
designed to reduce CO2 and other pollutants. Consistent with the self-sustaining concept
of the ETV program, should the private sector provide the resources, ETV will perform
additional verifications.
One area where ORD is already making a unique contribution is the development of
integrated tools using a systems-based approach. For example, several efforts are
underway to produce tools that will holistically examine the environmental impacts of the
production, utilization and disposal of biofuels.
In addition, the revised STS MYP will more clearly delineate the strategic choices and
the criteria for selection of programs and projects. New outreach activities across EPA
programs and regions are underway to help inform the strategic choices. As these
strategic choices are made, ORD will make better use of ORD-wide capabilities by
strengthening existing partnerships with other Agency programs such as continuing
linkages with Ecosystems Research Program and strengthening partnerships with the
Drinking Water and Global Change Research Programs. ORD collaborates extensively
with other federal agencies to minimize duplication of effort and ensure involvement of
academia in high-priority research areas. This collaboration includes efforts to influence
NSF and other federal agencies that provide funding to the outside academic community
for fundamental research.
Already at the forefront of developing LCA methodology, ORD will ensure that the
revised MYP accurately reflects the need for streamlined LCA approaches. Several
projects supported by CNS employ MFA methodologies. ORD is also cosponsoring with
NSF a special issue of the Journal of Industrial Ecology on applications of MFA. The
MYP will also be revised to ensure we explain the range of decision-making tools
available.
Strategic Issue #6: ORD needs to collaborate more extensively with outside
partners to avoid duplication and enhance research impact.
Currently, it does not appear as if extramural collaborations are planned on
techniques to better relate process outputs to environmental impacts.
10
-------
December 2008 Office of Research and Development's (ORD) Response to the
Board of Scientific Counselors (BOSC) Report on the Review of ORD's Science and
Technology for Sustainability Research Program (final report received April 2008)
It is not clear if the coordination has been successful in reaching a wider set of
stakeholders, such as NGOs, state agencies, etc. If possible, these efforts should
be encouraged.
Many of the results of the research under LTG 3 have not reached the user
community. Some outside partners appear dedicated to obtaining grant support
and/or license fees as opposed to utilization of the technology. Also, there is a
need to better under stand what has been done by academia and industry. While it
is clear that ORD has collaborated with and obtained input from others on
research objectives, especially to avoid duplication of effort, the Subcommittee
members thought that this is so critical to the acceptance and use of the
technologies developed that the Program should seek input from a number of
extramural groups to assist EPA. In fact, such communications could result in
partnerships and greater leveraging of the limited Program resources. Some of
the work is a duplication of previous or current work being done by others outside
of EPA. Larger industrial and manufacturing firms are underrepresented in
setting objectives and avoiding duplication. Hurdles should be lowered and/or
obstacles removed to insure that "those who really know "participate. It appears
that the U.S. Department of Energy (DOE) and others may be acting without
knowledge of the P2NT efforts. While not a complete replication, the projects and
programs seem to be uncoordinated with those of other agencies.
SBIR: Increase meeting of stakeholder needs. If the Program can better address
the internal Agency needs from the STSMYP, it will provide a valuable service
and be recognized more favorably. The goal of moving to a 100 percent cost
share basis needs to be carefully evaluated. Although this will better leverage
funds, it might miss important opportunities. This might be a future goal, but it
needs to be determined if this would result in missed opportunities for small
businesses. This could occur if they: (1) could not afford the assessment; and (2)
are not being funded for this purpose through the SBIR Program. Additional SBIR
opportunities in the broader set of sustainability concerns, such as land and water
uses, need to be explored. One example might be the design of storm water
handling systems in new developments. Certainly there are other opportunities as
well.
ORD is keenly aware of sustainability activities across the government and
internationally. ORD has always worked with outside organizations on aspects of tools
development. ORD also benefits from having a strong academic effort in this area and is
pursuing partnerships with other federal agencies that are poised to fund this research.
For example, the STS is conducting a demonstration project with the direct support of a
variety of local governments, including the regional metropolitan sewer district, which is
facing an EPA consent decree to solve the community's stormwater problem.
Coordinated strategic plans are currently being developed for certain high priority
sustainability areas such as biofuels. Both the extramural and intramural portions of the
11
-------
December 2008 Office of Research and Development's (ORD) Response to the
Board of Scientific Counselors (BOSC) Report on the Review of ORD's Science and
Technology for Sustainability Research Program (final report received April 2008)
Program are currently developing relationships with our federal partners (and others
investing in research) to positively influence their research and identify any gaps/niches
for future ORD investment. Over the next year, the Program will better characterize
activities underway, nationally and internationally, in the areas of metrics and decision
support tools to ensure our research is adding incremental value. We will capitalize on
the successful approach used to engage stakeholders in the ETV Program to gather this
information.
The SBIR program encourages but does not require cost sharing as part of the
commercialization focus of the program. An SBIR Phase II program review completed
this year showed that 73% of SBIR projects secure additional investment beyond their
SBIR awards. SBIR is structured to address technology priorities across EPA through the
Environmental Technology Council (ETC) and other mechanisms. Storm water
management technology has been among the sustainability-related topics addressed by
SBIR in recent years.
Strategic Issue #7: Develop annual goals that are better defined and quantifiable so
that impact can be determined.
The APGs should be provided in more quantifiable forms, generally in the form of
SMART (specific, measurable, achievable, relevant, and timely) goals. The goals are
written very generically, without sufficient measurable targets against which one can
evaluate performance.
ETV: The current outcomes analysis does not measure the effect of the ETV Program,
because it does not attempt to identify outcomes in the absence of the ETV Program.
Outcome measures stated in terms of numbers of verifications are probably better, even if
considered less relevant. These metrics are better linked to the question of number of
decision-makers/impacts. The metrics were deemed to be well-defined, but not well-
quantified.
ORD will make adjustments to the existing APG language and structure in order to
ensure that goals are more quantifiable and to better define the anticipated outcomes.
ETV is a critical component of the overall technology continuum, which has been
strongly endorsed by NACEPT. Over the years, ETV has produced many protocols that
are widely used across many industries to evaluate technologies and their environmental
implications. ETV offers independent and unbiased data which decision-makers feel
comfortable using for implementation of regulations and/or voluntary programs. The
ETV program tracks its verifications and protocols and can provide the BOSC annual
figures.
The Appendix consists of a table that provides more details on how we plan to respond to
each of the BOSC recommendations, including a timeline showing completions dates for
12
-------
December 2008 Office of Research and Development's (ORD) Response to the
Board of Scientific Counselors (BOSC) Report on the Review of ORD's Science and
Technology for Sustainability Research Program (final report received April 2008)
each action. We look forward to meeting with the BOSC in 2009 and reporting on the
activities described in this response.
13
-------
December 2008 Office of Research and Development's (ORD) Response to the Board of
Scientific Counselors (BOSC) Report on the Review of ORD's Science and Technology for
Sustainability Research Program (final report received April 2008)
Appendix. Table of Recommendations and Response/Actions
Issue
Recommendations
A clear definition of sustainability and a
framework for its application are
required to fund appropriate extramural
research programs and to determine
the efficacy of specific metric or
decision tools.
Response/Actions
While the STS programs adopts the
general definition of sustainability given
in E.O. 13423, more specific goals and
metrics must be defined for each media
or cross media element, as in the case of
sustainable production of biofuels. ORD
will coordinate a series of workshops to
facilitate discussions that will further
identify sustainability outcomes and
metrics in key areas such as sustainable
urban development, green building
design and sustainable agriculture, as for
example. These workshops will be
designed to engage key Agency officials
in Program Offices and Regions to obtain
their perspectives.
Time
Line
Sept 09
Definitions are needed for some terms
to improve clarity of Program elements
and responsibilities.
See response above.
Sept 09
Develop an outline for how metrics for
sustainability will be developed. This
should include criteria for assessing the
utility and predictability of metrics.
In the revised MYP, the general
approach for the development of
sustainability metrics will be described
including criteria to assess their utility.
Feb09
ORD is working with other federal
agencies to define a set of criteria and
indicators for sustainable biofuel
production. This work will be a model for
application and development of metrics
in other areas.
NovOS
Coordinate metric development with
other LTGs.
ORD will add language to the MYP to
ensure that appropriate linkages are
made between the metrics LTG and the
balance of the program. The revisions
will factor in the role of metrics and other
driving forces.
Feb09
Determine a strategy of how metrics
will be used.
ORD will coordinate a series of
workshops to facilitate a discussion to
further develop definitions of
sustainability concepts, including metrics.
Sept 09
-------
December 2008 Office of Research and Development's (ORD) Response to the Board of
Scientific Counselors (BOSC) Report on the Review of ORD's Science and Technology for
Sustainability Research Program (final report received April 2008)
Issue
Recommendations
Response/Actions
These workshops will be designed to
engage key Agency officials in Program
Offices and Regions to obtain their
perspectives. During these discussions,
ORD will address how metrics will be
used to support the sustainability
concepts identified and determine their
applicability to key Agency programs.
Biofuel work is a good case study. Here
the intent of using metrics is to define a
"dashboard" of key environmental, social
and economic measures for all agencies
to monitor. Where trends are going in
the wrong direction, collaborative federal
action would be initiated.
Time
Line
The metrics developed under the P2NT
Research Program have not pervaded
other programs.
While P2 is an important consideration in
other Agency programs, the P2NT
Research Program was not specifically
designed for or focused on metrics
development for other programs.
N/A
Going forward, an extramural program
based on the Technology for a
Sustainable Environment (TSE)
Program could be crafted to emphasize
metrics and how technologies move
toward improving the measures.
The STS program is pursuing
partnerships with NSF and other
extramural research agencies on topics
such as Green Building. In addition,
current projects funded under the
extramural CMS program are using
decision-making tools to move towards
identified sustainability outcomes at a
regional scale. Many of the projects
have a strong focus on metrics and some
are also incorporating technologies.
Beginning in October 2008, ORD will
initiate a monthly webinar accessible to
all of EPA on these projects. In addition,
ORD will sponsor a workshop in the
spring of 2010 to bring together internal
and external investigators to identify
lessons learned from the CMS program
and identify any specific outputs from the
ongoing projects that could be used to
enhance research efforts related to
metrics and decision support.
Oct. 08
Testing protocols [in real world
applications] should be established to
determine if the metrics are measuring
the intended functions, if they are
We agree metrics should be evaluated to
ensure they are moving society to a
more sustainable future. There is some
work underway to develop and test a set
FebOQ
-------
December 2008 Office of Research and Development's (ORD) Response to the Board of
Scientific Counselors (BOSC) Report on the Review of ORD's Science and Technology for
Sustainability Research Program (final report received April 2008)
Issue
Recommendations
consistent in their evaluation, if they
are sufficiently independent, and if they
can be effectively used to determine if
specific actions are driving society to
become more sustainable.
Response/Actions
of system metrics that represent the
most fundamental properties and
processes that must be preserved to
ensure the sustainability of a particular
geographical system or region (e.g., the
San Luis Valley community).
However, since these studies are data-
intensive and often resource-intensive,
the extent to which EPA alone can fund
and manage such activities is limited.
Therefore, at the present time, our efforts
are limited to geographic-specific studies
like the one above and a new effort in the
area of biofuels. The revised MYP will
reflect our current plans in these areas.
Time
Line
The predictability of the models should
be evaluated and sensitivity analyses
conducted.
See response above.
FebOQ
Sustainability targets need to be
identified so that appropriate metrics
can be designed and tested.
ORD will coordinate a series of
workshops to facilitate a discussion on
further developing definitions of
sustainability concepts, including metrics
and how to make them operational.
These workshops will be designed to
engage key Agency officials in Program
Offices and Regions to obtain their
perspectives. During these discussions,
ORD will address metrics and their
applicability to key Agency programs.
Sept 09
Critical experiments should be
designed that allow testing of
hypotheses within the realm of defined
metrics.
We agree that the intent of metrics is to
ensure they are moving society to a
more sustainable future. There is some
work underway to develop and test a set
of system metrics that represent the
most fundamental properties and
processes that must be preserved to
ensure the sustainability of a particular
geographical system or region (e.g., San
Luis Valley community).
However, since these studies are often
resource intensive and are data
intensive, the extent to which EPA alone
can fund and manage such activities is
limited. Therefore, at the present time,
our efforts are limited to geographic-
FebOQ
-------
December 2008 Office of Research and Development's (ORD) Response to the Board of
Scientific Counselors (BOSC) Report on the Review of ORD's Science and Technology for
Sustainability Research Program (final report received April 2008)
Issue
Recommendations
Response/Actions
specific studies like the one above and a
new effort in the area of biofuels. The
revised MYP will reflect our current plans
in these areas.
Time
Line
Evaluation of the metrics should be
done systematically and quantitatively.
Our new work on biofuels is setting a
model of how we might proceed to
addressing critical national issues. ORD
will use the biofuel example as a model
of how to work across EPA and the rest
of govern me nt to establish and
implement metrics that are systematic
and quantitative.
NovOS
A team that was better integrated
throughout EPA could draw on
additional resources that could
enhance their effectiveness.
ORD will expand the existing metrics
team to include other parts of the
Agency. For example, an internal EPA
team focused on sustainable biofuel
production has been organized and is
already discussing issues related to
metrics.
Jun 09
The relevance and impact of the Green
Technology Program (GTP) is less
apparent and this program needs to be
assessed internally to determine if it is
serving a function that is not being met
already by the private sector and
academia.
ORD will deemphasize in-house Green
Technology research (LTG 3) and modify
the MYP to reflect increased emphasis
on metrics (LTG 1) and decision support
tools (LTG 2). The MYP will also be
modified to indicate that extramural GTP
efforts will be focused on partnerships
with other funding agencies to help
identify important priorities.
FebOQ
All of the program elements and the
Green Technology Program in
particular are in need of refinement to
better address sustainability issues and
to demonstrate and articulate the role
that they play in contributing to
sustainable outcomes.
See response above.
FebOQ
Consideration should be given to
redirecting the Green Technology
Program or replacing it with an
extramural grants program.
See response above.
FebOQ
Green Technology: Carefully examine
the rationale for the selection of target
See response above.
FebOQ
-------
December 2008 Office of Research and Development's (ORD) Response to the Board of
Scientific Counselors (BOSC) Report on the Review of ORD's Science and Technology for
Sustainability Research Program (final report received April 2008)
Issue
Recommendations
areas/technologies to better address
market failures and tie outcome
measures to sustainable measures and
metrics.
Response/Actions
Time
Line
Results derived from the Green
Technology Program have not been
effectively communicated to larger
industrial enterprises.
The research results from the Program's
many projects have been the subject of
numerous presentations at national
professional meetings attended by
representatives of nearly all major
chemical producers. In the ten year
period from 1996 to 2006, the Green
Chemistry program published 384 peer
reviewed papers, 34% of which are listed
among the top 10% of all cited papers in
their field. Staff was elected as the
Second Vice-Chair of the American
Institute of Chemical Engineers'
Environmental Division, thus an
additional opportunity to highlight ORD
sponsored Green Chemistry research.
N/A
The P3 Program should improve the
solicitation/judging criteria to require a
clear statement by students as to the
effects articulated via sustainability
metrics or decision tools. A clear tie-in
with the goals of LTG 1 and LTG 2
could be developed.
The MYP will be revised to indicate that
the P3 program already requires student
teams to quantitatively and/or
qualitatively articulate the benefits of
their project in the social, environmental,
and economic dimensions, both at the
proposal stage and at the final report
stage. Results from ORD's STS
research program will be used to
enhance the sustainability criteria used in
future P3 solicitations.
N/A
The SBIR Program should increase its
use of sustainability metrics in selection
criteria and increase the linkage of
program outcomes to sustainability
metrics.
The MYP will also be modified to indicate
that this is an excellent opportunity to
focus on sustainability in the Small
Business Innovation Research (SBIR)
program. Like other extramural
programs, SBIR is in the stage of
identifying solicitation topics. SBIR
already releases "success story" reports
that quantify environmental benefits.
SBIR is structured to address technology
priorities across the Agency, many of
which have been sustainability-related in
recent years. In addition, the law that
authorizes SBIR requires that potential
for commercialization be a strong
N/A
-------
December 2008 Office of Research and Development's (ORD) Response to the Board of
Scientific Counselors (BOSC) Report on the Review of ORD's Science and Technology for
Sustainability Research Program (final report received April 2008)
Issue
Recommendations
Response/Actions
criterion for funding. Results from ORD's
STS research program will be used to
enhance the sustainability criteria used in
future SBIR solicitations.
Time
Line
The [P3] Program could benefit from a
more systematic evaluation of the
program outcomes, such as tracking of
careers of recipients to obtain
information that can be used to
measure impact as outcome. Thus, a
detailed analysis of the impacts on the
P3 Program on the student participants
is desirable.
Although the P3 Program is a relatively
young program (it was started in 2003),
NCER is working with ORD/ORMA to
assess the effectiveness of the program
relative to its role in (1) stimulating
sustainability in academic institutions, (2)
providing students with an opportunity to
work on a real-world problems and
thereby learn the value of teamwork and
diversity, and (3) develop technologies,
tools and processes that promote
sustainability.
2010
The Subcommittee recommends
integrating an implementation plan as
partoftheSTSMYP.
As part of the revision of the STS MYP,
ORD will include additional appropriate
language to better describe the planned
research. However, the MYP is not
intended to include specific
implementation details about the
research activities that ORD plans to
perform over the next 5 years. ORD
Laboratories and Centers typically
develop these implementation details
consistently with the MYP goals.
FebOQ
The two Annual Performance Goals
(APGs) do not seem to flow well into a
logical research plan with quantifiable
goals and objectives.
ORD will make adjustments to the
existing APG language and structure to
ensure that goals are more quantifiable
and to better define the anticipated
outcomes.
FebOQ
There needs to be significant
interaction between this LTG [1] and, in
particular, LTG 2, which are intimately
tied together. It should be assured that
there is integration and continuity
among the elements during the plan for
transition.
ORD will incorporate changes into the
MYP that will better explain how ORD
and its partners will integrate the various
research components described in the
STS MYP.
FebOQ
LTG 1 metrics should be used to inform
LTG 3 activities.
Language will be added to the MYP to
address this recommendation.
FebOQ
-------
December 2008 Office of Research and Development's (ORD) Response to the Board of
Scientific Counselors (BOSC) Report on the Review of ORD's Science and Technology for
Sustainability Research Program (final report received April 2008)
Issue
Recommendations
Geographic and landscape orientation
should be incorporated for local
implementation.
Response/Actions
ORD also recognizes that sustainability
will occur at various geographic scales.
Some efforts within STS and in other
programs already have activities that
address geographic-specific sustain-
ability issues. The MYP will be modified
to more clearly reference these efforts.
Time
Line
FebOQ
Economics and other social
dimensions should be incorporated as
part of feedback loops of process or
output evaluated decision-making.
Others in the Agency are components of
this work. ORD will remain abreast of
these activities and attempt, where
feasible, to incorporate results into our
decision tools.
N/A
The life cycle assessment (LCA)
programs, metrics, and procedures
developed under the Pollution
Prevention and New Technologies
(P2NT) Research Program are relevant
and important to the goals of EPA,
stakeholders, and the international
community. The STS Research
Program is positioned to move these
initiatives forward and is encouraged to
build on this strength.
Agree.
N/A
Ecological aspects should be
incorporated into the decision analysis
tools. Additional expertise might be
needed to cover ecological systems so
it would be wise to strengthen
collaborations with the ORD Ecology
Research Program.
Aspects of this recommendation are
already being incorporated into ORD's
Eco research program, and efforts to
coordinate the two programs will
continue.
N/A
Some program elements are small
components and lack a critical mass of
personnel. Currently, much of the work
being conducted by the STS Research
Program is eclipsed by the magnitude
and pace of advancements of industrial
and academic communities. Thus, in
developing the plan, the Program must
make strategic decisions on where it
can make an impact on the overall
field.
The MYP will be modified to reflect
integration or elimination of smaller
components. ORD has advanced
academic sustainability concepts by
funding the Technology for a Sustainable
Environment (TSE) and Collaborative
Science and Technology Network for
Sustainability (CNS) programs. Based
on these recommendations and those of
the Science Advisory Board, ORD has
already made some strategic
adjustments, to ensure high impact,
including focusing on the key emerging
issue of the sustainability of various
FebOQ
-------
December 2008 Office of Research and Development's (ORD) Response to the Board of
Scientific Counselors (BOSC) Report on the Review of ORD's Science and Technology for
Sustainability Research Program (final report received April 2008)
Issue
Recommendations
Response/Actions
biofuel production options and these will
be reflected in the revised MYP.
Time
Line
The potential impact of STS programs
is limited by lack of a critical mass and
resources. In developing the STS
Research Program, ORD must make
better use of capabilities across ORD.
The revised STS will more clearly
delineate the strategic program choices
made and the criteria for selection. The
ORD sustainability lead will conduct new
outreach activities across EPA programs
and regions to help inform the strategic
choices.
FebOQ
... is important to keep abreast of and
continue to lead the development of
LCA methodologies.
ORD is already at the forefront of LCA
methodology development.
N/A
Development of streamlined methods
is needed as part of the expansion of
LCA tools (e.g., make them user-
friendly) as well as integration of
material flow analysis (e.g., industrial
ecology concepts).
The MYP will be modified to ensure that
it reflects streamlined LCA approaches.
Several supported CMS projects employ
material flow analysis (MFA)
methodologies. ORD is also
cosponsoring with NSF a special issue of
the Journal of Industrial Ecology on
applications of MFA.
FebOQ
System-based methods are
indispensable for moving towards
sustainability. These are integrated in
the STS MYP, but need to be
integrated into tools.
ORD is already taking a systems-based
approach as it develops its integrated
tools. For example, several efforts are
underway to produce tools that will
holistically examine the environmental
impacts of biofuels. However, we will
make this more explicit in the revised
MYP.
FebOQ
Carefully examine the rationale for the
selection of target areas/technologies
to better address market failures and
tie outcome measures to sustainable
measures and metrics.
This is a very ambitious goal. Some STS
elements attempt to provide technologies
that promote sustainable choices,
particularly the environmental
implications of the production and
utilization and disposal of biofuels.
FebOQ
The ETV Program should encourage
an increased role in supporting
emerging markets in
trades/mitigation/offsets, such as
mercury/greenhouse gases, etc.
Several of the existing ETV centers have
or are currently verifying technologies in
emerging market areas. For example, in
the area of climate change, ETV has
verified several combined heat and
power units designed to reduce CO2 and
other pollutants.
N/A
-------
December 2008 Office of Research and Development's (ORD) Response to the Board of
Scientific Counselors (BOSC) Report on the Review of ORD's Science and Technology for
Sustainability Research Program (final report received April 2008)
Issue
Recommendations
An analysis should be conducted to
determine if there are emerging
markets in this trade/offset area that
have a barrier surrounding verification
issues.
Response/Actions
ORD's NRMRL has already engaged in
market analysis research to identify
where verification will be most useful.
Time
Line
N/A
...industrial sectors [need] to have tools
for streamlining LCAs that allow for
rapid evaluation of environmental
burdens.
The MYP will be modified to ensure that
it reflects streamlining LCA approaches.
FebOQ
The Program should incorporate
additional decision-making tools, such
as probabilistic risk assessment,
Bayesian networks, causal pathways,
and Multi-Criteria Decision Analysis
(Igor Linkov and others) in the research
program.
The MYP will be revised to explain the
range of decision-making tools being
applied.
FebOQ
Because the STS Research Program is
sparsely populated and not coordinated
with outside efforts, a strategic plan
that includes an awareness of what is
being done outside of the Agency,
including that of organizations outside
of the United States, and how ORD can
make a significant impact on the
science should be developed.
Language will be added to the MYP to
explain how areas in the STS are
coordinated with international research
and other outside efforts. ORD cannot
commit to developing a separate
strategic plan to describe these
relationships. Since the sustainability
concept transcends the STS, integration
must occur with other ORD MYPs. ORD
will lead a dialogue among its National
Program Directors (NPDs) and others to
determine how sustainability concepts
can be integrated into the design and
execution of all of its research programs.
However, in order for sustainability
concepts to become a priority, senior
Agency officials such as members of the
Science Policy Council will need to
participate and support this effort.
FebOQ
LTG 2 could be improved through
targeted extramural collaborations on
the development of new tools or
cooperation on the advancement of
existing tools or tools being developed
in the private sector.
A number of tools have been developed
or enhanced through the CMS program.
An example is the Energy & Materials
Flow & Cost Tracker (EMFACT), a free
materials management tool designed for
small business manufacturers. ORD has
worked extensively with outside
organizations to advance the
implementation of new tools. For
N/A
-------
December 2008 Office of Research and Development's (ORD) Response to the Board of
Scientific Counselors (BOSC) Report on the Review of ORD's Science and Technology for
Sustainability Research Program (final report received April 2008)
Issue
Recommendations
Response/Actions
example, to help implement TRACI (Tool
for the Reduction and Assessment of
Chemical and Other Environmental
Impacts), ORD has worked with
numerous organizations and programs
including the NSF International/American
National Standards institute and the
Leadership in Energy and Environmental
Design (LEED) green building rating
system of the U.S. Green Building
Council.
Time
Line
Efforts should be made to reach a
wider set of stakeholders, such as
nongovernmental organizations
(NGOs), state agencies, etc.
The revised MYP will identify specific
efforts ORD will conduct to better
characterize activities underway,
nationally and internationally, in the
areas of metrics and decision support
tools to ensure our research is adding
incremental value.
FebOQ
One example of a program with many
successful elements is the ETV
Program ... The Subcommittee would
like to recognize two program elements
that it considered to be of excellence.
These include: (1) the public outreach
component which brings early public
use; and (2) the clear team spirit of the
Program members. To find a balance
of speed and a team sense of "over-
accomplishment" is rare. ORD can be
rightly proud of this program element
and the impact that it has had. The
Subcommittee recommends this
program element for an ORD citation if
this has not been done already.
The ETV program was nominated by
ORD for and won an Agency Bronze
Medal in 2004 for its work to verify
homeland security technologies from
2002 through 2004. ORD agrees with
the BOSC and will consider submitting a
package recognizing these and other
components of the ETV program.
N/A
SBIR: Increase meeting of stakeholder
needs. If the Program can better
address the internal Agency needs
from the STS MYP, it will provide a
valuable service and be recognized
more favorably. The goal of moving to
a 100 percent cost share basis needs
to be carefully evaluated. Although this
will better leverage funds, it might miss
important opportunities. This might be
a future goal, but it needs to be
determined if this would result in
The SBIR program encourages but does
not require cost sharing as part of the
commercialization focus of the program.
An SBIR Phase II program review
completed this year showed that 73% of
SBIR projects secure additional
investment beyond their SBIR awards.
SBIR is structured to address technology
priorities across EPA through the
Environmental Technology Council
(ETC) and other mechanisms. Storm
water management technology has been
N/A
10
-------
December 2008 Office of Research and Development's (ORD) Response to the Board of
Scientific Counselors (BOSC) Report on the Review of ORD's Science and Technology for
Sustainability Research Program (final report received April 2008)
Issue
Recommendations
missed opportunities for small
businesses. This could occur if they:
(1) could not afford the assessment;
and (2) are not being funded for this
purpose through the SBIR Program.
Additional SBIR opportunities in the
broader set of sustainability concerns,
such as land and water uses, need to
be explored. One example might be
the design of storm water handling
systems in new developments.
Certainly there are other opportunities
as well.
Response/Actions
among the sustainability-related topics
addressed by SBIR in recent years.
Time
Line
The APGs should be provided in more
quantifiable forms, generally in the form
of SMART (specific, measurable,
achievable, relevant, and timely) goals.
The goals are written very generically,
without sufficient measurable targets
against which one can evaluate
performance. ARM 1, 2008 is well-
defined, but 2009 is nebulous and
could be refined.
ORD will make adjustments to the
existing APG language and structure to
ensure that goals are more quantifiable
and better define the outcomes
anticipated.
FebOQ
The actual outputs and outcomes could
be more clearly defined and
communicated to targeted sectors.
See response above.
FebOQ
The two Annual Performance Goals
(APGs) do not seem to flow well into a
logical research plan, with quantifiable
goals and objectives.
See response above.
FebOQ
ETV: The current outcomes analysis
does not measure the effect of the ETV
Program, because it does not attempt
to identify outcomes in the absence of
the ETV Program. Outcome measures
stated in terms of numbers of
verifications are probably better, even if
considered less relevant. These
metrics are better linked to the question
of number of decision-makers/impacts.
The metrics were deemed to be well-
defined, but not well-quantified.
Over the years, ETV has produced many
protocols that are widely used across
many industries to evaluate technologies
and their environmental implications.
ETV offers independent and unbiased
data which decision-makers feel
comfortable using for implementation of
regulations and/or voluntary programs.
The ETV program tracks its verifications
and protocols and can provide the BOSC
annual figures. On February 2, 200Q, the
ETV Program will be hosting a kick-off
meeting for the "Advanced ETV
European Effort to Support International
FebOQ
11
-------
December 2008 Office of Research and Development's (ORD) Response to the Board of
Scientific Counselors (BOSC) Report on the Review of ORD's Science and Technology for
Sustainability Research Program (final report received April 2008)
Issue
Recommendations
Response/Actions
Environmental Technology Verifications."
This meeting will be held in conjunction
with the 4th Annual International ETV
Working Group Meeting, bringing
together representatives from verification
programs in the United States, Canada,
and the European Union.
Time
Line
12
------- |