Improving EPA's Performance
with Program Evaluation
Project in Excellence and Leadership: New England Universities'Laboratories
Mid-Term Evaluation: Piloting Superior Environmental Performance in Labs
Series No. 8
By continuously evaluating its programs, EPA is able to capitalize on lessons learned and incorporate that
experience into other programs. This enables the Agency to streamline and modernize its operations while promoting
continuous improvement and supporting innovation. This series of short sheets on program evaluation is intended
to share both the results and benefits of evaluations conducted across the Agency, and share lessons learned
about evaluation methodologies in this evolving discipline. For more information contact EPA's Evaluation Support
Division at www.epa.gov/evaluate.
At a Glance
Evaluation Purpose
To garner lessons learned and highlight
opportunities to improve the overall
environmental performance of the universities
for the rest of the project.
Evaluation Type
Mid-term Outcome Evaluation
Publication Date
September 2002
Partners
EPA Office of Policy, Economics and
Innovation, EPA Region 1, Boston College,
University of Massachusetts Boston, University
of Vermont, Vermont Department of
Environmental Conservation, Massachusetts
Department of Environmental Protection, and
Nexus Environmental Partners
Contact
Suganthi Simon, OPEI (202) 566-7416
Background: Why was an evaluation
performed?
Over the last fifteen years, the regulated community has recognized
the difficulties in tailoring regulations to the unique environmental
and structural aspects of academic and research laboratories. The
Occupational Safety and Health Administration's (OSHA)
laboratory standard is written specifically for laboratories, while
the Resource Conservation and Recovery Act (RCRA) Subtitle C
makes no distinction among its many different regulated entities.
This dual and dissimilar regulatory scheme currently governing
labs has proven to be unwieldy.
In 1999, a consortium of university laboratories in New England
joined the U.S. Environmental Protection Agency's Project XL
(excellence and Leadership) to test an innovative program to reduce
regulatory inefficiencies and achieve better environmental
performance than what is required under the current regulatory
structure. Three universities—Boston College, University of
Massachusetts Boston (UMB), and University of Vermont
(UVM)—are testing the integration of RCRA hazardous waste
regulations with OSHA's performance-based Chemical Hygiene
NCEI
NATIONAL CENTER FOR
ENVIRONMENTAL INNOVATION
-------
Plan. This harmonization system requires the use of
performance-based criteria to effectively manage
laboratory wastes with an Environmental Management
Plan (EMP). The EMP is specifically tailored to the
research needs and processes of each university.
The new system focused on the following priority areas:
(1) increasing faculty, laboratory staff, and student
training to improve individual behavior in the
laboratory and overall environmental awareness of
staff and students; (2) generating pollution prevention
ideas; (3) reducing laboratory hazardous waste generation;
and (4) increasing chemical redistribution and reuse.
By December 2000, all three schools had implemented
their EMPs and had actively begun to track their
commitments and progress in meeting the stated goals
and objectives of the pilot project. In June 2001, the
three schools issued the first annual progress report
for the project, which yielded mixed results. The project
partners agreed to conduct a mid-term evaluation to
obtain a clear picture of why the schools were seeing
certain EMP elements take hold and why others seemed
to be having minimal results. The mid-term evaluation
discusses the results of the universities' efforts to
actively encourage chemical reuse and recycling,
enhance conformance with internal policies, increase
efficiency, and promote environmental stewardship
within laboratories.
Basic Evaluation Approach: How
did they do it?
The evaluation was conducted by a team comprising
staff from EPA's Office of Policy, Economics and
Innovation (OPEI), the universities, (Nexus
Environmental Partners), the Vermont Department of
Environmental Conservation (DEC), and the
Massachusetts Department of Environmental
Protection. The evaluation involved the seven steps
outlined below.
Step I: Develop an evaluation plan and evaluation
outline.
Step II: Use a logic model to lay out a framework
for understanding the project. The logic
model graphically represents the
relationship among program inputs,
outputs, and intended outcomes.
Step III: Develop a standard set of discussion
questions for use in conducting group
discussions with the schools.
Step IV: Collect qualitative data by conducting
group discussions with university
administrators, students, and faculty at
each of the three universities.
Step V: Collect and analyze compliance data from
project universities soon after EMP
implementation and gather compliance data
on non-project universities as baseline
comparison data to compare the first-year
audit results at the participating universities.
Conduct regular team conference calls on
the EMP development and implementation
phase with key environmental, health, and
safety staff from the three universities,
(Nexus Environmental Partners), and with
Vermont DEC.
Step VI:Analyze group discussion data and
information and prepare report findings and
recommendations.
Step VII: Share report findings and recommendations
with the evaluation team and chart a
communications plan for the evaluation
findings with senior managers and other
interested parties.
Approach for this Evaluation
Step I
Develop Evaluation Plan
Step II
Develop Logic Model
Step III
Develop Interview/Discussion Questions
Step IV
Conduct Interviews
StepV
Collect and Analyze Data
Step VI
Develop Findings and Recommendations
Step VII
Share Findings and Recommendations
-------
Evaluation Results: What was
learned?
The mid-term assessment of this project indicated that
the project showed great success in some important
areas: developing EMPs, training staff, increasing
awareness, shifting attitudes and behaviors, improving
the range of activities that determine compliance and
emergency preparedness, and demonstrating that the
environmental management system approach to
managing laboratory waste is slowly gaining hold and
making progress. At the same time, as fully described
in the report, the project has not shown the expected
successes in other areas—such as chemical reuse and
redistribution or pollution prevention—for a variety
of academic and cultural reasons. The lessons learned
highlight areas of great progress and areas that require
further thought, discussion, brainstorming, and action.
In the era of heightened awareness of domestic security
issues, colleges and universities can benefit from a more
holistic management scheme, such as the New England
Universities' Labs project, that stresses chemical
awareness, proper chemical handling and disposal, and
better laboratory housekeeping in general.
Evaluation Outcomes: What
happened as a result?
The final project agreement (FPA) for the New England
Universities' Laboratories Project XL originally expired
in 2003. EPA staff extended the FPA and will adjust
aspects of the pilot project based on the findings and
recommendations included in the evaluation. The
evaluation will be used to identify the strengths and
weaknesses of the New England Universities' Labs
program, offering suggestions for continuous
improvement and creating a system of learning within
EPA, the states, and the universities on laboratory
innovation. The evaluation will also be used to inform
a national dialogue on the potential for regulatory
reform for academic laboratories. Specifically EPA's
Office of Solid Waste and Emergency Response is
considering the pilot's experience as it considers the
development of a national rule that may regulate
hazardous wastes in college and university laboratories.
United States
Environmental Protection
Agency
Office of Policy,
Economics and Innovation
(1807T)
June 2003
EPA-100-F-03-012
------- |