OFFICE OF INSPECTOR GENERAL
REPORT OF REVIEW
SPECIAL REVIEW OF LESSONS LEARNED
FROM THE MASSACHUSETTS GRANT
FLEXIBILITY DEMONSTRATION
PROGRAM
JULY 24,1997
E1FMG 6-01-0071- 7400058
4;,
V
-------
Inspector General Division
Conducting the Audit:
Region Covered:
Offices Involved:
Eastern Audit Division
Boston, Massachusetts
Region 1
Office of Ecosystem
Protection
Office of Administration
& Resource Management
-------
3 $0
'a
h
Mix
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF THE INSPECTOR GENERAL
EASTERN AUDIT DIVISION
JOHN F. KENNEDY FEDERAL BUILDING
BOSTON, MASSACHUSETTS 02203-000 T
July 24, 1997
NEW YORK OFFICE:
290 BROADWAY
NEW YORK, NY 10007-1866
MEMORANDUM
SUBJECT: Special Review of the Lessons Learned From the Massachusetts Grant
Flexibility Demonstration Program
Report No. E1FMG6-01-0071-7400058
FROM: Paul D.
Divisional Inspector General for Audit
Eastern Division
TO: John DeVillars
Regional Administrator
New England Region
The Eastern Audit Division (EAD) conducted a special review to determine the lessons
learned from the Massachusetts Grant Flexibility Demonstration Program. We
performed the review at both the EPA New England Region, Boston, Massachusetts
(hereafter referred to as Region 1) and at the Massachusetts Department of
Environmental Protection (MADEP), Boston, Massachusetts.
Action Required
In accordance with EPA Order 2750, we ask that you provide us with a written
response to the review within ninety calender days of the date of the final report date.
The report contains issues and recommendations regarding the administration of the
demonstration program. For corrective actions planned but not completed by your
pesponse date, reference to milestone dates will assist this office in deciding whether to
close this report.
We have no objections to the release of this report to the public. This report does not
contain confidential or propriety information that cannot be released to the public.
This report contains matters that describe the issues the Office of Inspector General
(OIG) has identified, and corrective action the OIG recommends. This report represents
the opinion of the OIG. EPA managers will make final determination on matters in this
R»cycl«ifl«cycUW« . Printed with Vegetatote Oil Basod inks on 100% Ftecycfed Papw (40% PorfoMsumw)
-------
report according to the established EPA audit resolution procedures. Accordingly, the
matters in this report do not necessarily represent the final EPA position.
Should you or your staff have any questions about this report, please contact Steven
Weber, Team Leader, at (617) 565-3160.
Attachment
David Struhs, Commissioner
Department of Environmental Protection
The Commonwealth of Massachusetts
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
EXECUTIVE SUMMARY
PURPOSE The demonstration grant was a model for the future
Performance Partnership Grant (PPG) Program1. PPG's are
a part of the Environmental Protection Agency's (EPA)
continuing effort to increase State flexibility, improve
intergovernmental partnership, and help improve State and
tribal environmental protection capacity. EPA can use
lessons learned from the administration of the demonstration
grant to foster the success of future PPGs. ''" '"
The Office of Inspector General (OIG) completed a review
of the Massachusetts Demonstration Grant Flexibility
Program (also referred to as the Compliance Assurance
Demonstration Grant) to determine lessons learned. The
purpose of our review was to evaluate EPA's management
and oversight of the demonstration grant awarded to the
Massachusetts Department of Environmental Protection
(MADEP). We also discussed MADEP's concerns for the
effective implementation and management of the
demonstration grant at the State level. Specific objectives
were to:
verify if the pilot demonstration grant accomplished its
intended purpose;
determine if grant performance measures were
established and if so were they verifiable and
quantifiable with specific milestones to measure
outcomes;
determine if Region 1 developed oversight
procedures that allowed the State the flexibility the
1 President Clinton announced the Performance Partnership Grants Program on
March 16,1995, as part of the "Reinvent Environmental Regulation" program.
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
Agency promised, while ensuring they were meeting
grant reporting requirements;
evaluate and report the lessons learned to the
Agency; and _.;;_
compare the lessons learned with the Performance
Partnership Grant agreement for fiscal year 1997
."' dated, January 31,1997.
RESULTS-IN-BR1EF
Region 1 and MADEP need to carry out specific actions to
eliminate barriers that restrict State grant management
flexibility, and the achievement of improved environmental
outcomes. Elimination of these restrictive barriers will
promote the success of future Performance Partnership
Grant Programs.
The MADEP demonstration grant illustrated that barriers
existed that hindered the effective management of the
consolidated Federal funding concept. By eliminating these
barriers, the State should realize improved environmental
performance and administrative savings through greater
grant management flexibility, and the elimination of wasteful
paperwork.
The role of the OJG, in the review of demonstration grants or
future Performance Partnership Grants, is to evaluate the
grantee's progress toward addressing identified
environmental problems. Our review focused on how well
MADEP and Region 1 worked together to identify and
resolve any barriers or problems encountered while
attempting to reach goals and commitments. Ideally, EPA
should work with grantees to deal with problems as they
arise and develop the appropriate corrective action(s) to
eliminate the problems.
We completed a similar review of the lessons learned from "
the New Hampshire Grant Flexibility Demonstration Program
(Report number, E1FMG6-01-0031-6400102) on September/
Report N* E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
30,1996. We applied the lessons learned from our review
of the New Hampshire demonstration grant, where
appropriate, to complement and enhance our review of
Region 1's administration of the MADEP demonstration
grant.
The demonstration grant was considered successful by both
Region 1 and State officials because it provided positive
results that included:
multiple year, integrated enforcement activities across
media statutes, regulations and programs, (For
example, inspectors who had performed only water
inspections were trained now to perform water, air,
and Resource Conservation and Recovery Act
(RCRA) inspections.);
facility-wide inspection and enforcement actions;
fewer work plans associated with single-media
programs for water, waste, and air; and
advanced development and implementation of
multimedia compliance and enforcement programs,
(MADEP programs can provide enforcement and
compliance assistance to other States.)
The demonstration grant resulted in administrative and
technical efficiencies, however, the elimination of barriers
and more effective Region 1 grant management can
enhance future grants. The barriers we noted included:
performance measures were not verifiable or
quantifiable and did not provide adequate
accountability;
lack of a timely formal evaluation of the pilot by
; Region 1 to ensure lessons learned were
incorporated into PPG guidance;
iii Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
lack of a mechanism for measuring administrative
savings; and
MADEP problems in meeting Region 1 multimedia
reporting requirements.
Management deficiencies identified in the administration of-
the pilot demonstration grant were a direct result of poor
start-up planning. Different levels of understanding to
commit to the grant existed and there was never 100
- percent closure on what part of the grant work plan would or
would not be completed. The MADEP project officer agreed
with the region officials'comments concerning grant
management and stated, "more up-front planning is needed,
discussions and negotiations are needed between the
Region and MADEP prior to entering Performance
Partnership Grants."
Additionally, a decision memo was not prepared for the
MADEP pilot demonstration grant to aid in the grant's
management. The decision memo contained in the New
Hampshire pilot demonstration grant, was an in-depth
management tool that contained specific information for EPA
grant management. The New Hampshire decision memo
contained such items as: grant duration, components of the
work plan, application procedures, budget period, evaluation
plan requirements, environmental goals and indicators and
program and financial reporting requirements. The
development and use of a decision memo for the
..." , Massachusetts pilot demonstration grant would have helped
alleviate grant administration problems.
We believe the lack of "up-front" management planning by
responsible Region 1 officials directly contributed to the
problems associated with the grant's administration. Early
and more thorough management planning between Region
...... 1 and MADEP officials, before the start-up of the
: . demonstration grant, would have clarified questionable
areas and helped eliminate problems identified in this
review.
iv
Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
While MADEP's pilot demonstration grant encountered the
obstacles previously mentioned, we believe current
responsible Region 1 and MADEP management officials
have recognized these problems and have strengthened
their working relations to eliminate these obstacles.
Elimination of these barriers will improve future PPG
management.
RECOMMENDATIONS We recommend that the Regional Administrator, Region 1:
1) Work with MADEP to ensure all grant requirements, (i.e.,
performance measures, evaluations, reporting and financial
matters) are fully understood by all concerned parties before
the start-up of future pilot grant programs or PPGs.
2) Use "lessons learned" from the demonstration grant to
work with EPA's Headquarters to help formulate a
successful PPG program.
3) Provide MADEP with effective and timely guidance on
national reporting requirements. EPA should be flexible and
implement improved reporting procedures as appropriate.
4) Work with MADEP to develop program measures that are
verifiable and quantifiable with specific milestones to
measure outcomes. Also, work with MADEP to develop
environmental indicators.
5) Develop evaluation criteria for future demonstration/pilot
programs to ensure the Agency's decision making process
uses documented lessons learned.
REGION 1 AND
MADEP COMMENTS
In general, the Region and State agreed with our
recommendations, but provided additional clarification or
explanation of actions taken. Their responses are
summarized at the end of the Issues and Recommendations
section. The Region's and MADEP's responses are
included as Appendix A and B. The Region did not feel an
exit conference was necessary before issuance of the
Report Na E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
special report. The Region advised that they may request a
meeting subsequent to the issuance of the final report. We
concur with this arrangement.
vi
Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
TABLE OF CONTENTS
Page
EXECUTIVE SUMMARY i
RESULTS IN BRIEF ii
ISSUES AND RECOMMENDATIONS v
REGION 1 AND MADEP COMMENTS v
PURPOSE, SCOPE AND
METHODOLOGY , 1
BACKGROUND 3
ISSUES AND RECOMMENDATIONS 5
GRANT FLEXIBILITY IMPROVED 5
WORK PLAN MILESTONES AND ENVIRONMENTAL
INDICATORS LACKED MEASURABILITY 6
REGION 1 DID NOT EVALUATE DEMONSTRATION
GRANT 9
COST SAVINGS COULD NOT BE DETERMINED :.. 12
REPORTING ISSUES NEED CLARIFICATION 13
STATE CONCERNS 15
RECOMMENDATIONS ....i .-...' 16
AUDITEE COMMENTS 17
OIG EVALUATION 19
APPENDICES
APPENDIX I: REGION 1 COMMENTS 23
APPENDIX II: MADEP COMMENTS...' 30
APPENDIX III: DISTRIBUTION 33
vfi Report N« E1FMG6-014W71-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
This page left biank intentionally.
.-s .. _' <-T/\
.'. .. :_ '. --.-V
Report N« E1FMG6-01 -0071 -7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
SPECIAL REVIEW REPORT
E1FMG6-014)071 -7400058
PURPOSE. SCOPE We completed our review of the Massachusetts
AND METHODOLOGY Demonstration Grant Flexibility Program. The purpose of
our review was to evaluate EPA's management and
oversight of the demonstration grant awarded to MADEP.
We also discussed MADEP's concerns for the effective
implementation and management of the demonstration grant
at the State level. Specific objectives were to:
verify that the pilot demonstration grant accomplished
its intended purpose;
determine if grant performance measures were
' established and if so were they verifiable and
quantifiable with specific milestones to measure
outcomes;
determine if Region 1 developed oversight
procedures that allowed the State the flexibility
promised while ensuring grant reporting requirements
were met;
evaluate and report the lessons learned to the
Agency; and
compare the lessons learned with the Performance ,
Partnership Grant agreement for fiscal year 1997
dated, January 31,1997.
We conducted our review during the period May 1996
through October 1996. We evaluated the MADEP pilot
demonstration grant to determine what lessons could be
learned from the pilot program that could apply to future
PPGs. Following the completion of field work in October
1996, the Region provided us with the fiscal year (FY) 1997
Performance Partnership Grant agreement between Region
1 and MADEP, dated January 31, 1997. Based on our
Report N» E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
review of the agreement, we have made changes to the
report to reflect the current status of PPG implementation in
Region 1. We conducted our work in the Region 1 program
offices, and MADEP offices in Boston, Massachusetts.
To find out if the MADEP demonstration grant attained its
proposed objectives, we reviewed the regional project
officer's files and obtained a list of the grants included in the
demonstration program. We also reviewed decision memos
concerning the demonstration grant and grant evaluation
plans and the EPA interim PPG guidance. To obtain
clarification about the matters found during our file reviews,
we interviewed the regional project officer and other Region
1 officials. Through these interviews we determined if EPA
had made visits to MAOEP to discuss the administration and
oversight of the demonstration grant. In addition, we
determined if there were written evaluations of the
demonstration grant. We documented lessons learned and
reasons why the program was considered a success or not.
Through discussions with MADEP officials, we determined if
the demonstration grant gave the State more flexibility in the
use of grant funds, resulted in cost savings as anticipated,
and whether or not the State was ready to enter a PPG with
EPA at the current time. We discussed State officials'
concerns and areas they believed needed improvement
before entering a PPG. We determined through interviews
whether State officials believed the demonstration grant was
successful and the specific reasons for their beliefs.
We reviewed FY 1995 and 1996 demonstration grant work
plans to determine if the performance measures and time
specific milestones contained in the work plans were
measurable and realistic. Also, we reviewed State reports to
: determine if they met reporting requirements and if they
: , effectively managed the State's financial Federal fund
:., . . matching and reporting requirements.
We performed our review according to OIG Manual Chapter
150 for Special Reviews. Special reviews are short-term
Report N» E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
studies of EPA activities. They are not designed to be
statistical research studies or detailed audits. Rather, they
are information gathering studies that seek to identify issue
areas for top management attention. The goal of a special
review is to produce timely, constructive change, while
minimizing the resources invested in studying and
documenting the issue areas.
BACKGROUND As part of EPA's reinvention and State capacity building
efforts Region 1 awarded a pilot demonstration grant to
Massachusetts in fiscal year 1995. The pilot program
responded to the recommendations from the National
Performance Review regarding the Agency's need to
develop State and tribal environmental protection capacity,
increase flexibility, improve intergovernmental partnerships,
help States and tribes improve environmental performance,
and increase ability to achieve administrative savings by
reducing and streamlining the grants process.
Region 1 awarded a multimedia grant to MADEP consisting
of funds from its Clean Water Act (CWA) Section 106
(water), Clean Air Act (CAA) Section 105 (air).and (RCRA)
grants to support a multimedia/pollution prevention
inspection and compliance program. The pilot tested the
effectiveness of inspections by multimedia inspectors
checking for compliance with applicable air, water,
hazardous waste and toxic use reduction requirements. In
addition, "rt tested promotion of source reduction
opportunities through technical assistance and enforcement
as a strategy for achieving compliance. MADEP was very
committed to attempting to achieve compliance through this
multimedia/pollution prevention inspection strategy.
From a grant demonstration perspective, the purpose of the
pilot was to test the pooling of 13 to 16 percent of funds from
three large State program grants, air, water and RCRA as
described above, to fund an alternative multimedia approach
to achieving program objectives. This multimedia approach
raised concerns whether the costs could be identified to the
Report N* E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
specific applicable authorized statues. The grant was an
example of allowing the State to experiment with using a
portion of grant funds. The grant was to include an agreed
upon work plan, including a plan for measuring success.
From a programmatic perspective the purpose of the grant
was to test a variety of specific issues including the
adequacy of multimedia inspection protocol and practices,^.,
the potential for acceptable reporting of activities
. --:.. accomplished, and the achievement of source reduction Lvi* ^
through both regulatory enforcement and technical
assistance. To measure the success of the approach,
indicators were to be developed to measure the success of
enforcement and the use of source reduction and
environmental objectives of the program.
The demonstration grant was set up for a two year period,
October 1, 1994 through September 30,1996. As of August
14,1996, the total grant funds provided were $2,823,743;
Federal share, $2,112,300 and State share, $711,443.
Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
ISSUES AND RECOMMENDATIONS
GRANT FLEXIBILITY
IMPROVED
The demonstration grant resulted in administrative and
technical efficiencies, however, the Agency can improve the
success of future grants through the elimination of current
barriers and more effective grant management. The issues
identified during the review and our recommendations to
enhance the success of future grants are presented below.
The MADEP demonstration grant was considered successful
by both EPA Region 1 and State officials because it resulted
in both improved program administrative flexibility and
administrative efficiencies. The pilot grant demonstrated
administrative flexibility by showing that a recipient could
develop and carry out multiple year, integrated enforcement
activities across media statutes and programs.
Massachusetts developed and set up a multimedia
approach that incorporated inspections of water, air and
RCRA medias, under the MADEP Bureau of Waste
Prevention, into a single, facility wide inspection. MADEP
officials believe facility wide inspections improved the
regulatory process by replacing single-media inspections
with one multimedia inspection.
MADEP officials further stated that multimedia inspections >v
also provided greater flexibility in targeting polluters. t'^
Inspectors who previously performed single media , ;Q
inspections were trained to conduct multimedia inspections.,'
For example, one MADEP inspector who previously . :,
performed only water inspections is now trained to perform
water, air, and RCRA inspections.
The project also demonstrated that administrative savings
can be achieved through reducing the number of grant work
plans associated with single-media programs for water,
waste, and air. MADEP reduced its work plans from three to
one consolidated work plan that encompassed each of the
three media under the demonstration grant.
Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
From a programmatic viewpoint the demonstration pilot
provided increased flexibility in the performance of facility-
wide inspections. However, the MADEP grant financial
manager stated his workload had increased not decreased
as a result of the grant He advised that although the grant
was awarded in October 1994, funding was not awarded
until February 1995. He stated the delay added to his
administrative workload.
According to the MADEP grant financial manager, MADEP
invested a large amount of work and time to establish the
demonstration grant because only the compliance and ^
inspection portion from each of the three grants (air, water,
and RCRA) comprised the program. He also stated that he
received limited financial guidance from EPA in establishing
the grant. He concluded by stating that although they
pooled the demonstration grant funds, his workload and
administrative costs increased because he had to account
for more grants. He said he was now responsible for the
financial management of four rather than three grants.
Although the program initiatives improved grant flexibility, we
believe the area of financial management requires additional
attention to ensure reduction of administrative workload and
associated costs in future PPGs.
WORK PLAN
MILESTONES AND
ENVIRONMENTAL
INDICATORS LACKED
MEASURABILITY
The demonstration grant FY 1995 and FY 1996 work plans
did not contain environmental indicators or environmental
program results to measure the program's success.
Instead, the output measures contained in the work
plans consisted of a number of inspections, often referred to
as "bean counts," for a particular targeted area and the
associated milestones pertained to report dates,
presentation dates or planned meeting dates.
MADEP targeted inspections by sectors and geographic
areas that met their priorities, finding this approach more
effective than being driven by national EPA targets.
MADEP's targeting included many smaller-source initiatives
such as State printers, inspection support for their
Report N» E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
Environmental Results Program pilot, Boston Dudley Street
project, environmental justice initiatives, and waste
management at industrial parks and malls, etc.
MADEP was to develop environmental indicators to measure
the success of the demonstration grant. Environmental
indicators are measurable features that provide evidence of
environmental and ecosystem progress (quality) or evidence
of trends in quality. Environmental indicators measure
changes in air, water and land quality parameters and
human health. For example, measures could include such
items as the change in the percentage of population
exposed to substandard air or water.
An MADEP program official stated,"... there were some
measurable milestones in the work plans however, there
were no environmental measures. For example, one can't
say that due to the consolidation of these grants, air is 10
percent cleaner."
An MADEP technical team developed alternative multimedia
measures of success to evaluate program progress.
However, they never implemented these measures.
Inspectors planned regularly to report the results of these
measures of success into MADEP's Facility Master File
during FY 1996. In practice, however, the measures proved
difficult to calculate so that reporting has not occurred.
The MADEP project officer stated, "environmental indicators
and measures of success still require extensive work,
requirements for further development are acknowledged at -
State, regional and national levels." In addition, the Region
1 project officer stated, about measuring performance, "Up-
front joint agreement on outcome measures and the
reporting of those outcomes needs to be dear to both
parties to minimize later misunderstandings. Where direct
measures are, or the ability to report direct measures is
- missing, identification of surrogate measures and agreement
on reporting needs to be clear in the work plan. Both parties
need to understand what reporting is necessary."
Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
We are pleased that responsible MADEP and Region 1
officials are aware environmental indicators and measures of
success need further development and are working to
improve this process. We believe this is an area that
requires considerable attention before the implementation of
future PPGs.
Our review of the New Hampshire demonstration grant, also
overseen by Region 1, showed they made improvements in
the development of environmental indicators and
environmental program results to measure program
effectiveness. This improvement in grant management
directly resulted from the guidance provided to the State by
the Region 1 project officer for the New Hampshire
demonstration grant.
We believe it would be beneficial and encourage the Region
1 project officers for the Massachusetts and New Hampshire
demonstration grants to coordinate their efforts to improve
environmental performance measures contained in future
State grants. These measures provide the only means of
determining whether grant deliverables are producing
beneficial results.
The FY 1997 Performance Partnership Grant agreement
contained environmental indicators and milestones to
measure program progress. To measure progress made,
and determine program effectiveness, MADEP largely used
a percentage factor in the FY 1997 agreement For
example, the percentage of State waters that will be safe for
. . recreation.
However, the FY 1997 Performance Partnership Grant
agreement did not contain current baseline data for tracking
progress toward the established milestones on an interim or
final basis. Based on our review of the FY 1997 agreement,
determining progress under the agreement would be nearly
impossible. For example, the agreement states that by the
year 2005, all major Massachusetts facilities in any of the
8
Report N» E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
174 Federal categories of major industrial sources will meet
toxic air emission standards.
The FY 1997 agreement does not provide baseline data to
use for future evaluation. As a result, it is impossible to
track, verify, collect data and determine progress under the
agreement We believe Region 1 and MADEP have made a
first step under the PPG in developing environmental
performance measures to evaluate program results. We
encourage Region 1 and MADEP to continue to work
together to refine these measures to meet the joint goal of
accountability and program results.
REGION 1 DID NOT
EVALUATE
DEMONSTRATION
GRANT
Region T oversight of the MADEP demonstration grant
evaluation process needs improvement. The Region 1
project officer stated Region 1 had not performed an
evaluation of the MADEP demonstration grant to determine
program successes or weaknesses. She also stated Region
1 had not prepared and/or submitted reports to EPA
Headquarters on the progress of the program. Evaluation
of the success or weaknesses of a demonstration program is
an important ingredient in the development of the future
PPG program.
The EPA Grant Flexibility Initiative Steering Committee
focused on the question of evaluation. They developed
specific areas in each grant and/or approved work plan with
the State to address "measuring success." The categories
included four areas: environmental priorities
addressed/improvements sought, administrative efficiencies
gained, improved State/EPA working relationships, and
program results achieved. Each grant should include a
detailed description of measures.
Although Region 1 recognized the importance of the
evaluation process and developed a comprehensive
evaluation plan, dated October 26, 1994, for the
demonstration grant, they never implemented the plan. The
evaluation plan was very specific and required the
Report N» E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
preparation of evaluation reports by both the MADEP and
Region 1 on various phases of the grant's program results
and administrative efficiencies as prescribed by the EPA
Grant Flexibility Initiative Steering Committee.
A Headquarters EPA Grants Flexibility Steering Committee
member, for the MADEP demonstration grant, helped
develop and write the evaluation plan for the MADEP
:. . demonstration grant. However, he did not know they never
implemented it. He stated that Region 1 was primarily
responsible for overseeing the grant.
A Region 1 senior management official, who was actively _..
involved in the implementation of the MADEP demonstration
grant, stated although the evaluation plan was extensively *u
developed, Region 1 and MADEP could not agree on what .-
should be done. He stated the key MADEP player was the~"f
Assistant Commissioner. Unfortunately she left her position
and the evaluation plan was never approved. He concluded
by stating the primary causes of grant management
problems were poor EPA start-up management, and a lack
of Region 1 and MADEP program management agreement
The MADEP grant project officer also expressed concerns
about grant evaluation requirements. She stated that it was
her understanding Region 1 and MADEP would perform
separate evaluations, however, Region 1 never explicitly
explained when the evaluations would be performed or who
would perform them.
On March 29,1996, MADEP agreed to perform an
evaluation at the request of Region 1 as an amendment to
the FY 1996 work plan. The MADEP Director, Bureau of
Waste Prevention, Office of Program Integration, stated
MADEP agreed to perform the evaluation although MADEP
had made no previous requirement in the original grant
agreement to perform one. He stated MADEP agreed to the
March 29, 1996 amendment to perform the grant evaluation
as one prerequisite for Region 1 to release the remaining
grant funds. The MADEP grant financial manager said
10
Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
Region 1 withheld funds of $538,500. The withholding of
grant funds by Region 1 is addressed later in this review
under the section entitled, "State Concerns."
MADEP developed a plan to evaluate the demonstration
grant in the spring of 1996. It was an activity-based
evaluation plan that reflected the number of inspections
performed. The information reviewed by MADEP was
preliminary and a two-year evaluation of the plan was due In
November or December 1996.
MADEP did an independent preliminary evaluation of the
demonstration grant in July 1996. The MADEP project
officer stated in a letter of attachment to the evaluation,
."Please be advised that some of the materials are extremely
preliminary. Some were prepared for me by a very talented,
but very young, intern who had never before written for our
intended audience." Based on our review of MADEP's
preliminary evaluation, and the fact Region 1 never
performed an evaluation of the demonstration grant, we do
not believe the intent of the EPA Grant Flexibility Initiative
Steering Committee's evaluation process to measure the
success of the grant demonstration program was met.
We believe the Region 1 project officer should have
prepared periodic independent evaluation reports of the
MADEP demonstration grant and documented the program's
strengths and weaknesses. These evaluations would have__
timely measured the success of the program and met the
intent of the evaluation process as prescribed by the EPA ".C
Grant Flexibility Initiative Steering Committee. Also, MADEP.
and Region 1 officials needed clearer lines of " '
communication to ensure all participants fully understood
grant requirements before the implementation of PPGs.
We believe the Region 1 project officer for the MADEP
demonstration grant should contact the Region 1 project
officer for the New Hampshire demonstration grant program
to gain an understanding of the methods used to evaluate
the New Hampshire demonstration grant. We believe the
11 Report N» E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
coordinated efforts of regional grant project officers will
significantly enhance the success of future grants.
Although Region 1 developed a comprehensive evaluation
plan to report the lessons learned from the demonstration
program, they did not conduct the evaluations. Evaluations
identify the successes or failures of the demonstration grant
program and are an important ingredient in the development
of future PPGs.
The FY 1997 PPG agreement referred to evaluations. The
agreement stated, "DEP will prepare a brief summary report
by March 15, 1997. It will describe progress made towards
meeting environmental goals. The report will briefly
summarize the status of federal grant expenditures as of the
time of the report, on a grant basis only. The report will also
identify areas of environmental or public health needs which
should be considered for discussion with the public and the
EPA in developing the FY '98 Agreement"
We believe Region 1 should make periodic independent
evaluations of the FY 1997 PPG to measure program
success and to identify program weaknesses requiring
corrective actions. Although MADEP evaluations are useful
management tools to measure program results, independent
EPA program evaluations are essential to the success of
future PPGs.
COST SAVINGS
COULD NOT BE
DETERMINED
Demonstration grant administrative cost savings could not
be determined. The Region 1 project officer stated that
from the Federal perspective savings could not be
quantitatively measured, but from a qualitative viewpoint,
she believed there was a reduction in administrative
paperwork. The Region 1 project officer stated, "The project
demonstrated that savings could be achieved through
reducing the number of grant work plans associated with
single media programs for water, waste and air."
12
Report N« E1FMG6-01-0071-74000S8
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
State officials said there were no documented administrative
cost savings under the demonstration grant The MADEP
project officer stated that she did not know of any
program/technical cost savings. In addition, the MADEP
grant financial manager stated there were no cost savings
and, in fact, administrative costs increased due to the
implementation of the demonstration grant. As previously
discussed, MADEP required a large amount of work and
time to financially establish the demonstration grant because
only the compliance and inspection portion of the three
grants (air, water, and RCRA) comprised the program. The
MADEP grant financial manager said he did not feel
confident that in the future financial administrative cost
savings would materialize.
REPORTING ISSUES EPA requirements for the reporting of data was a significant
NEED CLARIFICATION problem in the administration of the MADEP demonstration
grant. The Region 1 project officer and the MADEP project
officer agreed that EPA needed to give grant partners clear,
succinct and timely information on minimum national
reporting requirements before the implementation of PPGs.
MADEP encountered several difficulties in meeting EPA's
expectations for accomplishing its multimedia reporting
under the demonstration grant. MADEP conducted facility
wide inspections, accounting for and reporting the results as
one multimedia activity (air, water, and RCRA). However,
EPA required MADEP to report the results to multiple
categorical reporting systems.
The MADEP project officer stated, "MADEP is accounting for
and reporting information in a multimedia data base to
include, air, water and RCRA activities as one entity.
Although we use one data base for all three activities,
Region 1 is requiring information be reported categorically.
This has created a great deal of extra work for us and we
don't believe this level of reporting is necessary." She
concluded by stating, "EPA has not changed its categorical
13 Report Nfl E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
grant reporting requirements to meet the intent of the
. multimedia demonstration grant."
Also, the MADEP project officer stated MADEP encountered
difficulties in meeting EPA's expectation for reporting results
for the demonstration grant. Specifically, MADEP
encountered problems in providing a categorical breakout of
inspection information, level of reporting detail, data entry
- * procedures, data entry codes for national reporting systems,
- scope of data requested, and procedures for reconciliation
'" discrepancies.
The Region 1 project officer stated, "A lesson learned for the
PPG would be that EPA's structure is not compatible to
multimedia grants. EPA wants this multimedia program, but._
can't get away from categorical grant reporting
requirements. EPA must do multimedia prioritizing by :-
targeting and deciding the content of the work to be ->
performed. EPA and MADEP want to work in partnership
but until EPA can overcome its own obstacles it will be
difficult."
The Region 1 project officer also stated that grant reporting
problems should be resolved before the implementation of
PPGs. She stated the following reporting problems currently
exist, "MADEP's multimedia Facility Master File (FMF)
system is not directly compatible with national compliance
and enforcement reporting systems, which increases time
and costs and degrades quality assurance. Specific
problems included: redundant manual entry, interim software
for computer uploads, and manual sorting of FMF data for
reporting to EPA are required. Additionally, all enforcement
case reporting was compiled by DEP's Demo Grant
Coordinator and submitted in hardcopy in the absence of an
automated system."
: - Grant reporting is a major PPG concern the Agency must
resolve at the earliest possible date to ensure the success of
future PPGs. The FY 1997 PPG agreement made reference
to reporting requirements. The agreement states, "Normal
14
Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
reporting of program data required by federal programs will
continue under this agreement as DEP and EPA continue
discussions about State reporting requirements to national
databases." We believe the problems associated with grant
reporting is a significant issue that EPA and the State must
resolve before the implementation of future PPGs. We are
encouraged that the Region 1 and the MADEP staff have
recognized the above reporting problems and have met to
work on resolving them.
STATE CONCERNS MADEP program management officials expressed a concern
over the late receipt of Federal funds required to manage
the demonstration grant program.
The MADEP grant financial manager stated that during the
grant period EPA withheld payment for $538,500 due to
unresolved program issues between MADEP and EPA. As
a result, as of June 30,1996, actual grant costs exceeded
Federal funds by approximately $200,000 causing the State
to finance the grant for that amount temporarily. He said the ;
grant could not have continued without the temporary use of
these State funds.
According to the Region 1 project officer, funds were not
withheld. They did however require MADEP to address the
four remaining issues as an amendment to the FY1996
work plan before they finalized the FY 1996 demonstration
grant. The four issues were: (1) a commitment to perform
evaluations; (2) a commitment to review continuous
emission monitoring reports; (3) an additional breakout of
inspection numbers; and (4) inspection estimates for waste
generators. She said the completion of these issues was
essential to the successful implementation of the
demonstration grant.
In a March 29, 1996, letter from the MADEP Assistant
.-; . Commissioner, Bureau of Waste Prevention, to the Region 1
Director, Environmental Stewardship, the MADEP Assistant
Commissioner agreed to perform the four outstanding issues
15 Report N* E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
as an amendment to the FY1996 work plan. He concluded
the letter by saying he trusted that these commitments
should be sufficient to allow the EPA-New England grant
manager to release the remaining funds. Region 1 made a
final grant award for $538,'500 to MADEP, effective August
14,1996. We believe this is an area where better planning
and more timely communication between Region 1 and
MADEP officials would have alleviated State financial
concerns in the management of the demonstration grant.
EPA staff should use the lessons learned from the
Massachusetts demonstration grant to set up improved ./:!:
management procedures before entering into future PPGs.
We believe the current Region 1 and MADEP project officers
have recognized the obstacles and barriers we identified in
the demonstration grant project. We encourage them to
continue to work together to eliminate these barriers to help
ensure the success of future PPGs.
RECOMMENDATIONS We recommend that the Regional Administrator, Region 1:
1) Work with MADEP to ensure all grant requirements, (i.e.,
performance measures, evaluations, reporting and financial
matters) are fully understood by all concerned parties before
the start-up of future pilot grant programs or PPGs.
2) Use "lessons learned" from the demonstration grant to
work with EPA's Headquarters staff to help formulate a
successful PPG program.
3) Provide MADEP with effective and timely guidance on
national reporting requirements. EPA should be flexible and
implement improved reporting procedures as appropriate.
4) Work with MADEP to develop program measures that are
verifiable and quantifiable with specific milestones to
measure environmental outcomes. Also, work with MADEP
on guidance for developing environmental indicators.
16
Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
5) Develop evaluation criteria for future demonstration/pilot
programs to ensure the Agency's decision making process
uses documented lessons learned.
REGIONAL The Region's response stated that it was incumbent that our
RESPONSE review of the lessons learned acknowledge the experimental
nature of the Demonstration Grant. The Region understood
that Massachusetts was undertaking a state-of-the-art
approach, unique nationally.
The Region disagreed with our assessment that there was a
lack of "management buy-in." The response stated that the
"senior managers at Region I were and are committed to
improving the delivery of federal funds to Massachusetts
using the grant authority demonstration." However, the
Region noted that "a problem during the early stages of the
project was the difficulty of promoting the buy-in of many
participants of both agencies who were wedded to the
traditional ways of doing business."
With regard to performance measures, the regional
response stated the demonstration grant work plans
contained quantifiable activity commitments on the number
of inspections to be performed for each initiative verified in
semiannual reporting. Additionally, MADEP committed to
develop innovative and quantifiable measures of
environmental performance success. However, as the OIG
reported, the absence of reportable measures hindered
report implementation.
The regional management also believed our draft report
implied that environmental measures developed for the New
Hampshire demonstration grant were transferable to the
MADEP demonstration grant. They said this was not
possible. The New Hampshire demonstration grant dealt
with environmental quality programs and monitoring for
which measures were being developed. The MADEP
demonstration grant pertained to compliance and
enforcement objectives and monitoring, for which specific
measurements have not been developed.
17 Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
Further, the regional response stated that the financial
issues needed to be placed in historical context. EPA
discussed financial requirements and cost allocation with the
State over a two year period. MADEP was aware of the
situation, but "concluded that a separate grant account was*-
both feasible and an acceptable level of administrative cost
. " to achieve its desired program flexibility."
Regarding work plan milestones and environmental
indicators the Region indicated there had been a
commitment to develop success measures. Also, the
'-'- FY1996 grant work plan included semiannual reporting
milestones. They agreed, however, that our conclusion was
correct that, the measures proved difficult to calculate so
reporting had not occurred.
Finally, according to the regional response the draft report's
conclusion that they performed no evaluation and they
submitted no progress reports to Headquarters was taken
out of context and is inaccurate. Regional program staff met
with MADEP personnel and reviewed semiannual data that
they entered into national databases. However, the two-
year nature of the demonstration program constrained the
feasibility of performing evaluations until the follow-up
inspections had occurred by the end of the second year.
Until that time, data to compare against the first year
baseline inspections was not available.
The regional response also notes that throughout the grant
work plans, MADEP committed to performing evaluations.
Unfortunately, changes in key MADEP management
..--..- occurred and the "Region had reason to believe that the
commitment to perform the demonstration evaluation had
slipped through MA DEP cracks during transition, without a
resource commitment made to ensure the evaluations would
be performed."
In March 1996, regional management took action to assure
that MADEP met its evaluation responsibilities. In early
October 1996, MADEP requested and was granted an
18
Report N« E1FMG6-01 -0071 -7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
extension of the demonstration grant to March 31,1997.
MADEP recently completed the evaluation of the two-year
demonstration and the Region 1 team is reviewing it. The
final evaluation report was issued in April 1997.
STATE RESPONSE
PIG EVALUATION
The State officials in their response stated they did not
intend to reduce administrative costs, and anticipated that .
some costs might increase due to the complex nature of this
grant. No mechanism for measuring administrative costs
existed as this was a relatively unimportant concern to the
design of the grant.
The State officials were also concerned over our
"characterization" that it failed to report accomplishments to
EPA, and that they did not use non-traditional measures.
According to the State they regularly reported the traditional
program output accomplishments: number of inspections,
and number and types of enforcement actions. MADEP
even manipulated its data to present multimedia work in a
manner that would be easy to interpret by single program
analysis. Also, MADEP used additional nonperformance
measures.
Finally, State officials said they looked forward to continuing
to implement many of the draft report recommendations in
combination with their own assessment of desired
improvements as they initiated plans for the FY1998 PPG.
The MADEP demonstration grant was a model for future
PPGs and Performance Partnership agreements (PPAs).
The MADEP was to use the "lessons learned" from the
demonstration grant to improve the overall quality of work
plans and measurements of success for both environmental
outcomes and outputs. Environmental indicators were to be
developed to measure the success of enforcement and the
use of source reduction and environmental objectives of the
program. We agree with Region 1 officials that the grant
work plan contained quantifiable activity commitments on the
number of inspections to be performed for each initiative.
19
Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
However, the MADEP did not implement or report the
measures of success they developed.
We are encouraged that the Region and MADEP have taken
the first step by developing and including performance
measures in the FY 1997 PPA to meet program
accountability and environmental results. We further believe
that although the environmental factors developed for the
New Hampshire demonstration grant may not be directly
transportable to the MADEP demonstration grant, they can
still be of significant value to future MADEP environmental
planning. We encourage the Region 1 project officers for
the Massachusetts and New Hampshire demonstration
grants to work together to improve environmental measures
and results.
We have clarified our position on management buy-in to the
MADEP demonstration program. We did not mean to imply
that there was a lack of EPA-Region 1 senior management
buy-in to the demonstration grant. We acknowledge that
senior management at both EPA-Region 1 and MADEP
played a significant role in the implementation of the MADEP
demonstration grant. However, Region 1 does acknowledge
that during the early stages of the project it was difficult to
obtain the buy-in of many participants of both agencies. We
believe better up-front planning between Region 1 and
MADEP management officials would have clarified
questionable areas, and helped eliminate grant
administrative problems/misunderstandings identified in this"--
-. - review.
A primary goal of the demonstration grants initiative was to
reduce administrative costs by reducing and streamlining the
grants process. In this case, the administrative costs were
not reduced but increased by creating an additional grant.
The MADEP grant financial manager stated that although
Region 1 awarded the grant in October 1994, MADEP did
not receive funding until February 1995. He also stated the
delayed receipt of grant funds further increased his
administrative workload because costs which had been
20
Report Na E1FMG6-01 -0071 -7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXEBHJTY PROGRAM
temporarily charged to the existing grants (Air, Water and
RCRA) had to be transferred to the new demonstration
grant. We believe more effective communication and up-
front planning between Region 1 and MADEP officials could
have resulted in decreased administrative costs and
alleviated state financial concerns in the management of the
demonstration grant.
As we noted in the narrative section of our report, attention
needs to be given to developing environmental indicators.
Environmental indicators are measurable features that
- . . provide evidence of environmental ecosystem progress
(quality) or evidence of trends in quality. Environmental
indicators measure changes in air, water, and land quality
parameters and human health. For example, measures
could include such items as the change in the percentage of
population exposed to substandard air or water. MADEP
was to develop environmental indicators to measure the
success of the demonstration grant. An MADEP technical
team developed alternative multimedia measures of success
to evaluate program progress. However, the measures
proved difficult to calculate so that reporting has not
occurred. While MADEP committed to developing
measures, they did not carry the process through to
conclusion.
In addition, we believe Region 1's lack of an independent
evaluation is a significant area of concern. Although
MADEP performed a preliminary evaluation (prepared by an
intern), we do not believe this relieved the Region of its
responsibility. The Region 1 project officer should have
prepared independent evaluation reports and documented
the program's strengths and weaknesses. The EPA Grant
Flexibility Steering Committee prescribed the areas to be
evaluated and contained in the Region 1 evaluation plan
which was never implemented. EPA was to use the results
of these reviews in the PPG program development
Moreover, we disagree that the two-year nature of the grant
constrained the flexibility of performing an evaluation. We
21
Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
believe they could have conducted periodic evaluations for
such areas as: documented cost savings, documented,
paperwork reduction, and whether the program achieved
multimedia/multiprogram objectives, such as ecosystem
protection, pollution prevention, and environmental justice.
These areas were contained in the Region 1 evaluation
plan. The Agency must identify and resolve significant
problem areas such as the reporting of multimedia data at
the earliest possible date to ensure the success of future
PPGs.
Further, it is significant to recognize that EPA's
demonstration grant (and PPG authority) was not a true
block grant to States. EPA continued to have fiduciary
responsibility for the State's results in carrying out Federal
mandates. In the MADEP demonstration grant, 75 percent
of the funds were provided by the Federal government and
25 percent by the State. In view of EPA's fiduciary
responsibility and based upon the fact that this is a pilot
project under EPA's Grant Flexibility Initiative to be managed
under Federal administrative guidelines, we do not believe
independent EPA grant evaluations are unreasonable.
Finally, we have clarified our position on MADEP's
agreement to perform an evaluation of the demonstration
grant. We believe more timely communication between
Region 1 and MADEP officials would have alleviated grant
evaluation concerns. While MADEP and Region 1 were
both aware of the ultimate necessity of evaluating activities,
they did not negotiate a formal set of questions until March
- 1996. This led to MADEP's independent preliminary
evaluation in July 1996 and Region 1's final grant award in
August 1996.
22
Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
APPENDIX I
REGION I RESPONSE
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
REGION I
JOHN F. KENNEDY FEDERAL BUILDING
\ wo*0" BOSTON, MASSACHUSETTS 02203-0001
~.__ . 11 ,f , «- RECK3HM. ADMINISTRATOR
DAT8S April 16, 1997
SDBJi Comments on Draft special Review of the Lessons Learned From " '---
the Massachusetts Grant Flexibility Demonstration Program
Report No. E1FMC6-01-0071-XXXXXXX
rs^XN/^sNA/V'-' *-.
strator
FROM: John P.
Regional Admini
TOt Paul D. McXechnie
Divisional Inspector General
Eastern Audit Division
We have reviewed the draft special report on the Lessons Learned
From the Massachusetts Grant Flexibility Demonstration Program,
received March 10, 1997.
We appreciate your recognition of the contribution of the
demonstration grant effort to positive grant and program
flexibitity and your recognition of the steps we have already
taken to deal with early problems associated with the development
of the demonstration project. I was surprised with your comments
on page iv regarding the lack of "management buy-in". I can -
assure you that the senior managers in EPA-Region I were and
are committed to improving the delivery of federal funds to
Massachusetts using this grant authority demonstration.
While the use of a multi-program workgroup caused delays and
difficulties in getting timely and constructive input into the
process of negotiating and^.finalizing the intial workplan, we
believe we have made considerable progress in achieving cultural
change since the Massachusetts demonstration grant began. So too
has the state. In addition, we have reorganized our planning and
grant activities around State Units, providing a strong structural
contribution to integrated planning and action.
Please be clear however, that Senior Management buy-in was the
reason this project even got started in the first place. We
have forged a path for what is now a national movement toward
consolidated grants. And the reason we were able to do so was
because of the time, energy and attention that senior management
at EPA-Region I and the Commonwealth devoted to making this
happen.
23 Report N» E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
-2-
Attached ia the Office of Environmental Stewardship's response
to the review. I have also attached a letter from the Massachu-
setts Departaent of Environmental Protection that outlines their
comments on the Draft Special Review. If you have any questions
or concerns regarding this response, please call ae or Susan
Kulstad at (617) 565-3228.
You nay want to schedule the exit conference after the MA DBP
presentation of the demonstration evaluation on April 30, 1997
(1:00-4:00 p.«. in the 10th Floor DOL Conference Room). We look
forward to discussion of these areas during the exit conference.
Enclosures
24
Report N« E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
Attachment
p. 1 of 5
Response to "Draft Special Review of the Lessons Learned
From the MA Grant Flexibility Demonstration Program"
OIG Report No. E1FMG6-01-0071-XXXXXXXXX
Dated March 7, 1997
From the Office of Environmental Stewardship
It is incumbent upon the Environmental Protection Agency (EPA) that
a review of the lessons learned acknowledges the experimental
nature of the Massachusetts Compliance Assurance Demonstration. By
granting funding under the statutory provisions for a
'demonstration' grant, EPA understood that Massachusetts was
undertaking a state-of-the-art approach, unique nationally. In
organizing their compliance assurance program, Massachusetts
piloted a number of major innovations, including a premise that
focused Bore on aggregate threats from smaller sources than it did
on major sources of pollution; an integrated, multimedia inspection
protocol; coordinated action to support assistance initiatives; a
pollution prevention emphasis; and an effort to develop and
measure the environmental performance of inspected facilities.
The Massachusetts demonstration grant was underway before the
development of the Performance Partnership program, although
many of the concepts later adopted into Performance Partnership
incorporated ideas first conceived of in the Massachusetts
demonstration. It is within this context that your review of
lessons learned from the demonstration needs to be framed.
COMMENTS
Performance measures were not verifiable or ouaatiftable..,,*i
(p. iii. Executive Summary; page 8) The demonstration grant work
plans contain quantifiable activity commitments on the number of
inspections that would be performed for each initiative.
Verification of tho*« commitments appeared in the semiannual
reporting committed to in the work plans, which simultaneously
acted as a commitment to supply data for national compliance and
enforcement reporting. Additionally, MA OEP committed to develop
innovative and quantifiable measure of success on the environmental
performance of inspected facilities which it did submit, but which
are not yet being reported (as the OIG draft report notes) because
of practical difficulties "in implementation.
In discussion later in the report at page 8, the draft report seems
to imply that environmental measures developed for the Kew
Hampshire demonstration grant are transportable to the MA DEP
demonstration grant. They are not. The nature of the Kew
Hampshire demonstration encompassed the implementation of
environmental quality programs and monitoring, which lends itself
to the use of the environmental measurement being developed
nationally and regionally. In contrast, the MA Compliance
25
Report N« E1FMG6-01-0071-74000S8
"V '""'
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
Attachment
p. 2 of 5
Assurance Demonstration Grant pertains to compliance and
enforcement objectives and monitoring, for which no comparable
environmental measurements have yet been developed. For this
reason, performance measurement of compliance assurance nationally
as well as regional!/ continues to be largely reliant upon activity
measurement until better measures can be developed and implemented
in practice.
fT1{te
did pot receive m
buv-*ii>.*i
{Exec. Sum, p. iv) He believe that the project had a high level of
management buy-in at both EPA and KA DEP. However, a problem
during the early stages of the project was the difficulty of
promoting the buy-in of many participants at both agencies who were
wedded to traditional ways of doing business. At EPA, the use of
a aulti-prograa workgroup caused delays and difficulties in getting
timely and constructive input into the process of negotiating and
finalizing the initial work plan. we believe we have made
considerable progress in achieving cultural change since the
Massachusetts demonstration grant began. In addition, we have
reorganized our planning and grant activities around State Units,
providing a strong structural contribution to integrated planning
and action.
*[HA PEP grant
stated that ha received limited
financial guidance from EPA in establishing the grant. *t
(p. 6, para 3) The conclusions that EPA gave limited financial
guidance concerning the grant and that the administrative costs
increased because of the need to set up an additional grant account
needs to be placed into context by reviewing the history leading up
to the grant. Almost two years before MA DEP applied for the
grant, EPA discussed with MA DEP the financial requirements
concerning multimedia inspections supported by more than one
categorical grant. EPA offered to work with MA DEP to develop a
method to allocate the costs to each categorical grant as required.
After sporadic efforts by MA DEP, EPA was told that MA DEP had
concluded that such allocation was infeasible, contrary to the
management intention of the program, and far too burdensome to be
considered. It was explained to XA DEP that if a multimedia
inspection grant were made that the allocation problem would be
avoided, but that an additional grant account would need to be set
up. KA DEP clearly decided to proceed with full knowledge of this
requirement. Unlike development and use of an allocation method,
KA DEP concluded that a separate grant account was both feasible
and an acceptable level of administrative cost to achieve its
desired program flexibility.
"The Demonstration Grant FY 1995 aad FY199S vorfc plans did net
contain environmental indicators er environmental prffgrfflB
results.. . .*t (pp. 6, last par. and pp.7, par 4.) The FY95 grant
work plan did include a commitment for the development of measures
of success at page 6, as follows:
26
Report N» E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
Attachment p. 3 of 5
"1.8. Measures of Success: During FY95 and FY96, MA DEP
will be developing and piloting a number of new measures to
evaluate the success of MA DEP activity. MA OEP plans to
create, teat, and share with Region Z six measures for
pollution reduction in FY9S, and soae neasures for pollution
prevention in FY96. These will include measures at the
facility level which can be linked to specific actions taken
by HA OEP, as well as broader, statewide environnental
indicators of the aggregate benefits of MA OEP activity. Note
that for each of HA DEPs proposed compliance assurance .
initiatives proposed below, there are (or will be} specific
measures and criteria for evaluating its [sic] success."
Similarly, the FY96 grant work plan did include semiannual
reporting milestones for 'Compliance Inspection (and Enforcement),
Reporting' under Output I, Section J that entailed reporting on
"...DEPs new neasures of success (MOS)* that were introduced in the
FY9S grant work plan. See reporting milestones for section J on
page 7, and they are repeated under Output II, Section A on
Enforcement Reporting, on page 8.
However, the OIG report is accurate on page 7, paragraph 4, where
it concludes that, "In practice, however, the measures proved
difficult to calculate so that reporting has not occurred."
Region I Did MotJv&Luata Demonstration Oranft (PP 9-12}
Frograa Progress Was Periodically Assessed & Reported to HQt
On page 9, paragraph 3 of the draft OIG report, facts are reported
out of context, and the conclusion that no evaluation was performed
and no progress reports were submitted to HQ is inaccurate.
The Region I project officer stated that the basis for ongoing
progress evaluation by EPA was the compliance and enforcement data
submitted according to the grant work plan semiannual reporting
commitments. Region I technical program experts did regularly
review and net with MA DEP to discuss the semiannual data that was
reported, and the data was entered into the national EPA databases,
serving as the means for reporting to HQ. (However, as the OIG
draft report indicates, problems were encountered in MA DEPs ...
meeting of the reporting commitments.) Only upon IG questioning as
to whether further evaluations were made, did the Region I project
officer respond as cited on page 9 of the draft report.
Periodic Evaluation of Program Approach: (page 9}
As the Region I project officer pointed out during the meeting with
OIG, the two-year nature of the demonstration constrained the
feasibility of performing evaluations "...to determine program
successes or weaknesses.* Until the follow-up inspections had
occurred (by the end of the second year}, data to compare against
the first-year baseline inspections was not available. Therefore,
evaluation of DEPs program approaches was not timely until after -
the demonstration had been conducted.
27 Report N» E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
Attachment
p. 4 of 5
A MA DSP Evaluation of tne Demonstration Z* & Grant Work plan
Comaitaent (pp. 10-11)i The draft OZG report states incorrectly
at the top of page 11 that, 'The MA DEP Director... agreed to
parfont the evaluation although MA DEP had Bade no previous
commitment to do so,* and goes on to introduce a contingent
issue about EPA withholding of grant funds.
In the FY95 grant work plan, HA DEP states on pages 1-2 that its
'Basic Principal of this Compliance Assurance Strategy* is that "DEP
has a responsibility to continually evaluate the effectiveness of
the various compliance assurance tools and strategies and to usa
the results of these evaluations in [sic] make future decisions on
the allocations of resources.* Later on page 2, in paragraph 3,
it says, "DEP is proposing to test different combinations of
inspections, enforcement, compliance assistance, incentives, and
penalties, and ability of these combinations to assure compliance."
After discussion on the strategies to be tested, this section
concludes at the top of page 3 that DEP will, '...use statistical
sampling and alternative measures of success as tools for
evaluating the effectiveness of various initiatives.*
The FY95 grant work plan goes on to describe each two-year
initiative, and under many of the initiatives outlines the
evaluation that will be performed. For example, under 'Follow-up
to the FY94 National Priority Good Guys* on pages 7-8, DEP commits
to '...inspect [major sources having a strong compliance record that
were skipped in FY94] in order to test assure compliance and to
evaluate the impact of skipping them for one year." Under
'Statistically-Eased Targeting At Auto fueling Facilities,* DEP
commits on page 8 to inspect a randomly selected, statistically
significant number of facilities in FY95 followed by FY96
inspection of an equal number and type to '...evaluate the benefits
of the approach, and extrapolate results up to the entire universe
for the SIP.* As a third example, under "Statistically-Based
Targeting of Printers With Amnesty* on page 9, DEP commits to
inspect a statistically significant random sample in FY95,
undertake compliance assistance, perform follow-up inspections of
a comparable sample in FY96, and "...evaluate the benefits of the
approach, and extrapolate results up to the entire universe." An
evaluation step is included as a commitment for nost of the
initiatives within the FY9S grant work plan, which outlined the
2-year demonstration.
These FY9S 2-year demonstration commitments were the premise for
Region Is March 1, 1996 follow-up request and negotiation of a MA
DEP evaluation. MA DEP personnel changes for both the positions of
the Bureau of Waste Prevention Director and MA Demo Grant Project
Manager had occurred since FY9S, and Region I had reason to believe
that the commitment to perform the demonstration evaluation had
slipped through MA DEP cracks during transition, without a resource
commitment made to ensure that the evaluations would be performed. *
28
Report Na E1FMG6-01 -0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
Attachment p. 5 of 5
For this reason, Region I informed the Bwp Director in its March 1
letter of the central importance of the evaluation to the demon-
stration and indicated that reassurance of HA DEPs FY95 grant work
plan commitment with staff assignment to and schedule for
completion of the demonstration evaluation vas needed for release
of the remaining grant funds. In a written reply and amendment
dated March 29, 1996, MA DEP fulfilled Region Is request.
OIO conclusion that ao evaluation of the success of the
demonstration vas ever conducted is premature» At pages 11-12, the
draft OIG report points to a preliminary (and partial) submission
by MA DEP in July 1996 and the absence of a Region I evaluation in
spite of a '...comprehensive evaluation plan to report the lessons
learned from the demonstration program...* to make its conclusion
that no evaluation of the success of the demonstration vas ever
conducted.
In early October 1997, MA DEP requested and vas granted an
extension of the demonstration grant to March 31, 1997. During
this time, MA DEP is completing the evaluation of the two-year
demonstration and Region Is technical team is reviewing this .
evaluation. MA DEP. briefed Region I senior managers on the
evaluation of the results on March 26th, and will be providing
Region I with copies of their final evaluation report, which
incorporates EPAs review comments, at a staff briefing scheduled
for April 30th.
29 Report N" E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
APPENDIX II
MADEP RESPONSE
DEP COMMENTS ON IG REPORT NO E1FMG6-01-0071
DEP does not refer to the subject grant as "Demonstration
Grant Flexibility Program", but as the Compliance Assurance
Demonstration Grant. The IG should use the latter title for the
activities studied or both titles should be referenced early in <
the report so as not to confuse readers when reviewing any
associated documents produced by DEP. '
The Performance Partnership Grant Program was not developed
at the time the Compliance Assurance Demonstration Grant was '
initiated in September, 1994. Therefore, the Compliance Assurance1
Demonstration Grant could be said to serve as a model for the PPG, -
but was not designed to be a prelude, as the text suggests. - '
Correction of all references to the Demonstration Grant as "a -'<
prelude to" the Performance Partnership Grant is warranted. '
The IG evaluated administrative cost savings, but DEP did not
intend to reduce its administrative costs, and anticipated that
some costs might increase due to the complex structure of this
grant. No mechanism for measuring administrative costs existed as *
this was a relatively unimportant concern to the design of the
grant. Still, the IG states, for example, (page 2) "...we
determined if the demonstration grant.. .resulted in cost savings
as planned...* And, on page 6 text indicates "...the MADEP grant
financial manager stated his workload had increased not decreased
as a result of the grant." Finally, page 13 includes two lengthy
paragraphs in a section titled Cost Savings Could Not Be
Determined. While DEP has no problem with stating this fact, the
report must not indicate or suggest that any cost savings were
anticipated. DEP requests that all suggestions of planning or
performance barriers or failures related to administrative cost
savings issues be removed from the report.
Page 3 requires some minor wording changes in order to
clarify one of the intended purposes of the Grant. A suggested
redline/strikeout version of one sentence is, "In addition, it
tested promotion pit uaed source reduction opportunities aehi-owsd
through technicar~assistance and enforcement as a strategy for
achieving compliance." Although subtle, the distinction to be
made here is that DEP promotes source reduction as a strategy for
compliance; regulated facilities use source reduction as a
strategy for achieving compliance.
The section discussing Grant Flexibility Improved (page 5)
requires some significant correction to the third paragraph. It
is not accurate to say that "...multimedia inspections also
provided greater flexibility in targeting polluters." Rather, it
was the Demonstration Grant itself which provided this
flexibility. The next sentence (beginning "Inspectors who
previously...") more correctly should conclude the prior paragraph
{following "...with one multimedia inspection.1) A replacement
for that sentence might read as follows: "Flexible targeting has
allowed Massachusetts to experiment with expanding the scope of
30
Report N» E1FMG6-Q1-0071-7400058
. -:. At
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
inspections which were previously limited to national priority '
sources to include smaller facilities which may have larger
aggregate non-compliance problems and environmental impact."
It is not helpful to the readers' understanding to provide
unverified quotes, particularly when attributed position titles
are not found in the organization. Similarly, the second full
paragraph, first sentence, beginning "According to the MADEP grant
financial manager,..." is, additionally, difficult to understand
in meaning. The report should revise for clarity of meaning or
delete that sentence. .
The first sentence of the second paragraph on page 7 should
be revised as follows, "MADEP |ttempted was to develop
environmental indicators ^'ri conj;uhct,ton'with to meaourc the
oiiceesa of the demonstration grant." This would correct the
auditors' mistaken interpretation of the development of --.;
"alternative measures of success", a project which would have been
more fully undertaken had DEP been provided with additional
resources (as stated in DEP's FY95 grant workplan) . Two
paragraphs later, revisions should be made as follows: Inopcotoga
planned regularly to report the reoulto of -^hcsc measures of-
succc&g into MADBP'o Facility Maoteef File during FYOfr. In
practice, howaver, the measures proved difficult to calculateao
that reporting haa not occurred. These "alternative measures" are
more appropriately characterized as "program outcome" measures
than environmental indicators. The development of environmental
indicators, per se, was in its infancy, and was being discussed at
DEP but had not yet been undertaken at the time of the
Demonstration Grant. DEP would be happy to discuss the
differences batween environmental indicators and program outcome
measures with the IG, and to provide more details on the state of
development of either at the time of the grant throughout the U.S.
and in Massachusetts, if necessary, to ensure greater clarity in
the IG's report.
Of still greater concern is the characterization that
Massachusetts failed to report accomplishments to EPA, and that no
non-traditional measures were used. In fact, Massachusetts
regularly reported its traditional program output accomplishments:
numbers of inspections, and numbers and types of enforcement
actions. DEP even manipulated its data to present multimedia work
in a manner that would be easy to interpret by single program
analyses. And, DEP used additional non-traditional performance
measures, notably the numbers of facilities at which some type of
unpermitted, unlicensed, or unregistered activity was discovered,
the numbers of facilities where enforcement actions included
violations in more than one single medium program, and analyses of
traditional and non-traditional measures broken out by the
flexible targeting initiative categories in addition to the
traditional single medium programs. Again, if further
understanding is required by the IG staff in order to more fully
convey these points, DEP would be pleased to make the necessary
time available.
31
Report N" E1FMG6-01 -0071-7400068
" " " " * ,-"
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
Most serious of all concerns, is the reference (page 9} to an
"EPA Grant Flexibility Initiative Steering Committee" without
providing any details as to its dates of activity or membership.
DEP had no information about this committee or its work prior to
reading the draft "Lessons Learned" report. The IG characterizes
this Committee's report as determining that four areas of
measurement are necessary:
* environmental priorities addressed/improvements sought;
* administrative efficiencies gained;
* improved State/EPA working relationship; and
* program results achieved.
Page 11 of the IG draft report states that, "...we do not believe
the intent of the EPA Grant Flexibility Initiative Steering . ,
Committee's evaluation process to measure the success of the
grant's demonstration program was met.' DEP is gravely concerned
by this statement. Given that DEP did not have the benefit of the
committee's recommendations, and that the IG acknowledges that
they reviewed our preliminary evaluation of the grant, DEP is
forced to conclude that there must be some very basic
misunderstanding of the Compliance Assurance Demonstration Grant
and DEP's evaluation of our performance under the grant. The .
preliminary evaluation provided clearly demonstrates that the .
first and last items above; this letter clarifies that improvement
of administrative efficiencies was irrelevant to evaluation of
this grant; and DEP believes that improvement' of the state/EPA ;
relationship is implicitly included by virtue of the ongoing
communications and negotiations between DEP and EPA-New England t.
throughout the grant period. To characterize DEP's evaluation of
the grant as a complete failure based upon committee ,;
recommendations not provided to DEP, but which are certainly
present in DEP's analysis, is a complete misrepresentation both of
DEP's intent, and our performance of the analysis activity.
32 Report N* E1FMG6-01-0071-7400058
-------
SPECIAL REVIEW OF MASSACHUSETTS DEMONSTRATION GRANT FLEXIBILITY PROGRAM
APPENDIX Hi
DISTRIBUTION
Office of Inspector General
Inspector General (2441)
Deputy Assistant Inspector General for
Internal and Performance Audits (2421)
Deputy Assistance inspector General for
Acquisition and Assistance Audits (2421)
EPA Headquarters Office
Assistant Administrator for Administration
And Resources Management (3101)
Comptroller (3301)
Associate Administrator for Regional Operations and
State/Locai Relations (1501)
Agency Audit Followup Coordinator (3304)
Agency Audit Followup Official (3101)
Director for Program and Policy Coordination
Office (3102)
Office of Congressional Liaison (1302)
Office of Public Affairs (1701)
Headquarters Library (3304)
EPA Region 1
Assistant Regional Administrator
Director, Office of Ecosystem Protection
Manager, Strategic Planning Office
Regional Audit Liaison Coordinator
33 Report Na E1FMG6-01-0071-7400058
-------
.v: ''. "' "3G
.' ' - ' ' -» -,'^^J
:..-.Q6C
------- |