SffiA
United States
Environmental Protection
Agency
Office of Policy
(1807T)
October 2013
EPA-100-K-15-001
Formative Evaluation of
the OSWER Community
Engagement Initiative
Final Report
October 28, 2013
-------
ACKNOWLEDGEMENTS
The Formative Evaluation of the OSWER Community Engagement Initiative was developed
for the U.S. Environmental Protection Agency's Office of Policy under Contract EP-W-10-
002 between EPA and Industrial Economics, Inc. (lEc) of Cambridge, MA. The evaluation
team was comprised of the Office of Policy's Evaluation Support Division (ESD), the Office
of Solid Waste and Emergency Response (OSWER), and lEc. Angela Helman, Peter
Courtright, and Josh Wolff represented lEc. lEc proposed the evaluation methodology,
collected and analyzed data, summarized findings, and developed this evaluation report
under the oversight of ESD and with guidance from the entire evaluation project team.
Michelle Mandolia, the Work Assignment Manager, provided oversight on behalf of ESD.
Ellen Manges and Jackie Harwood from OSWER provided overall guidance and direction.
Special thanks go to the many interviewees from EPA, state environmental agencies, and
community groups, who provided critical information that informed the evaluation.
This report was developed under the Program Evaluation Competition, sponsored by
EPA's Office of Policy. To access copies of this or other EPA program evaluations, please
go to ESD's Web site at http://www.epa.gov/evaluate
-------
TABLE OF CONTENTS
ACKNOWLEDGEMENTS i
EXECUTIVE SUMMARY ES-1
CHAPTER 1 | INTRODUCTION AND EVALUATION SCOPING
Evaluation Scoping 1
Exclusion of the Underground Storage Tank (UST) Program 2
Superfund, Brownfields, and RCRA Corrective Action Programs Have Different Structures
That Affect Data Availability and Collection 3
Survey Work was Impractical 4
Evaluation Questions 4
Logic Models 5
CHAPTER 2 | METHODOLOGY
Review of Existing Data 10
Interviews 10
Thematic Analysis of Interview Data 12
Strengths and Weaknesses of Methodology 12
CHAPTER 3 | FINDINGS
Question 1: What Are the Requirements and Drivers for Community Involvement (CI)
within the Superfund, RCRA Corrective Action (CA), and Brownfields Programs? 14
Summary of Methods to Address Question 1 14
Findings on Requirements 14
Findings on Additional Drivers 15
Question 2: What Is the Baseline of Current OSWER Technical Assistance and
Information Dissemination Activities 15
Summary of Methods to Address Question 2 16
Findings on Frequency of Practice and Program-to-Program Variability 16
Superfund 16
Brownfields 17
RCRACA 18
Findings on Proportion of Communities that Receive Formal Assistance through TAG.
TASC, or TAB 18
Superfund 18
-------
RCRACA 18
Brownfields 20
Findings on the Selection Criteria and on-the-Ground Process for Accessing TAG and TASC 20
TAG 20
TASC 20
Findings on Community Assistance Provided Outside of Formal Programs 21
Superfund 21
Brownfields 21
RCRACA 22
Findings on Unmet TA or ID Needs 22
Superfund 22
Brownfields 23
RCRACA 23
Question 3: How Can Superfund Community Involvement Plans (CIPs) Be Used to
Improve Technical Assistance and Delivery of Information? 23
Summary of Methods to Address Question 3 23
Findings on CIP Implementation 24
Findings on CIP Revisions 24
Findings on CIPs as a Tool to Ensure Effective Technical Assistance and Information
Dissemination 24
Question 4: What Is the Baseline of Customer Satisfaction with OSWER Technical
Assistance and Information Dissemination Activities? 25
Summary of Methods to Address Question 4 25
Findings on Interview Satisfaction 25
Findings on Suggestions for Improving Services 27
Superfund 28
Brownfields 26
RCRACA 29
Question 5: What Measures Can be Used to Assess the Effectiveness and Tangible
Outcomes of OSWER Technical Assistance and Information Dissemination Activities
Across the Lifecycle of Site Planning, Remediation, and Reuse? 29
Question 6: How Can these Measures be Used to Improve OSWER Technical Assistance
and Information Dissemination Activities? 29
Summary of Methods to Address Question 5 and 6 29
Interview Findings of Relevance for Developing Measures 29
Approach to Developing Menu of Potential Measures 31
Menu of Potential Measures 31
Output Measures 31
Short-term Outcomes 33
Long-term Outcomes 33
111
-------
APPENDICES:
Appendix A: Methodology Document
Appendix B: Interview Summary Count
IV
-------
ACRONYM LIST
CEI: Community Engagement Initiative
CAG: Community Advisory Group
CARE: Community Action for a Renewed Environment
CPRC: Conflict Prevention and Resolution Center
CIP: Community Involvement Plan
ESD: Evaluation Support Division
HQ: Headquarters
ID: Information Dissemination
lEc: Industrial Economics, Incorporated
NEPA: National Environmental Policy Act
NCP: National Contingency Plan
OEJ: Office of Environmental Justice
OSWER: Office of Solid Waste and Emergency Response
OW: Office of Water
PRP: Potentially Responsible Party
RCRA CA: Resource Conservation and Recovery Act Corrective Action
RP: Responsible Party
TA: Technical Assistance
TAB: Technical Assistance for Brownfields
TAG: Technical Assistance Grants
TASC: Technical Assistance Services for Communities
TBA: Targeted Brownfields Assessment
UST: Underground Storage Tank
-------
EXECUTIVE SUMMARY
In December 2009, EPA's Office of Solid Waste and Emergency Response (OSWER)
announced the Community Engagement Initiative (CEI) to enhance OSWER HQ and
Regional offices' engagement with local communities and other stakeholders, and to
help stakeholders meaningfully participate in decision-making processes related to the
cleanup and reuse of contaminated sites. The CEI is comprised of 16 actions that affect
many aspects of OSWER's work. This evaluation focuses on Action 7 and Action 13A,
which address improving technical assistance (TA) and information dissemination (ID),
respectively.1 The CEI defines technical assistance (TA) as the provision of services,
resources, and training focused on increasing community understanding of the relevant
science, regulations and policy related to environmental issues.2 Information
dissemination (ID) refers to communication from EPA or state agencies to communities
regarding environmental issues; CEI focuses on delivering the right information, in the
right place, at the right time into existing guidance, training, and Agency outreach
efforts.3 Furthermore, this evaluation focuses on Superfund, RCRA Corrective Action
(CA), and Brownfields programs within OSWER.
CEI includes a commitment to evaluate the effectiveness of OSWER program community
engagement activities. OSWER envisions highly effective community engagement
becoming a standard business practice across all of its programs, and furthermore wants
to share lessons on successful community engagement throughout the Agency. As such,
OSWER requested a formative evaluation for the CEI program. OSWER's goals for the
evaluation were to establish a baseline of current community satisfaction with EPA's TA
and ID practices, and to consider the feasibility of potential measures for tracking TA
and ID moving forward.
EVALUATION QUESTIONS
The evaluation questions that we used to guide this evaluation are as follows:
1. What are the requirements and drivers for community involvement within the
Superfund, RCRA Corrective Action, and Brownfields programs?
2. What is the baseline of current OSWER technical assistance and information
dissemination activities, with respect to:
a. Frequency of practice and program-to program variability?
b. Proportion of communities that receive formal assistance through TAG, TASC, or
other formal program?4
1 ID is also sometimes referred to as Delivery of Information, or Dol.
2 EPA CEI, Action 7- Evaluate and Improve EPA Technical Assistance Processes, Work Group Report on Recommendations,
available at: http://www.epa.gOV/oswer/engagementinitiative/7 action recommendation report.pdf
3 EPA CEI, Action 13A- Delivery of Information, Summary of Draft Work Group Report, available at:
http://www.epa.gov/oswer/engagementinitiative/13a action summary report.pdf
4 Formal technical assistance refers to TA that is delivered under the auspices of a TA program such as TAG, TASC, and TAG;
information TA refers to TA delivered outside of a program.
ES-1
-------
c. Selection criteria and on-the-ground process for accessing TAG and TASC (for
Superfund and RCRA CA only)?
d. Community assistance that is provided outside of formal programs (including
helping to set up CAGs and providing ad-hoc assistance)?
e. Areas of unmet TA or ID need?
3. How can Superfund community involvement plans (CIPs) be used to improve
technical assistance and delivery of info?
a. Does every site have a CIP?
b. What information is available on the implementation of CIPs?
c. Are CIPs revised over time?
d. Do CIPs, as they are currently used, ensure effective technical assistance and
information delivery throughout the life of the project? Why or why not?
4. What is the baseline of customer satisfaction with OSWER technical assistance and
information dissemination activities?
5. What measures can be used to assess the effectiveness and tangible outcomes of
OSWER technical assistance and information dissemination activities across the
lifecycle of site planning, remediation, and reuse?
6. How can these measures be used to improve OSWER technical assistance and
information dissemination activities?
METHODOLOGY
The methodology for this evaluation combines a thematic analysis of qualitative data
gathered from interviews with a review and synthesis of existing documentation. The
complete methodology is included in Appendix A; it was completed in July of 2012.
Important methodological updates that occurred after finalizing the evaluation
methodology are discussed in Chapter 2.
lEc reviewed existing documentation and data to inform our responses to evaluation
questions, as discussed in the methodology. We reviewed the following data sources:
Documentation of community engagement requirements;
Site inventory data for cleanup programs;
Performance measurement resources (for Questions 5 and 6); and
Existing satisfaction interviews.
lEc conducted 46 interviews with EPA HQ and regional contacts from Superfund,
Brownfields, RCRA CA, as well as state RCRA CA contacts in Arkansas, California, Illinois,
Missouri, New York, and South Carolina. lEc also conducted satisfaction interviews with
contacts from community groups that had received technical assistance through the
Superfund TAG program, and analyzed these data in conjunction with previous
satisfaction interviews of service recipients conducted for the TASC contract. lEc
conducted a thematic analysis of interview data, and synthesized interview data with
ES-2
-------
information gleaned from the document review as applicable, to develop evaluation
findings.
STRENGTHS AND WEAKNESSES OF METHODOLOGY
The methodology employed has a number of strengths, including coverage of all three
programs: Superfund, RCRA CA, and Brownfields. Notably, the methodology employed
for this evaluation was sufficient for addressing most of the research questions,
including all of Questions 1, 3, 5, and 6; and most of Question 2. Within Question 2 in
particular, because lEc was able interview one contact for each of the three programs, in
every region, we are able to accurately characterize the baseline of EPA's technical
assistance and information dissemination activities. However, because we only
interviewed six states that implement RCRA CA, and only had information on a few
additional states from existing documentation, we cannot be sure if our findings on
frequency of practice, and community assistance provided outside of formal programs,
are representative of state RCRA CA programs overall.
For Question 4-establishing a baseline of satisfaction with technical assistance and
information dissemination among communities-a key strength of the evaluation was
the ability to combine analysis of new satisfaction interviews with existing interviews.
We also achieved geographic diversity in our interviews. However, the methodology for
Question 4 was limited by resources available to conduct interviews, and difficulties in
identifying community contacts to interview, particularly within the RCRA CA program.
Subsequently, a key limitation of our approach is that results for Question 4 on
satisfaction with technical assistance and information dissemination cannot be
extrapolated to all communities served by OSWER programs. Ideally, lEc would have
conducted surveys or interviews using a statistically valid sample of those served for
each OSWER program, to develop a true baseline of satisfaction with technical
assistance and information dissemination criteria.5 However, this approach would have
required extensive resources to develop contact data (in particular for RCRA CA
communities), and to develop and administer the survey or requisite number of
interviews. In addition, programs expressed reservations about broadly surveying
community groups affiliated with their programs, citing information burden.
Finally, the method by which lEc selected interviewees has inherent strengths and
weaknesses. For the TASC contract, lEc was able to analyze all applicable data collected
for Superfund to-date, which is a methodological strength. In addition, we were able to
randomly select Superfund TAG grantees to interview. In contrast, as discussed above,
lEc had to rely on TAB grantees to identify Brownfields community groups to interview,
and we had to rely on RCRA CA regional and state interviewees to identify RCRA CA
community groups to interview. Relying on these parties to identify interviewees can be
a source of bias, although we have no evidence or suspicions that contacts cherry picked
community contacts to participate in this evaluation.
s In addition to surveying a statistically valid sample of those served, it would also be methodologically preferable to survey
multiple stakeholders associated with each site. One individual interviewee may not represent the perspective of all local
stakeholders.
ES-3
-------
SUMMARY OF FINDINGS
High-level findings from the evaluation include the following:
The EPA Superfund and Brownfields programs appear to have robust systems in
place for delivering technical assistance and information to communities,
including clear mandates and guidance, formal programs and mechanisms for
delivering technical assistance, and adequate levels of EPA staffing and
resources.
RCRA CA appears to be meeting its mandate of providing information
dissemination to community groups as required. However, in comparison to the
Superfund and Brownfields programs, RCRA CA lacks many inputs helpful in
ensuring the delivery of technical assistance to communities, including a lack of:
regulatory mandates; adequate resources and staffing at EPA and state
agencies; and up-to-date, program-specific guidance. Unmet community needs
appear to be higher within the RCRA CA program than other programs, and
satisfaction with technical assistance provided by states to communities under
RCRA CA was rated lower on average by interviewees than for the other two
programs.
Compared to the other programs, it is more difficult to characterize the needs of
RCRA CA communities nationally, and to track progress in meeting those needs,
because the program is largely delegated to states, and EPA currently lacks
mechanisms for collecting community engagement data from states. Thus, if
EPA were to conduct regular tracking of measures of unmet needs and
customer satisfaction, the Agency would need to work with states to implement
a data collection system. In contrast, existing data collection systems employed
by Superfund and Brownfields could potentially be augmented to track
suggested measures.
Findings Summary by Evaluation Question
1. What are the requirements and drivers for community involvement within the
Superfund, RCRA Corrective Action, and Brownfields programs?
Community involvement requirements are documented in the National Contingency
Plan for Superfund and Brownfields.6 All Superfund sites must have a Community
Involvement Plan (CIP), and CIPs must be in place before remediation commences. The
Brownfields Program is required to provide community involvement opportunities for
sites that receive cleanup grants from EPA, as opposed to assessment grants, where
community involvement is not required. RCRA CA's community involvement
requirements are codified in a public participation manual for the program dating back
to 1996. RCRA CA is only required to provide public notice during key phases of the
corrective action process; TA is not required. Beyond requirements, additional drivers of
6 NCR [40 CFR 300.430(2)(ii)]: states the following intent regarding community involvement: "(A) Ensure the public
appropriate opportunities for involvement in a wide variety of site-related decisions, including site analysis and
characterization, alternatives analysis, and selection of remedy; and (B) Determine, based on community interviews,
appropriate activities to ensure such public involvement."
ES-4
-------
community involvement across the three programs include community demand and
environmental justice concerns.
2. What is the baseline of current OSWER technical assistance and information
dissemination activities, with respect to:
a. Frequency of practice and program-to program variability? Superfund has the
most extensive formal TA and ID practices of the three OSWER programs. CIPs
are developed for each site and EPA has dedicated personnel in each region to
assist Superfund communities. Superfund communities have access to both TAG
and TASC. The Brownfields program requires its cleanup grantees, which are
typically state and local governments, to provide TA, and the Brownfields
program also administers the TAB program. RCRA CA complies with ID
requirements, butTA is not required, and as such, TA is provided on a case-by-
case basis based on perceived need and community interest. Because resource
limitations precluded interviews with more than six states, it is important to
understand that this evaluation cannot provide comprehensive information on
RCRA CA community involvement at the state level.
b. Proportion of communities that receive formal assistance through TAG, TASC, or
other formal program ?
Use of formal TA is summarized in Exhibit ES-1 below.
ES-1. Use of Formal Technical Assistance Programs
PROGRAM
Superfund
RCRA CA
Brownfields
FORMAL USE OF TA
Site count = 13,662 (1652 NPL site + 12,010 non-NPL sites)
350 TAGs awarded to and 56 TASC projects used at Superfund
sites
Site Count = 3747
Formal TA is rarely used at RCRA CA sites
Only seven RCRA CA sites have used TASC assistance
No official program site count because all communities eligible
833 Brownfields communities have received TAB assistance
since 2008
c. Selection criteria and on-the-ground process for accessing TAG and TASC (for
Superfund and RCRA CA only)?
As a grant program, the TAG program has formal selection criteria covering: the
types of groups that are eligible to receive TAG funds; group administrative and
management experience; group past performance with federal grants; and
group legal incorporation requirements. In cases where multiple parties from
the same community compete for TAG funds, EPA uses secondary criteria to
make a selection; these include community representativeness, communication
planning, and the potential for measurable environmental results.
ES-5
-------
Community groups do not apply for TASC directly; they receive access to the
TASC contract services based on recommendations from regional staff to EPA
HQ. HQ and regional interviewees within the Superfund and RCRA CA programs
stated that there are no formal selection criteria for OSWER programs to access
TASC. However, communities that are selected for TASC funds typically share
one or more of the following conditions:
A lack of other sources of funding for community TA needs;
A high level of community interest in the cleanup process;
Environmental justice concerns;
Short-term technical assistance needs; and/or
Superfund communities that are not interested in forming a TAG group.
d. Community assistance that is provided outside of formal programs (including
helping to set up CAGs and providing ad-hoc assistance)?
The Superfund program routinely provides TA outside of formal programs, in
addition to providing extensive support within formal programs. Within the
Brownfields program, communities with EPA cleanup grants receive assistance
through the grantee (typically a local government entity). Nearly all TA provided
to RCRA CA communities is conducted outside of formal programs. Regions and
states administering RCRA CA make decisions about where to offer TA based on
community requests and staff assessment of need.
e. Areas of unmet TA or ID need?
Most interviewees identified some unmet TA and/or ID needs. However, as
shown in Exhibit ES-2 below, interviewees noted a somewhat higher incidence
of unmet needs among RCRA CA communities than among Superfund or
Brownfields communities.
Exhibit ES-2. Unmet ID and TA needs
RCRA CA
Superfund
Brownfields
NOTED ANY UNMET TA NEEDS
HQ, 8 Regions
HQ, 6 Regions
HQ, 4 Regions
NOTED ANY UNMET ID NEEDS
7 Regions
7 Regions
5 Regions
Moreover, RCRA CA HQ and eight of 10 regional contacts indicated that resource
limitations are a barrier to providing TA under RCRA CA; in contrast, only four
Brownfields contacts and two Superfund contacted indicated that lack of resources is an
impediment to delivering TA.
3. How can Superfund community involvement plans (CIPs) be used to improve
technical assistance and delivery of info?
ES-6
-------
a. Does every site have a CIP? According to all Superfund interviewees, every
Superfund site has a CIP.
b. What information is available on the implementation ofCIPs? According to all
Superfund regional interviewees, CIPS are tracked and maintained by regional
staff. CIP updates are variable. From the perspective of tenured EPA staff, CIPs
are most valuable at the beginning of the Superfund process. From the
perspective of community members and new EPA staff, CIPs and updates have
value throughout the cleanup process. Interviewees noted that CIPs may be
improved by more frequent, targeted revisions and by making them more
readable by community members.
c. Are CIPs revised over time? According to Superfund interviewees, CIPs are
revised overtime but revisions sometimes lag project milestones.
d. Do CIPs, as they are currently used, ensure effective technical assistance and
information delivery throughout the life of the project? Why or why not? Six of
10 regional interviewees said that CIPs are not an important tool for ongoing
technical assistance and information delivery. The general sentiment among
Superfund regional interviewees is that CIPs are helpful to organize community
engagement at the beginning of the Superfund process, but are not used by EPA
staff as a resource over time. EPA tends to set aside the document after the
start of the project, instead relying on ad-hoc communication with community
members.
Because CIP updates often lag behind project changes, they may not ensure
effective technical assistance and information delivery throughout the life of the
project given lagging updates. However, as discussed under Evaluation Question
4 findings, satisfaction with technical assistance and information dissemination
is high among Superfund community contacts participating in this evaluation
that have received a TAG, and across the Superfund community contacts that
have used the TASC contract.
e. What is the baseline of customer satisfaction with OSWER technical assistance
and information dissemination activities? Community contacts rated their
overall satisfaction with assistance services provided a scale of one to six; with a
score of one being very dissatisfied to a score of 6 being very satisfied. As shown
in Exhibit ES-3 below, interviewees noted consistently high satisfaction overall
scores for Superfund and Brownfields programs, while scores for RCRA CA were
more mixed. However, regarding RCRA CA, it should be noted that at the lower
end, only two interviews rated their overall satisfaction as a "1" with assistance
provided by RCRA CA states; the majority of RCRA CA sites provided favorable
overall ratings for assistance received. Satisfaction rates are very similar
between recipients of TAG grants and users of the TASC contract. Common
suggestions from community contacts for improving TA included making
outreach materials more accessible by using visual aids and plainer language;
and reaching out to community members often or more actively.
ES-7
-------
Exhibit ES-3. Customer Satisfaction Score Summary
80
70
sj 60
01
2 50
a
£ 40
| 30
8 20
01
°- 10
0
Overall Score Distribution by Program
RCRA
(9 Interviews)
I Superfund (9 Interviews)
I Brownfields (8 Interviews)
Very
Unsatisfied
3 4
Likert Scale Score
Very
Satisfied
4. What measures can be used to assess the effectiveness and tangible outcomes of
OSWER technical assistance and information dissemination activities across the
lifecycle of site planning, remediation, and reuse?
5. How can these measures be used to improve OSWER technical assistance and
information dissemination activities?
lEc developed a potential menu of output, short-term outcome, and long-term
outcome measures for tracking the progress of community engagement in OSWER
programs. We focused output measures on tracking unmet needs, and outcome
measures on gauging community satisfaction. lEc also considered other types of
measures, but honed in on the measures presented in Exhibit ES-4 after considering
feasibility factors including:
The ability to define objective measure(s);
The need to develop and implement new information collection
infrastructure;
Feedback from interviewees on potential measures; and
lEc's professional judgment regarding the likelihood of the measure being
accepted and successfully implemented by the OSWER programs.
It is important to note that lEc does not suggest that EPA consider or adopt all of the
measures in Exhibit ES-4.
ES-8
-------
Exhibit ES-4. lEc Recommendations for a Menu of Potential Measures
OUTPUT MEASURES
Proportion of Superfund communities that applied for TAG assistance, but have not
received assistance
Proportion of Brownfields communities that requested TAB or other form of K6
assistance, but have not received assistance
Proportion of Superfund and RCRA CA communities that were recommended by a
region for TASC assistance, but have not received assistance
Proportion of RCRA CA communities that have asked a state or region for any form of
assistance, but have not receive assistance
Number/Percent of RCRA CA communities that have received TA
Number/Percent EPA Regions that have offered TA/ID training in the past year
SHORT-TERM OUTCOME MEASURES
The number/proportion of community members filing complaints related to the
provision of TA with Superfund/Brownfields/RCRA CA regional staff annually.
LONG-TERM OUTCOME MEASURES
Proportion of TAG/TASC/TAB communities that have received a community
satisfaction survey that are "satisfied" with the information provided by EPA
throughout the cleanup process (tracked individually for each program).
Proportion of TAGs/TASC/TAB/RCRA CA communities that have received a community
satisfaction survey that are "satisfied" with the assistance provided by their technical
advisor (each of the four measures tracked individually).
ES-9
-------
CHAPTER 1 | INTRODUCTION AND EVALUATION SCOPING
In December 2009, EPA's Office of Solid Waste and Emergency Response (OSWER)
announced the Community Engagement Initiative (CEI) to enhance OSWER HQ and
Regional offices' engagement with local communities and other stakeholders, and to
help stakeholders meaningfully participate in decision-making processes related to the
cleanup and reuse of contaminated sites. CEI includes a commitment to evaluate the
effectiveness of OSWER program community engagement activities. OSWER envisions
highly effective community engagement becoming a standard business practice across
all of its programs, and furthermore wants to share lessons on successful community
engagement throughout the Agency. As such, OSWER requested a formative evaluation
for the CEI program. A formative evaluation is a prospective evaluation designed to help
newer programs establish baseline conditions, and establish measures to track program
implementation and progress over time.
The CEI is comprised of 16 actions that affect many aspects of OSWER's work. Exhibit 1-
1 summarizes these actions.7 This evaluation focuses on Action 7 and Action 13A
(highlighted in Exhibit 1-1), which address improving technical assistance (TA) and
information dissemination (ID), respectively. The CEI defines technical assistance (TA) as
the provision of services, resources, and training focused on increasing community
understanding of the relevant science, regulations and policy related to environmental
issues.8 Information dissemination (ID) refers to communication from EPA or state
agencies to communities regarding environmental issues; CEI focuses on delivering the
right information, in the right place, at the right time into existing guidance, training,
and Agency outreach efforts.9 Unlike some other CEI Actions that are program-specific,
Actions 7 and 13A are cross-program actions that can help communities participate in
OSWER processes more effectively. OSWER's goals for the evaluation were to establish a
baseline of current satisfaction with EPA's TA and ID practices, and to consider the
feasibility of potential measures for tracking TA and ID moving forward.
EPA contracted with Industrial Economics (lEc) to conduct the evaluation under the
oversight of the Agency's Evaluation Support Division (ESD). ESD, lEc, and
representatives from OSWER comprised the evaluation team. lEc prepared this
evaluation report, and the terms "we" and "our" in this report refer to the authors at
lEc.
EVALUATION SCOPING
lEc conducted a scoping task to inform this project. First, we reviewed publicly-available
information for each OSWER cleanup program (Superfund, Brownfields, RCRA Corrective
7 See OWSER's website about the Community Engagement Initiative at http://www.epa.gov/oswer/engagementinitiative/,
last visited July 30, 2013.
8 EPA CEI, Action 7- Evaluate and Improve EPA Technical Assistance Processes, Work Group Report on Recommendations,
available at: http://www.epa.gOV/oswer/engagementinitiative/7 action recommendation report.pdf
9 EPA CEI, Action 13A- Delivery of Information, Summary of Draft Work Group Report, available at:
http://www.epa.gov/oswer/engagementinitiative/13a action summary report.pdf
-------
Action, and the Underground Storage Tank Program) to identify the formal and informal
processes through which the programs conduct TA and ID. This consisted of reviewing
program websites, guidance, regulations, and grant RFPs for program policies on TA or
ID. Then we conducted interviews with contacts at EPA HQfor each program to sharpen
our understanding of the framework through which these programs conduct TA and ID,
as well as to determine additional TA and ID policies that we did not see in our review of
publicly-available information. We also used these initial calls to determine the most
effective research methods to answer our evaluation questions.
Exhibit 1-1. Community Engagement Initiative (CEI) Actions
Action 1
Action 2
Action 3
Action 4
Action 5
Action 6
Action 7
Action 8
Action 9
Action 10
Action 1 1
Action 12
Action 13
Action 14
Action 15
Action 16
Decision-Making Processes and Guidance
Underground Storage Tank (LIST) Programs
Best Community Engagement Practices for RCRA
OSWER Regulation and Guidance Development
Community Engagement Policies and Activities related to Enforcement
CERCLA Enforcement Involving Federal Facilities
EPA Technical Assistance Processes
Community Action for a Renewed Environment (CARE) Program
Brownfields Area-Wide Planning Pilot Program
Public Health Information on OSWER Projects
Risk Communication Processes and Comprehensive Education Program
Sampling and Testing Results
Delivery of Information:
13A: At-Riskand Remote Communities
13B: Superfund Repositories
Community Engagement Training Program
Measures of Effectiveness and Annual Report
Environmental Workforce Development and Job Training Program
Another aspect of our scoping work was to determine how concurrent efforts within
EPA relate to aspects of the evaluation. With many CEI initiatives in progress, the
evaluation team was concerned about overlap between this evaluation and other
efforts, particularly within RCRA Corrective Action (RCRA CA). Thus, we spoke to
programs about the status of ongoing CEI initiatives and scoped this evaluation
accordingly. The results of our scoping task are as follows:
Exclusion of the Underground Storage Tank (UST) Program
lEc recommended excluding the UST program from the evaluation based on scoping
research. State UST programs do not conduct much TA; most UST sites are not
complicated or technically challenging enough to require it. In rare instances of a highly
contaminated tank site with community exposure, states assess the need for public
participation on a case-by-case basis. UST is aware of only one state that has funding for
conducting TA. Similarly, state UST programs typically do not conduct ID activities,
except in rare cases of community exposures.
-------
In cases of public exposures, states must notify affected communities, as reflected in
EPA UST regulations. However, even in this case, the regulations do not require much
interaction; they are summarized below (emphases are lEc's):
(a) The public should be notified of every confirmed release that requires a
corrective action plan.
(b) "The implementing agency must ensure that site release information and
decisions concerning the corrective action plan are made available to the public
for inspection upon request."
(d) "Before approving a corrective action plan, the implementing agency may
hold a public meeting to consider comments on the proposed corrective action
plan if there is sufficient public interest, or for any other reason."10
It should be noted that EPA does not have funding for conducting TA at UST sites, nor
does the Agency provide guidance on the topic. One exception is Indian country, where
in some cases, EPA Regional staff communicate information regarding the UST program
to the tribal governments, who are then responsible for informing their communities
and the affected public.
Thus, the evaluation team agreed not to include UST in this evaluation; EPA HQ UST
contacts concurred with this decision. Factors in this decision included:
General consensus that UST sites are low-risk and low priority;
Lack of TA and ID activity at UST sites (except in cases of release);
EPA's weak regulatory and policy framework regarding TA and ID at UST sites
(TA is not required; and only limited ID is required); and
Limited evaluation resources.
However, it should be noted that because the UST program was omitted from the
evaluation, no judgment has been made about the about the effectiveness of the
program's community engagement activities.
Superfund, Brownfields, and RCRA Corrective Action Programs Have
Different Structures that Affect Data Availability and Collection
Superfund works directly with communities affected by Superfund sites, has ultimate
authority over the site remediation process, and retains control over community
engagement as part of that process. Superfund offers a grant program directly to
affected communities, the Technical Assistance Grant (TAG), and Superfund site
managers can also take advantage of the Technical Assistance Services for Communities
(TASC) Contract administered by OSWER (use of TAG and TASC is discussed in detail in
Chapter 3). Superfund sites are required to have a community involvement plan for each
site. As such, EPA Regional staff have direct contact with community organizations
involved in Superfund cleanup activities. This structure differs significantly from the
other OSWER programs.
10 40 CFR Ch. 1 (7-1 -05 Edition) S 280.67 Public Participation provisions from Subpart FRelease Response and Corrective
Action for LIST Systems Containing Petroleum or Hazardous Substances (http://www.epa.gov/swerust1/fedlaws/280_f.pdf)
-------
The Brownfields program provides grants to local governments and non-profit
organizations to cleanup and reuse sites, and community engagement is required as
part of cleanup grants. So, recipients of EPA brownfields grants provide technical
assistance to community groups; EPA does not provide or administer this assistance. In
addition, the EPA Brownfields program administers the Brownfields Training, Research,
and Technical Assistance K6 Grant program, which among other things, provides funding
to academic and research organizations to work with community organizations on
public participation issues. The Technical Assistance for Brownfields (TAB) program is
the main component of the K6 grant program focused intently on TA and ID. Under TAB,
EPA provides grant support to four regional organizations, which in turn provide
technical assistance to Brownfields communities.
The RCRA Corrective Action (CA) program is delegated to 43 states. In delegated states,
at most RCRA CA sites, states take the lead on cleanup, including community
engagement (although EPA regions sometimes provide TA at sites in delegated states).
The RCRA CA program has official guidance on public participation, but it has not been
updated since 1996.n EPA does not track state-led RCRA CA community engagement
activities, and does not have direct contacts with community organizations involved in
site cleanup activities, except for the minority of sites where EPA is implementing the
cleanup, or in the few cases where a RCRA CA community has been provided with
assistance through the TASC contract. Thus, it was important for lEc to interview state
RCRA CA coordinators for this evaluation.
Survey Work was Impractical
Both the Superfund and Brownfields contacts expressed reservations about surveying
community groups affiliated with their program, but were more amenable to interviews.
In addition, Superfund has already done a significant amount of work on satisfaction for
TASC, using a well-constructed interview guide that asks participants to rate specific
aspects of TA and ID. Upon reviewing the satisfaction questionnaire used by Superfund
for TASC, and talking with Superfund's TASC contractor, Skeo, lEc concluded that asking
specific and customized questions (e.g., about specific workshops, materials, and
communications) is the best strategy for ensuring that participants provide feedback on
the TA and ID process, as opposed to cleanup or redevelopment outcomes, which EPA
typically cannot control. However, one cannot ask customized questions via a survey.
Thus, lEc recommended conducting interviews for Superfund TAG, Brownfields, and
RCRA CA using the same tool as Skeo, which provided us with consistent information to
compare across programs.
EVALUATION QUESTIONS
The evaluation questions that we used to guide this evaluation are as follows:
1. What are the requirements and drivers for community involvement within the
Superfund, RCRA Corrective Action, and Brownfields programs?
11 RCRA Public Participation Manual, 1996 Edition, available at:
http://www.epa.gov/osw/hazard/tsd/permit/pubpart/manual.htni
-------
2. What is the baseline of current OSWER technical assistance and information
dissemination activities, with respect to:
a. Frequency of practice and program-to program variability?
b. Proportion of communities that receive formal assistance through TAG, TASC, or
other formal program?
c. Selection criteria and on-the-ground process for accessing TAG and TASC (for
Superfund and RCRA CA only)?
d. Community assistance that is provided outside of formal programs (including
helping to set up CAGs and providing ad-hoc assistance)?
e. Areas of unmet TA or ID need?
3. How can Superfund community involvement plans (CIPs) be used to improve
technical assistance and delivery of info?
a. Does every site have a CIP?
b. What information is available on the implementation of CIPs?
c. Are CIPs revised over time?
d. Do CIPs, as they are currently used, ensure effective technical assistance and
information delivery throughout the life of the project? Why or why not?
4. What is the baseline of customer satisfaction with OSWER technical assistance and
information dissemination activities?
5. What measures can be used to assess the effectiveness and tangible outcomes of
OSWER technical assistance and information dissemination activities across the
lifecycle of site planning, remediation, and reuse?
6. How can these measures be used to improve OSWER technical assistance and
information dissemination activities?
LOGIC MODELS
To illustrate the various components of community engagement in OSWER, and
specifically the ID and TA tasks within the CEI, EPA and lEc developed three logic models
(i.e., graphical representations of the relationships between program inputs, outputs,
and intended changes in knowledge/attitude, behavior, and condition). The first logic
model (Exhibit 1), illustrates the design of the community engagement process within
OSWER. It does not represent how community engagement necessarily works at all
sites; instead, the model represents the process through which community engagement
is designed to function. The next two logic models (Exhibits 2 and 3), outline the design
of CEI Actions 7 (TA) and 13A (ID), which are the foci of this evaluation. These two logic
models illustrate the conceptual design of how these Actions will be implemented to
achieve their expected results. All three models are needed to understand the
community involvement process and how Actions 7 and 13A are intended to improve
the process.
-------
The key components of the models include:
Resources staff, contractor support, partners, and funds dedicated to the
program.
Activities the specific procedures or processes used to achieve program
goals. OSWER's community engagement activities consist of developing site-
specific Community Engagement Plans and continually updating the plans as
community needs change. For the CEI Actions 7 and 13A, activities focus on
examining current ID and TA practices for opportunities for expansion and
revision.
Outputs the immediate products that result from activities, which are often
used to measure short-term progress. OSWER's community engagement
outputs consist of holding public meetings, generating documents for public
consumption, and providing technical assistance to communities. Example
outputs of the CEI Actions are community-centric training modules, brochures,
and expansion of existing formal technical assistance programs.
Target Audiences groups and individuals targeted by community
engagement activities and outputs. For example, OSWER's community
engagement audiences include community members, community-based
organizations, and both local and tribal governments. In addition to these
groups, some activities and outputs of CEI Actions 7 and 13A target EPA staff
and delegated state programs.
Short-Term Outcomes changes in knowledge, awareness, attitudes,
understanding, and skills resulting from program outputs that are causally
linked to community engagement. For example, CEI Action 7 is designed to
increase stakeholders' awareness of EPA's technical assistance programs.
Intermediate Outcomes changes in behavior resulting from changes in
knowledge and attitude. For example, CEI Action 7 is designed to increase
stakeholders' use of EPA's technical assistance programs.
Long-Term Outcomes the overarching goals of the program, which in the
case of the CEI Actions 7 and 13A is to ensure the community has a more
informed voice in decisions at contaminated sites within their community.
Finally, the logic model makes note of external factors that are beyond the direct
control of EPA's community engagement program, but may influence program
outcomes. For example, external factors for Action 13A may include available resources
and lack of community organization.
-------
Exhibit 1-2. LOGIC MODEL FOR COMMUNITY ENGAGEMENT IN OSWER
Goal: To facilitate community members' participation in government decisions on land cleanup, land reuse, and emergency response.
Resources
OSWER staff
State staff and
delegated programs
Superfund guidance
on how to create a
Community
Involvement Plan
EPA Information
Access Strategy
"Partnering with
Communities: Federal
Models of Partnering
with Communities"
Funding for grants
and contractors
EPA Community
Involvement Training
Conference
Community
Involvement Toolbox
Brownfields
Conference
Community
Involvement
Activities
Outputs
Target Audienc
Develop and
implement project-
specific Community
Engagement Plans
* based on current
EPA community-
engagement best
practices and
existing case
Regular public
^ meetings to discuss _
site issues and
obtain feedback
Informational
documents and/or
^ a website to
present
environmental
issues at site
Out-stationed EPA
or delegated state
employee to serve
as the point of
contact with
community
members
Continuously react
to site
developments and
update plan
Access to formal
technical
assistance
programs that
" provide financial
resources and
independent
technical advice
(e.g. TAGS, TASCs,
^ Community
I Members
Community-
based
Local Government
Tribal
Governments
Short-Term Intermediate"! Long-Term
Outcomes Outcomes Outcomes
Increased
community
understanding of
^ environmental
issues at the site
and project
actions to address
these issues
Increased
community
_^. understanding of
how its
involvement can
shape the project
Increased
opportunityto
provide OSWER
^^ and/or delegated
Community
members
engage with
OSWER and/or
^ delegated state
to their level of
interest
throughout the
lifetime of the
OSWER and/or
delegated state
^ incorporates the
community's
vision into
OSWER project
outcomes are
aligned with the
needs and goals
^ expressed by the
community
throughout the
engagement
process, to the
\extent
Projects
experience fewer
delays due to
^ differences in
qtakphnlrlpr
project " """""'. t
expectations for
site cleanup and
rai ICQ
state with
feedback about
community
External Factors: Budget constraints, limited existing community-engagement best practices, limited flexibility to incorporate community desires
-------
Exhibit 1-3. LOGIC MODEL FOR CEI ACTION 7: EVALUATE AND IMPROVE EPA TECHNICAL ASSISTANCE PROCESSES
Goal: To improve and broaden the availability of technical assistance to communities so that community members can better understand site issues, and
participate in an informed way during the decision-making process.
Resources
OSWER staff
EPA's Conflict
Prevention and
Resolution Center
Existing EPA technical
assistance programs
(e.g. TAGS, TASCs,
TAPS, TABs)
Existing EPA best
practices for technical
assistance
Activities Outputs Target Audience Short-Term
1 1 1 1 Outcomes
"^7 ' "V" "V
Promote OSWER's
technical assistance Brochure and
capabilities to website to educate .,
fr. increase ^. communities about i
communities' access technical assistance
to and use of opportunities
existing technical
assistance
^ Technical assistance
,
^ technical assistance ~~
resou rces
~ "Partners Program"
Expanded availability
^ ofTechnical
Assistance Plans and
TASCs
Tool kit for local
communities that
may be receiving
.. Community
Members
^ Community-based
Organizations
^ Local Government
Tribal
muedbeu
community
stakeholder
"^ awareness of
available technical
assistance
programs
Broader availability
of technical
x^ assistance
programs to
Governments community
stakeholder groups
Mid-Level HQ and ' Teaerany-
regional program transferred property
managers discuss property or
^ opportunities to
identify and resolve
technical assistance
issues and problems
Assess existing
technical assistance
^ practices to identify
Formerly Used
Defense Sites ^ ^^ Increased
^ Guidance regarding ~~
EPA expectations
^f crrt oidii ^^
Delegated State
^ staff implementing
awareness among
EPA and delegated
state staff of
^ OSWER technical
assistance
and best practices OSWER programs programs and
on technical associated best
assistance practices
gaps and best
practices
Intermediate
Outcomes
Long-Term Outcomes
~^F~
Increased use of
available OSWER
technical
"^ assistance
programs among
communitv
stakeholders
Increased
implementation
of technical
Community
stakeholder Community
groups have a stakeholder
deeper groups have a
understanding of more informed
^ the relevant _^. voice in
science, decisions
regulation, and related to
policy related to environmental
environmental issues at
issues at OSWER OSWER sites
sites
-^ assistance best
practices across
OSWER
programs
External Factors: Limited resources, some communities may not be a good candidate for technical assistance because of: (1) a lack of organizational structure, (2) a
lack of agreement among community stakeholders, and/or (3) assistance saturation within community.
-------
Exhibit 1-4. LOGIC MODEL FOR CEI ACTION 13A: EVALUATE AND IMPROVE DELIVERY OF INFORMATION - AT-RISK AND REMOTE
COMMUNITIES
Goal: To develop options for improvement in how information is delivered to a-risk and remote communities and to enhance community members' knowledge and
ability to meaningfully participate in decision-making processes.
Resources
OSWER staff
Current EPA
relationships with
leaders of at-risk and
remote communities
and tribes
Existing EPA
information
dissemination key
tools
Existing EPA
information
dissemination best
practices
Activities
Coordinate with OEI
on information
delivery mechanisms
for At-Risk and
Remote
Commu nities
Coordinate with
tribes that are close
to OSWER sites to
determine if there
are ways to improve
information delivery
to them
Hold video
teleconferences with
at-risk and remote
communities
Establish a national
workgroup to
identify existing
information delivery
mechanisms to
identify gaps and
best practices
Outputs
V
Improved
information
delivery methods
using best
practices, key tools,
^ and innovative
ideas to increase
transparency and
stakeholder
certainty in the site
cleanup process
A
Community-centric
training module on
^ the delivery of
information to
remote and at-risk
communities
Target Audience
>r
At-Risk and
Remote
Community
Members
At-Risk and
Remote
Community-Based
Organizations
At-Risk and
Remote Local
Governments
At-Risk and
Remote Tribal
Governments
EPA Staff
Delegated State
^ staff implementing
OSWER programs
Short-Term
Outcomes
At-risk and remote
communities
_- receive the right
^ information, in the
right place, at the
right time
Increased
awareness amongst
EPA and delegated
<^ state staff of
"^ OSWER information
delivery
mechanisms and
best practices
I Intermediate
Outcomes
At-risk and
remote
community
members are
more informed
-^ about H
environmental
issues and
actions at
contaminated
sites.
More
stakeholder
confidence
^ among at-risk
and remote
communities in
the site cleanup
process
j
^^^
Increased
implementation
^ of information
1 delivery best
practices across
OSWER programs
Long-Term
Outcomes
1 3 C
"
At-risk and remote
community
groups have a
^ more informed
voice in decisions
related to
environmental
issues at OSWER
sites
External Factors: Available resources (especially for travel, in-person meetings at affected communities, and translation), lack of community organization.
-------
CHAPTER 2 | METHODOLOGY
The methodology for this evaluation combines a thematic analysis of qualitative data
gathered from interviews with a review and synthesis of existing documentation. The
complete methodology is included in Appendix A; it was completed in July of 2012.
Important methodological updates that occurred after finalizing the evaluation
methodology are discussed in this Chapter.
REVIEW OF EXISTING DATA
lEc reviewed existing documentation and data to inform our responses to evaluation
questions, as discussed in the methodology. We reviewed the following data sources:
Documentation of community engagement requirements;
Site inventory data for cleanup programs;
Performance measurement resources (for Questions 5 and 6); and
Existing community satisfaction interviews (as discussed above).
Use of these documents and data is discussed in more detail in the context of Findings in
Chapter 3.
INTERVIEWS
lEc conducted 46 interviews with EPA HQ and regional contacts from Superfund,
Brownfields, RCRA CA, as well as state RCRA CA contacts in Arkansas, California, Illinois,
Missouri, New York, and South Carolina. Exhibit 2-1 below summarizes these interviews.
EPA HQ contacts helped us to identify EPA regional contacts; and regional RCRA CA
contacts helped to identify state contacts.
Exhibit 2-1. Interviews with EPA and State Contacts
INTERVIEWS
EPAHQ
EPA Regions
RCRA CA State
Other
TOTAL
COUNT
7
30
6
3
46
DESCRIPTION
Interviewed HQ staff: Superfund (2), Brownfields (2),
and RCRA CA (3)
Interviewed Superfund, Brownfields, and RCRA CA HQ
staff in all 10 regions (1 interview for each region)
Interviewed staff from 6 delegated states
Interviewed CPRC contact, TASC contractor,
Brownfields K6 grantee
lEc also conducted satisfaction interviews with contacts from community groups that
had received technical assistance. As discussed in the methodology, we conducted these
interviews using the format developed and implemented by another contractor for use
10
-------
in measuring customer satisfaction under the TASC contract. A summary of satisfaction
interviews is included in Exhibit 2-2 below.
Exhibit 2-2. Satisfaction Interview Summary
INTERVIEWS
Superfund TASC
Superfund TAG
RCRA CA
Brownfields TAB
andK6
TOTAL
COUNT
22
9
7
8
46
DESCRIPTION
(Another contractor previously conducted interviews
in 22 communities)
Interviewed TAG recipients in 8 communities
Interviewed recipients of state technical assistance in
7 communities. No contacts from recipients of EPA-led
RCRA CA sites were available for interv/ew.
Interviewed 7 recipients of technical assistance under
TAB in 7 communities and 1 recipient of K6 assistance
in 1 community
lEc identified satisfaction interviewees in various ways. The Superfund program
maintains a list of TAG recipients; we randomly selected one primary and one backup
TAG recipient to contact in each region. We subsequently conducted nine TAG
interviews (a TAG interviewee could not be reached in one region). The Brownfields
program does not maintain a list of communities that have received assistance under
TAB, and the intensiveness of TAB assistance varies by community group. Hence, we
asked all four TAB grantees and the one K6 grantee that does community engagement
work to identify community contacts for satisfaction interviews. Specifically, we asked
the TAB grantees to identify contacts from community groups that received a
substantial degree of assistance. Because the TAB grantees are regional, this approach
provided geographical diversity. For RCRA CA, we asked EPA regional and state contacts
interviewed to help identify community contacts, as there is no single list of RCRA CA
communities that have received technical assistance at either the EPA or state level.
We analyzed results from lEc's satisfaction interviews in conjunction with existing
Superfund TASC satisfaction interviews. We were able to analyze 22 existing TASC
interviews in addition to the satisfaction interviews that lEc conducted.12 A notable
difference in lEc interviews and previously conducted satisfaction interviews is that
some of the previous interviews contained scores from multiple community contacts,
while all of lEc's scores are from a single contact for each site.
12 Of the 56 Superfund communities that have accessed TASC, only 22 had satisfaction interviews that could be used. The
main reason is that community satisfaction interviews are conducted towards the end of the cleanup process, and many of
the 56 sites have cleanup ongoing. Also, some of the satisfaction interviews previously conducted were joint interviews
with an EPA representative and a community contact, but the contractor provided only the EPA representative scoring for
some of these interviews. We excluded these communities from our analysis.
11
-------
Thematic Analysis of Interview Data
lEc conducted a thematic analysis of interviews. We developed interview guides (see
Methodology document in Appendix A of this report) to guide each interview. We
developed a thematic analysis workbook in MS Excel that contained one worksheet for
each evaluation question. We cross-walked each interview question with applicable
evaluation question(s) and recorded this coding within the spreadsheet for each
evaluation question. Then, we recorded interviewee responses to each interview
questions in the applicable spreadsheets. All responses to the same interview question
were recorded on the same row. Once we entered all interview data, we reviewed
responses to each interview question one by one, and identified key themes for each
interview questions. For many interview questions, we counted the number of times
interviewees provided the same response. We summarized an overall response to each
interview question in the spreadsheet, noting areas of consensus and conflict, and
noting if there were clear schisms between different types of interviewees. We then
summarized responses for the interview questions associated with the same evaluation
question, and brought in information gleaned from the document review as applicable,
to develop findings for our evaluation questions.
STRENGTHS AND WEAKNESSES OF METHODOLOGY
The methodology employed has a number of strengths, including coverage of all three
programs: Superfund, RCRA CA, and Brownfields. Notably, the methodology employed
for this evaluation was sufficient for addressing most of the research questions,
including all of Questions 1, 3, 5, and 6; and most of Question 2. Within Question 2 in
particular, because lEc was able interview one contact for each of the three programs, in
every region, we are able to accurately characterize the baseline of EPA's technical
assistance and information dissemination activities. However, because we only
interviewed six states that implement RCRA CA, and only had information on a few
additional states from existing documentation, we cannot be sure if our findings on
frequency of practice, and community assistance provided outside of formal programs,
are representative of state RCRA CA programs overall.
For Question 4-establishing a baseline of satisfaction with technical assistance and
information dissemination among communities-a key strength of the evaluation was
the ability to combine analysis of new satisfaction interviews with existing interviews.
We also achieved geographic diversity in our interviews. However, the methodology for
Question 4 was limited by resources available to conduct interviews, and difficulties in
identifying community contacts to interview, particularly within the RCRA CA program.
Subsequently, a key limitation of our approach is that results for Question 4 on
satisfaction with technical assistance and information dissemination cannot be
extrapolated to all communities served by OSWER programs. Ideally, lEc would have
conducted surveys or interviews using a statistically valid sample of those served for
each OSWER program, to develop a true baseline of satisfaction with technical
assistance and information dissemination criteria.13 However, this approach would have
13 In addition to surveying a statistically valid sample of those served, it would also be methodologically preferable to survey
multiple stakeholders associated with each site. One individual interviewee may not represent the perspective of all local
stakeholders.
12
-------
required extensive resources to develop contact data (in particular for RCRA CA
communities), and to develop and administer the survey or requisite number of
interviews. In addition, programs expressed reservations about broadly surveying
community groups affiliated with their programs, citing information burden.
Finally, the method by which lEc selected interviewees has inherent strengths and
weaknesses. For the TASC contract, lEc was able to analyze all applicable data collected
for Superfund to-date, which is a methodological strength. In addition, we were able to
randomly select Superfund TAG grantees to interview. In contrast, as discussed above,
lEc had to rely on TAB grantees to identify Brownfields community groups to interview,
and we had to rely on RCRA CA regional and state interviewees to identify RCRA CA
community groups to interview. Relying on these parties to identify interviewees can be
a source of bias, although we have no evidence or suspicions that contacts cherry picked
community contacts to participate in this evaluation.
13
-------
CHAPTERS | FINDINGS
This section presents evaluation findings by evaluation question. High-level findings
from the evaluation include the following:
The EPA Superfund and Brownfields programs appear to have robust systems in
place for delivering technical assistance and information to communities,
including clear mandates and guidance, formal programs and mechanisms for
delivering technical assistance, and adequate levels of EPA staffing and
resources.
RCRA CA appears to be meeting its mandate of providing information
dissemination to community groups as required. However, in comparison to the
Superfund and Brownfields programs, RCRA CA lacks many inputs helpful in
ensuring the delivery of technical assistance to communities, including a lack of:
regulatory mandates; adequate resources and staffing at EPA and state
agencies; and up-to-date, program-specific guidance. Unmet community needs
appear to be higher within the RCRA CA program than other programs, and
satisfaction with technical assistance provided by states to communities under
RCRA CA was rated lower on average by interviewees than for the other two
programs.
Compared to the other programs, it is more difficult to characterize the needs of
RCRA CA communities nationally, and to track progress in meeting those needs,
because the program is largely delegated to states, and EPA currently lacks
mechanisms for collecting community engagement data from states. Thus, if
EPA were to conduct regular tracking of measures of unmet needs and
customer satisfaction, the Agency would need to work with states to
implement a data collection system. In contrast, existing data collection systems
employed by Superfund and Brownfields could potentially be augmented to
track suggested measures.
QUESTION 1: WHAT ARE THE REQUIREMENTS AND DRIVERS FOR
COMMUNITY INVOLVEMENT (Cl) WITHIN THE SUPERFUND, RCRA
CORRECTIVE ACTION (CA), AND BROWNFIELDS PROGRAMS?
Summary of Methods to Address Question 1
Review of documents to identify requirements
Interviews with staff at OSWER HQ and regions to explore knowledge of
requirements and collect information on additional Cl drivers
Findings on Requirements
Each Superfund site must develop a community involvement plan (CIP) and update it as
necessary. The National Contingency Plan (NCP) requires developing a CIP "based on
community interviews and other relevant information, specifying the community
14
-------
relations activities that the lead agency expects to undertake during the remedial
response." The NCP specifies that the CIP must be in place before remedial investigation
field activities start "to the extent practicable." 14 Requirements for TA and ID under
Superfund match the perception of requirements from Superfund staff; Superfund HQ
contacts and all 10 Superfund regional contacts interviewed indicated that CIPs are
required.
Requirements for Brownfields found in the National Contingency Plan dictate that, for
cleanup grants only, EPA must develop community relations plans and make cleanup
plans publicly available. In addition, according to Brownfields HQ interviewees, National
Environmental Policy Act (NEPA) requirements applicable to brownfields include holding
public meetings, answering comments, and making grant proposals publicly available.
Part of the scoring for Brownfields grant applications includes evaluation criteria for
community engagement. Brownfields HQ correctly indicated that technical assistance
and information dissemination are required for cleanup grants. In addition, eight out of
10 Brownfields regional interviewees indicated that this activity is required.
Under RCRA CA, a facility owner/operator is required to provide public notice during key
phases of the CA process, as documented in the 1996 RCRA Public Participation Manual.
Technical assistance is not required. However, seven of 10 RCRA CA regional contacts,
and four out of six RCRA CA state contacts, indicated that the program is required to
provide both technical assistance and information dissemination to affected
communities.
Findings on Additional Drivers
In addition to requirements discussed above, several interviewees noted additional
drivers of community involvement:
Six interviewees cited community demand as a driver.
Six interviewees cited environmental justice, including helping disadvantaged
and tribal areas.
Two interviewees cited EPA initiatives and priorities, such as the Targeted
Brownfields Assessment (TBA) program and region-specific priorities.
Two regional contacts cited benefits to EPA of providing technical assistance
and information dissemination as an additional driver. Specifically, one regional
contact indicated that long-term stewardship at cleanup sites requires pro-
active community and civic involvement. Another region said: "If the region
postpones adequate community engagement, it will receive complaint letters
and will establish a lack of trust" that can inhibit cleanup and redevelopment.
M I Community Involvement Plans, p. 7, available at: http://www.epa.gov/superfund/communitv/pdfs/toolkit/ciplans.pdf.
NCP [40 CFR 300.430(2)(ii)] states the following intent regarding community involvement: "(A) Ensure the public
appropriate opportunities for involvement in a wide variety of site-related decisions, including site analysis and
characterization, alternatives analysis, and selection of remedy; and (B) Determine, based on community interviews,
appropriate activities to ensure such public involvement."
15
-------
QUESTION 2: WHAT IS THE BASELINE OF CURRENT OSWER TECHNICAL
ASSISTANCE AND INFORMATION DISSEMINATION ACTIVITIES WITH
RESPECT TO:
a. Frequency of practice and program-to program variability?
b. Proportion of communities that receive formal assistance through TAG, TASC, or
TAB?
c. Selection criteria and on-the-ground process for accessing TAG and TASC (for
Superfund and RCRA CA only)?
d. Community assistance that is provided outside of formal programs (including
helping to set up CAGs and providing ad-hoc assistance)?
e. Areas of unmet TA or ID need?
Summary of Methods to Address Question 2
Interviews with HQ and regional OSWER staff in all three cleanup programs
Interviews with six delegated RCRA CA states
Interviews with TAB grantees
Review of data and documents provided by OSWER
Review of publicly available information
Findings on Frequency of Practice and Program-to-Program Variability
OSWER administers three formal programs to provide TA to communities affected by
contaminated sites: TAG, TASC, and TAB. As shown in Exhibit 3-1 below, Superfund
communities are eligible for TAG, and Brownfields communities can access TAB, and all
OSWER communities are eligible for TASC.15
Superfund
The Superfund program has the most extensive formal TA and ID practices of the
OSWER programs. Most notably, the Superfund awards TAG grants to community-based
organizations at eligible Superfund sites. Groups awarded TAGs up to $50,000 can
contract with independent technical advisors to interpret and explain technical
information to the community. 350 TAGs have been awarded since their inception in
1988. Superfund communities are also eligible for TASC assistance, with 56 Superfund
communities receiving TASC assistance since the program's inception in 2008.
15 Although all OSWER communities are eligible for TASC, lEc notes confusion within EPA about whether RCRA CA sites in
particular are eligible for TASC funding if there is no link to Superfund. RCRA CA staff members in EPA regions are confused
about this issue. For example, according to one interviewee, one region submitted several RCRA CA sites to EPA HQ for
TASC funding, but all were declined because they had no link to Superfund. The disconnect may be that the RCRA CA
program needs to contribute funds to TASC in order to use the contract, which may be a limitation precluding wider use by
RCRA CA sites.
16
-------
Exhibit 3-1. Eligibility and Basic Setup of OSWER TA Programs
PROGRAM
TAG (Since 1988)
TASC (Since 2008)
TAB (Since 2008)
ELIGIBILITY
Superfund
All OSWER
programs
Brownfields
BASIC SETUP
Community groups apply for grant from EPA.
Community groups hire their own advisor to provide
technical assistance.
OSWER maintains a contract with a firm that provides
Cl to community groups. Community groups access the
contract by contacting their region or the region
recommends communities to HQ directly.
OBLR maintains a long-term grant with four
organizations that provide Cl to community groups.
Community groups access the contract by contacting
the TAB grantee for their region.
Furthermore, the Superfund program employs 80 Community Involvement Coordinators
(CICs), with between two and 10 CICs in each region. CICs provide a liaison between
project managers and the community, and provide opportunities for two-way
communication throughout the life of a project. Every Superfund site is also required to
produce a CIP at the beginning of the Superfund process to identify how to best engage
the community throughout the cleanup. CIPs are discussed in more detail under
Question 3.
In contrast to Superfund, which administers technical assistance directly to community
groups, assistance to community groups is delegated under the RCRA CA and
Brownfields programs as discussed below.
Brownfields
Brownfields community groups can receive technical assistance from the TAB grantee
assigned to their region. EPA has awarded TAB grants to the New Jersey Institute of
Technology for Regions 1, 2, and 3; Enterprise Corporation of the Delta, Inc. for Regions
4 and 6; Kansas State University for Regions 5 and 7; and the Center for Creative Land
Recycling for Regions 8, 9, and 10. 833 Brownfields communities have received
assistance through TAB, with assistance lasting anywhere from a five minute phone call
to a multiple-year relationship. The TAB grantees assist communities with: community
outreach and mediation; grant writing; environmental planning; training; regulatory
facilitation; identifying funding sources; technical presentations; summarizing
stakeholder comments and concerns; technical reports; and site inventories.
However, most technical assistance in Brownfields is not provided through TAB, but by
the recipients of Brownfields grants, which are typically local governments. As discussed
under Question 1, Brownfields cleanup grantees are required to provide technical
assistance. Moreover, OBLR awards points for community engagement plans as part of
evaluating most types of grant applications, including Assessment Grants, Cleanup
Grants, Multi-Purpose Pilot Grants, and Revolving Loan Funds Grants. Once the grants
are awarded, EPA does not directly track community engagement activities by
Brownfields grantees, but community engagement is part of grant reporting.
17
-------
RCRA CA
RCRA CA has standard guidelines for engaging the community in its 1996 Guide to
Public Participation. However, requirements for TA or ID under the RCRA CA program
are limited. Under RCRA CA, EPA or the delegated state agency is required by statute to
issue a public notice in the newspaper and hold a period of public comment when key
project milestones are reached, such as the proposed remedy and final remedy
decision. All regional and state staff interviewed indicated that they comply with these
ID requirements.
RCRA CA sites are eligible for TASC assistance. Seven RCRA CA sites have received TASC
assistance since 2008. Most TA and ID activities within RCRA CA are conducted by RCRA
CA regional and/or state staff on a site-by-site basis based on perceived need. Every
region and state interviewed holds public meetings, but only at select sites with high
levels of community interest. It should be noted that the number of RCRA CA
community groups receiving TA is not tracked centrally for either EPA or state sites.
Some states have developed TA approaches for select RCRA CA sites. For example,
Mississippi's RCRA CA program has an EJ coordinator who provides training to staff to
identify EJ communities within the RCRA CA universe, conduct face-to-face community
meetings, and use an electronic notification system for RCRA CA communities within the
state. South Carolina's RCRA CA program staff report that they meet with RCRA CA
facility staff early in the process to encourage them to engage with the community;
South Carolina staff noted that facilities can be persuaded that community engagement
will help to build trust.
Findings on Proportion of Communities that Receive Formal
Assistance through TAG, TASC, or TAB
Superfund
Superfund communities are eligible for formal technical assistance through both TAG
and TASC. Of the total universe of 13,662 Superfund sites (1652 NPL sites and 12,010
non-NPL sites), 350 TAGs have been awarded. Thus, approximately 2.6 percent of
Superfund sites have had a TAG group. As shown in Exhibit 3-2 below, 56 Superfund
communities had received TASC assistance since 2008, or just 0.4 percent of all
Superfund sites. It should be noted that the TAG program has been available since 1988,
whereas the TASC contract became available much latter, in 2008.
RCRA CA
Formal TA is rarely used at RCRA CA sites. Seven RCRA CA communities have received
TASC assistance since 2008. The RCRA CA universe contains 3,747 sites, so just a tiny
fraction of RCRA CA communities have received TASC assistance. The breakdown of the
authority types of these 3,747 sites is shown below in Exhibit 3-3.
18
-------
Exhibit 3-2. TASC Assistance by Program
PROGRAM
TASC COMMUNITY COUNT
(CURRENT AS OF JANUARY 2013)
Superfund
56
NPL
37
Non-NPL
14
Federal Facility
OEJ
5
29
RCRA CA
CARE
Historically Black Colleges and Universities
OW
TOTAL
105
Exhibit 3-3. Authority Type of RCRA CA Sites
RCRA CA FACILITY AUTHORITY
Total facilities in need of corrective action
Under EPA authority only
Under state authority only
Under EPA authority and under state
authority at different times
Under state authority, and under joint
EPA/state authority, at different times
Under joint EPA/state authorities only
Under EPA authority, state authority, and
joint authorities at different times
Sites for which authority information is not
available in RCRA Info
Uncategorized
RCRA CA SITE INVENTORY (FROM
RCRA INFO MAY 2013)
3747
1485
1174
808
37
31
13
186
13
19
-------
Brownfields
833 communities have received TAB assistance from one of the four TAB grantees since
2008. However, there is no official list of brownfields in the U.S., since most U.S.
communities have brownfields sites, and EPA considers all communities to be eligible
for Brownfields grant assistance. Thus, it is not possible to determine the proportion of
communities with brownfields that have received TAB assistance.
Findings on the Selection Criteria and on-the-Ground process for
Accessing TAG and TASC
TAG
Communities typically learn about the availability of TAGs through site Remedial Project
Managers (RPMs) and CICs at public meetings and/or through e-mail and standard mail
distributions. EPA also distributes fact sheets regarding TAG and has information about
TAG on the EPA website.
Once a community group applies for a TAG, EPA conducts an Eligibility and
Organizational Capability Review to determine whether the applicant meets all the basic
requirements of administering a TAG. These requirements include:
The applicant must contain groups of individuals that may be affected by a
release or threatened release at a Superfund site;
The applicant cannot be a Potentially Responsible Party (PRP), academic
institution, or political subdivision; and
The applicant must have the organizational capability to administer the TAG.
Our interviews with Superfund regional staff indicated that TAGs are almost always
awarded to groups that meet these eligibility requirements. Resource constraints are
rare; only one region reported cutting TAG funds to due resource constraints. Two
regions stated that some communities do not apply for TAG because the paperwork
required to administer the EPA grant is too burdensome.
In some instances, more than one group applies for a TAG at a single site. This situation
occurs at less than 10 percent of the sites according to HQ. In these instances, EPA
conducts a Programmatic Criteria Evaluation to decide between applicants. This review
consists of determining the extent to which the TAG group: represents the community;
plans to use a technical advisor throughout the response; plans to communicate with
the broader community; and identifies potential environmental results that can be
measured.
TASC
Community groups do not apply to TASC directly; they receive access to the TASC
contract based on recommendations from regional staff to EPA HQ. Once EPA notifies a
community that it has been selected for TASC, the community writes an informal letter
or e-mail to their EPA region stating they are interested in funding. Since communities
do not apply for TASC directly, we heard from six RCRA CA regions that they do not
inform any community members about TASC services. However, all 10 Superfund
20
-------
regions and four RCRA stated that communities may hear about TASC through project
managers, CICs, the EPA website, and/or fact sheets.
HQand regional interviewees within the Superfund and RCRA CA programs stated that
there are no formal selection criteria for OSWER programs to access TASC. However,
communities that are selected for TASC funds typically share one or more of the
following conditions:
A lack of other sources of funding for community TA needs;
A high level of community interest in the cleanup process;
Environmental justice concerns;
Short-term technical assistance needs; and/or
Superfund communities that are not interested in forming a TAG group.
Based on interviewee opinions, there appears to be adequate TASC funding for
Superfund sites, but not for RCRA CA sites. Superfund HQ and eight of ten regions
interviewed stated that there is no lack of TASC funds to support the number of
communities recommended to receive funding. The two Superfund regions that
reported a lack of TASC funds pointed to examples of certain communities that did not
receive as much funding as the region had requested. In contrast, four RCRA CA regions
and HQ stated that RCRA CA sites were unable to get TASC funds for sites unless they
had a link to Superfund, and as noted above, only seven RCRA CA sites have accessed
TASC.
Findings on Community Assistance Provided Outside of Formal
Programs
Superfund
All ten Superfund regions interviewed stated that technical assistance to communities is
provided outside of formal programs. The most common response among regions (eight
out of 10 interviewed) was that RPMs and CICs are available to provide community
members with ongoing technical assistance as needs arise. In addition, four regions
mentioned holding special public meetings with communities as needed.
Brownfields
As discussed above, the majority of TA and ID to communities in the Brownfields
program is provided outside of TAB by cleanup grant recipients. For Brownfields cleanup
grants, the grantee must undertake the following steps to ensure the community is
informed and involved in the cleanup process:
Designate a community relations spokesperson.
Prepare a draft Community Relations Plan and submit to EPA to review prior to
the 'Analysis of Brownfield Cleanup Alternatives' is complete.
o The Community Relations Plan must outline steps to "provide
reasonable notice of proposed cleanup, opportunity for involvement,
21
-------
response to comments, and administrative records that are available to
the public."
Establish an information repository and maintain an administrative record for
the site.16
RCRA CA
As discussed previously, only seven RCRA CA sites have received TASC assistance, which
is the only formal TA program available to RCRA CA communities. Thus, nearly all TA
provided to RCRA CA communities has been outside of formal programs. EPA regions
and delegated states that administer RCRA CA make decisions on where to offer
targeted TA based on community requests and staff assessments of need.
The most common form of technical assistance reported is holding special or regular
public meetings, which was mentioned by nearly all states and regions interviewed. In
addition, all six states interviewed and four regions mentioned that they post site
information online for high profile sites. Other forms of technical assistance provided by
RCRA CA staff include workshops, programs for EJ communities, and encouraging PRPs
to engage the community. As noted above, one delegated state reports that
encouraging facilities to engage the community early in the RCRA CA process will help
build trust and facilitate community engagement throughout the lifetime of the project.
Other methods of distributing information to RCRA CA communities mentioned by
interviewees include newsletters, surveys, fact sheets, and social media.
Findings on Unmet TA or ID Needs
Most interviewees were able to identify some unmet TA and/or ID needs. However, as
shown in Exhibit 3-4 below, when asked to characterize unmet needs in their region,
RCRA CA interviewees noted a somewhat higher incidence of unmet needs among RCRA
CA communities than among Superfund or Brownfields communities.
Exhibit 3-4. Unmet ID and TA needs
RCRA CA
Superfund
Brownfields
NOTED UNMET TA NEEDS
HQ, 8 Regions
HQ, 6 Regions
HQ, 4 Regions
NOTED UNMET ID NEEDS
7 Regions
7 Regions
5 Regions
Superfund
Four regions reported that there are no identifiable unmet TA needs within Superfund.
The most common cause of perceived TA gaps identified by Superfund regions are a lack
of resources and staff (two of 10 regions) and that TAGs are too labor intensive to
administer (two of 10 regions). Other issues with TA noted by Superfund regions include
that more regions should be pursuing Technical Assistance Plans, which require PRPs to
provide TA; and that communities are pressuring HQ directly, instead of their own
16 These steps are documented in the EPA Brownfields Cleanup Grant Checklist: Major Programmatic Tasks. However,
Brownfields contacts did speak to the extent of implementation during interviews for this evaluation.
22
-------
regions, to access TA. Notably, Superfund HQ contacts did not mention any resource
limitations affecting TA and ID within the program.
Three Superfund regions reported there are no identifiable gaps in Superfund's
approach to ID. The most common issues identified with regard to Superfund's
approach to ID are that tailored outreach activities and electronic records work better
than traditional public notices (two regions and HQ); only vocal communities receive
adequate attention (two regions); and there are too few resources for ID (two regions).
Individual regions also stated that EPA falls short of delivering high quality information
to English-as-a-Second-Language (ESL) communities and immigrant communities, and
that CIPs should include references to TA options available to communities.
Brownfields
Most Brownfields interviewees suggested that there are no gaps in Brownfields'
approach to TA (six regions) and ID (five regions and HQ). Four Brownfields regions and
HQ indicated that any perceived gaps in TA are the result of resource limitations. Three
regions stated that Brownfields' approach to ID falls short of reaching rural communities
and/or communities that have not won grants. One region reported that the staff
members that manage the cooperative agreements lack knowledge, while another
stated that it is too difficult to post pertinent information for communities to EPA's
website.
RCRA CA
RCRA CA interviewees noted a lack of resources to conduct TA and ID as a cause of
perceived gaps more frequently than either of the other two cleanup programs. Eight of
10 RCRA CA regions and HQ contacts stated that a lack of resources is the cause of
unmet TA needs in the program, while six of 10 regions reported resource limitations as
a gap in RCRA CA's approach to ID.
Other areas of concern identified by RCRA CA interviewees included a lack of staff
training in community engagement, and that traditional forums such as public meetings
are not as effective as going to community-sponsored events, like fairs and celebrations.
QUESTION 3: HOW CAN SUPERFUND COMMUNITY INVOLVEMENT PLANS
(CIPS) BE USED TO IMPROVE TECHNICAL ASSISTANCE AND DELIVERY OF
INFO?
This question had a number of sub-questions:
Does every site have a CIP?
What information is available on the implementation of CIPs?
Are CIPs revised over time?
Do CIPs, as they are currently used, ensure effective technical assistance and
information delivery throughout the life of the project? Why or why not?
Summary of Methods to Address Question 3
Interviews with 10 Superfund regional staff
23
-------
Findings on CIP Implementation
As noted above under Question 1, all Superfund sites have a CIP. According to all
Superfund regional interviewees, CIPS are tracked and maintained by regional staff. CIP
updates are variable. From the perspective of tenured EPA staff, CIPs are most valuable
at the beginning of the Superfund process. From the perspective of community
members and new EPA staff, CIPs and updates have value throughout the cleanup
process. CIPs may be improved by more frequent, targeted revisions and by making
them more readable by community members. CIP implementation is a broad topic that
encompasses the other two sub-questions; thus, more information on implementation
is in the sections below.
Finding on CIP Revisions
EPA's official CIP guidance says that documents should be updated as needed:
"The NCP requires that the CIP be reviewed prior to the
initiation of the remedial design to determine whether it
should be revised to describe further public involvement
activities. Yet, there is no standard rule about when to
update or completely revise the CIP. Because the CIP should
be a living document that is referred to regularly, it makes
sense that information will be continuously added or updated."17
However, there appears to be confusion about the CIP guidance and the recommended
revision schedule among Superfund regional contacts. Only two interviewees referred
to the guidance when asked about CIP revision frequency, and they both mistakely
indicated that the guidance contained a specific time frame for CIP revisions. One region
said updates are prescribed every two years by the guidance, while another said every
five years.
The reality is that average revision frequencies for CIPs vary widely from site to site;
interviewees indicated that revisions occur once per year for some sites to once every
five to seven years for others. Major impetuses for revision include community demand
and project milestones. Five interviewees said revisions occur "as needed." Of these five
interviewees, three interviewees cited significant milestones or phase changes of the
cleanup process as impetuses for CIP revisions. Five interviewees said that revisions
occur too infrequently; three of these contacts indicated that revisions only occur in
response to community demand.
Findings on CIPs as a Tool to Ensure Effective Technical Assitance and
Information Dissemination
Six of 10 interviewees said that CIPs are not an important tool for ongoing technical
assistance and information delivery. The general sentiment among Superfund regional
interviewees is that CIPs are helpful to organize community engagement at the
beginning of the Superfund process, but are not used by EPA staff as a resource over
time. EPA tends to set aside the document after the start of the project, instead relying
on ad-hoc communication with community members.
17 Community Involvement Plans, p. 7, available at: http://www.epa.gov/superfund/communitv/pdfs/toolkit/ciplans.pdf
24
-------
CIP updates often lag behind project changes. Thus, CIPs may not ensure effective
technical assistance and information delivery throughout the life of the project given
lagging updates. However, as discussed under Evaluation Question 4 findings,
satisfaction with technical assistance and information dissemination is high among
Superfund community contacts participating in this evaluation that have received a TAG,
and across the Superfund community contacts that have used the TASC contract.
CIPS appear to be important for other reasons, including gathering community contact
information; providing an ongoing reference tool for community members; and serving
as training materials for new Superfund staff.
Superfund regional contacts provided several suggestions for improving CIPs. Many
interviewees suggested format changes including:
Two interviewees suggested that CIPs be shortened, or that updates be placed
into an addendum.
Two interviewees suggested making CIPs easier to read, perhaps by using fact
sheet summaries.
One interviewee noted that CIPs have not kept up with changing
communication methods and social media; the format of the CIPs as stand-
alone documents could be reconsidered with social media in mind.
Two interviewees recommended more regular updates. However, one interviewee
cautioned against "obsessive" updates, pointing out that only certain parts of CIPs need
updating overtime.
QUESTION 4: WHAT IS THE BASELINE OF CUSTOMER SATISFACTION WITH
OSWER TECHNICAL ASSISTANCE AND INFORMATION DISSEMINATION
ACTIVITIES?
Summary of Methods to Address Question 4
Satisfaction interviews with community contacts (see summary table Exhibit 2-2
in the Methods chapter).
Findings on Interviewee Satisfaction
Interviewees rated their overall satisfaction with assistance services provided on a scale
of one to six; with a score of one being very dissatisfied to a score of 6 being very
satisfied (see satisfaction interview guide in the methodology in Appendix A). As shown
in Exhibit 3-5 below, interviewees noted consistently high satisfaction overall scores for
Superfund and Brownfields programs, while scores for RCRA CA were more mixed.
However, regarding RCRA CA, it should be noted that at the lower end, only two
interviews rated their overall satisfaction as a "1" with assistance provided by RCRA CA
states (see bottom graph in Exhibit 3-5); the majority of RCRA CA sites provided
favorable overall ratings for assistance received.
25
-------
Exhibit 3-5. Satisfaction Score Summary
80
70
60
50
£
= 40
° 30
8 20
01
°- 10
0
Very
Unsatisfied
Overall Score Distribution by Program
RCRA
(9 Interviews)
I Superfund (9 Interviews)
I Brownfields (8 Interviews)
3 4
Likert Scale Score
Very
Satisfied
22
20
18
16
14
= 12
10
8
6
4
2
0
Very
Unsatisfied
Overall Score Distribution
I Brownfields
RCRA
I Superfund
3 4
Likert Scale Score
Very
Satisfied
26
-------
Illustrative interviewee elaborations on overall satisfaction scores are provided in
Exhibit 3-6 below.
Exhibit 3-6. Satisfaction Ratings Explanations from Interviewees
Summary of Score
"6" Elaborations
Summary of Score
"5" Elaborations
Summary of Score
"4" Elaborations
Summary of Score
"1" Elaborations
Excellent service that goes above and beyond requirements
High level of expertise
Excellent record of interfacing with the community often and
in plain language
Affords community a high degree of influence over events at
the site
Has helped community secure grants
Displays ability to work effectively with all parties: community,
state, Responsible Party (RP), EPA
Generally high quality service
Good communication; solid expertise conveyed in plain
language
Should reach out to community more often
Would be rated 6, but community hasn't worked long enough
with them
Outreach is passive; delegated RCRA CA only provides
information to citizens that ask for it
RCRA CA does not help citizens tackle short term issues (e.g.,
declining home equity), instead focusing on the decades-long
cleanup
Too much control is put in the hands of the RP
EPA (RCRA CA staff) is not straightforward about
administrative issues, and claim to have inadequate manpower
to tackle problems
Delegated RCRA CA state agency doesn't listen to relevant
parties: not the other state agencies, not the community, and
not EPA
lEc compared overall satisfaction ratings for TAG and TASC among Superfund
community contacts and found remarkably similar overall levels of satisfaction with
these two mechanisms for providing assistance, as shown in Exhibit 3-7 below.
Findings on Interviewee Suggestions For Improving Services
Community contact suggestions common to all three programs include making outreach
materials more accessible by using visual aids and plainer language, and reaching out to
community members more often and/or more actively. Additional suggestions include:
bolstering technical expertise of advisors, improving EPA's website, and assisting with
EPA grant processes.
27
-------
Exhibit 3-7. Comparison of Overall Satisfaction Scores for TAG and
TASC
STATISTIC
Median
Mean
Minimum
Maximum
TAG
6
5.8
5
6
TASC
5.7
5.5
4
6
Superfund
Most Superfund community interviewees stated that technical advisors did not need to
make improvements. The small number of suggestions provided by interviewees on
improving technical assistance by providers include: using more visual aids during
presentations; developing a more local presence; and bolstering expertise on wildlife
impact issues. All of these suggestions were mentioned by only one individual.
Interviewees had more suggestions for EPA to improve on community outreach:
Reach out to communities more often (3 respondents).
Create a more transparent decision-making process (2 respondents).
Acquire more contamination science expertise for residents to access (1
respondent).
Avoid giving residents unrealistic expectations (1 respondent).
Brownfields
Two Brownfields interviewees said their advisors did not need any improvement. Other
interviewees provided suggestions on how how Brownfields technical advisors could
improve their services:
Engage the community more deeply. Use them as a resource to discover site
histories (2 respondents).
Assist with grant applications (2 respondents).
Draft higher quality reports and visuals (1 respondent).
Brownfields community contacts also had suggestions for EPA on improving community
outreach:
Improve Brownfields website search-ability and update schedule (2
respondents).
Simplify the grant process (2 respondents).
Increase project officers' level of technical expertise (1 respondent).
Distribute more success stories and metrics (1 respondent).
28
-------
RCRA CA
Three RCRA CA community interviews had no suggestions for technical advisor
improvements; others provided the following input:
Communicate in plainer language (3 interviewees).
Distribute outreach more promptly, frequently, or actively (3 interviewees).
Spend more time listening to community needs, including shorter term issues
like declining property values (1 interviewee).
Most RCRA CA satisfaction interviewees had no experience working with EPA, and did
not offer suggestions for EPA regarding community outreach.
QUESTION 5: WHAT MEASURES CAN BE USED TO ASSESS THE
EFFECTIVENESS AND TANGIBLE OUTCOMES OF OSWER TECHNICAL
ASSISTANCE AND INFORMATION DISSEMINATION ACTIVITIES ACROSS THE
LIFECYCLE OF SITE PLANNING, REMEDIATION, AND REUSE?
QUESTION 6: HOW CAN THESE MEASURES BE USED TO IMPROVE OSWER
TECHNICAL ASSISTANCE AND INFORMATION DISSEMINATION ACTIVITIES?
Summary of Methods to Address Questions 5 and 6:
Reviewed CEI logic models and related program materials.
Reviewed Guidelines for Measuring the Performance of EPA Partnership
Programs, developed by the National Center for Environmental Innovation,
dated June 2006.
Reviewed the Partnerships for Environmental Public Health Evaluation Metrics
Manual, dated October 18, 2010.
Reviewed the types of measures currently used by OSWER programs and other
EPA community-based programs related to community involvement.
Conducted interviews with Superfund, Brownfields, and RCRA CA HQ and
regional staff; included interview questions on potential measures.
Considered feasibility factors for metrics implementation.
Interview Findings of Relevance for Developing Measures
lEc asked HQand regional program staff from each program what constitutes successful
TA and/or ID, as a starting point to thinking about meaures. The most frequent response
was that successful engagement consists of tailoring materials to meet community
needs and/or to ensure the community is satisfied with the information provided; 12 of
30 regional interviews (four from each program) provided this response. Other respones
were far less frequent. Three Superfund interviewees equated unbiased TA with
successful TA. lEc's interpretation of this response, based on other feedback provided by
29
-------
Superfund interviewees on TAG, is that some regional saff are concerned that technical
advisors hired under TAG are biased towards providing information that their clients
(community groups) want to hear, rather than providing unbiased information. Three
Brownfields interviewees indicated that successfull TA is tied to successful project
outcomes.
lEc also asked HQ and regional contacts about current metrics used to track community
engagement for each program. We learned that in some regions, Superfund staff track
output metrics, but not outcome metrics. Examples of output metrics tracked by some
Superfund rgions include : funding spent, number of public notices, number of
meetings, number of TASCs, number of CAGs, number of TAGs, number of people on
mailing lists, number of meeting attendees, amount of media coverage, requests
received for TA. Only two RCRA CA regions reported tracking output metrics: one
regional tracks the number of sites that they have designated as high profile sites, but
does not store these data post-remediation. The other region tracks the engagement
tools that are used and how often they are used. According to our interviewees, the
Brownfields program does not track community involvement metrics, but some
interviewees referred lEcto site-level success stories generated by HQ, some of which
have information on communit involvement. Finally, as noted previously in this report,
EPA's TASC contractor Skeo tracks customer satisfaction at sites that have used the
TASC contract.
Finally, lEc asked interviewees "What other metrics might help assess the extent to
which the TA and/or ID efforts conducted by EPA are successful?" Responses are as
follows:
Most Brownfields interviewees (seven of 10 Regions and HQ) do not believe
any new metrics should be created. The other three Brownfields regions
suggested closely tracking specific resources used at individual sites.
Measuring customer satisfaction was the most common response from RCRA
CA and Superfund interviewees. Four Superfund regions suggested satisfaction
surveys for TAGs, similar to those conducted by Skeo for TASC. Four RCRA CA
regions also suggested satisfaction interviews.
Four regional interviewees (1 from Superfund, 2 brom Brownfields, and 1 from
RCRA CA) said that one metric to gage the success of TA/ID is to count the
frequency of complaints filed by community groups.
One Superfund regional contact and one RCRA CA regional contact suggested
measuring community member learning from assistance received.
One Superfund regional contact suggested an efficiency metric by comparing
resources spent to satisfaction rates. Similarly, one Brownfields regional
contact suggested comparing resources spent to project outcomes.
30
-------
Approach to Developing Menu of Potential Measures
lEc considered the following types of measures for inclusion in a menu of potential
measures for OSWER to use to track progress on TA and ID moving forward:
Output measures:
Measures of unmet demand for Cl services
Conventional output measures for Cl work: numbers of meetings, engagements,
et cetera (variations on measures currently used by some Superfund regions)
Output measures for ID, such as frequency of community notices
Measures of EPA training
Outcome measures:
Measures of community satisfaction
Measures of EPA use of best practices
We developed this universe of potential types of measures based on:
Close review of the CEI logic models and theory of change
Existing community involvement measures used by OSWER programs
Review of the types of measures currently used by OSWER programs and other
EPA community-based programs related to community involvement
References in applicable metrics manuals and literature reviewed
We then narrowed down the types of measures to include in a menu based on:
Feasibility factors including:
o The ability to define objective measure(s)
o The need to develop and implement new information collection
infrastructure
Feedback from interviewees on potential measures (discussed above)
lEc's professional judgment regarding the likelihood of the measure being
accepted and successfully implemented by the OSWER programs
Menu of Potential Measures
lEc developed menus of potential measures for outputs, short-term outcomes, and long-
term outcomes discussed below. It is important to note that these are menus from
which to select measures; we do not recommend adopting all of these measures.
Output Measures
The output menu is presented in Exhibit 3-8 below.
31
-------
Exhibit 3-8. lEc Recommendations for a Menu of Potential Output
Measures
1. Proportion of Superfund communities that applied for TAG assistance, but have not
received assistance
2. Proportion of Brownfields communities that requested TAB or other form of K6
assistance, but have not received assistance
3. Proportion of Superfund and RCRA CA communities that were recommended by a
region for TASC assistance, but have not received assistance
4. Proportion of RCRA CA communities that have asked a state or region for any form
of assistance, but have not received assistance
5. Number/Percent of RCRA CA communities that have received TA
6. Number/Percent EPA Regions that have offered TA/ID training in the past year
lEc's rationale for these output measures is as follows:
The first four measures address unmet needs. Several regions identified assigning
resources where they are needed as successful aspect of TA. Brownfields and RCRA CA
interview contacts also frequently identified a lack of funding as an impediment to TA.
Thus, the proportion of communities that have either applied for, or been identified by
regional staff, as needing technical assistance will give EPA management an idea of how
much additional resources are needed and where they are needed . Note that lEc did
not frame the measures as "proportion of communities that are known to have unmet
needs" because "unmet needs" is not measurable language. We included a specific
measure on RCRA CA communities receiving TA (#5) because this is where the gap
seems greatest. Thus, it would be reasonable for EPA to focus additional output
measures on RCRA CA in particular. Finally, some interviewees indicated that more
training would be helpful in delivering quality TA, which led to the development of #6
on frequency of training.
lEc does not suggest any output measures for information dissemination. Based on our
interview data , it appears that required ID, such as CIPs and public notices for RCRA CA
are already widely implemented. Since ID should be tailored to community needs, there
is no single measure that can be universally applied to all sites because sites do not
apply for ID in and of itself.
Many regional contacts across the three OSWER programs advised against any
additional tracking of conventional output measures such as number of meetings,
number of attendees, number of public notices, et cetera. These metrics are already
tracked by Superfund in some regions, but contacts indicate they cannot be used to
improve TA and/or ID activities. We agree with this conclusion, and do not recommend
additional tracking of these types of "bean counting" measures.
32
-------
Short-term Outcomes
Based on the criteria stated above, lEc has not identified many options for a short-term
outcomes measure for the CEI. The one measure that we recommend EPA consider is
presented below in Exhibit 3-9.
Exhibit 3-9.lEc Recommendation for Potential Short-Term Outcome
Measures
1. The number/proportion of community members filing complaints related to the
provision of TA with Superfund/Brownfields/RCRA CA regional staff annually.
Several regional interviewees indicated that tracking trends in complaints is a viable
approach to measuring program success, and some regional staff currently track
complaints informally. If EPA works to continually improve the overall quality and
availability of TA, complaints should decrease over time. However, tracking complaints
may pose challenges. For example, EPA would have to develop guidance to define the
boundaries of a community member complaint about TA (in contrast to complaints
about other remediation or redevelopment issues), and regions would need to be very
careful about coding complaints. EPA may also need to develop or augment a data
collection system to track this measure, in particular for RCRA CA, which has the
complication of state delegation in most states, and does not have an existing data
collection infrastructure for tracking assistance provide to communities. A limitation of
this measure is that it would not account for those who are dissatisfied but do not file a
complaint.
Theoretically, EPA's utilization of best practices would be a superior short-term measure
of success of the CEI program compared to tracking complaints. However, we cannot
recommend tracking the use of best practices because: best practices would need to be
rigorously analyzed and defined prior to tracking, and EPA has learned from experience
that this first step can be quite difficult; tracking of best practices is not in place for any
of the three OSWER programs; and the data collection approach required to build and
administer a system for tracking use of best practices would be more complex than for
other measures.
Long-term Outcomes
The long-term outcome menu is presented in Exhibit 3-9 below.
Exhibit 3-10.lEc Recommendations for a Menu of Potential Long-Term
Outcome Measures
1. Proportion of TAG/TASC/TAB communities that have received a community
satisfaction survey that are "satisfied" with the information provided by EPA
throughout the cleanup process (tracked individually for each program).
2. Proportion of TAGs/TASC/TAB/RCRA CA communities that have received a
community satisfaction survey that are "satisfied" with the assistance provided by
their technical advisor (tracked individually for each program).
33
-------
Throughout the interview process, lEc heard consistent support for satisfaction
surveying, and we think it is the best approach for understanding the long-term success
of TA provided under the CEI. EPA could adopt both of these measures, or could choose
to adopt only the second measure that focuses on satisfaction with TA provided (as
opposed to information provided by EPA). We would recommend that each community
that receives formal technical assistance be provided with a satisfaction survey. With
the exception of RCRA CA, we do not recommend surveing individuals from
communities that received only informal assistance, as it will be difficult to identify an
appropriate individual(s) to be surveyed (this was a key problem for lEc in conducting
this evaluation, as discussed previously).
Skeo Solutions has already developed a community satisfaction format for TASC which
asks communities to rank various components of technical assistance, which lEc used
for conducting satisfaction surveys with TAG recipients. We would suggest revising
Skeo's format slightly moving forward to ensure every number on the scale has an
associated value. Currently, the 1 to 6 scale implemented by Skeo only defines 1 as "not
at all satisfied" and 6 as "very satisfied." Using this scale, we would interpret a score of
4 or more to be "satisfied" but it would be preferable to define each rating for the
community member taking the survey.
Furthermore, we suggest that Skeo discontinue its practice of interviewing community
contacts and EPA contacts about satisfaction within the same interview; community
contacts should be interviewed separately from EPA contacts, and have their scores
tracked seperately. However, we encourage more use of Skeo's practice of interviewing
multiple contacts for each community, as this provides a diversity of viewpoints.
EPA could also include questions to probe on community member learning within
community satisfaction surveys. However, we do not think that measuring community
member learning is a substitute for measuring satisfaction. Moreover, while community
contacts can easily and accurately report their level of satisfaction, self-reported data on
learning may suffer from biases, and it is not feasible in this context to admister surveys
that objectively probe knowledge gained by community members through the TA
process.
Finally, as noted above, EPA would need to develop new data infrastructure to track
assistance provided to RCRA CA communities, and work with states to implement a
reporting system. Again, this is because the vast majority of assistance given to RCRA CA
communities is not provided under the auspices of a formal program, and TA is typically
administered by state agencies not EPA.
34
-------
APPENDIX A: INTERVIEW GUIDES
lEc proposes to conduct interviews with the following individuals as the main source of
new data collection for the evaluation:
1. Superfund HQ contacts and contractor Skeo (2-3 interviews)
2. Superfund Regional contacts (10 interviews)
3. RCRA CA HQ contacts (2 interviews)
4. RCRA CA Regional contacts (lOinterviews)
5. RCRA CA state contacts (6 interviews)
6. Brownfields HQ contacts (2 interviews)
7. Brownfields Regional contacts (10 interviews)
8. TAB Grantees (4 interviews)
9. CPRC (1 interview)
10. Community satisfaction interviews
a. Superfund (10 interviews with TAG grant recipients)
b. RCRA CA (12-15 interviews, including 2-3 sites where the EPA Region
conducted community engagement instead of the state)
c. Brownfields TAB and K6 grants (the majority of interviews will be
conducted with communities that have received assistance through TAB
program; we will also interview any community that has received TA
through K6, but our current understanding is that there may be only one
or two applicable K6 grants)
lEc will conduct all interviews by phone. We will make initial contact through an
introductory email to explain the evaluation, provide the relevant interview guide, and
suggest dates and times for an interview.
The initial email should also address confidentiality issues. We suggest sharing all names
of interviewees with EPA, but not publishing names and positions in the final report.
More importantly, we suggest reporting results in aggregate; for example, the final report
may say that "X# of six EPA Regional RCRA CA interviewees identified instances
where community TA needs went unmet." We suggest not attributing statements to
individual interviewees without prior consent. This confidentiality approach will increase
the likelihood of interviewee candor.
35
-------
lEc developed a series of interview guides to ensure consistency in conducting
interviews; these guides appear starting on the following page. Within the context of
those guides, we may depart from the written script during interviews as appropriate to
obtain additional detail or clarification, or follow up on a topic raised by the interviewee.
The interview guides in this appendix indicate the evaluation question(s) addressed by
each interview question; we will delete this text in the version of the guides provided to
interviewees.
lEc will commence analysis of information collected during interviews once interviews
are complete. lEc may code interview responses and/or conduct a thematic analysis of
interview responses. Our analysis will focus on identifying trends, such as areas of
consensus or sharp disagreement between interviewees. We will look for such trends both
within and between interviewee categories, with a particular view to differences between
EPA staff and individuals outside the agency. We will also highlight any areas in which
interviewee responses diverged significantly from our expectations. Where there are
significant points of disagreement between interviewees, or between our expectations and
the overall results, we will note potential contributing factors.
We will conduct a quantitative analysis of responses to the community satisfaction
interviews, and will provide, for example, average community satisfaction scores, as well
as scores broken down by EPA cleanup program, TA program (e.g., TAG, TASC, TAB),
and TA service (e.g. facilitation, written materials). We will conduct this analysis using
both lE's community satisfaction interviews as well as data from interviews previously
conducted by Skeo.
36
-------
INTERVIEW GUIDE: SUPERFUND HQ CONTACTS AND CONTRACTOR (SKEO)
Background Questions
1. Can you briefly summarize your position within the Superfund program?
2. Can you briefly summarize your activities and responsibilities related to
technical assistance and information dissemination at Superfund sites?
Questions on TASC
3. What are the communication methods through which communities typically
hear aboutTASC?
a. Some examples of communication methods may be Internet sites or e-
mail.
Addresses Evaluation Question(s): 2a
4. What are the communication methods through which you communicate TASC
availability and guidelines to Regions?
Addresses Evaluation Question(s): 2a
5. What are the key factors that determine whether the EPA Region will use the
TASC contract at a specific site?
a. For example, does the level of contamination of the site or the
demographics of the community factor into the decision on whether
TASC funds are used? If so, how so?
Addresses Evaluation Question(s): 2a
6. Does EPA maintain existing documentation on selection criteria for TASC funds?
a. If so, could you provide us with documentation on this issue?
b. Are communities asked to "apply" for TASC?
Addresses Evaluation Question(s): 2c
7. Are there more communities that meet the criteria for receiving TASC assistance
than funds available?
a. If so, how are award decisions made between qualifying groups?
b. If so, do you track the universe of sites that meet the criteria for
receiving TASC that have not received it?
Addresses Evaluation Question(s): 2c
37
-------
8. (For Skeo only) As of January of 2012, you had indicated that the TASC contract
had been used at 27 Superfund sites. Does this number need to be updated?
Addresses Evaluation Question(s): 4
9. (For Skeo only) As of January of 2012, Skeo completed 50 evaluations of TASC;
have you completed more?
a. If yes, please share the additional evaluations with us.
Addresses Evaluation Question(s): 4
Questions on TAGs
10. What are the communication methods through which communities typically
hear aboutTAGs?
a. Some examples of communication methods may be internet sites or e-
mail.
Addresses Evaluation Question(s): 2a
11. What are the key factors that determine whether communities receive TAGs?
a. For example, does the size of the community group or the
demographics of the community factor into the decision on whether
certain community groups are awarded TAGs? If so, how so?
Addresses Evaluation Question(s): 2a
12. Does EPA maintain existing documentation on selection criteria for TAG funds,
including scoring sheets for RFPs?
a. If so, could you provide us with documentation on this issue?
Addresses Evaluation Question(s): 2c
13. Are there more communities that meet the criteria for receiving TAGs than
funds available?
a. If so, how are award decisions made between qualifying groups?
Addresses Evaluation Question(s): 2c
14. Are there sites where two or more community groups compete for only one
TAG?
a. If so, how often does this occur?
b. If so, how are award decisions made between groups?
38
-------
Addresses Evaluation Question(s): 2c
15. Do TAG recipients share information with other community groups and
stakeholders?
a. Does EPA encourage them to share information? If yes, how?
Addresses Evaluation Question(s): 2a
16. Have you seen tensions arise between TAG grantees and other community
groups and stakeholders?
a. If yes, can you give an example of the type of issues that arise?
Addresses Evaluation Question(s): 2a
17. Do you maintain information on the types of TA that have been provided to
each grantee?
a. If so, can you provide this information to us?
Addresses Evaluation Question(s): 4
18. We plan on conducting community satisfaction surveys with 10 TAG recipients;
are there any communities that you recommend in particular?
a. If yes, why?
Addresses Evaluation Question(s): 4
Other Questions
19. Have you identified any gaps in EPA's technical assistance coverage at
Superfund sites?
a. If so, please identify these gaps.
Addresses Evaluation Question(s): 2e
20. Have you identified any gaps in EPA's information dissemination coverage at
Superfund sites?
a. If so, please identify these gaps.
Addresses Evaluation Question(s): 2e
21. Please describe the types of in-kind community assistance that EPA provides at
Superfund sites, outside of formal programs such as TAGs and TASC?
a. How does EPA HQ define and promote in-kind TA to Regions?
39
-------
i. Is guidance available on providing in-kind TA?
ii. Do you make a distinction between project-related community
engagement efforts and in-kind TA provided during a project?
Addresses Evaluation Question(s): 2d
22. Does EPA HQ track how each Region conducts TA? For example, do you track:
a. The number of Community Involvement Coordinators?
b. The number of CIPs?
c. The use of project contractors?
d. The general approach that each Region uses to conduct TA?
Addresses Evaluation Question(s): 2a
23. One of the goals of the evaluation is to develop measures for EPA to use moving
forward to assess the success of OSWER's TA and ID practices. In your opinion,
what constitutes successful TA and/or ID?
a. Do you currently track any metrics that might help assess the extent to
which the TA and/or ID efforts conducted by EPA are successful?
b. Are there any metrics that you are not currently tracking that might help
assess the extent to which the TA and/or ID efforts conducted by EPA
are successful?
Addresses Evaluation Question(s): 5 and 6
24. Do you have any other thoughts that you would like to share with us to inform
our project?
40
-------
INTERVIEW GUIDE: SUPERFUND REGIONAL CONTACTS
Background Questions
1. Can you briefly summarize your position within the Superfund program?
2. Can you briefly summarize your activities and responsibilities related to
technical assistance and information dissemination at Superfund sites?
Questions on CIPs
3. Are CIPs developed for each Superfund site in your Region?
a. If not, explain why CIPs are not developed for each site.
Addresses Evaluation Question(s): 2a, 3
4. How frequently are CIPs revised within your Region to meet the changing needs
of the community?
Addresses Evaluation Question(s): 2a, 3
5. Are the information dissemination practices identified in the CIP always carried
out when deciding what information to share with the community and what
communication method through which to share it?
a. If not, please elaborate on why these practices are not always followed.
Addresses Evaluation Question(s): 2a
6. How do you rate CIPs as a tool to ensure effective technical assistance and
information delivery throughout the lifetime of the project?
a. If not, what improvements could be made to the process?
Addresses Evaluation Question(s): 3
Questions on TASC
7. What are the communication methods through which communities typically
hear aboutTASC?
a. Some examples of communication methods may be internet sites or e-
mail.
Addresses Evaluation Question(s): 2a
8. What are the key factors that determine whether the EPA Region will use the
TASC contract at a specific site?
41
-------
a. For example, does the level of contamination of the site or the
demographics of the community factor into the decision on whether
TASC funds are used? If so, how so?
Addresses Evaluation Question(s): 2a
9. Does EPA maintain existing documentation on selection criteria for TASC funds?
a. If so, could you provide us with documentation on this issue?
Addresses Evaluation Question(s): 2c
10. Are there more communities that meet the criteria for receiving TASC assistance
than funds available?
a. If so, how are award decisions made between qualifying groups?
b. Are communities ever asked to "apply" for TASC?
Addresses Evaluation Question(s): 2c
Questions on TAGs
11. What are the communication methods through which communities typically
hear aboutTAGs?
a. Some examples of communication methods may be internet sites or e-
mail.
Addresses Evaluation Question(s): 2a
12. What are the key factors that determine whether communities receive TAGs?
a. For example, does the size of the community group or the
demographics of the community factor into the decision on whether
certain community groups are awarded TAGs? If so, how so?
Addresses Evaluation Question(s): 2a
13. Does EPA maintain existing documentation on selection criteria for TAG funds,
including scoring sheets for RFPs?
a. If so, could you provide us with documentation on this issue?
Addresses Evaluation Question(s): 2c
14. Are there more communities that meet the criteria for receiving TAGs than
funds available?
a. If so, how are award decisions made between qualifying groups?
42
-------
Addresses Evaluation Question(s): 2c
15. Do TAG recipients share information with other community groups and
stakeholders?
a. Does EPA encourage them to share information? If yes, how?
Addresses Evaluation Question(s): 2a
16. Have you seen tensions arise between TAG grantees and other community
groups and stakeholders?
a. If yes, can you give an example of the type of issues that arise?
Addresses Evaluation Question(s): 2a
17. (Ask only if HQ has not provided the information) Do you maintain information
on the types of TA that have been provided to each grantee?
a. If yes, can you provide this information to us?
Addresses Evaluation Question(s): 4
Other Questions
18. Have you identified any gaps in EPA's technical assistance coverage at
Superfund sites in your Region?
a. If so, please identify these gaps.
Addresses Evaluation Question(s): 2e
19. Have you identified any gaps in EPA's information dissemination coverage at
Superfund sites in your Region?
a. If so, please identify these gaps.
Addresses Evaluation Question(s): 2e
20. Do you provide any in-kind community assistance at Superfund sites, outside of
formal programs in your Region?
a. If so, please elaborate on the types of in-kind assistance you provide.
Addresses Evaluation Question(s): 2d
21. One of the goals of the evaluation is to develop measures for EPA to use moving
forward to assess the success of OSWER's TA and ID practices. In your opinion,
what constitutes successful TA and/or ID?
43
-------
a. Do you currently track any metrics that might help assess the extent to
which the TA and/or ID efforts conducted by EPA are successful?
b. Are there any metrics that you are not currently tracking that might help
assess the extent to which the TA and/or ID efforts conducted by EPA
are successful?
Addresses Evaluation Question(s): 5 and 6
22. To what extent does your Region use the EPA Conflict Resolution and Prevention
Center? Please elaborate.
Addresses Evaluation Question(s): 2a
23. Do you have any other thoughts that you would like to share with us to inform
our project?
44
-------
INTERVIEW GUIDE: RCRA CORRECTIVE ACTION HQ CONTACTS
Introductory Questions
1. Can you briefly summarize your position within RCRA Corrective Action (CA)?
2. Can you briefly summarize your activities and responsibilities related to
technical assistance and information dissemination at RCRA CA sites?
3. Is EPA required to provide ID and TA to communities?
a. If so, what are the requirements?
b. Are there additional drivers to provide ID and TA outside of formal
requirements?
i. If so, what are these drivers?
Addresses Evaluation Question(s): 1
Questions on TASC
4. What are the communication methods through which RCRA CA communities
typically hear about TASC?
a. Some examples of communication methods may be internet sites or e-
mail.
Addresses Evaluation Question(s): 2a
5. What are the key factors that determine whether the EPA Region will use the
TASC contract at a specific RCRA CA site?
a. For example, does the level of contamination of the site or the
demographics of the community factor into the decision on whether
TASC funds are used? If so, how so?
Addresses Evaluation Question(s): 2a
6. Does EPA maintain existing documentation on selection criteria for TASC funds?
a. If so, could you provide us with documentation on this issue?
b. Are communities ever asked to "apply" for TASC?
Addresses Evaluation Question(s): 2c
7. Only five RCRA CA sites have received TASC assistance. Are there more
communities that meet the criteria for receiving TASC assistance than funds
available?
45
-------
a. If so, how are award decisions made between qualifying groups?
Addresses Evaluation Question(s): 2c
General Questions
8. How does EPA track community involvement activities within RCRA CA, if at all?
a. Does EPA track community involvement for sites where EPA leads these
activities (in non-delegated states, and/or in states where the Region
and state have a work sharing relationship for certain sites)?
b. Please provide any documentation you have that tracks community-
involvement activities within RCRA.
Addresses Evaluation Question(s): 1
9. Do you provide any in-kind community assistance at RCRA CA sites, outside of
formal programs?
a. If so, please elaborate on the types of in-kind assistance you provide.
Addresses Evaluation Question(s): 2d
10. Have you identified any gaps in EPA's technical assistance coverage at RCRA CA
sites?
a. If so, please identify these gaps.
Addresses Evaluation Question(s): 2e
11. How does HQ communicate and/or promote community engagement with the
Regions?
Addresses Evaluation Question(s): 2a
12. How does EPA conduct community engagement at EPA-lead sites?
a. Is there guidance from HQto the Regions?
13. Have you identified any gaps in EPA's information dissemination coverage at
RCRA CA sites?
a. If so, please identify these gaps.
Addresses Evaluation Question(s): 2e
14. Which EPA Regions and states do you recommend that we talk with as part of
this evaluation? We plan on talking with 4-6 Regions and six states. As we noted
in our email to you, we would like to talk to talk with an EPA staff member in
Region 4, but we would like your opinion on which Regions and delegated states
46
-------
would provide us with the most information on ID and TA practices, as well as
TASC usage. The only states we do not want to speak with are CT, MS, SC, and
MA, because the RCRA Compendium provided information on programs in these
states.
15. One of the goals of the evaluation is to develop measures for EPA to use moving
forward to assess the success of OSWER's TA and ID practices. In your opinion,
what constitutes successful TA and/or ID?
a. Do you currently track any metrics that might help assess the extent to
which the TA and/or ID efforts conducted by EPA are successful?
b. Are there any metrics that you are not currently tracking that might help
assess the extent to which the TA and/or ID efforts conducted by EPA
are successful?
Addresses Evaluation Question(s): 5 and 6
16. Do you have any other thoughts that you would like to share with us to inform
our project?
47
-------
INTERVIEW GUIDE: RCRA CORRECTIVE ACTION REGIONAL CONTACTS
Introductory Questions
1. Can you briefly summarize your position within RCRA Corrective Action?
2. Can you briefly summarize your activities and responsibilities related to
technical assistance and information dissemination at RCRA CA sites?
3. Is EPA required to provide ID and TA to communities?
a. If so, what are the requirements?
b. Are there additional drivers to provide ID and TA outside of formal
requirements?
i. If so, what are these drivers?
Addresses Evaluation Question(s): 1
Questions on Standard Community Involvement Practices at EPA-Led Sites
The following questions pertain only to EPA-led sites; does your Region lead cleanup at
any RCRA CA sites? (If not, skip section)
4. Does the Region provide public notice at RCRA CA sites?
a. If not, what are the key factors in determining whether a public notice is
issued?
Addresses Evaluation Question(s): 2a
5. Does the Region provide an opportunity for public comment at RCRA CA sites?
a. If not, what are the key factors in determining whether there is an
opportunity for public comment?
Addresses Evaluation Question(s): 2a
6. Does the Region hold public meetings to address the community's concerns at
RCRA CA sites?
a. If not, does the Region hold public meetings to address the community's
concerns at RCRA CA sites?
i. If so, please provide examples of sites for which your Region
held public meetings and,
ii. Provide the key factors in deciding whether to hold public
meetings.
48
-------
Addresses Evaluation Question(s): 2a
7. Does EPA take the lead on community engagement at state-lead sites?
a. If so, how is the lead for community engagement decided?
8. Are there other community involvement practices that your Region uses when
directly implementing corrective action, specifically involving information
dissemination or technical assistance?
a. If so, please elaborate on:
iii. The extent to which these practices implemented.
iv. Examples of how community engagement plays out at your
sites.
v. The key factors in deciding whether to implement these
practices at specific sites.
Addresses Evaluation Question(s): 2a
Questions on TASC
9. What are the communication methods through which communities typically
hear aboutTASC?
a. Some examples of communication methods may be internet sites or e-
mail.
Addresses Evaluation Question(s): 2a
10. What are the key factors that determine whether the EPA Region will use the
TASC contract at a specific RCRA CA site?
b. For example, does the level of contamination of the site or the
demographics of the community factor into the decision on whether
TASC funds are used? If so, how so?
Addresses Evaluation Question(s): 2a
11. Does EPA maintain existing documentation on selection criteria for TASC funds?
c. If so, could you provide us with documentation on this issue?
Addresses Evaluation Question(s): 2c
12. Are there more communities that meet the criteria for receiving TASC assistance
than funds available?
a. If so, how are award decisions made between qualifying groups?
49
-------
b. Are communities ever asked to "apply" for TASC?
Addresses Evaluation Question(s): 2c
General Questions
13. Does the Region track community engagement activities at state-led sites?
d. If yes, what information is available on frequency of engagement, types
of engagement, etc.? Can you share it with us?
14. Do you provide any in-kind community assistance, or assistance outside of
formal grant programs, at RCRA CA sites that are led by the states?
e. If so, please elaborate on the types of in-kind assistance you provide.
Addresses Evaluation Question(s): 2d
15. We plan on conducting interviews with 6 delegated states; are there any states
that you recommend in particular? (The only states we do not want to speak
with are CT, MS, SC, and MA because of research already available on programs
in those states).
a. If yes, why?
Addresses Evaluation Question(s): 4
16. We plan on conducting community satisfaction surveys with 9-12 communities
in delegated states and 2-3 in non-delegated states. We will be using the same
format that Superfund's contractor, Skeo Solutions, has been using for years to
assess communities' satisfaction with TASC assistance. This format consists of
asking communities their level of satisfaction with the technical assistance
they've received on a scale of 1 to 6, with 6 meaning very satisfied and 6
meaning very unsatisfied. Are there any communities that you recommend in
particular?
a. If yes, why?
Addresses Evaluation Question(s): 4
17. Have you identified any gaps in EPA's technical assistance coverage at RCRA CA
sites in your Region?
a. If so, please identify these gaps.
Addresses Evaluation Question(s): 2e
18. Have you identified any gaps in EPA's information dissemination coverage at
RCRA CA sites in your Region?
b. If so, please identify these gaps.
50
-------
Addresses Evaluation Question(s): 2e
19. One of the goals of the evaluation is to develop measures for EPA to use moving
forward to assess the success of OSWER's TA and ID practices. In your opinion,
what constitutes successful TA and/or ID?
a. Do you currently track any metrics that might help assess the extent to
which the TA and/or ID efforts conducted by EPA are successful?
b. Are there any metrics that you are not currently tracking that might help
assess the extent to which the TA and/or ID efforts conducted by EPA
are successful?
Addresses Evaluation Question(s): 5 and 6
20. Does your Region use the EPA Conflict Resolution and Prevention Center? Why
or why not?
Addresses Evaluation Question(s): 2a
21. Do you have any other thoughts that you would like to share with us to inform
our project?
51
-------
INTERVIEW GUIDE: RCRA CORRECTIVE ACTION STATE CONTACTS
Introductory Questions
1. Can you briefly summarize your position as it relates to RCRA Corrective Action
within your state?
2. Can you briefly summarize your activities and responsibilities related to public
involvement at RCRA CA sites?
3. Is EPA required to provide ID and TA to communities?
a. If so, what are the requirements?
b. Are there additional drivers to provide ID and TA outside of formal
requirements?
i. If so, what are these drivers?
Addresses Evaluation Question(s): 1
Questions on Standard Community Involvement Practices at State-Lead Sites
4. Does the state provide public notice at every RCRA CA site?
a. If not, what are the key factors in determining whether a public notice is
issued?
Addresses Evaluation Question(s): 2a
5. Does the state provide an opportunity for public comment at every RCRA CA
site?
a. If not, what are the key factors in determining whether there is an
opportunity for public comment?
Addresses Evaluation Question(s): 2a
6. Does the state hold public meetings to address the community's concerns at
every RCRA CA site?
a. If not, does the state hold public meetings to address the community's
concerns at any RCRA CA sites?
i. If so, please provide examples of sites for which your state held
public meetings and,
ii. Provide the key factors in deciding whether to hold public
meetings.
Addresses Evaluation Question(s): 2a
52
-------
7. Does the state have any formal TA or ID programs for communities?
a. If so, please elaborate on each formal TA or ID program.
Addresses Evaluation Question(s): 2a
8. Do you work with EPA to conduct community engagement at state-lead sites?
Addresses Evaluation Question(s): 2a
9. Do you provide any in-kind community assistance, or assistance outside of
formal grant programs, at RCRA CA sites, outside of formal programs in your
state?
a. If so, please elaborate on the types of in-kind assistance you provide.
Addresses Evaluation Question(s): 2d
10. Are there other community involvement practices that your state takes when
directly implementing corrective action, specifically involving information
dissemination or technical assistance?
a. If so, please provide:
i. To what extent are these practices implemented.
ii. If they are not universally implemented, examples of sites at
which they were implemented, and
iii. The key factors in deciding whether to implement these
practices at specific sites.
Addresses Evaluation Question(s): 2a
11. We plan on conducting community satisfaction surveys several communities.
We will be using the same format that Superfund's contractor, Skeo Solutions,
has been using for years to assess communities' satisfaction with TASC
assistance. This format consists of asking communities their level of satisfaction
with the technical assistance they've received on a scale of 1 to 6, with 6
meaning very satisfied and 6 meaning very unsatisfied. Are there any
communities in your state that you recommend in particular?
a. If yes, why?
Addresses Evaluation Question(s): 4
12. Do you have any other thoughts that you would like to share with us to inform
our project?
53
-------
INTERVIEW GUIDE: BROWNFIELDS HQ CONTACTS
Introductory Questions
1. Can you briefly summarize your position within Brownfields?
2. Can you briefly summarize your activities and responsibilities related to public
involvement at Brownfields sites?
3. Is EPA required to provide ID and TA to communities?
a. If so, what are the requirements?
b. Are there additional drivers to provide ID and TA outside of formal
requirements?
i. If so, what are these drivers?
Addresses Evaluation Question(s): 1
4. Does Brownfields directly provide any in-kind community assistance at
Brownfields sites, outside of grantee-led activities?
a. If so, please elaborate on the types of in-kind assistance you provide.
Addresses Evaluation Question(s): 2d
5. We understand that OBLR uses scoring criteria for community involvement
within grant RFPs. How does EPA know that community involvement activities
identified in grant applications are carried out?
a. Are community involvement activities covered in regular grant
reporting?
i. If yes, could you share data available?
b. If a community does not carry out involvement activities specified, does
EPA take any action?
Addresses Evaluation Question(s): 1
6. Outside of grant reporting, has EPA collected any qualitative information from
grantees about community involvement that you can share with us?
Addresses Evaluation Question(s): 2a
7. Have you identified any gaps in EPA's technical assistance coverage at
Brownfields sites?
a. If so, please identify these gaps.
54
-------
Addresses Evaluation Question(s): 2e
8. Have you identified any gaps in EPA's information dissemination coverage at
Brownfields sites?
a. If so, please identify these gaps.
Addresses Evaluation Question(s): 2e
9. One of the goals of the evaluation is to develop measures for EPA to use moving
forward to assess the success of OSWER's TA and ID practices. In your opinion,
what constitutes successful TA and/or ID?
a. Do you currently track any metrics that might help assess the extent to
which the TA and/or ID efforts conducted by EPA are successful?
b. Are there any metrics that you are not currently tracking that might help
assess the extent to which the TA and/or ID efforts conducted by EPA
are successful?
Addresses Evaluation Question(s): 5 and 6
10. Do you have any other thoughts that you would like to share with us to inform
our project?
55
-------
INTERVIEW GUIDE: BROWNFIELDS REGIONAL CONTACTS
Introductory Questions
1. Can you briefly summarize your position within Brownfields?
2. Can you briefly summarize your activities and responsibilities related to public
involvement at Brownfields sites?
3. Is EPA required to provide ID and TA to communities?
a. If so, what are the requirements?
b. Are there additional drivers to provide ID and TA outside of formal
requirements?
i. If so, what are these drivers?
Addresses Evaluation Question(s): 1
4. Does Brownfields directly provide any in-kind community assistance, or
assistance outside of formal grant programs, at Brownfields sites?
a. If so, please elaborate on the types of in-kind assistance you provide.
Addresses Evaluation Question(s): 2d
5. We understand that OBLR uses scoring criteria for community involvement
within grant RFPs. How does EPA know that community involvement activities
identified in grant applications are carried out?
a. Are community involvement activities covered in regular grant
reporting?
i. If yes, could you share data available?
b. If a community does not carry out involvement activities specified, does
EPA take any action?
Addresses Evaluation Question(s): 1
6. Outside of grant reporting, has EPA collected any qualitative information from
grantees about community involvement that you can share with us?
Addresses Evaluation Question(s): 2a
7. What are the methods of referral that you typically employ to communicate the
availability of TAB to communities in your Region?
a. For each method, how often do you typically refer a community to the
TAB grantee using this method of referral?
Addresses Evaluation Question(s): 2a, 2b
56
-------
8. Have you identified any gaps in EPA's technical assistance coverage at
Brownfields sites?
a. If so, please identify these gaps.
Addresses Evaluation Question(s): 2e
9. Have you identified any gaps in EPA's information dissemination coverage at
Brownfields sites?
a. If so, please identify these gaps.
Addresses Evaluation Question(s): 2e
10. One of the goals of the evaluation is to develop measures for EPA to use moving
forward to assess the success of OSWER's TA and ID practices. In your opinion,
what constitutes successful TA and/or ID?
a. Do you currently track any metrics that might help assess the extent to
which the TA and/or ID efforts conducted by EPA are successful?
b. Are there any metrics that you are not currently tracking that might help
assess the extent to which the TA and/or ID efforts conducted by EPA
are successful?
Addresses Evaluation Question(s): 5 and 6
11. Do you have any other thoughts that you would like to share with us to inform
our project?
57
-------
INTERVIEW GUIDE: TAB GRANTEES
Note: We may further customize this guide based on research on specific TAB grantees.
1. How many communities has your organization provided technical assistance
(TA) to since it was awarded the TAB grant?
Addresses Evaluation Question(s): 2a and 2b
2. What kinds of TA does your organization typically provide?
a. For example, for help obtaining a grant, or help with technical matters in
relation to a Brownfields site, etc.
Addresses Evaluation Question(s): 2a and 2b
3. How long does the assistance typically last?
Addresses Evaluation Question(s): 2a and 2b
4. What are the criteria that you use to select which communities receive TA and
which do not?
a. When resources are not available to provide assistance to all
communities seeking it, how do you determine which groups to assist?
Addresses Evaluation Question(s): 2c
5. Through which communication methods do you promote the availability of your
services to communities?
Addresses Evaluation Question(s): 2a
6. lEc is planning on talking with up to 10 communities served by TAB grantees to
administer a community satisfaction survey. We will be using the same format
that Superfund's contractor, Skeo Solutions, has been using for years to assess
communities' satisfaction with TASC assistance. This format consists of asking
communities their level of satisfaction with the technical assistance they've
received on a scale of 1 to 6, with 6 meaning very satisfied and 6 meaning very
unsatisfied. Are there particular communities that you have conducted
extensive TA with?
a. If so, please provide:
i. The names of these communities
ii. A point of contact and contact information for each
iii. A brief description of the assistance provided
Addresses Evaluation Question(s): 4
58
-------
INTERVIEW GUIDE: CONFLICT RESOLUTION PREVENTION CENTER (CRPC)
1. How many sites have you provided assistance to?
a. What types of sites do you typically provide assistance to?
Addresses Evaluation Question(s): 2a
2. What types of assistance do you typically provide to regions that contact you?
Addresses Evaluation Question(s): 2a
3. What communication methods do you use to promote your services to the
Regions?
Addresses Evaluation Question(s): 2a
4. Do you ever turn down requests due to resource limitations?
Addresses Evaluation Question(s): 2c
5. What are your criteria for providing services at a site?
Addresses Evaluation Question(s): 2c
6. Do you have materials that track the number of sites that have contacted the
CRPC?
a. If so, do these materials include the date range in which you provided
assistance to each site?
b. If so, do these materials outline the type of assistance provided to each
site?
c. Please provide available documentation.
Addresses Evaluation Question(s): 2a
7. In your opinion, what constitutes successful conflict resolution?
a. Do you currently track any metrics that might help assess the extent to
which the conflict efforts conducted by the CRPC are successful?
b. Are there any metrics that you are not currently tracking that might help
assess the extent to which the conflict resolution efforts conducted by
the CRPC are successful?
Addresses Evaluation Question(s): 5 and 6
59
-------
INTERVIEW GUIDE: GENERAL COMMUNITY SATISFACTION TEMPLATE
lEc adapted this general template from the existing Skeo community satisfaction
interview guides used for specific TASC sites.
1. Using a scale of 1 to 6, please rate how satisfied you are with the quality of
assistance provided by [name of individual technical advisor or technical
advisory groupl's facilitation efforts for the [name of meeting], where "1" means
you are "very dissatisfied" and "6" means you are "very satisfied." Please
elaborate on your level of satisfaction.
2. Using a scale of 1 to 6, please rate how satisfied you are with the quality of the
presentations given by [name of individual technical advisor or technical
advisory group] as part of the [name of meeting]. "1" means "not at all satisfied"
and "6" means "very satisfied." Please elaborate on your level of satisfaction.
3. Using a scale of 1 to 6, please indicate how satisfied you are with [name of
individual technical advisor or technical advisory groupl's efforts to identify
potential issues of community concern in the [formal decision-making
document, such as a consent decree]. "1" means "not at all satisfied" and "6"
means "very satisfied." Please elaborate on your level of satisfaction.
4. Using a scale of 1 to 6 where "1" means you are "very dissatisfied" and "6"
means you are "very satisfied/' please indicate how satisfied you are with the
quality of [name of document or other resource] prepared by [name of TAG
technical advisor or technical advisory group]. Please elaborate on your level of
satisfaction.
5. Using a scale of 1 to 6, please rate how satisfied you are with the overall quality
of assistance provided by [name of technical advisor or technical advisory group]
to support [community name]. "1" means you are "very dissatisfied" and "6"
means you are "very satisfied." Please elaborate on your level of satisfaction.
6. How, if at all, could [name of technical advisor or technical advisory group] have
improved the quality of services it provided in support of the fname(s) of
pro ject/document/meeting]. Please elaborate on your level of satisfaction.
7. Have you identified any areas in which EPA's technical assistance or information
dissemination is lacking?
60
-------
EXAMPLE INTERVIEW GUIDE: SITE-SPECIFIC COMMUNITY SATISFACTION TEMPLATE
FOR BROWNFIELDS TAB COMMUNITY
Site Information
Name: Oak Grove Neighborhood
Location: Kansas City, Kansas
Assisted Organization: Oak Grove Neighborhood Association (Clintel Betts)
Assisting Organization: Kansas State University (KSU) (Wendy Griswold)
Start Date: January 2001
Project Narrative: http://www.engg.ksu.edu/CHSR/outreach/tab/sites/oakgrove.html
1. Using a scale of 1 to 6, please rate how satisfied you are with the quality of
assistance provided by Wendy Griswold's facilitation efforts for the landfill reuse
workshop, where "1" means you are "very dissatisfied" and "6" means you are
"very satisfied." Please elaborate on your level of satisfaction.
2. Using a scale of 1 to 6, please rate how satisfied you are with the quality of the
presentations given by Wendy Griswold as part of the landfill reuse workshop.
"1" means "not at all satisfied" and "6" means "very satisfied." Please elaborate
on your level of satisfaction.
3. Using a scale of 1 to 6 where "1" means you are "very dissatisfied" and "6"
means you are "very satisfied/' please indicate how satisfied you are with the
quality of the regional park development poster prepared by Wendy Griswold.
Please elaborate on your level of satisfaction.
4. Using a scale of 1 to 6, please rate how satisfied you are with the overall quality
of assistance provided by Wendy Griswold to support Oak Grove neighborhood.
"1" means you are "very dissatisfied" and "6" means you are "very satisfied."
Please elaborate on your level of satisfaction.
5. How, if at all, could Wendy Griswold have improved the quality of services she
provided in support of the Oak Grove landfill reuse project. Please elaborate on
your level of satisfaction.
6. Have you identified any areas in which EPA's technical assistance or information
dissemination is lacking?
61
-------
APPENDIX B: INTERVIEW SUMMARY COUNT
Superfund
Superfund HQ: 2
Superfund Regional: 10
Satisfaction Interviews with TAG Recipients: 9
RCRA CA
RCRA CA HQ: 3
RCRA CA Regional: 10
Delegated States: 6
Satisfaction Interviews: 9
Brownfields
Brownfields HQ: 2
Brownfields Regional: 10
TAB and other K6 Grantees: 8
Satisfaction Interviews with TAB and other K6 Community Contacts: 8
Other Interviews
Skeo Solutions: 1
CPRC: 1
62
------- |