SffiA
United States
Environmental Protection
Agency
Office of Policy
(1807T)
October 2013
EPA-100-K-15-001
           Formative Evaluation of
           the OSWER Community
           Engagement Initiative
           Final Report
           October 28, 2013

-------
ACKNOWLEDGEMENTS
The Formative Evaluation of the OSWER Community Engagement Initiative was developed
for the U.S. Environmental Protection Agency's Office of Policy under Contract EP-W-10-
002 between EPA and Industrial Economics, Inc. (lEc) of Cambridge, MA. The evaluation
team was comprised of the Office of Policy's Evaluation Support Division (ESD), the Office
of Solid Waste and Emergency Response (OSWER), and lEc. Angela Helman,  Peter
Courtright, and Josh Wolff represented lEc. lEc proposed the evaluation methodology,
collected and analyzed data, summarized findings, and developed this evaluation report
under the oversight of ESD and with guidance from the entire evaluation project team.
Michelle Mandolia, the Work Assignment Manager, provided oversight on behalf of ESD.
Ellen Manges and Jackie Harwood from OSWER provided overall guidance and direction.
Special thanks go to the many interviewees from EPA, state environmental agencies, and
community groups, who provided critical information that informed the evaluation.
This report was developed under the Program Evaluation Competition, sponsored by
EPA's Office of Policy. To access copies of this or other EPA program evaluations, please
go to ESD's Web site at http://www.epa.gov/evaluate

-------
TABLE OF CONTENTS


ACKNOWLEDGEMENTS  i

EXECUTIVE SUMMARY  ES-1

CHAPTER 1  |   INTRODUCTION AND EVALUATION  SCOPING
Evaluation Scoping  1
   Exclusion of the Underground Storage Tank (UST) Program 2
   Superfund, Brownfields, and RCRA Corrective Action Programs Have Different Structures
   That Affect Data Availability and Collection 3
   Survey Work was Impractical 4
Evaluation Questions 4
Logic Models 5

CHAPTER 2  |   METHODOLOGY
Review  of Existing Data 10
Interviews 10
   Thematic Analysis of Interview Data 12
Strengths and Weaknesses of Methodology 12

CHAPTER 3  |   FINDINGS
Question 1: What Are the Requirements and Drivers for Community Involvement (CI)
within the Superfund, RCRA Corrective Action (CA), and Brownfields Programs? 14
   Summary of Methods to Address Question 1  14
   Findings on Requirements 14
   Findings on Additional Drivers  15
Question 2: What Is the Baseline of Current OSWER Technical Assistance and
Information Dissemination Activities 15
   Summary of Methods to Address Question 2  16
   Findings on Frequency of Practice and Program-to-Program Variability 16
      Superfund  16
      Brownfields 17
      RCRACA 18
   Findings on Proportion of Communities that Receive Formal Assistance through TAG.
   TASC, or TAB  18
      Superfund  18

-------
      RCRACA 18
      Brownfields 20
   Findings on the Selection Criteria and on-the-Ground Process for Accessing TAG and TASC 20
      TAG 20
      TASC 20
   Findings on Community Assistance Provided Outside of Formal Programs 21
      Superfund 21
      Brownfields 21
      RCRACA 22
   Findings on Unmet TA or ID Needs 22
      Superfund 22
      Brownfields 23
      RCRACA 23
Question 3: How Can Superfund Community Involvement Plans (CIPs) Be Used to
Improve Technical Assistance and Delivery of Information?  23
   Summary of Methods to Address Question 3 23
   Findings on CIP Implementation 24
   Findings on CIP Revisions  24
   Findings on CIPs as a Tool to Ensure Effective Technical Assistance and Information
   Dissemination 24
Question 4: What Is the Baseline of Customer Satisfaction with  OSWER Technical
Assistance and Information Dissemination Activities? 25
   Summary of Methods to Address Question 4 25
   Findings on Interview Satisfaction 25
   Findings on Suggestions for Improving Services 27
      Superfund 28
      Brownfields 26
      RCRACA 29
Question 5: What Measures Can be Used to Assess the Effectiveness and Tangible
Outcomes of OSWER Technical Assistance and Information Dissemination Activities
Across the Lifecycle of Site Planning, Remediation, and Reuse? 29
Question 6: How Can these Measures be Used to Improve  OSWER Technical Assistance
and Information Dissemination Activities?  29
   Summary of Methods to Address Question 5 and 6 29
   Interview Findings of Relevance for Developing Measures 29
   Approach to Developing Menu of Potential Measures 31
   Menu of Potential Measures 31
      Output Measures 31
      Short-term Outcomes 33
      Long-term Outcomes 33
                                                                                    111

-------
APPENDICES:

Appendix A: Methodology Document
Appendix B: Interview Summary Count
                                                                       IV

-------
ACRONYM LIST


CEI: Community Engagement Initiative
CAG: Community Advisory Group
CARE: Community Action for a Renewed Environment
CPRC: Conflict Prevention and Resolution Center
CIP: Community Involvement Plan
ESD: Evaluation Support Division
HQ: Headquarters
ID: Information Dissemination
lEc: Industrial Economics, Incorporated
NEPA: National Environmental Policy Act
NCP: National Contingency Plan
OEJ: Office of Environmental Justice
OSWER: Office of Solid Waste and Emergency Response
OW: Office of Water
PRP: Potentially Responsible Party
RCRA CA: Resource Conservation and Recovery Act Corrective Action
RP: Responsible Party
TA: Technical Assistance
TAB: Technical Assistance for Brownfields
TAG: Technical Assistance Grants
TASC: Technical Assistance Services for Communities
TBA: Targeted Brownfields Assessment
UST: Underground Storage Tank

-------
EXECUTIVE SUMMARY
In December 2009, EPA's Office of Solid Waste and Emergency Response (OSWER)
announced the Community Engagement Initiative (CEI) to enhance OSWER HQ and
Regional offices' engagement with local communities and other stakeholders,  and to
help stakeholders meaningfully participate in decision-making processes related to the
cleanup and reuse of contaminated sites. The CEI is comprised of 16 actions that affect
many aspects of OSWER's work. This evaluation focuses on Action 7 and Action 13A,
which address improving technical assistance (TA) and information dissemination (ID),
respectively.1 The CEI defines technical assistance (TA) as the provision of services,
resources, and training focused on increasing community understanding of the relevant
science, regulations and policy related to environmental  issues.2 Information
dissemination (ID) refers to communication from EPA or  state agencies to communities
regarding environmental issues; CEI focuses on delivering the right information, in the
right place, at the right time into existing guidance, training, and Agency outreach
efforts.3  Furthermore, this evaluation focuses on Superfund, RCRA Corrective  Action
(CA), and Brownfields programs within OSWER.
CEI  includes a commitment to evaluate the effectiveness of OSWER program community
engagement activities.  OSWER envisions highly effective community engagement
becoming a standard business practice across all of its programs,  and furthermore wants
to share  lessons on successful community engagement throughout the Agency. As such,
OSWER requested a formative evaluation for the CEI program. OSWER's goals  for the
evaluation were to establish a baseline of current community satisfaction with EPA's TA
and ID practices, and to consider the feasibility of potential measures for tracking TA
and ID moving forward.

EVALUATION  QUESTIONS
The evaluation questions that we used to guide this evaluation are as follows:
1.  What are the requirements and drivers  for community involvement within the
    Superfund, RCRA Corrective Action, and Brownfields programs?
2.  What is the baseline of current OSWER technical assistance and information
    dissemination activities, with respect to:
    a.  Frequency of practice and program-to program variability?
    b.  Proportion of communities that receive formal assistance through TAG, TASC, or
       other formal program?4
1 ID is also sometimes referred to as Delivery of Information, or Dol.
2 EPA CEI, Action 7- Evaluate and Improve EPA Technical Assistance Processes, Work Group Report on Recommendations,
 available at: http://www.epa.gOV/oswer/engagementinitiative/7 action recommendation report.pdf
3 EPA CEI, Action 13A- Delivery of Information, Summary of Draft Work Group Report, available at:
 http://www.epa.gov/oswer/engagementinitiative/13a action summary report.pdf
4 Formal technical assistance refers to TA that is delivered under the auspices of a TA program such as TAG, TASC, and TAG;
 information TA refers to TA delivered outside of a program.
                                                                              ES-1

-------
    c.  Selection criteria and on-the-ground process for accessing TAG and TASC (for
       Superfund and RCRA CA only)?
    d.  Community assistance that is provided outside of formal programs (including
       helping to set up CAGs and providing ad-hoc assistance)?
    e.  Areas of unmet TA or ID need?
3.   How can Superfund community involvement plans (CIPs) be used to improve
    technical assistance and delivery of info?
    a.  Does every site have a CIP?
    b.  What information is available on the implementation of CIPs?
    c.  Are CIPs revised over time?
    d.  Do CIPs, as they are currently used, ensure effective technical assistance and
       information delivery throughout the life of the project? Why or why not?
4.   What is the baseline of customer satisfaction with OSWER technical assistance and
    information dissemination activities?
5.   What measures can be used to assess the effectiveness and tangible outcomes of
    OSWER technical assistance and information dissemination activities across the
    lifecycle of site planning, remediation, and reuse?
6.   How can these measures be used to improve OSWER technical assistance and
    information dissemination activities?

METHODOLOGY
The methodology for this evaluation combines a thematic analysis of qualitative data
gathered from interviews with a review and synthesis of existing documentation. The
complete methodology is included in Appendix A; it was completed in July of 2012.
Important methodological updates that occurred after finalizing the evaluation
methodology are discussed in Chapter 2.
lEc reviewed existing documentation and data to inform our responses to evaluation
questions, as discussed in the methodology. We reviewed the following data sources:
    •   Documentation of community engagement requirements;
    •   Site inventory data for cleanup programs;
    •   Performance measurement resources (for Questions 5 and 6); and
    •   Existing satisfaction interviews.
lEc conducted 46 interviews with EPA HQ and regional contacts from Superfund,
Brownfields, RCRA CA, as well as state RCRA CA contacts in Arkansas, California, Illinois,
Missouri, New York, and South Carolina. lEc also conducted satisfaction interviews with
contacts from community groups that had received technical assistance through the
Superfund TAG program, and analyzed these data in conjunction with previous
satisfaction interviews of service recipients conducted for the TASC contract. lEc
conducted a thematic analysis of interview data, and synthesized interview data with
                                                                            ES-2

-------
information gleaned from the document review as applicable, to develop evaluation
findings.

STRENGTHS AND WEAKNESSES OF METHODOLOGY
The methodology employed has a number of strengths, including coverage of all three
programs:  Superfund, RCRA CA, and Brownfields. Notably, the methodology employed
for this evaluation was sufficient for addressing most of the research questions,
including all of Questions 1, 3, 5, and 6; and most of Question 2. Within Question 2 in
particular, because lEc was able interview one contact for each of the three programs, in
every region, we are able to accurately characterize the baseline of EPA's technical
assistance and information dissemination activities. However, because we only
interviewed six states that implement RCRA CA, and only had information on a few
additional states from existing documentation, we cannot be sure if our findings on
frequency of practice, and community  assistance provided outside of formal programs,
are representative of state RCRA CA programs overall.
For Question 4-establishing a baseline of satisfaction with technical assistance and
information dissemination among communities-a key strength of the evaluation was
the ability to combine analysis of new satisfaction interviews with existing interviews.
We also achieved geographic diversity  in our interviews. However, the methodology for
Question 4 was limited by resources available to conduct interviews, and difficulties in
identifying community contacts to interview, particularly within the RCRA CA program.
Subsequently, a key limitation of our approach is that results for Question 4 on
satisfaction with technical assistance and information dissemination cannot be
extrapolated to all communities served by OSWER programs. Ideally, lEc would have
conducted surveys or interviews using  a statistically valid sample of those served for
each OSWER program, to develop a true baseline of satisfaction with technical
assistance and information dissemination criteria.5 However, this approach would have
required extensive resources to develop contact data (in particular for RCRA CA
communities), and to develop and administer the survey or requisite number of
interviews. In addition, programs expressed reservations about broadly surveying
community groups affiliated with their programs, citing information burden.
Finally, the method by which lEc selected interviewees has inherent strengths and
weaknesses. For the TASC contract, lEc was able to analyze all applicable data collected
for Superfund to-date, which is a methodological strength. In addition, we were able to
randomly select Superfund TAG grantees to interview. In contrast, as discussed above,
lEc had to rely on TAB grantees to identify Brownfields community groups to interview,
and we had to rely on RCRA CA regional and state interviewees to identify RCRA CA
community groups to interview. Relying on these parties to identify interviewees can be
a source of bias,  although we have no evidence or suspicions that contacts cherry picked
community contacts to participate in this evaluation.
s In addition to surveying a statistically valid sample of those served, it would also be methodologically preferable to survey
 multiple stakeholders associated with each site. One individual interviewee may not represent the perspective of all local
 stakeholders.
                                                                             ES-3

-------
SUMMARY OF FINDINGS
High-level findings from the evaluation include the following:
    •   The EPA Superfund and Brownfields programs appear to have robust systems in
       place for delivering technical assistance and information to communities,
       including clear mandates and guidance, formal programs and mechanisms for
       delivering technical assistance, and adequate levels of EPA staffing and
       resources.
    •   RCRA CA appears to be meeting its mandate of providing information
       dissemination to community groups as required. However, in comparison to the
       Superfund and Brownfields programs, RCRA CA lacks many inputs helpful in
       ensuring the delivery of technical assistance to communities, including a lack of:
       regulatory mandates; adequate resources and staffing at EPA and state
       agencies; and up-to-date, program-specific guidance. Unmet community needs
       appear to be higher within the RCRA CA program than other programs, and
       satisfaction with technical assistance provided by states to communities under
       RCRA CA was rated lower on average by interviewees than for the other two
       programs.
    •   Compared to the other programs, it is more difficult to characterize the needs of
       RCRA CA communities nationally, and to track progress in meeting those needs,
       because the program is largely delegated to states, and EPA currently lacks
       mechanisms for collecting community engagement data from  states. Thus, if
       EPA were to conduct regular tracking of measures of unmet needs and
       customer satisfaction, the Agency would need to work with states to implement
       a data collection system. In contrast, existing data collection systems employed
       by Superfund and Brownfields could potentially be augmented to track
       suggested measures.

Findings Summary by Evaluation  Question
1.   What are the requirements and drivers for community involvement within the
    Superfund, RCRA Corrective Action, and Brownfields programs?
Community involvement requirements are documented in the National Contingency
Plan for Superfund and Brownfields.6 All Superfund sites must have a Community
Involvement Plan (CIP), and CIPs must be in place before remediation commences. The
Brownfields Program is required to provide community involvement opportunities for
sites that receive cleanup grants from EPA, as opposed to assessment grants, where
community involvement is not required. RCRA CA's community involvement
requirements are codified in a public participation manual for the program dating back
to 1996. RCRA CA is only required to  provide public notice during key phases of the
corrective action process; TA is not required. Beyond requirements, additional drivers of
6 NCR [40 CFR 300.430(2)(ii)]: states the following intent regarding community involvement: "(A) Ensure the public
 appropriate opportunities for involvement in a wide variety of site-related decisions, including site analysis and
 characterization, alternatives analysis, and selection of remedy; and (B) Determine, based on community interviews,
 appropriate activities to ensure such public involvement."
                                                                             ES-4

-------
community involvement across the three programs include community demand and
environmental justice concerns.
2.   What is the baseline of current OSWER technical assistance and information
    dissemination activities, with respect to:
    a.  Frequency of practice and program-to program variability? Superfund has the
       most extensive formal TA and ID practices of the three OSWER programs. CIPs
       are developed for each site and EPA has dedicated personnel in each region to
       assist Superfund communities. Superfund communities have access to both TAG
       and TASC. The Brownfields program requires its cleanup grantees, which are
       typically state and local governments, to provide TA, and the Brownfields
       program also administers the TAB program. RCRA CA complies with ID
       requirements, butTA is not required, and as such, TA is provided on a case-by-
       case basis based on perceived need and community interest. Because resource
       limitations precluded interviews with more than six states, it is important to
       understand that this evaluation cannot provide comprehensive information on
       RCRA CA community involvement at the state level.
    b.  Proportion of communities that receive formal assistance through TAG, TASC, or
       other formal program ?
Use of formal TA is summarized in Exhibit ES-1 below.
ES-1.  Use of  Formal Technical Assistance Programs
PROGRAM
Superfund
RCRA CA
Brownfields
FORMAL USE OF TA
Site count = 13,662 (1652 NPL site + 12,010 non-NPL sites)
• 350 TAGs awarded to and 56 TASC projects used at Superfund
sites
Site Count = 3747
• Formal TA is rarely used at RCRA CA sites
• Only seven RCRA CA sites have used TASC assistance
• No official program site count because all communities eligible
• 833 Brownfields communities have received TAB assistance
since 2008
    c.  Selection criteria and on-the-ground process for accessing TAG and TASC (for
       Superfund and RCRA CA only)?
       As a grant  program, the TAG program has formal selection criteria covering: the
       types of groups that are eligible to receive TAG funds; group administrative and
       management experience; group past performance with federal grants; and
       group legal incorporation requirements. In cases where multiple parties from
       the same community compete for TAG funds, EPA uses secondary criteria to
       make a selection; these include community representativeness, communication
       planning, and the potential for measurable environmental results.
                                                                          ES-5

-------
       Community groups do not apply for TASC directly; they receive access to the
       TASC contract services based on recommendations from regional staff to EPA
       HQ. HQ and regional interviewees within the Superfund and RCRA CA programs
       stated that there are no formal selection criteria for OSWER programs to access
       TASC. However, communities that are selected for TASC funds typically share
       one or more of the following conditions:
          •   A lack of other sources of funding for community TA needs;
          •   A high level of community interest in the cleanup process;
          •   Environmental justice concerns;
          •   Short-term technical assistance needs; and/or
          •   Superfund communities that are not interested in forming a TAG group.
    d.  Community assistance that is provided outside of formal programs (including
       helping to set up CAGs and providing ad-hoc assistance)?
       The Superfund program routinely provides TA outside of formal programs, in
       addition to providing extensive support within formal programs. Within the
       Brownfields program, communities with EPA cleanup grants receive assistance
       through the grantee (typically a local government entity). Nearly all TA provided
       to RCRA CA communities is conducted outside of formal programs. Regions and
       states administering RCRA CA make decisions about where to offer TA based on
       community requests and staff assessment of need.
    e.  Areas of unmet TA or ID need?
       Most interviewees identified some unmet TA and/or ID needs. However, as
       shown in Exhibit ES-2 below, interviewees noted a somewhat higher incidence
       of unmet needs among RCRA CA communities than among Superfund or
       Brownfields communities.
Exhibit  ES-2. Unmet ID and TA needs

RCRA CA
Superfund
Brownfields
NOTED ANY UNMET TA NEEDS
HQ, 8 Regions
HQ, 6 Regions
HQ, 4 Regions
NOTED ANY UNMET ID NEEDS
7 Regions
7 Regions
5 Regions
Moreover, RCRA CA HQ and eight of 10 regional contacts indicated that resource
limitations are a barrier to providing TA under RCRA CA; in contrast, only four
Brownfields contacts and two Superfund contacted indicated that lack of resources is an
impediment to delivering TA.
3.   How can Superfund community involvement plans (CIPs) be used to improve
    technical assistance and delivery of info?
                                                                          ES-6

-------
a.   Does every site have a CIP? According to all Superfund interviewees, every
    Superfund site has a CIP.
b.   What information is available on the implementation ofCIPs? According to all
    Superfund regional interviewees, CIPS are tracked and maintained by regional
    staff. CIP updates are variable. From the perspective of tenured EPA staff, CIPs
    are most valuable at the beginning of the Superfund process. From the
    perspective of community members and new EPA staff, CIPs and updates have
    value throughout the cleanup process. Interviewees noted that CIPs may be
    improved by more frequent, targeted revisions and by making them more
    readable by community members.
c.   Are CIPs revised over time? According to Superfund interviewees, CIPs are
    revised overtime but revisions sometimes lag project milestones.
d.   Do CIPs, as they are currently used, ensure effective technical assistance and
    information delivery throughout the life of the project? Why or why not? Six of
    10 regional interviewees said that CIPs are not an important tool for ongoing
    technical assistance and information delivery. The general sentiment among
    Superfund regional interviewees is that CIPs are helpful to organize community
    engagement at the beginning of the Superfund process, but are not used by EPA
    staff as a resource over time. EPA tends to set aside the document after the
    start of the project, instead relying on ad-hoc communication with community
    members.
    Because CIP updates often lag behind project changes, they may not ensure
    effective technical assistance and information delivery throughout the life of the
    project given lagging updates. However, as discussed under Evaluation Question
    4 findings, satisfaction with technical assistance and information dissemination
    is high among Superfund community contacts participating in this evaluation
    that have received a TAG, and across the Superfund community contacts that
    have used the TASC contract.
e.   What is the baseline of customer satisfaction with OSWER technical assistance
    and information dissemination activities? Community contacts rated their
    overall  satisfaction with assistance services provided a scale of one to six; with a
    score of one being very dissatisfied to a score of 6 being very satisfied. As shown
    in Exhibit ES-3 below, interviewees noted consistently high satisfaction overall
    scores for Superfund and Brownfields programs, while scores for RCRA CA were
    more mixed. However, regarding RCRA CA, it should be noted that at the lower
    end, only two  interviews rated their overall satisfaction as a "1" with assistance
    provided by RCRA CA states; the majority of RCRA CA sites provided favorable
    overall  ratings for assistance received. Satisfaction rates are very similar
    between recipients of TAG grants and users of the TASC contract. Common
    suggestions from community contacts for improving TA included making
    outreach materials more accessible by using visual aids and plainer language;
    and reaching out to community members often or more actively.
                                                                        ES-7

-------
Exhibit  ES-3. Customer Satisfaction Score Summary
     80
     70
sj  60
01
2  50
a
£  40
|  30
8  20
01
°-  10
   0
                        Overall Score Distribution by Program
         RCRA
                                                                             (9 Interviews)
                                                                I Superfund    (9 Interviews)

                                                                I Brownfields   (8 Interviews)
       Very
    Unsatisfied
                              3       4
                           Likert Scale Score
  Very
Satisfied
              4.   What measures can be used to assess the effectiveness and tangible outcomes of
                  OSWER technical assistance and information dissemination activities across the
                  lifecycle of site planning, remediation, and reuse?
              5.   How can these measures be used to improve OSWER technical assistance and
                  information dissemination activities?
                  lEc developed a potential menu of output, short-term outcome, and long-term
                  outcome measures for tracking the progress of community engagement in OSWER
                  programs. We focused output measures on tracking unmet needs, and outcome
                  measures on gauging community satisfaction.  lEc also considered other types of
                  measures, but honed in on the measures presented in Exhibit ES-4 after considering
                  feasibility factors including:
                     •   The ability to define objective measure(s);
                     •   The need to develop and implement new information collection
                         infrastructure;
                     •   Feedback from interviewees on potential measures; and
                     •   lEc's professional judgment regarding the likelihood of the measure being
                         accepted and successfully implemented by the OSWER programs.
                  It is important to note that lEc does not suggest that EPA consider or adopt all of the
                  measures in Exhibit ES-4.
                                                                                         ES-8

-------
Exhibit ES-4. lEc Recommendations for a Menu of Potential Measures
OUTPUT MEASURES
    •   Proportion of Superfund communities that applied for TAG assistance, but have not
        received assistance
    •   Proportion of Brownfields communities that requested TAB or other form of K6
        assistance, but have not received assistance
    •   Proportion of Superfund and RCRA CA communities that were recommended by a
        region for TASC assistance, but have not received assistance
    •   Proportion of RCRA CA communities that have asked a state or region for any form of
        assistance, but have not receive  assistance
    •   Number/Percent of RCRA CA communities that have received TA
    •   Number/Percent EPA Regions that have offered TA/ID training in the past year

SHORT-TERM OUTCOME  MEASURES
    •   The number/proportion of community members filing complaints related to the
        provision of TA with Superfund/Brownfields/RCRA CA regional staff annually.

LONG-TERM  OUTCOME MEASURES
    •   Proportion of TAG/TASC/TAB communities that have received a community
        satisfaction survey that are "satisfied" with the information  provided by EPA
        throughout the cleanup process (tracked individually for each program).
    •   Proportion of TAGs/TASC/TAB/RCRA CA communities that have received a community
        satisfaction survey that are "satisfied" with the assistance provided by their technical
        advisor (each of the four measures tracked individually).
                                                                                 ES-9

-------
CHAPTER 1   |  INTRODUCTION AND EVALUATION SCOPING
In December 2009, EPA's Office of Solid Waste and Emergency Response (OSWER)
announced the Community Engagement Initiative (CEI) to enhance OSWER HQ and
Regional offices' engagement with local communities and other stakeholders, and to
help stakeholders meaningfully participate in decision-making processes related to the
cleanup and reuse of contaminated sites. CEI includes a commitment to evaluate the
effectiveness of OSWER program community engagement activities. OSWER envisions
highly effective community engagement becoming a standard business practice across
all of its programs, and furthermore wants to share lessons on successful community
engagement throughout the Agency. As such, OSWER requested a formative  evaluation
for the CEI program. A formative evaluation is a prospective evaluation designed to help
newer programs establish baseline conditions, and establish measures to track program
implementation  and progress over time.
The CEI is comprised of 16 actions that affect many aspects of OSWER's work. Exhibit 1-
1 summarizes these actions.7 This evaluation focuses on Action 7 and Action 13A
(highlighted in Exhibit 1-1), which address improving technical assistance (TA) and
information dissemination (ID), respectively. The CEI defines technical  assistance (TA) as
the provision of services, resources, and training focused on increasing community
understanding of the relevant science, regulations and  policy related to environmental
issues.8 Information dissemination (ID) refers to communication from EPA or state
agencies to communities regarding environmental issues; CEI focuses on delivering the
right information, in the right place, at the right time into existing guidance, training,
and Agency outreach efforts.9  Unlike some other CEI Actions that are program-specific,
Actions 7 and 13A are cross-program actions that can help communities participate in
OSWER processes more effectively. OSWER's goals for the evaluation were to establish a
baseline of current satisfaction with EPA's TA and ID practices, and to consider the
feasibility of potential measures for tracking TA and ID  moving forward.
EPA contracted with Industrial  Economics (lEc) to conduct the evaluation under the
oversight of the Agency's Evaluation Support Division (ESD). ESD, lEc, and
representatives from OSWER comprised the evaluation team. lEc prepared this
evaluation report, and the terms "we" and "our" in this report refer to the authors at
lEc.

EVALUATION SCOPING
lEc conducted  a scoping task to inform this project. First, we reviewed  publicly-available
information for each OSWER cleanup  program (Superfund,  Brownfields, RCRA Corrective
7 See OWSER's website about the Community Engagement Initiative at http://www.epa.gov/oswer/engagementinitiative/,
 last visited July 30, 2013.
8 EPA CEI, Action 7- Evaluate and Improve EPA Technical Assistance Processes, Work Group Report on Recommendations,
 available at: http://www.epa.gOV/oswer/engagementinitiative/7 action recommendation report.pdf
9 EPA CEI, Action 13A- Delivery of Information, Summary of Draft Work Group Report, available at:
 http://www.epa.gov/oswer/engagementinitiative/13a action summary report.pdf

-------
Action, and the Underground Storage Tank Program) to identify the formal and informal
processes through which the programs conduct TA and ID. This consisted of reviewing
program websites, guidance, regulations, and grant RFPs for program policies on TA or
ID. Then we conducted interviews with contacts at EPA HQfor each program to sharpen
our understanding of the framework through which these programs conduct TA and ID,
as well as to determine additional TA and ID policies that we did not see in our review of
publicly-available information. We also used these initial calls to determine the most
effective research methods to answer our evaluation questions.

Exhibit 1-1. Community Engagement Initiative (CEI)  Actions

Action 1
Action 2
Action 3
Action 4
Action 5
Action 6
Action 7
Action 8
Action 9
Action 10
Action 1 1
Action 12
Action 13
Action 14
Action 15
Action 16

Decision-Making Processes and Guidance
Underground Storage Tank (LIST) Programs
Best Community Engagement Practices for RCRA
OSWER Regulation and Guidance Development
Community Engagement Policies and Activities related to Enforcement
CERCLA Enforcement Involving Federal Facilities
EPA Technical Assistance Processes
Community Action for a Renewed Environment (CARE) Program
Brownfields Area-Wide Planning Pilot Program
Public Health Information on OSWER Projects
Risk Communication Processes and Comprehensive Education Program
Sampling and Testing Results
Delivery of Information:
13A: At-Riskand Remote Communities
13B: Superfund Repositories
Community Engagement Training Program
Measures of Effectiveness and Annual Report
Environmental Workforce Development and Job Training Program
Another aspect of our scoping work was to determine how concurrent efforts within
EPA relate to aspects of the evaluation. With many CEI initiatives in progress, the
evaluation team was concerned about overlap between this evaluation and other
efforts, particularly within RCRA Corrective Action (RCRA CA). Thus, we spoke to
programs about the status of ongoing CEI initiatives and scoped this evaluation
accordingly. The results of our scoping task are as follows:

Exclusion of the  Underground Storage Tank (UST) Program
lEc recommended excluding the UST program from the evaluation based on scoping
research. State UST programs do not conduct much TA; most UST sites are not
complicated or technically challenging enough to require it. In rare instances of a highly
contaminated tank site with community exposure, states assess the need for public
participation on a case-by-case basis. UST is aware of only one state that has funding for
conducting TA. Similarly, state UST programs typically do not conduct ID activities,
except in rare cases of community exposures.

-------
In cases of public exposures, states must notify affected communities, as reflected in
EPA UST regulations. However, even in this case, the regulations do not require much
interaction; they are summarized below (emphases are lEc's):
    •    (a) The public should be notified of every confirmed release that requires a
        corrective action plan.
    •    (b) "The implementing agency must ensure that site release information and
        decisions concerning the corrective action plan are made available to the public
        for inspection upon request."
    •    (d) "Before approving a corrective action plan, the implementing agency may
        hold a public meeting to consider comments on the proposed corrective action
        plan if there is sufficient public interest, or for any other reason."10
It should be noted that EPA does not have funding for conducting TA  at UST sites, nor
does the Agency provide guidance on the topic. One exception is Indian country, where
in some cases, EPA Regional staff communicate information regarding the UST program
to the tribal governments, who are then responsible for informing their communities
and the affected public.
Thus, the evaluation team agreed not to include UST in this evaluation; EPA HQ UST
contacts concurred with this decision. Factors in this decision included:
    •    General consensus that UST sites are low-risk and low priority;
    •    Lack of TA and ID activity at UST sites (except in cases of release);
    •    EPA's weak regulatory and policy framework regarding TA and ID at UST sites
        (TA is not required; and only limited ID is required); and
    •    Limited evaluation resources.
However, it should be noted that because the UST program was omitted from the
evaluation, no judgment has been made about the about the effectiveness of the
program's community engagement  activities.

Superfund, Brownfields, and RCRA Corrective Action Programs Have
Different Structures that Affect Data Availability and Collection
Superfund works directly with communities affected by Superfund sites, has ultimate
authority over the site remediation  process, and retains control over  community
engagement as part of that process. Superfund offers a grant program directly to
affected communities, the Technical Assistance Grant (TAG), and Superfund site
managers can also take advantage of the Technical Assistance Services for Communities
(TASC) Contract administered by OSWER (use of TAG and TASC is discussed in detail in
Chapter 3). Superfund sites are required to have a community involvement plan for each
site. As such, EPA Regional staff have direct contact with community organizations
involved in Superfund cleanup activities. This structure differs significantly from the
other OSWER programs.
10 40 CFR Ch. 1 (7-1 -05 Edition) S 280.67 Public Participation provisions from Subpart F—Release Response and Corrective
 Action for LIST Systems Containing Petroleum or Hazardous Substances (http://www.epa.gov/swerust1/fedlaws/280_f.pdf)

-------
The Brownfields program provides grants to local governments and non-profit
organizations to cleanup and reuse sites, and community engagement is required as
part of cleanup grants. So, recipients of EPA brownfields grants provide technical
assistance to community groups; EPA does not provide or administer this assistance. In
addition, the EPA Brownfields program administers the Brownfields Training, Research,
and Technical Assistance K6 Grant program, which among other things, provides funding
to academic and research organizations to work with community organizations on
public participation issues. The Technical Assistance for Brownfields (TAB) program is
the main component of the K6 grant program focused intently on TA and ID. Under TAB,
EPA provides grant support to four regional organizations, which in turn provide
technical assistance to Brownfields communities.
The RCRA Corrective Action (CA) program is delegated to 43 states. In delegated states,
at most RCRA CA sites, states take the lead on cleanup, including community
engagement (although EPA regions sometimes provide TA at sites in delegated states).
The RCRA CA program has official guidance on public participation, but it has not been
updated since 1996.n EPA does not track state-led RCRA CA community engagement
activities, and does not have direct contacts with community organizations involved in
site cleanup activities, except for the minority of sites where EPA is implementing the
cleanup, or in the few cases where a RCRA CA community has been provided with
assistance through the TASC contract. Thus, it was important for lEc to interview state
RCRA CA coordinators for this evaluation.

Survey Work was Impractical
Both the Superfund and Brownfields contacts expressed reservations about surveying
community groups affiliated with their program, but were more amenable to interviews.
In addition, Superfund has already done a significant amount of work on satisfaction for
TASC, using a well-constructed interview guide that asks participants to rate specific
aspects of TA and  ID. Upon reviewing the satisfaction questionnaire used by Superfund
for TASC, and talking with Superfund's TASC contractor, Skeo, lEc concluded that asking
specific and customized questions (e.g., about specific workshops, materials, and
communications) is the best strategy for ensuring that participants provide feedback on
the TA  and ID process, as opposed to cleanup or redevelopment outcomes, which EPA
typically cannot control. However, one cannot ask customized questions via a survey.
Thus, lEc recommended conducting interviews for Superfund TAG, Brownfields, and
RCRA CA using the same tool as Skeo, which provided us with consistent information to
compare across programs.

EVALUATION QUESTIONS
The evaluation questions that we used to guide this evaluation are as follows:
1. What are the requirements and drivers for community involvement within the
   Superfund, RCRA Corrective Action, and Brownfields programs?
11 RCRA Public Participation Manual, 1996 Edition, available at:
http://www.epa.gov/osw/hazard/tsd/permit/pubpart/manual.htni

-------
2.  What is the baseline of current OSWER technical assistance and information
    dissemination activities, with respect to:
    a.  Frequency of practice and program-to program variability?
    b.  Proportion of communities that receive formal assistance through TAG, TASC, or
       other formal program?
    c.  Selection criteria and on-the-ground process for accessing TAG and TASC (for
       Superfund and RCRA CA only)?
    d.  Community assistance that is provided outside of formal programs (including
       helping to set up CAGs and providing ad-hoc assistance)?
    e.  Areas of unmet TA or ID need?
3.  How can Superfund community involvement plans (CIPs) be used to improve
    technical assistance and delivery of info?
    a.  Does every site have a CIP?
    b.  What information is available on the implementation of CIPs?
    c.  Are CIPs revised over time?
    d.  Do CIPs, as they are currently used, ensure effective technical assistance and
       information delivery throughout the life of the project? Why or why not?
4.  What is the baseline of customer satisfaction with OSWER technical assistance and
    information dissemination activities?
5.  What measures can be used to assess the effectiveness and tangible outcomes of
    OSWER technical assistance and information dissemination activities across the
    lifecycle of site planning, remediation, and reuse?
6.  How can these measures be used to improve OSWER technical assistance and
    information dissemination activities?

LOGIC MODELS
To illustrate the various components of community engagement in OSWER, and
specifically the ID and TA tasks within the CEI, EPA and lEc developed three logic models
(i.e., graphical representations of the relationships  between program inputs,  outputs,
and intended changes in knowledge/attitude, behavior, and condition). The first logic
model (Exhibit  1),  illustrates the design of the community engagement process within
OSWER. It does not represent how community engagement necessarily works at all
sites; instead, the  model represents the process through which community engagement
is designed to function. The next two logic models (Exhibits 2 and 3), outline the design
of CEI Actions 7 (TA) and 13A (ID), which are the foci of this evaluation. These two logic
models illustrate the conceptual design of how these Actions will be implemented to
achieve their expected results. All three models are needed to understand the
community involvement process and how Actions 7 and 13A are intended to  improve
the process.

-------
The key components of the models include:
    •   Resources —  staff, contractor support, partners, and funds dedicated to the
       program.
    •   Activities — the specific procedures or processes used to achieve program
       goals. OSWER's community engagement activities consist of developing site-
       specific Community Engagement Plans and continually updating the plans as
       community needs change. For the CEI Actions 7 and 13A, activities focus on
       examining current ID and TA practices for opportunities for expansion and
       revision.
    •   Outputs — the immediate products that  result from activities, which are often
       used to measure short-term  progress. OSWER's community engagement
       outputs consist of holding public meetings, generating documents for public
       consumption, and providing technical assistance to communities. Example
       outputs of the CEI Actions are community-centric training modules, brochures,
       and expansion  of existing formal technical  assistance programs.
    •   Target Audiences — groups and individuals targeted by community
       engagement activities and outputs. For example, OSWER's community
       engagement audiences include community members, community-based
       organizations, and both local and tribal governments. In addition to these
       groups, some activities and outputs of CEI  Actions 7 and 13A target EPA staff
       and delegated state programs.
    •   Short-Term Outcomes — changes in knowledge, awareness, attitudes,
       understanding, and skills resulting from program outputs that are causally
       linked to community engagement. For example, CEI Action 7 is designed to
       increase stakeholders' awareness of EPA's technical assistance programs.
    •   Intermediate Outcomes —  changes in behavior resulting from changes in
       knowledge and attitude. For  example, CEI Action 7 is designed to increase
       stakeholders' use of EPA's technical assistance programs.
    •   Long-Term Outcomes —  the overarching goals of the program, which in the
       case of the CEI  Actions 7 and 13A is to ensure the community has a more
       informed voice in decisions at contaminated sites within their community.
Finally, the logic model makes note of external factors that are beyond the direct
control of EPA's community engagement program, but may influence program
outcomes. For example, external factors for Action 13A may include available resources
and lack of community organization.

-------
 Exhibit 1-2.  LOGIC MODEL FOR COMMUNITY ENGAGEMENT IN OSWER

Goal: To facilitate community members' participation in government decisions on land cleanup, land reuse, and emergency response.

   Resources
    OSWER staff
   State staff and
 delegated programs

 Superfund guidance
  on how to create a
    Community
  Involvement Plan
  EPA Information
  Access Strategy
  "Partnering with
Communities: Federal
 Models of Partnering
 with Communities"

  Funding for grants
  and contractors

  EPA Community
 Involvement Training
    Conference
    Community
 Involvement Toolbox
    Brownfields
    Conference
    Community
    Involvement
   Activities
     Outputs
                                                Target Audienc
     Develop and
  implement project-
  specific Community
   Engagement Plans
*  based on current
   EPA community-
   engagement best
     practices and
     existing case
                           Regular public
                      ^  meetings to discuss _
                           site issues and
                          obtain feedback
                           Informational
                         documents and/or
                       ^    a website to
                             present
                           environmental
                            issues at site
                         Out-stationed EPA
                         or delegated state
                         employee to serve
                           as the point of
                           contact with
                            community
                             members

Continuously react
     to site
developments and
   update plan
   Access to formal
      technical
     assistance
    programs that
"  provide financial
    resources and
    independent
   technical advice
  (e.g. TAGS, TASCs,
                        ^    Community   	
                          I   Members
                             Community-
                               based
                          Local Government
                                                      Tribal
                                                   Governments
Short-Term Intermediate"! Long-Term
Outcomes Outcomes Outcomes
Increased
community
understanding of
^ environmental
issues at the site
and project
actions to address
these issues



Increased
community
_^. understanding of 	
how its
involvement can
shape the project
Increased
opportunityto
provide OSWER
^^ and/or delegated 	
Community
members
engage with
OSWER and/or
^ delegated state 	
to their level of
interest
throughout the
lifetime of the






OSWER and/or
delegated state
•^ incorporates the 	
community's
vision into
OSWER project
outcomes are
aligned with the
needs and goals
^ expressed by the
community
throughout the
engagement
process, to the
\extent


Projects
experience fewer
delays due to
^ differences in
qtakphnlrlpr
project " """""'. t
expectations for
site cleanup and
rai ICQ
state with
feedback about
community
  External Factors:  Budget constraints, limited existing community-engagement best practices, limited flexibility to incorporate community desires

-------
  Exhibit 1-3.  LOGIC MODEL FOR CEI ACTION 7:  EVALUATE AND IMPROVE EPA TECHNICAL ASSISTANCE PROCESSES
Goal: To improve and broaden the availability of technical assistance to communities so that community members can better understand site issues, and
participate in an informed way during the decision-making process.

     Resources
     OSWER staff
     EPA's Conflict
    Prevention and
   Resolution Center
  Existing EPA technical
   assistance programs
   (e.g. TAGS, TASCs,
     TAPS, TABs)
    Existing EPA best
  practices for technical
      assistance
Activities Outputs Target Audience Short-Term
1 	 	 1 1 	 	 1 Outcomes
"^7 	 ' "V" "V

Promote OSWER's
technical assistance Brochure and
capabilities to website to educate 	 .,
fr. increase 	 ^. communities about — i
communities' access technical assistance
to and use of opportunities
existing technical
assistance
^ Technical assistance


,
^ technical assistance ~~
resou rces





~ "Partners Program"

Expanded availability
^ ofTechnical
Assistance Plans and
TASCs

Tool kit for local
communities that
may be receiving
.. Community
Members


^ Community-based
Organizations


^ Local Government —

Tribal 	
muedbeu
community
stakeholder
"^ awareness of
available technical
assistance
programs


Broader availability
of technical
x^ assistance
programs to
Governments community
stakeholder groups
Mid-Level HQ and '• Teaerany-
regional program transferred property
managers discuss property or
^ opportunities to —
identify and resolve
technical assistance
issues and problems



Assess existing
technical assistance
^ practices to identify —
Formerly Used
Defense Sites ^ ^^ Increased



^ Guidance regarding ~~
EPA expectations
^f crrt oidii ^^


Delegated State
^ staff implementing —
awareness among
EPA and delegated
state staff of
•^ OSWER technical 	
assistance
and best practices OSWER programs programs and
on technical associated best
assistance practices
gaps and best
practices
Intermediate
Outcomes
Long-Term Outcomes
~^F~





Increased use of
available OSWER
technical
"^ assistance
programs among
communitv
stakeholders








Increased
implementation
of technical


Community
stakeholder Community
groups have a stakeholder
deeper groups have a
understanding of more informed
^ the relevant _^. voice in
science, decisions
regulation, and related to
policy related to environmental
environmental issues at
issues at OSWER OSWER sites
sites




-^ assistance best —
practices across
OSWER
programs


    External Factors: Limited resources, some communities may not be a good candidate for technical assistance because of: (1) a lack of organizational structure, (2) a
    lack of agreement among community stakeholders, and/or (3) assistance saturation within community.

-------
 Exhibit 1-4.  LOGIC MODEL FOR CEI ACTION 13A: EVALUATE AND IMPROVE DELIVERY OF INFORMATION - AT-RISK AND REMOTE
 COMMUNITIES

Goal: To develop options for improvement in how information is delivered to a-risk and remote communities and to enhance community members' knowledge and
ability to meaningfully participate in decision-making processes.

     Resources
      OSWER staff
      Current EPA
    relationships with
   leaders of at-risk and
   remote communities
       and tribes
      Existing EPA
      information
    dissemination key
        tools
      Existing EPA
      information
    dissemination best
       practices
Activities

Coordinate with OEI
on information
delivery mechanisms
for At-Risk and
Remote
Commu nities



Coordinate with
tribes that are close
to OSWER sites to
	 determine if there 	
are ways to improve
information delivery
to them

Hold video
teleconferences with
at-risk and remote
communities


Establish a national
workgroup to
identify existing
— information delivery 	
mechanisms to
identify gaps and
best practices


Outputs
V







Improved
information
delivery methods
using best 	
practices, key tools,
— ^ and innovative
ideas to increase
transparency and
stakeholder
certainty in the site
cleanup process
A






Community-centric
training module on
^ the delivery of
information to
remote and at-risk
communities

Target Audience
>r

At-Risk and
Remote
Community
Members



At-Risk and
Remote
Community-Based
Organizations


At-Risk and
	 Remote Local 	
Governments


At-Risk and
	 Remote Tribal 	
Governments




EPA Staff 	
Delegated State
— ^ staff implementing 	
OSWER programs

Short-Term
Outcomes








At-risk and remote
communities
_- receive the right
•^ information, in the
right place, at the
right time










Increased
awareness amongst
EPA and delegated
<^ state staff of
"^ OSWER information
delivery
mechanisms and
best practices
I Intermediate
Outcomes


At-risk and
remote
community
members are

more informed
-^ about H —
environmental
issues and
actions at
contaminated
sites.

More
stakeholder
confidence
•^ among at-risk 	
and remote
communities in
the site cleanup
process

j
^^^
Increased
implementation
^ of information
1 delivery best
practices across
OSWER programs

Long-Term
Outcomes
1 	 3 C 	
"














At-risk and remote
community

groups have a
^ more informed
voice in decisions
related to
environmental
issues at OSWER
sites





  External Factors: Available resources (especially for travel, in-person meetings at affected communities, and translation), lack of community organization.

-------
CHAPTER 2 |   METHODOLOGY
The methodology for this evaluation combines a thematic analysis of qualitative data
gathered from interviews with a review and synthesis of existing documentation. The
complete methodology is included in Appendix A; it was completed in July of 2012.
Important methodological updates that occurred after finalizing the evaluation
methodology are discussed in this Chapter.

REVIEW OF EXISTING DATA
lEc reviewed existing documentation and data to inform our responses to evaluation
questions, as discussed in the methodology. We reviewed the following data sources:
    •   Documentation of community engagement requirements;
    •   Site inventory data for cleanup programs;
    •   Performance measurement resources (for Questions 5 and 6); and
    •   Existing community satisfaction interviews (as discussed above).
Use of these documents and  data is discussed in more detail in the context of Findings in
Chapter 3.

INTERVIEWS
lEc conducted 46 interviews with EPA HQ and regional contacts from Superfund,
Brownfields, RCRA CA, as well as state RCRA CA contacts in Arkansas, California, Illinois,
Missouri, New York, and South Carolina. Exhibit 2-1 below summarizes these interviews.
EPA HQ contacts helped us to identify EPA regional contacts; and regional RCRA CA
contacts helped to identify state contacts.

Exhibit 2-1.  Interviews  with  EPA and State Contacts
INTERVIEWS
EPAHQ
EPA Regions
RCRA CA State
Other
TOTAL
COUNT
7
30
6
3
46
DESCRIPTION
Interviewed HQ staff: Superfund (2), Brownfields (2),
and RCRA CA (3)
Interviewed Superfund, Brownfields, and RCRA CA HQ
staff in all 10 regions (1 interview for each region)
Interviewed staff from 6 delegated states
Interviewed CPRC contact, TASC contractor,
Brownfields K6 grantee

lEc also conducted satisfaction interviews with contacts from community groups that
had received technical assistance. As discussed in the methodology, we conducted these
interviews using the format developed and implemented by another contractor for use
                                                                           10

-------
in measuring customer satisfaction under the TASC contract. A summary of satisfaction
interviews is included in Exhibit 2-2 below.

Exhibit  2-2. Satisfaction Interview Summary
INTERVIEWS
Superfund TASC
Superfund TAG
RCRA CA
Brownfields TAB
andK6
TOTAL
COUNT
22
9
7
8
46
DESCRIPTION
(Another contractor previously conducted interviews
in 22 communities)
Interviewed TAG recipients in 8 communities
Interviewed recipients of state technical assistance in
7 communities. No contacts from recipients of EPA-led
RCRA CA sites were available for interv/ew.
Interviewed 7 recipients of technical assistance under
TAB in 7 communities and 1 recipient of K6 assistance
in 1 community

lEc identified satisfaction interviewees in various ways. The Superfund program
maintains a list of TAG recipients; we randomly selected one primary and one backup
TAG recipient to contact in each region. We subsequently conducted nine TAG
interviews (a TAG interviewee could not be reached in one region). The Brownfields
program does not maintain a list of communities that have received assistance under
TAB, and the intensiveness of TAB assistance varies by community group. Hence, we
asked all four TAB grantees and the one K6 grantee that does community engagement
work to identify community contacts for satisfaction interviews. Specifically, we asked
the TAB grantees to identify contacts from community groups that received a
substantial degree of assistance. Because the TAB grantees are regional, this approach
provided geographical diversity. For RCRA CA, we asked  EPA regional and state contacts
interviewed to help identify community contacts, as there is no single list of RCRA CA
communities that have received technical assistance at either the EPA or state level.
We analyzed results from lEc's satisfaction interviews in conjunction with existing
Superfund TASC satisfaction interviews. We were able to analyze 22 existing TASC
interviews in addition to the satisfaction interviews that lEc conducted.12 A notable
difference in lEc interviews and previously conducted satisfaction interviews is that
some of the previous interviews contained scores from multiple community contacts,
while all of lEc's scores are from a single contact for each site.
12 Of the 56 Superfund communities that have accessed TASC, only 22 had satisfaction interviews that could be used. The
 main reason is that community satisfaction interviews are conducted towards the end of the cleanup process, and many of
 the 56 sites have cleanup ongoing. Also, some of the satisfaction interviews previously conducted were joint interviews
 with an EPA representative and a community contact, but the contractor provided only the EPA representative scoring for
 some of these interviews. We excluded these communities from our analysis.
                                                                                  11

-------
Thematic Analysis of Interview Data
lEc conducted a thematic analysis of interviews. We developed interview guides (see
Methodology document in Appendix A of this report) to guide each interview. We
developed a thematic analysis workbook in MS Excel that contained one worksheet for
each evaluation question. We cross-walked each interview question with applicable
evaluation question(s) and recorded this coding within the spreadsheet for each
evaluation question. Then, we recorded  interviewee responses to each interview
questions in the applicable spreadsheets. All responses to the same interview question
were recorded on the same  row. Once we entered all interview data, we reviewed
responses to each interview question one by one, and  identified key themes for each
interview questions. For many interview questions, we counted the number of times
interviewees provided the same response. We summarized an overall response to each
interview question in the spreadsheet, noting areas of consensus and conflict, and
noting if there were clear schisms between different types of interviewees. We then
summarized responses for the interview questions associated with the same evaluation
question, and brought in information gleaned from the document review as applicable,
to develop findings for our evaluation  questions.

STRENGTHS AND WEAKNESSES OF METHODOLOGY
The methodology employed has a number of strengths, including coverage of all three
programs: Superfund, RCRA CA, and Brownfields. Notably, the methodology employed
for this evaluation was sufficient for addressing most of the research questions,
including all of Questions 1,  3, 5, and 6; and most of Question 2. Within Question 2 in
particular, because lEc was able interview one contact for each of the three programs, in
every region, we are able to accurately characterize the baseline of EPA's technical
assistance and information dissemination activities. However, because we only
interviewed six states that implement RCRA CA, and only had information on a few
additional states from existing documentation, we cannot be sure if our findings on
frequency of practice, and community assistance provided outside of formal programs,
are representative of state RCRA CA programs overall.
For Question 4-establishing a baseline of satisfaction with technical assistance and
information dissemination among communities-a key strength of the evaluation was
the ability to combine analysis of new  satisfaction interviews with existing interviews.
We also achieved geographic diversity in our interviews. However, the methodology for
Question 4 was limited by resources available to conduct interviews, and difficulties in
identifying community contacts to interview, particularly within the RCRA CA program.
Subsequently, a key limitation of our approach is that results for Question 4 on
satisfaction with technical assistance and information dissemination cannot be
extrapolated to all communities served by OSWER programs. Ideally, lEc would have
conducted surveys or interviews using a  statistically valid sample of those served for
each OSWER program, to develop a true baseline of satisfaction with technical
assistance and information dissemination criteria.13 However, this approach would have
13 In addition to surveying a statistically valid sample of those served, it would also be methodologically preferable to survey
 multiple stakeholders associated with each site. One individual interviewee may not represent the perspective of all local
 stakeholders.


                                                                              12

-------
required extensive resources to develop contact data (in particular for RCRA CA
communities), and to develop and administer the survey or requisite number of
interviews. In addition, programs expressed reservations about broadly surveying
community groups affiliated with their programs, citing information burden.
Finally, the method by which lEc selected interviewees has inherent strengths and
weaknesses. For the TASC contract, lEc was able to analyze all applicable data collected
for Superfund to-date, which is a methodological strength. In addition, we were able to
randomly select Superfund TAG grantees to interview. In contrast, as discussed above,
lEc had to  rely on TAB grantees to identify Brownfields community groups to interview,
and we had to rely on RCRA CA regional and state interviewees to identify RCRA CA
community groups to interview.  Relying on these parties to identify interviewees can be
a source of bias, although we have no evidence or suspicions that contacts cherry picked
community contacts to participate in this evaluation.
                                                                              13

-------
CHAPTERS |  FINDINGS
This section presents evaluation findings by evaluation question. High-level findings
from the evaluation include the following:
   •   The EPA Superfund and Brownfields programs appear to have robust systems in
       place for delivering technical assistance and information to communities,
       including clear mandates and guidance, formal programs and mechanisms for
       delivering technical assistance, and adequate levels of EPA staffing and
       resources.
   •   RCRA CA appears to be meeting its mandate of providing information
       dissemination to community groups as required. However, in comparison to the
       Superfund and Brownfields programs, RCRA CA lacks many inputs helpful in
       ensuring the delivery of technical assistance to communities, including a lack of:
       regulatory mandates; adequate resources and staffing at EPA and state
       agencies; and up-to-date, program-specific guidance. Unmet community needs
       appear to be higher within the RCRA CA program than other programs, and
       satisfaction with technical assistance provided by states to communities under
       RCRA CA was rated lower on average by interviewees than for the other two
       programs.
   •   Compared to the other programs, it is more difficult to characterize the needs of
       RCRA CA communities nationally, and to track progress in meeting those needs,
       because the program is largely delegated to states, and EPA currently lacks
       mechanisms for collecting community engagement data from states. Thus, if
       EPA were to conduct regular tracking of measures of unmet needs and
       customer satisfaction, the Agency would need to work with states to
       implement a data collection system. In contrast, existing data collection systems
       employed by Superfund and Brownfields could potentially be augmented to
       track suggested measures.

QUESTION 1: WHAT ARE  THE  REQUIREMENTS AND  DRIVERS  FOR
COMMUNITY INVOLVEMENT  (Cl) WITHIN THE SUPERFUND, RCRA
CORRECTIVE ACTION (CA),  AND  BROWNFIELDS PROGRAMS?	

Summary of Methods  to Address Question 1
   •   Review of documents to identify requirements
   •   Interviews with staff at OSWER HQ and regions to explore knowledge of
       requirements and collect information on additional Cl drivers

Findings on Requirements
Each Superfund site must develop a community involvement plan (CIP) and update it as
necessary. The National Contingency Plan (NCP) requires developing a CIP "based on
community interviews and other relevant information, specifying the community
                                                                          14

-------
relations activities that the lead agency expects to undertake during the remedial
response." The NCP specifies that the CIP must be in place before remedial investigation
field activities start "to the extent practicable." 14 Requirements for TA and ID under
Superfund match the perception of requirements from Superfund staff; Superfund HQ
contacts and all 10 Superfund regional contacts interviewed indicated that CIPs are
required.
Requirements for Brownfields found in the National Contingency Plan dictate that, for
cleanup grants only,  EPA must develop community relations plans and make cleanup
plans publicly available. In addition, according to Brownfields HQ interviewees, National
Environmental Policy Act (NEPA) requirements applicable to brownfields include holding
public meetings, answering comments, and making grant proposals publicly available.
Part of the scoring for Brownfields grant applications includes evaluation criteria for
community engagement. Brownfields HQ correctly indicated that technical assistance
and information dissemination are required for cleanup grants. In addition, eight out of
10 Brownfields regional interviewees indicated that this activity is required.
Under RCRA CA, a facility owner/operator is required to provide public notice during key
phases of the CA process, as documented in the 1996 RCRA Public Participation Manual.
Technical assistance  is not required. However, seven of 10 RCRA CA regional contacts,
and four out of six RCRA CA state contacts, indicated that the program is required to
provide both technical  assistance and information dissemination to affected
communities.

Findings on Additional Drivers
In addition to requirements discussed above, several interviewees noted additional
drivers of community involvement:
    •   Six interviewees cited community demand as a driver.
    •   Six interviewees cited environmental justice, including helping disadvantaged
        and tribal areas.
    •   Two interviewees cited EPA initiatives and priorities, such as the Targeted
        Brownfields Assessment (TBA) program and region-specific priorities.
    •   Two regional contacts cited benefits to EPA of providing technical  assistance
        and information dissemination as an additional driver. Specifically, one regional
        contact indicated that long-term stewardship at cleanup sites requires pro-
        active community and civic involvement. Another region said: "If the region
        postpones adequate community engagement, it will receive complaint letters
        and will establish a lack of trust" that can inhibit cleanup and redevelopment.
M I Community Involvement Plans, p. 7, available at: http://www.epa.gov/superfund/communitv/pdfs/toolkit/ciplans.pdf.
 NCP [40 CFR 300.430(2)(ii)] states the following intent regarding community involvement: "(A) Ensure the public
 appropriate opportunities for involvement in a wide variety of site-related decisions, including site analysis and
 characterization, alternatives analysis, and selection of remedy; and (B) Determine, based on community interviews,
 appropriate activities to ensure such public involvement."
                                                                                 15

-------
QUESTION 2: WHAT IS THE BASELINE OF CURRENT OSWER TECHNICAL
ASSISTANCE AND INFORMATION DISSEMINATION ACTIVITIES WITH
RESPECT TO:	

    a.  Frequency of practice and program-to program variability?
    b.  Proportion of communities that receive formal assistance through TAG, TASC, or
       TAB?
    c.  Selection criteria and on-the-ground process for accessing TAG and TASC (for
       Superfund and RCRA CA only)?
    d.  Community assistance that is provided outside of formal programs (including
       helping to set up CAGs and providing ad-hoc assistance)?
    e.  Areas of unmet TA or ID need?

Summary of Methods to Address Question  2
    •  Interviews with HQ and regional OSWER staff in all three cleanup programs
    •  Interviews with six delegated RCRA CA states
    •  Interviews with TAB grantees
    •  Review of data and documents provided by OSWER
    •  Review of publicly available information

Findings on Frequency  of Practice and Program-to-Program Variability
OSWER administers three formal programs to provide TA to  communities affected by
contaminated sites: TAG, TASC, and TAB. As shown in Exhibit 3-1 below, Superfund
communities are eligible for TAG, and Brownfields communities can access TAB, and all
OSWER communities are eligible for TASC.15

Superfund
The Superfund program has the most extensive formal TA and ID practices of the
OSWER programs. Most notably, the Superfund awards TAG grants to community-based
organizations at eligible Superfund sites. Groups awarded TAGs up to $50,000 can
contract with independent technical advisors to interpret and explain technical
information to the community. 350 TAGs have been awarded since their inception in
1988. Superfund communities are also eligible for TASC assistance, with 56 Superfund
communities receiving TASC assistance since the program's inception in 2008.
15 Although all OSWER communities are eligible for TASC, lEc notes confusion within EPA about whether RCRA CA sites in
 particular are eligible for TASC funding if there is no link to Superfund. RCRA CA staff members in EPA regions are confused
 about this issue. For example, according to one interviewee, one region submitted several RCRA CA sites to EPA HQ for
 TASC funding, but all were declined because they had no link to Superfund. The disconnect may be that the RCRA CA
 program needs to contribute funds to TASC in order to use the contract, which may be a limitation precluding wider use by
 RCRA CA sites.


                                                                              16

-------
Exhibit 3-1. Eligibility and Basic Setup of OSWER TA Programs
PROGRAM
TAG (Since 1988)
TASC (Since 2008)
TAB (Since 2008)
ELIGIBILITY
Superfund
All OSWER
programs
Brownfields
BASIC SETUP
Community groups apply for grant from EPA.
Community groups hire their own advisor to provide
technical assistance.
OSWER maintains a contract with a firm that provides
Cl to community groups. Community groups access the
contract by contacting their region or the region
recommends communities to HQ directly.
OBLR maintains a long-term grant with four
organizations that provide Cl to community groups.
Community groups access the contract by contacting
the TAB grantee for their region.
Furthermore, the Superfund program employs 80 Community Involvement Coordinators
(CICs), with between two and 10 CICs in each region. CICs provide a liaison between
project managers and the community, and provide opportunities for two-way
communication throughout the life of a project. Every Superfund site is also required to
produce a CIP at the beginning of the Superfund process to identify how to best engage
the community throughout the cleanup. CIPs are discussed in more detail under
Question 3.
In contrast to Superfund, which administers technical assistance directly to community
groups, assistance to community groups is delegated under the RCRA CA and
Brownfields programs as discussed below.

Brownfields
Brownfields community groups can receive technical assistance from the TAB grantee
assigned to their region. EPA has awarded TAB grants to the New Jersey Institute of
Technology for Regions 1, 2, and 3; Enterprise Corporation of the Delta, Inc. for Regions
4 and 6; Kansas State University for Regions 5 and 7; and the Center for Creative Land
Recycling for Regions 8, 9, and 10. 833 Brownfields communities have received
assistance through TAB, with assistance lasting anywhere from a five minute phone call
to a multiple-year relationship. The TAB grantees assist communities with: community
outreach and mediation; grant writing; environmental planning; training; regulatory
facilitation; identifying funding sources; technical presentations; summarizing
stakeholder comments and concerns; technical reports; and site inventories.
However, most technical assistance in Brownfields is not provided through TAB, but  by
the recipients of Brownfields grants, which are typically local governments. As discussed
under Question 1, Brownfields cleanup grantees are required to provide technical
assistance. Moreover, OBLR awards points for community engagement plans as part of
evaluating most types of grant applications, including Assessment Grants, Cleanup
Grants, Multi-Purpose Pilot Grants, and  Revolving Loan Funds Grants. Once the grants
are awarded, EPA does not directly track community engagement activities by
Brownfields grantees, but community engagement is part of grant reporting.
                                                                            17

-------
RCRA CA
RCRA CA has standard guidelines for engaging the community in its 1996 Guide to
Public Participation. However, requirements for TA or ID under the RCRA CA program
are limited. Under RCRA CA, EPA or the delegated state agency is required by statute to
issue a  public notice in the newspaper and hold a period of public comment when key
project milestones are reached, such as the proposed remedy and final remedy
decision. All regional and state staff interviewed indicated that they comply with these
ID requirements.
RCRA CA sites are eligible for TASC assistance. Seven RCRA CA sites have received TASC
assistance since 2008. Most TA and ID activities within RCRA CA are conducted by RCRA
CA regional and/or state staff on a site-by-site basis based on perceived need. Every
region and state interviewed holds public meetings, but only at select sites with high
levels of community interest. It should be noted that the number of RCRA CA
community groups receiving TA is not tracked centrally for either EPA or state sites.
Some states have developed TA approaches for select RCRA CA sites. For example,
Mississippi's RCRA CA program has an EJ coordinator who provides training to staff to
identify EJ communities within the RCRA CA universe, conduct face-to-face community
meetings, and use an electronic notification system for RCRA CA communities within the
state. South Carolina's RCRA CA program staff report that they meet with RCRA CA
facility staff early in the process to encourage them to engage with the community;
South Carolina staff noted that facilities can be persuaded that community engagement
will help to build trust.

Findings on Proportion of Communities that Receive  Formal
Assistance through TAG,  TASC, or TAB

Superfund
Superfund communities are eligible for formal technical assistance through both TAG
and TASC. Of the total universe of 13,662 Superfund sites (1652 NPL sites and  12,010
non-NPL sites), 350 TAGs have been awarded. Thus, approximately 2.6  percent of
Superfund sites have had a TAG group. As shown in Exhibit 3-2 below, 56 Superfund
communities had received TASC assistance since 2008, or just 0.4 percent of all
Superfund sites. It should be noted that the TAG program has  been available since 1988,
whereas the TASC contract became available much latter, in 2008.

RCRA CA
Formal TA is rarely used at RCRA CA sites. Seven RCRA CA communities have received
TASC assistance since 2008. The RCRA CA universe  contains 3,747 sites, so just a tiny
fraction of RCRA CA communities have received TASC assistance. The breakdown of the
authority types of these 3,747 sites is shown below in Exhibit 3-3.
                                                                            18

-------
Exhibit 3-2. TASC Assistance by Program
               PROGRAM
   TASC COMMUNITY COUNT
(CURRENT AS OF JANUARY 2013)
 Superfund
           56
 NPL
           37
 Non-NPL
            14
 Federal Facility
 OEJ
            5
           29
 RCRA CA
 CARE
 Historically Black Colleges and Universities
 OW
 TOTAL
           105
Exhibit 3-3. Authority Type of RCRA CA Sites
RCRA CA FACILITY AUTHORITY
Total facilities in need of corrective action
Under EPA authority only
Under state authority only
Under EPA authority and under state
authority at different times
Under state authority, and under joint
EPA/state authority, at different times
Under joint EPA/state authorities only
Under EPA authority, state authority, and
joint authorities at different times
Sites for which authority information is not
available in RCRA Info
Uncategorized
RCRA CA SITE INVENTORY (FROM
RCRA INFO MAY 2013)
3747
1485
1174
808
37
31
13
186
13
                                                                         19

-------
Brownfields
833 communities have received TAB assistance from one of the four TAB grantees since
2008. However, there is no official list of brownfields in the U.S., since most U.S.
communities have brownfields sites, and EPA considers all communities to be eligible
for Brownfields grant assistance. Thus, it is not possible to determine the proportion of
communities with brownfields that have received TAB assistance.

Findings on  the Selection Criteria and on-the-Ground process for
Accessing TAG  and TASC

TAG
Communities typically learn about the availability of TAGs through site Remedial Project
Managers (RPMs) and CICs at public meetings and/or through e-mail and standard mail
distributions. EPA also distributes fact sheets regarding TAG and has information about
TAG on the EPA website.
Once a community group applies for a TAG, EPA conducts an Eligibility and
Organizational  Capability Review to determine whether the applicant meets all the basic
requirements of administering a TAG.  These requirements include:
    •   The applicant must contain groups of individuals that may be affected  by a
        release or threatened release at a Superfund site;
    •   The applicant cannot be a Potentially Responsible Party (PRP), academic
        institution, or political subdivision; and
    •   The applicant must have the organizational capability to administer the TAG.
Our interviews with Superfund regional staff indicated that TAGs are almost always
awarded to groups that  meet these eligibility requirements. Resource constraints are
rare; only one region reported cutting TAG funds to due resource constraints. Two
regions stated that some communities do not apply for TAG because the paperwork
required to administer the EPA grant is too burdensome.
In some instances, more than one group applies for a TAG  at a single site. This situation
occurs at less than 10 percent of the sites according to HQ. In these instances, EPA
conducts a Programmatic Criteria Evaluation to decide between applicants. This review
consists of determining the extent to which the TAG group: represents the community;
plans to use a technical advisor throughout the response; plans to communicate with
the broader community; and identifies potential environmental results that can be
measured.

TASC
Community groups do not apply to TASC directly; they receive access to the TASC
contract based on recommendations from regional staff to EPA HQ. Once EPA notifies a
community that it has been selected for TASC, the community writes an informal letter
or e-mail to their EPA region stating they are interested in funding. Since communities
do not apply for TASC directly, we heard from six RCRA CA regions that they do not
inform any community members about TASC services. However, all 10 Superfund
                                                                            20

-------
regions and four RCRA stated that communities may hear about TASC through project
managers, CICs, the EPA website, and/or fact sheets.
HQand regional interviewees within the Superfund and RCRA CA programs stated that
there are no formal selection criteria for OSWER programs to access TASC. However,
communities that are selected for TASC funds typically share one or more of the
following conditions:
    •   A lack of other sources of funding for community TA needs;
    •   A high level of community interest in the cleanup process;
    •   Environmental justice concerns;
    •   Short-term technical assistance needs; and/or
    •   Superfund communities that are not interested in forming a TAG group.
Based on interviewee opinions, there appears to be adequate TASC funding for
Superfund sites, but not for RCRA CA sites. Superfund HQ and eight of ten regions
interviewed stated that there is no lack of TASC funds to support the number of
communities recommended to receive funding. The two Superfund regions that
reported a lack of TASC funds pointed  to examples of certain communities that did not
receive as much funding as the region  had requested. In contrast, four RCRA CA regions
and HQ stated that RCRA CA sites were unable to get TASC funds for sites unless they
had a link to Superfund, and as noted above, only seven RCRA CA sites have accessed
TASC.

Findings on  Community Assistance Provided  Outside of Formal
Programs

Superfund
All ten Superfund regions interviewed  stated that technical assistance to communities is
provided outside of formal programs. The most common response  among regions (eight
out of 10 interviewed) was that RPMs  and CICs are available to provide community
members with ongoing technical assistance as needs arise. In addition, four regions
mentioned holding special public meetings with communities as needed.

Brownfields
As discussed above, the majority of TA and ID to communities in the Brownfields
program is provided outside of TAB by cleanup grant recipients. For Brownfields cleanup
grants, the grantee must undertake the following steps to ensure the community is
informed and involved in the cleanup process:
    •   Designate a community relations spokesperson.
    •   Prepare a draft Community Relations Plan and submit to EPA to review prior to
       the 'Analysis of Brownfield Cleanup Alternatives' is complete.
           o   The Community Relations Plan must outline steps to "provide
              reasonable  notice of proposed cleanup, opportunity for involvement,
                                                                           21

-------
              response to comments, and administrative records that are available to
              the public."
    •   Establish an information repository and maintain an administrative record for
       the site.16

RCRA CA
As discussed previously, only seven RCRA CA sites have received TASC assistance, which
is the only formal TA program available to RCRA CA communities. Thus, nearly all TA
provided to RCRA CA communities has been outside of formal programs. EPA regions
and delegated states that administer RCRA CA make decisions on where to offer
targeted TA based on community requests and staff assessments of need.
The most common form of technical assistance reported is holding special or regular
public meetings, which was mentioned by nearly all states and regions interviewed. In
addition, all six states interviewed and four regions mentioned that they post site
information online for high profile sites. Other forms of technical assistance provided by
RCRA CA staff include workshops, programs  for EJ communities, and encouraging PRPs
to engage the community. As noted above, one delegated state reports that
encouraging facilities to engage the community early in the RCRA CA process will help
build trust and facilitate community engagement throughout the  lifetime of the project.
Other methods of distributing information to RCRA CA communities mentioned by
interviewees include newsletters, surveys, fact sheets, and social media.

Findings on Unmet TA or ID Needs
Most interviewees were able to identify some unmet TA and/or ID needs. However, as
shown in Exhibit 3-4 below, when asked to characterize unmet needs in their region,
RCRA CA interviewees noted a somewhat higher incidence of unmet needs among RCRA
CA communities than among Superfund or Brownfields communities.
Exhibit  3-4. Unmet ID and TA needs

RCRA CA
Superfund
Brownfields
NOTED UNMET TA NEEDS
HQ, 8 Regions
HQ, 6 Regions
HQ, 4 Regions
NOTED UNMET ID NEEDS
7 Regions
7 Regions
5 Regions
Superfund
Four regions reported that there are no identifiable unmet TA needs within Superfund.
The most common cause of perceived TA gaps identified by Superfund regions are a lack
of resources and staff (two of 10 regions) and that TAGs are too labor intensive to
administer (two of 10 regions). Other issues with TA noted by Superfund regions include
that more regions should be pursuing Technical Assistance Plans, which require PRPs to
provide TA; and that communities are pressuring HQ directly, instead of their own
16 These steps are documented in the EPA Brownfields Cleanup Grant Checklist: Major Programmatic Tasks. However,
 Brownfields contacts did speak to the extent of implementation during interviews for this evaluation.
                                                                             22

-------
regions, to access TA. Notably, Superfund HQ contacts did not mention any resource
limitations affecting TA and ID within the program.
Three Superfund regions reported there are no identifiable gaps in Superfund's
approach to ID. The most common issues identified with regard to Superfund's
approach to ID are that tailored outreach activities and electronic records work better
than traditional public notices (two regions and HQ); only vocal communities receive
adequate attention (two regions); and there are too few resources for ID (two regions).
Individual regions also stated that EPA falls short of delivering high quality information
to English-as-a-Second-Language  (ESL) communities and immigrant communities, and
that CIPs should include references to TA options available to communities.

Brownfields
Most Brownfields interviewees suggested that there are no gaps in Brownfields'
approach to TA (six regions) and ID (five regions and HQ). Four Brownfields regions and
HQ indicated that any perceived gaps in TA are the result of resource limitations. Three
regions stated that Brownfields' approach to ID falls short of reaching rural communities
and/or communities that have not won grants. One region reported that the staff
members that manage the cooperative agreements lack knowledge, while another
stated that it is too difficult to post pertinent information for communities to EPA's
website.

RCRA CA
RCRA CA interviewees noted a lack of resources to conduct TA and ID as a cause of
perceived gaps more frequently than either of the other two  cleanup programs. Eight of
10 RCRA CA regions and HQ contacts stated that a lack of resources is the cause of
unmet TA needs in the program, while six of 10 regions reported resource limitations as
a gap in RCRA CA's approach to ID.
Other areas of concern identified  by RCRA CA interviewees included a lack of staff
training in community engagement, and that traditional forums such as public meetings
are not as effective as going to community-sponsored events, like fairs and celebrations.

QUESTION  3:  HOW CAN SUPERFUND COMMUNITY INVOLVEMENT PLANS
(CIPS) BE  USED TO  IMPROVE TECHNICAL ASSISTANCE AND DELIVERY  OF
INFO?	
This question had a number of sub-questions:
    •   Does every site have a CIP?
    •   What information is available on the implementation of CIPs?
    •   Are CIPs revised over time?
    •   Do CIPs, as they are currently used, ensure effective technical assistance and
       information delivery throughout the life of the project? Why or why not?

Summary of Methods to Address Question 3
    •   Interviews with 10 Superfund regional staff
                                                                            23

-------
Findings on  CIP Implementation
As noted above under Question 1, all Superfund sites have a CIP. According to all
Superfund regional interviewees, CIPS are tracked and maintained by regional staff. CIP
updates are variable.  From the perspective of tenured EPA staff, CIPs are most valuable
at the beginning of the Superfund process. From the perspective of community
members and new EPA staff, CIPs and updates have value throughout the cleanup
process. CIPs may be improved by more frequent, targeted revisions and by making
them more readable by community members. CIP implementation is a broad topic that
encompasses the other two sub-questions; thus, more information on implementation
is in the sections below.

Finding on CIP Revisions
EPA's official CIP guidance says that documents should be updated as needed:
        "The NCP requires that the CIP be reviewed prior to the
       initiation of the remedial design to determine whether it
       should be revised to describe further public involvement
       activities. Yet, there is no standard rule about when to
       update or completely revise the CIP. Because the CIP should
       be a living document that is referred to regularly, it makes
       sense that information will be continuously added or updated."17
However, there appears to be confusion about the CIP guidance and the recommended
revision schedule among Superfund regional contacts. Only two interviewees referred
to the guidance when asked about CIP revision frequency, and they both mistakely
indicated that the guidance contained a specific time frame for CIP revisions. One region
said updates are prescribed every two years by the guidance, while another said every
five years.
The reality is that average revision frequencies for CIPs vary widely from site to site;
interviewees indicated that revisions occur once per year for some sites to once every
five to seven years for others. Major impetuses for revision include community demand
and project milestones. Five interviewees said revisions occur "as needed." Of these five
interviewees, three interviewees cited significant milestones or phase changes of the
cleanup process as impetuses for CIP revisions. Five interviewees said that revisions
occur too infrequently; three of these contacts indicated that revisions only occur in
response to community demand.
Findings on  CIPs as a Tool to  Ensure Effective Technical Assitance  and
Information  Dissemination
Six of 10 interviewees said that CIPs are not an important tool for ongoing technical
assistance and  information delivery. The general sentiment among Superfund regional
interviewees is that CIPs are helpful to organize community engagement at the
beginning of the Superfund process, but are not used by EPA staff as a resource over
time. EPA tends to set aside the document after the start of the project, instead relying
on ad-hoc communication with community members.
17 Community Involvement Plans, p. 7, available at: http://www.epa.gov/superfund/communitv/pdfs/toolkit/ciplans.pdf


                                                                             24

-------
CIP updates often lag behind project changes. Thus, CIPs may not ensure effective
technical assistance and information delivery throughout the life of the project given
lagging updates. However, as discussed under Evaluation Question 4 findings,
satisfaction with technical assistance and information dissemination is high among
Superfund community contacts participating in this evaluation that have received a TAG,
and across the Superfund community contacts that have used the TASC contract.
CIPS appear to be important for other reasons, including gathering community contact
information; providing an ongoing reference tool for community members; and serving
as training materials for new Superfund staff.
Superfund regional contacts provided several suggestions for improving CIPs. Many
interviewees suggested format changes including:

   •   Two interviewees suggested that CIPs be shortened, or that updates be placed
       into an addendum.

   •   Two interviewees suggested making CIPs easier to read, perhaps by using fact
       sheet summaries.

   •   One interviewee noted that CIPs have not kept up with changing
       communication  methods and social media; the format of the CIPs as stand-
       alone documents could be reconsidered with social media in mind.
Two interviewees recommended  more regular updates. However, one interviewee
cautioned against "obsessive" updates, pointing out that only certain parts of CIPs need
updating overtime.


QUESTION  4:  WHAT  IS THE  BASELINE OF CUSTOMER SATISFACTION WITH
OSWER TECHNICAL ASSISTANCE AND INFORMATION DISSEMINATION
ACTIVITIES?	

Summary of Methods to Address  Question 4

   •   Satisfaction interviews with community contacts (see summary table Exhibit 2-2
       in the Methods chapter).

Findings on Interviewee Satisfaction
Interviewees rated their overall satisfaction with assistance services provided on a scale
of one to six; with a score of one being very dissatisfied to a score of 6 being very
satisfied (see satisfaction interview guide in the methodology in Appendix A). As shown
in Exhibit 3-5 below, interviewees noted consistently high satisfaction overall scores for
Superfund and Brownfields programs, while scores for RCRA CA were more mixed.
However, regarding RCRA CA, it should be noted that at the lower end, only two
interviews rated their overall satisfaction as a "1" with assistance provided by  RCRA CA
states (see bottom graph in Exhibit 3-5); the majority of RCRA CA sites provided
favorable overall ratings for assistance received.
                                                                            25

-------
Exhibit  3-5.   Satisfaction Score Summary
     80
     70
     60
    50
   £
   = 40
   ° 30
   8 20
   01
   °- 10
      0

       Very
    Unsatisfied
                        Overall Score Distribution by Program
RCRA
                     (9 Interviews)
        I Superfund    (9 Interviews)

        I Brownfields   (8 Interviews)
                             3        4
                           Likert Scale Score
  Very
Satisfied
     22
     20
     18
     16
     14
   = 12
     10
      8
      6
      4
      2
      0
       Very
    Unsatisfied
                        Overall Score Distribution
            I Brownfields
            RCRA
            I Superfund
                              3       4
                            Likert Scale Score
   Very
 Satisfied
                                                                                           26

-------
Illustrative interviewee elaborations on overall satisfaction scores are provided in
Exhibit 3-6 below.

Exhibit 3-6.  Satisfaction Ratings Explanations from Interviewees
  Summary of Score
   "6" Elaborations
  Summary of Score
   "5" Elaborations
  Summary of Score
   "4" Elaborations
  Summary of Score
   "1" Elaborations
Excellent service that goes above and beyond requirements
High level of expertise
Excellent record of interfacing with the community often and
in plain language
Affords community a high degree of influence over events at
the site
Has helped community secure grants
Displays ability to work effectively with all parties: community,
state, Responsible Party (RP), EPA
Generally high quality service
Good communication; solid expertise conveyed in plain
language
Should reach out to community more often
Would be rated 6, but community hasn't worked long enough
with them
Outreach is passive; delegated RCRA CA only provides
information to citizens that ask for it
RCRA CA does not help citizens tackle short term issues (e.g.,
declining home equity), instead focusing on the decades-long
cleanup
Too much control is put in the hands of the RP
EPA (RCRA CA staff) is not straightforward about
administrative issues, and claim to have inadequate manpower
to tackle problems
Delegated  RCRA CA state agency doesn't listen to relevant
parties: not the other state agencies,  not the community, and
not EPA
lEc compared overall satisfaction ratings for TAG and TASC among Superfund
community contacts and found remarkably similar overall levels of satisfaction with
these two mechanisms for providing assistance, as shown in Exhibit 3-7 below.

Findings  on Interviewee Suggestions  For Improving Services
Community contact suggestions common to all three programs include making outreach
materials more accessible by using visual aids and plainer language, and reaching out to
community members more often and/or more actively. Additional suggestions include:
bolstering technical expertise of advisors, improving EPA's website, and assisting with
EPA grant processes.
                                                                                27

-------
Exhibit 3-7. Comparison of Overall Satisfaction  Scores  for TAG  and
TASC
STATISTIC
Median
Mean
Minimum
Maximum
TAG
6
5.8
5
6
TASC
5.7
5.5
4
6
Superfund
Most Superfund community interviewees stated that technical advisors did not need to
make improvements. The small number of suggestions provided by interviewees on
improving technical assistance by providers include: using more visual aids during
presentations; developing a more local presence; and bolstering expertise on wildlife
impact issues. All of these suggestions were mentioned by only one individual.
Interviewees had more suggestions for EPA to improve on community outreach:
     •   Reach out to communities more often  (3 respondents).
     •   Create a more transparent decision-making process (2 respondents).
     •   Acquire more contamination science expertise for residents to access (1
        respondent).
     •   Avoid giving residents unrealistic expectations (1 respondent).

Brownfields
Two Brownfields interviewees said their advisors did not need any improvement. Other
interviewees provided suggestions on how how Brownfields technical advisors could
improve their services:
     •   Engage the community more deeply. Use them as a resource to discover site
        histories (2 respondents).
     •   Assist with grant applications (2 respondents).
     •   Draft higher quality reports and visuals (1 respondent).
Brownfields community contacts also had suggestions for EPA on improving community
outreach:
    •  Improve Brownfields website search-ability and update schedule (2
       respondents).
    •  Simplify the grant process (2 respondents).
    •  Increase project officers' level of technical expertise (1 respondent).
    •  Distribute more success stories and metrics  (1 respondent).
                                                                             28

-------
RCRA CA
Three RCRA CA community interviews had no suggestions for technical advisor
improvements; others provided the following input:

    •   Communicate in plainer language (3 interviewees).

    •   Distribute outreach more promptly, frequently, or actively (3 interviewees).

    •   Spend more time listening to community needs, including shorter term issues
        like declining property values (1 interviewee).

Most RCRA CA satisfaction interviewees had no experience working with EPA, and did
not offer suggestions for EPA regarding community outreach.


QUESTION 5: WHAT MEASURES CAN  BE USED TO ASSESS THE
EFFECTIVENESS AND  TANGIBLE OUTCOMES OF  OSWER TECHNICAL
ASSISTANCE AND  INFORMATION DISSEMINATION ACTIVITIES ACROSS THE
LIFECYCLE  OF SITE PLANNING, REMEDIATION,  AND  REUSE?
QUESTION 6: HOW CAN THESE MEASURES BE USED TO IMPROVE OSWER
TECHNICAL ASSISTANCE AND INFORMATION DISSEMINATION ACTIVITIES?

Summary of Methods to Address Questions 5 and  6:
    •   Reviewed CEI logic models and related program materials.
    •   Reviewed Guidelines for Measuring the Performance of EPA Partnership
        Programs, developed by the National Center for Environmental Innovation,
        dated June 2006.
    •   Reviewed the Partnerships for Environmental Public Health Evaluation Metrics
        Manual, dated October 18, 2010.
    •   Reviewed the types of measures currently used by OSWER programs and other
        EPA community-based programs related to community involvement.
    •   Conducted interviews with Superfund, Brownfields, and RCRA CA HQ and
        regional staff; included interview questions on potential measures.
    •   Considered feasibility factors for metrics implementation.

Interview Findings of Relevance for Developing Measures
lEc asked HQand regional program staff from each program what constitutes successful
TA and/or ID, as a starting point to thinking about meaures. The most frequent response
was that successful engagement consists of tailoring materials to meet community
needs and/or to ensure the community is satisfied with the information provided; 12 of
30 regional interviews (four from each program) provided this response. Other respones
were far less frequent. Three Superfund interviewees equated unbiased TA with
successful TA. lEc's interpretation of this response, based on other feedback provided by
                                                                        29

-------
Superfund interviewees on TAG,  is that some regional saff are concerned that technical
advisors hired under TAG are biased towards providing information that their clients
(community groups) want to hear, rather than providing unbiased information. Three
Brownfields interviewees indicated that successfull TA is tied to successful project
outcomes.
lEc also asked HQ and regional contacts about current metrics used to track community
engagement for each program. We learned that in some regions, Superfund staff track
output metrics, but not outcome metrics. Examples of output metrics tracked by some
Superfund rgions include : funding spent, number of public notices, number of
meetings, number of TASCs, number of CAGs, number of TAGs, number of people on
mailing lists, number of meeting attendees, amount of media coverage, requests
received for TA. Only two RCRA CA regions reported tracking output metrics: one
regional tracks the number of sites that they have designated as high profile sites, but
does not store these data post-remediation. The other region tracks the engagement
tools that are  used and how often they are used. According to our interviewees, the
Brownfields program does not track community involvement metrics, but some
interviewees referred lEcto site-level success stories generated by HQ, some of which
have information on communit involvement. Finally, as noted previously in this report,
EPA's TASC contractor Skeo tracks customer satisfaction at sites that have used the
TASC contract.
Finally, lEc asked interviewees "What other metrics might  help assess the extent to
which the TA and/or ID efforts conducted by EPA are successful?" Responses are as
follows:
    •   Most Brownfields interviewees (seven of 10 Regions and HQ) do not believe
        any new metrics should be created. The other three Brownfields regions
        suggested closely tracking specific resources used at individual sites.
    •   Measuring customer satisfaction was the most common  response from RCRA
        CA and Superfund interviewees. Four Superfund regions  suggested satisfaction
        surveys for TAGs, similar to those conducted by Skeo for TASC. Four RCRA CA
        regions also suggested satisfaction interviews.
    •   Four regional interviewees (1 from Superfund, 2 brom Brownfields, and 1 from
        RCRA CA) said that one metric to gage the success of TA/ID is to count the
        frequency of complaints filed by community groups.
    •   One Superfund regional contact and one RCRA CA regional contact suggested
        measuring community member learning from assistance received.
    •   One Superfund regional contact suggested an efficiency metric by comparing
        resources spent to satisfaction rates. Similarly, one Brownfields regional
        contact suggested comparing resources spent to project outcomes.
                                                                            30

-------
Approach to Developing Menu  of Potential Measures
lEc considered the following types of measures for inclusion in a menu of potential
measures for OSWER to use to track progress on TA and ID moving forward:
Output measures:
    •   Measures of unmet demand for Cl services
    •   Conventional output measures for Cl work: numbers of meetings, engagements,
       et cetera (variations on measures currently used by some Superfund regions)
    •   Output measures for ID, such  as frequency of community notices
    •   Measures of EPA training
Outcome measures:
    •   Measures of community satisfaction
    •   Measures of EPA use of best practices
We developed this universe of potential types of measures based on:
    •  Close review of the CEI logic  models and theory of change
    •  Existing community involvement measures used by OSWER programs
    •  Review of the types of measures currently used by OSWER programs and other
       EPA community-based programs related to community involvement
    •  References in applicable metrics manuals and literature reviewed
We then narrowed down the types of measures to include in a menu based on:
    •  Feasibility factors including:
           o  The ability to define objective measure(s)
           o  The need to develop  and implement new information collection
              infrastructure
    •  Feedback from interviewees on potential measures (discussed above)
    •  lEc's professional judgment regarding the likelihood of the measure being
       accepted and successfully implemented by the OSWER programs
Menu of Potential Measures
lEc developed menus of potential measures for outputs, short-term  outcomes, and long-
term outcomes discussed below. It is important to note that these are menus from
which to select measures; we do not recommend adopting all of these measures.
Output Measures
The output menu is presented in  Exhibit 3-8 below.
                                                                          31

-------
Exhibit 3-8.  lEc  Recommendations  for a Menu of Potential Output
Measures
     1.  Proportion of Superfund communities that applied for TAG assistance, but have not
        received assistance
     2.  Proportion of Brownfields communities that requested TAB or other form of K6
        assistance, but have not received assistance
     3.  Proportion of Superfund and RCRA CA communities that were recommended by a
        region for TASC assistance, but have not received assistance
     4.  Proportion of RCRA CA communities that have asked a state or region for any form
        of assistance, but have not received assistance
     5.  Number/Percent of RCRA CA communities that have received TA
     6.  Number/Percent EPA Regions that have offered TA/ID training in the past year
lEc's rationale for these output measures is as follows:
The first four measures address unmet needs. Several regions identified assigning
resources where they are needed as successful aspect of TA. Brownfields and RCRA  CA
interview contacts also frequently identified a lack of funding as an impediment to TA.
Thus, the proportion of communities that have either applied for, or been identified by
regional staff, as needing technical assistance will give EPA management an idea of how
much additional resources are needed and where they are needed . Note that lEc did
not frame the measures as "proportion of communities that are known to have unmet
needs" because "unmet needs" is not measurable language. We included a specific
measure on RCRA CA communities receiving TA (#5) because this is where the gap
seems greatest. Thus, it would be reasonable for EPA to focus additional  output
measures on RCRA CA  in particular. Finally, some interviewees indicated that more
training would be helpful in delivering quality TA, which led to the development of #6
on frequency of training.
lEc does not suggest any output measures for information dissemination. Based on our
interview data , it appears that required ID, such as CIPs and public notices for RCRA CA
are already widely implemented. Since ID should be tailored to community needs, there
is no single measure that can be universally applied to all sites because sites do not
apply for ID in and of itself.
Many regional contacts across the three OSWER programs advised against any
additional tracking of conventional output measures such as number of meetings,
number of attendees, number of public notices, et cetera. These metrics are already
tracked by Superfund in some regions, but contacts indicate they cannot be used to
improve TA and/or ID activities. We agree with this conclusion, and do not recommend
additional tracking of these types of "bean counting" measures.
                                                                              32

-------
Short-term Outcomes
Based on the criteria stated above, lEc has not identified many options for a short-term
outcomes measure for the CEI. The one measure that we recommend EPA consider is
presented below in Exhibit 3-9.

Exhibit 3-9.lEc  Recommendation  for Potential Short-Term Outcome
Measures
     1.   The number/proportion of community members filing complaints related to the
         provision of TA with Superfund/Brownfields/RCRA CA regional staff annually.
Several regional interviewees indicated that tracking trends in complaints is a viable
approach to measuring program success, and some regional staff currently track
complaints informally. If EPA works to continually improve the overall quality and
availability of TA, complaints should decrease over time. However, tracking complaints
may pose challenges. For example, EPA would have to develop guidance to define the
boundaries of a community member complaint about TA (in contrast to complaints
about other remediation or redevelopment issues), and regions would need to be very
careful about coding complaints. EPA may also need to develop  or augment a data
collection system to track this measure, in particular for RCRA CA, which has the
complication of state delegation in most states, and does not have an existing data
collection infrastructure for tracking assistance provide to communities. A limitation of
this measure is that it would not account for those who are dissatisfied but do not file a
complaint.
Theoretically, EPA's utilization of best practices would be a superior short-term  measure
of success of the CEI program compared to tracking complaints.  However, we cannot
recommend tracking the use of best practices because: best practices would  need to be
rigorously analyzed and defined prior to tracking, and EPA has learned from experience
that this first step can be quite difficult; tracking of best practices is not in place for any
of the three OSWER programs; and the data collection approach required to build and
administer a system for tracking use of best practices would be more complex than for
other measures.

Long-term  Outcomes
The long-term outcome menu is presented in Exhibit 3-9 below.

Exhibit 3-10.lEc Recommendations for a Menu of  Potential Long-Term
Outcome Measures
     1.  Proportion of TAG/TASC/TAB communities that have received a community
        satisfaction survey that are "satisfied" with the information provided by EPA
        throughout the cleanup process (tracked individually for each program).
     2.  Proportion of TAGs/TASC/TAB/RCRA CA communities that have received a
        community satisfaction survey that are "satisfied" with the assistance provided by
        their technical advisor (tracked individually for each program).
                                                                              33

-------
Throughout the interview process, lEc heard consistent support for satisfaction
surveying, and we think it is the best approach for understanding the long-term success
of TA provided under the CEI. EPA could adopt both of these measures, or could choose
to adopt only the second measure that focuses on satisfaction with TA provided (as
opposed to information provided by EPA). We would recommend that each community
that receives formal technical assistance be provided with a satisfaction survey. With
the exception of RCRA CA, we do not recommend surveing individuals from
communities that received only informal assistance,  as it will be difficult to identify an
appropriate individual(s) to be surveyed (this was a key problem for lEc in conducting
this evaluation, as discussed previously).
Skeo Solutions has already developed a community satisfaction format for TASC which
asks communities to rank various components of technical assistance, which lEc used
for conducting satisfaction surveys with TAG recipients. We would suggest revising
Skeo's format slightly moving forward to ensure every number on the scale has an
associated value. Currently, the 1 to 6 scale implemented by Skeo only defines 1 as "not
at all satisfied" and 6 as "very satisfied." Using this scale, we would interpret a score of
4 or more to be "satisfied" but it would be  preferable to define each rating for the
community member taking the survey.
Furthermore, we suggest that Skeo discontinue its practice of interviewing community
contacts and EPA contacts about satisfaction within the same interview; community
contacts should be interviewed separately from EPA contacts, and have their scores
tracked seperately. However, we encourage more use of Skeo's practice of interviewing
multiple contacts for each community, as this provides a diversity of viewpoints.
EPA could also include questions to probe  on community member learning within
community satisfaction surveys. However, we do not think that measuring community
member learning is a  substitute for measuring satisfaction. Moreover, while community
contacts can easily and accurately report their level of satisfaction, self-reported data on
learning may suffer from biases, and it is not feasible in this context to admister surveys
that objectively probe knowledge gained by community members through the TA
process.
Finally, as noted above, EPA would need to develop new data infrastructure to track
assistance provided to RCRA CA communities, and work with states to implement a
reporting system. Again, this is because the vast majority of assistance given to RCRA CA
communities is not provided under the auspices of a formal program,  and TA is typically
administered by state agencies not EPA.
                                                                              34

-------
APPENDIX A:   INTERVIEW GUIDES
lEc proposes to conduct interviews with the following individuals as the main source of
new data collection for the evaluation:
    1.  Superfund HQ contacts and contractor Skeo (2-3 interviews)
    2.  Superfund Regional contacts (10  interviews)
    3.  RCRA CA HQ contacts (2 interviews)
    4.  RCRA CA Regional contacts (lOinterviews)
    5.  RCRA CA state contacts (6 interviews)
    6.  Brownfields HQ contacts (2 interviews)
    7.  Brownfields Regional contacts (10 interviews)
    8.  TAB Grantees (4 interviews)
    9.  CPRC (1 interview)
    10. Community satisfaction interviews
           a.  Superfund (10 interviews with TAG grant recipients)
           b.  RCRA CA (12-15 interviews, including 2-3 sites where the EPA Region
              conducted community engagement instead of the state)
           c.  Brownfields TAB and K6 grants (the majority of interviews will be
              conducted  with communities that have received assistance through TAB
              program; we will also interview any community that has received TA
              through K6, but our current understanding is that there may be only one
              or two applicable K6 grants)
lEc will conduct all interviews by phone. We will make initial contact through an
introductory email to explain the evaluation, provide the relevant interview guide, and
suggest dates and times for an interview.
The initial email should also address confidentiality issues. We suggest sharing all names
of interviewees with EPA, but not publishing names and positions in the final report.
More importantly, we suggest reporting results in aggregate; for example, the final report
may say that "X# of six EPA Regional RCRA CA interviewees identified instances
where community TA needs went unmet." We suggest not attributing statements to
individual interviewees without prior consent. This confidentiality approach will increase
the  likelihood of interviewee candor.
                                                                            35

-------
lEc developed a series of interview guides to ensure consistency in conducting
interviews; these guides appear starting on the following page. Within the context of
those guides, we may depart from the written script during interviews as appropriate to
obtain additional detail or clarification, or follow up on a topic raised by the interviewee.
The interview guides in this appendix indicate the evaluation question(s) addressed by
each interview question; we will delete this text in the version of the guides provided to
interviewees.
lEc will commence analysis of information collected during interviews once interviews
are complete. lEc may code interview responses and/or conduct a thematic analysis of
interview responses. Our analysis will focus on identifying trends, such as areas of
consensus or sharp disagreement between interviewees. We will look for such trends both
within and between interviewee categories, with a particular view to differences between
EPA staff and individuals outside the agency. We will also highlight any areas in which
interviewee responses diverged significantly from our expectations. Where there are
significant points of disagreement between interviewees, or between our expectations and
the overall results, we will note potential contributing factors.
We will conduct a quantitative analysis of responses to the community satisfaction
interviews, and will provide, for example, average community satisfaction scores, as well
as scores broken down by EPA cleanup program, TA program (e.g., TAG, TASC, TAB),
and TA service (e.g. facilitation, written materials).  We will conduct this analysis using
both lE's community satisfaction interviews as well as data from interviews previously
conducted by Skeo.
                                                                               36

-------
INTERVIEW GUIDE:  SUPERFUND HQ CONTACTS AND CONTRACTOR (SKEO)

Background Questions
    1.  Can you briefly summarize your position within the Superfund program?

    2.  Can you briefly summarize your activities and responsibilities related to
       technical assistance and information dissemination at Superfund sites?

Questions on TASC
    3.  What are the communication methods through which communities typically
       hear aboutTASC?

           a.  Some examples of communication methods may be Internet sites or e-
              mail.

       Addresses Evaluation Question(s): 2a

    4.  What are the communication methods through which you communicate TASC
       availability and guidelines to Regions?

       Addresses Evaluation Question(s): 2a
    5.  What are the key factors that determine whether the EPA Region will use the
       TASC contract at a specific site?

           a.  For example, does the level of contamination of the site or the
              demographics of the community factor into the decision on whether
              TASC funds are used? If so, how so?

       Addresses Evaluation Question(s): 2a

    6.  Does EPA maintain existing documentation on selection criteria for TASC funds?

           a.  If so, could you provide us with documentation on this issue?

           b.  Are communities asked to "apply"  for TASC?

       Addresses Evaluation Question(s): 2c

    7.  Are there more communities that meet the criteria for receiving TASC assistance
       than funds available?

           a.  If so, how are award decisions made between qualifying groups?

           b.  If so, do you track the universe of sites that meet the criteria for
              receiving TASC that have not received it?

       Addresses Evaluation Question(s): 2c
                                                                            37

-------
    8.  (For Skeo only) As of January of 2012, you had indicated that the TASC contract
       had been used at 27 Superfund sites. Does this number need to be updated?
       Addresses Evaluation Question(s): 4
    9.  (For Skeo only) As of January of 2012, Skeo completed 50 evaluations of TASC;
       have you completed more?
           a.  If yes, please share the additional evaluations with us.
       Addresses Evaluation Question(s): 4

Questions on TAGs

    10. What are the communication methods through which communities typically
       hear aboutTAGs?

           a.  Some examples of communication methods may be internet sites or e-
              mail.

       Addresses Evaluation Question(s): 2a

    11. What are the key factors that determine whether communities receive TAGs?

           a.  For example, does the size of the community group or the
              demographics of the community factor into the decision on whether
              certain community groups are awarded TAGs? If so, how so?

       Addresses Evaluation Question(s): 2a

    12. Does EPA maintain  existing documentation on selection criteria for TAG funds,
       including scoring sheets for RFPs?

           a.  If so, could  you provide us with documentation on this issue?

       Addresses Evaluation Question(s): 2c

    13. Are there more communities that meet the criteria for receiving TAGs than
       funds available?

           a.  If so, how are award decisions made between qualifying groups?

       Addresses Evaluation Question(s): 2c

    14. Are there sites where two or more community groups compete for only one
       TAG?

           a.  If so, how often does this occur?

           b.  If so, how are award decisions made between groups?
                                                                            38

-------
       Addresses Evaluation Question(s): 2c

    15. Do TAG recipients share information with other community groups and
       stakeholders?
           a.   Does EPA encourage them to share information?  If yes, how?

       Addresses Evaluation Question(s): 2a

    16. Have you seen tensions arise between TAG grantees and other community
       groups and stakeholders?
           a.   If yes, can you give an example of the type of issues that arise?

       Addresses Evaluation Question(s): 2a
    17. Do you maintain information on the types of TA that have been provided to
       each grantee?
           a.   If so, can you provide this information to us?

       Addresses Evaluation Question(s): 4

    18. We plan on conducting community satisfaction surveys with 10 TAG recipients;
       are there any communities that you recommend in particular?
           a.   If yes, why?

       Addresses Evaluation Question(s): 4

Other Questions

    19. Have you identified any gaps in EPA's technical assistance coverage at
       Superfund sites?

           a.   If so, please identify these gaps.

       Addresses Evaluation Question(s): 2e

    20. Have you identified any gaps in EPA's information dissemination coverage at
       Superfund sites?

           a.   If so, please identify these gaps.

       Addresses Evaluation Question(s): 2e

    21. Please describe the types of in-kind community assistance that EPA provides at
       Superfund sites, outside of formal  programs such as TAGs and TASC?

           a.   How does EPA HQ define and promote in-kind TA to Regions?
                                                                             39

-------
               i.  Is guidance available on providing in-kind TA?
              ii.  Do you make a distinction between project-related community
                  engagement efforts and in-kind TA provided during a project?
    Addresses Evaluation Question(s): 2d

22. Does EPA HQ track how each Region conducts TA? For example,  do you track:

       a.  The number of Community Involvement Coordinators?
       b.  The number of CIPs?
       c.  The use of project contractors?
       d.  The general approach that each Region uses to conduct TA?
    Addresses Evaluation Question(s): 2a
23. One of the goals of the evaluation is to develop measures for EPA to use moving
    forward to assess the success of OSWER's TA and ID practices.  In your opinion,
    what constitutes successful TA and/or ID?

       a.  Do you currently track any metrics that might help assess the extent to
           which the TA and/or ID efforts conducted by EPA are successful?

       b.  Are there any metrics that you are not currently tracking that might help
           assess the extent to which the TA and/or ID efforts conducted by EPA
           are successful?

    Addresses Evaluation Question(s): 5 and 6

24. Do you have any other thoughts that you would like to share with us to inform
    our project?
                                                                         40

-------
INTERVIEW GUIDE:  SUPERFUND REGIONAL CONTACTS

Background Questions
    1.  Can you briefly summarize your position within the Superfund program?

    2.  Can you briefly summarize your activities and responsibilities related to
       technical assistance and information dissemination at Superfund sites?

Questions on CIPs
    3.  Are CIPs developed for each Superfund site in your Region?

           a.  If not,  explain why CIPs are not developed for each site.

       Addresses Evaluation Question(s): 2a, 3

    4.  How frequently are CIPs revised within your Region to meet the changing needs
       of the community?

       Addresses Evaluation Question(s): 2a, 3

    5.  Are the information dissemination practices identified in the CIP always carried
       out when deciding what information to share with the community and what
       communication method through which to share it?

           a.  If not,  please elaborate on why these practices are not always followed.

       Addresses Evaluation Question(s): 2a

    6.  How do you rate CIPs as a tool to ensure effective technical assistance and
       information delivery throughout the lifetime of the project?

           a.  If not,  what improvements could be made to the process?

       Addresses Evaluation Question(s): 3

Questions on TASC
    7.  What are the communication methods through which communities typically
       hear aboutTASC?

           a.  Some examples of communication methods may be internet sites or e-
              mail.

       Addresses Evaluation Question(s): 2a

    8.  What are the key factors that determine whether the EPA Region will use the
       TASC contract  at a specific site?
                                                                             41

-------
           a.  For example, does the level of contamination of the site or the
              demographics of the community factor into the decision on whether
              TASC funds are used? If so, how so?

       Addresses Evaluation Question(s): 2a

    9.  Does EPA maintain existing documentation on selection criteria for TASC funds?

           a.  If so, could you provide us with documentation on this issue?

       Addresses Evaluation Question(s): 2c

    10. Are there more communities that meet the criteria for receiving TASC assistance
       than funds available?

           a.  If so, how are award decisions made between qualifying groups?

           b.  Are communities ever asked to "apply" for TASC?

       Addresses Evaluation Question(s): 2c

Questions on TAGs

    11. What are the communication methods through which communities typically
       hear aboutTAGs?

           a.  Some examples of communication methods may be internet sites or e-
              mail.

       Addresses Evaluation Question(s): 2a

    12. What are the key factors that determine whether communities receive TAGs?

           a.  For example, does the size of the community group or the
              demographics of the community factor into the decision on whether
              certain community groups are awarded TAGs?  If so, how so?

       Addresses Evaluation Question(s): 2a

    13. Does EPA maintain existing documentation on selection criteria for TAG funds,
       including scoring sheets for RFPs?

           a.  If so, could you provide us with documentation on this issue?

       Addresses Evaluation Question(s): 2c

    14. Are there more communities that meet the criteria for receiving TAGs than
       funds available?

           a.  If so, how are award decisions made between qualifying groups?
                                                                            42

-------
       Addresses Evaluation Question(s): 2c

    15. Do TAG recipients share information with other community groups and
       stakeholders?
           a.   Does EPA encourage them to share information? If yes, how?

       Addresses Evaluation Question(s): 2a

    16. Have you seen tensions arise between TAG grantees and other community
       groups and stakeholders?
           a.   If yes, can you give an example of the type of issues that arise?
       Addresses Evaluation Question(s): 2a
    17. (Ask only if HQ has not provided the information) Do you maintain information
       on the types of TA that have been provided to each grantee?
           a.   If yes, can you provide this information to us?

       Addresses Evaluation Question(s): 4
Other Questions
    18. Have you identified any gaps in EPA's technical assistance coverage at
       Superfund sites in your Region?

           a.   If so, please identify these gaps.

       Addresses Evaluation Question(s): 2e

    19. Have you identified any gaps in EPA's information dissemination coverage at
       Superfund sites in your Region?

           a.   If so, please identify these gaps.

       Addresses Evaluation Question(s): 2e

    20. Do you provide any in-kind community assistance at Superfund sites, outside of
       formal programs in your Region?

           a.   If so, please elaborate on the types of in-kind assistance you provide.

       Addresses Evaluation Question(s): 2d

    21. One of the goals of the evaluation is to develop measures for EPA to use moving
       forward to assess the success of OSWER's TA and ID practices.  In your opinion,
       what constitutes successful TA and/or ID?
                                                                             43

-------
       a.  Do you currently track any metrics that might help assess the extent to
           which the TA and/or ID efforts conducted by EPA are successful?

       b.  Are there any metrics that you are not currently tracking that might help
           assess the extent to which the TA and/or ID efforts conducted by EPA
           are successful?

   Addresses Evaluation Question(s): 5 and 6
22. To what extent does your Region use the EPA Conflict Resolution and Prevention
   Center? Please elaborate.

   Addresses Evaluation Question(s): 2a

23. Do you have any other thoughts that you would like to share with us to inform
   our project?
                                                                          44

-------
INTERVIEW GUIDE:  RCRA CORRECTIVE ACTION HQ CONTACTS

Introductory Questions
    1.  Can you briefly summarize your position within RCRA Corrective Action (CA)?

    2.  Can you briefly summarize your activities and responsibilities related to
       technical assistance and information dissemination at RCRA CA sites?

    3.  Is EPA required to provide ID and TA to communities?
           a.  If so, what are the requirements?
           b.  Are there additional drivers to provide ID and TA outside of formal
              requirements?
                   i.   If so, what are these drivers?
       Addresses Evaluation Question(s): 1

Questions on TASC
    4.  What are the communication methods through which RCRA CA communities
       typically hear about TASC?

           a.  Some examples of communication methods may be internet sites or e-
              mail.

       Addresses Evaluation Question(s): 2a

    5.  What are the key factors that determine whether the EPA Region will use the
       TASC contract at a specific RCRA CA site?

           a.  For example, does the level of contamination of the site or the
              demographics of the community factor into the decision on whether
              TASC funds are used?  If so, how so?

       Addresses Evaluation Question(s): 2a

    6.  Does EPA maintain existing documentation on selection criteria for TASC funds?

           a.  If so, could you provide us with documentation on this issue?

           b.  Are communities ever asked to "apply" for TASC?

       Addresses Evaluation Question(s): 2c

    7.  Only five RCRA CA sites have received TASC assistance. Are there more
       communities that meet the criteria for receiving TASC assistance than funds
       available?
                                                                            45

-------
          a.   If so, how are award decisions made between qualifying groups?

      Addresses Evaluation Question(s): 2c

General Questions
   8.  How does EPA track community involvement activities within RCRA CA, if at all?

          a.   Does EPA track community involvement for sites where EPA leads these
              activities (in non-delegated states, and/or in states where the Region
              and state have a work sharing relationship for certain sites)?

          b.   Please provide any documentation you have that tracks community-
              involvement activities within RCRA.

      Addresses Evaluation Question(s): 1

   9.  Do you provide any in-kind community assistance at RCRA CA sites, outside of
      formal programs?

          a.   If so, please elaborate on the types of in-kind assistance you provide.

      Addresses Evaluation Question(s): 2d

   10. Have you identified any gaps in EPA's technical assistance coverage at RCRA CA
      sites?

          a.   If so, please identify these gaps.

      Addresses Evaluation Question(s): 2e

   11. How does HQ communicate and/or promote community engagement with the
      Regions?

      Addresses Evaluation Question(s): 2a
   12. How does EPA conduct community engagement at EPA-lead sites?

          a.   Is there guidance from HQto the Regions?

   13. Have you identified any gaps in EPA's  information dissemination coverage at
      RCRA CA sites?

          a.   If so, please identify these gaps.

      Addresses Evaluation Question(s): 2e

   14. Which EPA Regions and states do you  recommend that we talk with as part of
      this evaluation? We plan on talking with 4-6 Regions and six states. As we noted
      in our email to you, we would like to talk to talk with an EPA staff member in
      Region 4, but we would like your opinion on which Regions and delegated states
                                                                            46

-------
   would provide us with the most information on ID and TA practices, as well as
   TASC usage. The only states we do not want to speak with are CT, MS, SC, and
   MA, because the RCRA Compendium provided information on programs in these
   states.
15. One of the goals of the evaluation is to develop measures for EPA to use moving
   forward to assess the success of OSWER's TA and ID practices.  In your opinion,
   what constitutes successful TA and/or ID?

       a.  Do you currently track any metrics that might help assess the extent to
           which the TA and/or ID efforts conducted by EPA are successful?

       b.  Are there any metrics that you are not currently tracking that  might help
           assess the extent to which the TA and/or ID efforts conducted by EPA
           are successful?

   Addresses Evaluation Question(s): 5 and 6
16. Do you have any other thoughts that you would like to share with us to inform
   our project?
                                                                        47

-------
INTERVIEW GUIDE:  RCRA CORRECTIVE ACTION REGIONAL CONTACTS

Introductory Questions
    1.  Can you briefly summarize your position within RCRA Corrective Action?

    2.  Can you briefly summarize your activities and responsibilities related to
       technical assistance and information dissemination at RCRA CA sites?

    3.  Is EPA required to provide ID and TA to communities?
           a.  If so, what are the requirements?
           b.  Are there additional drivers to provide ID and TA outside of formal
              requirements?
                   i.   If so, what are these drivers?
       Addresses Evaluation Question(s):  1

Questions on Standard Community Involvement  Practices at EPA-Led Sites
The following questions pertain only to EPA-led sites; does your Region lead cleanup at
any RCRA CA sites? (If not, skip section)
    4.  Does the Region provide public notice at RCRA CA sites?

           a.  If not, what are the key factors in determining whether a public notice is
              issued?

       Addresses Evaluation Question(s): 2a

    5.  Does the Region provide an opportunity for public comment at RCRA CA sites?

           a.  If not, what are the key factors in determining whether there is an
              opportunity for public comment?

       Addresses Evaluation Question(s): 2a

    6.  Does the Region hold public meetings to address the community's concerns at
       RCRA CA sites?

           a.  If not, does the Region hold public meetings to address the community's
              concerns at RCRA CA sites?

                   i.   If so, please provide examples of sites for which your Region
                      held public meetings and,

                   ii.   Provide the key factors in deciding whether to hold public
                      meetings.
                                                                             48

-------
       Addresses Evaluation Question(s): 2a

    7.  Does EPA take the lead on community engagement at state-lead sites?

           a.  If so, how is the lead for community engagement decided?

    8.  Are there other community involvement practices that your Region uses when
       directly implementing corrective action, specifically involving information
       dissemination or technical assistance?

           a.  If so, please elaborate on:

                  iii.  The extent to which these practices implemented.

                  iv.  Examples of how community engagement plays out at your
                     sites.

                  v.  The key factors in deciding whether to implement these
                     practices at specific sites.

       Addresses Evaluation Question(s): 2a

Questions on TASC
    9.  What are the communication methods through which communities typically
       hear aboutTASC?

           a.  Some examples of communication methods may be internet sites or e-
              mail.

       Addresses Evaluation Question(s): 2a

    10. What are the key factors that determine whether the EPA Region will use the
       TASC contract at  a specific RCRA CA site?

           b.  For example,  does the level of contamination of the site or the
              demographics of the community factor into the decision on whether
              TASC funds are used? If so, how so?

       Addresses Evaluation Question(s): 2a

    11. Does EPA maintain  existing documentation on selection  criteria for TASC funds?

           c.  If so, could  you provide us with documentation on this issue?

       Addresses Evaluation Question(s): 2c

    12. Are there more communities that meet the criteria for receiving TASC assistance
       than funds available?

           a.  If so, how are award decisions made between qualifying groups?
                                                                            49

-------
           b.   Are communities ever asked to "apply" for TASC?

       Addresses Evaluation Question(s): 2c
General Questions
    13. Does the Region track community engagement activities at state-led sites?

           d.   If yes, what information is available on frequency of engagement, types
               of engagement, etc.? Can you share it with us?

    14. Do you provide any in-kind community assistance, or assistance outside of
       formal grant programs, at RCRA CA sites that are led by the states?

           e.   If so,  please elaborate on the types of in-kind assistance you provide.

       Addresses Evaluation Question(s): 2d

    15. We plan on conducting interviews with 6 delegated states; are there any states
       that you recommend in particular?  (The only states we do not want to speak
       with are CT, MS, SC, and MA because of research already available on programs
       in those states).
           a.   If yes, why?

       Addresses Evaluation Question(s): 4

    16. We plan on conducting community satisfaction surveys with 9-12 communities
       in delegated states and 2-3 in non-delegated states. We will be using the same
       format that Superfund's contractor, Skeo Solutions, has been using for years to
       assess communities' satisfaction with TASC assistance. This format consists of
       asking communities their level  of satisfaction with the technical assistance
       they've received on a scale of 1 to 6, with 6 meaning very satisfied and 6
       meaning very unsatisfied.  Are there any communities that you recommend in
       particular?
           a.   If yes, why?

       Addresses Evaluation Question(s): 4

    17. Have you identified any gaps in EPA's technical  assistance coverage at RCRA CA
       sites in your Region?

           a.   If so,  please identify these gaps.

       Addresses Evaluation Question(s): 2e

    18. Have you identified any gaps in EPA's information dissemination coverage at
       RCRA CA sites in your Region?

           b.   If so,  please identify these gaps.
                                                                              50

-------
   Addresses Evaluation Question(s): 2e

19. One of the goals of the evaluation is to develop measures for EPA to use moving
   forward to assess the success of OSWER's TA and ID practices.  In your opinion,
   what constitutes successful TA and/or ID?

       a.  Do you currently track any metrics that might help assess the extent to
           which the TA and/or ID efforts conducted by EPA are successful?

       b.  Are there any metrics that you are not currently tracking that might help
           assess the extent to which the TA and/or ID efforts conducted by EPA
           are successful?

   Addresses Evaluation Question(s): 5 and 6
20. Does your Region use the EPA Conflict Resolution and Prevention Center? Why
   or why not?

   Addresses Evaluation Question(s): 2a

21. Do you have any other thoughts that you would like to share with us to  inform
   our project?
                                                                          51

-------
INTERVIEW GUIDE:  RCRA CORRECTIVE ACTION STATE CONTACTS

Introductory Questions
    1.  Can you briefly summarize your position as it relates to RCRA Corrective Action
       within your state?

    2.  Can you briefly summarize your activities and responsibilities related to public
       involvement at RCRA CA sites?

    3.  Is EPA required to provide ID and TA to communities?
           a.  If so, what are the requirements?
           b.  Are there additional drivers to provide ID and TA outside of formal
              requirements?
                   i.   If so, what are these drivers?
       Addresses Evaluation Question(s):  1

Questions on Standard Community Involvement  Practices at State-Lead Sites
    4.  Does the state provide public notice at every RCRA CA site?

           a.  If not, what are the key factors in determining whether a public notice is
              issued?

       Addresses Evaluation Question(s):  2a

    5.  Does the state provide an opportunity for public comment at every RCRA CA
       site?

           a.  If not, what are the key factors in determining whether there is an
              opportunity for public comment?

       Addresses Evaluation Question(s):  2a

    6.  Does the state hold  public meetings to address the community's concerns at
       every RCRA CA site?

           a.  If not, does the state hold public meetings to address the community's
              concerns at any RCRA CA sites?

                   i.   If so, please provide examples of sites for which your state held
                      public meetings and,

                  ii.   Provide the key factors in deciding whether to hold public
                      meetings.

       Addresses Evaluation Question(s):  2a
                                                                             52

-------
7.   Does the state have any formal TA or ID programs for communities?

       a.  If so, please elaborate on each formal TA or ID program.

    Addresses Evaluation Question(s): 2a

8.   Do you work with EPA to conduct community engagement at state-lead sites?
    Addresses Evaluation Question(s): 2a

9.   Do you provide any in-kind community assistance, or assistance outside of
    formal grant programs,  at RCRA CA sites, outside of formal programs in your
    state?

       a.  If so, please elaborate on the types of in-kind  assistance you provide.

    Addresses Evaluation Question(s): 2d

10. Are there other community involvement practices that your state takes when
    directly implementing corrective action, specifically involving information
    dissemination or technical assistance?

       a.  If so, please provide:

                i.  To what extent are these practices implemented.

               ii.  If they are not universally implemented, examples of sites at
                  which they were implemented, and

              iii.  The key factors in deciding whether to implement these
                  practices at specific sites.

    Addresses Evaluation Question(s): 2a

11. We plan on conducting community satisfaction surveys several communities.
    We will be using the same format that Superfund's contractor, Skeo Solutions,
    has been using for years to assess communities' satisfaction with TASC
    assistance. This format consists of asking communities their level of satisfaction
    with the technical assistance they've  received on a scale of 1 to 6, with 6
    meaning very satisfied and 6 meaning very unsatisfied. Are there any
    communities in your state that you recommend in particular?
       a.  If yes, why?

    Addresses Evaluation Question(s): 4

12. Do you have any other thoughts that you would like to share with us to inform
    our project?
                                                                          53

-------
INTERVIEW GUIDE:   BROWNFIELDS HQ CONTACTS

Introductory Questions
    1.  Can you briefly summarize your position within Brownfields?

    2.  Can you briefly summarize your activities and responsibilities related to public
       involvement at Brownfields sites?

    3.  Is EPA required to provide ID and TA to communities?
           a.   If so, what are the requirements?
           b.   Are there additional drivers to provide ID and TA outside of formal
               requirements?
                   i.  If so, what are these drivers?
       Addresses Evaluation Question(s): 1
    4.  Does Brownfields directly provide any in-kind community assistance at
       Brownfields sites, outside of grantee-led activities?

           a.   If so, please elaborate on the types of in-kind assistance you provide.

       Addresses Evaluation Question(s): 2d

    5.  We understand that OBLR uses scoring criteria for community involvement
       within grant RFPs. How does EPA know that community involvement activities
       identified in grant applications are carried out?

           a.   Are community involvement activities covered  in regular grant
               reporting?

                   i.  If yes, could you share data available?

           b.   If a community does not carry out involvement activities specified, does
               EPA take any action?

       Addresses Evaluation Question(s): 1

    6.  Outside of grant reporting, has EPA collected any qualitative information from
       grantees about community involvement that you can share with  us?

       Addresses Evaluation Question(s): 2a

    7.  Have you identified any gaps in EPA's technical assistance coverage at
       Brownfields sites?

           a.   If so, please identify these gaps.
                                                                              54

-------
    Addresses Evaluation Question(s): 2e

8.   Have you identified any gaps in EPA's information dissemination coverage at
    Brownfields sites?

       a.  If so, please identify these gaps.

    Addresses Evaluation Question(s): 2e

9.   One of the goals of the evaluation is to develop measures for EPA to use moving
    forward to assess the success of OSWER's TA and ID practices.  In your opinion,
    what constitutes successful TA and/or ID?

       a.  Do you currently track any metrics that might help assess the extent to
           which the TA and/or ID efforts conducted by EPA are successful?

       b.  Are there any metrics that you are not currently tracking that might help
           assess the extent to which the TA and/or ID efforts conducted by EPA
           are successful?

    Addresses Evaluation Question(s): 5 and 6
10. Do you have any other thoughts that you would like to share with us to inform
    our project?
                                                                          55

-------
INTERVIEW GUIDE:  BROWNFIELDS REGIONAL CONTACTS

Introductory Questions
    1.  Can you briefly summarize your position within Brownfields?

    2.  Can you briefly summarize your activities and responsibilities related to public
       involvement at Brownfields sites?

    3.  Is EPA required to provide ID and TA to communities?
           a.  If so, what are the requirements?
           b.  Are there additional drivers to provide ID and TA outside of formal
              requirements?
                   i.   If so, what are these drivers?
       Addresses Evaluation Question(s): 1
    4.  Does Brownfields directly provide any in-kind community assistance, or
       assistance outside of formal grant programs, at Brownfields sites?

           a.  If so, please elaborate on the types of in-kind assistance you  provide.

       Addresses Evaluation Question(s): 2d

    5.  We understand that OBLR uses scoring criteria for community involvement
       within grant RFPs. How does EPA know that community involvement activities
       identified in grant applications are carried out?

           a.  Are community involvement activities covered in regular grant
              reporting?

                   i.   If yes, could you share data available?

           b.  If a community does not carry out involvement activities specified, does
              EPA take any action?

       Addresses Evaluation Question(s): 1

    6.  Outside of grant reporting, has EPA collected any qualitative information from
       grantees about community involvement that you can share with us?

       Addresses Evaluation Question(s): 2a

    7.  What are the methods of referral that you typically employ to communicate the
       availability of TAB to communities in your Region?

           a.  For each method, how often do you typically refer a community to the
              TAB grantee using this method of referral?

       Addresses Evaluation Question(s): 2a, 2b
                                                                             56

-------
8.   Have you identified any gaps in EPA's technical assistance coverage at
    Brownfields sites?

       a.  If so, please identify these gaps.

    Addresses Evaluation Question(s): 2e

9.   Have you identified any gaps in EPA's information dissemination coverage at
    Brownfields sites?

       a.  If so, please identify these gaps.

    Addresses Evaluation Question(s): 2e

10. One of the goals of the evaluation is to develop measures for EPA to use moving
    forward to assess the success of OSWER's TA and ID practices.  In your opinion,
    what constitutes successful TA and/or ID?

       a.  Do you currently track any metrics that might help assess the extent to
           which the TA and/or ID efforts conducted by EPA are successful?

       b.  Are there any metrics that you are not currently tracking that might help
           assess the extent to which the TA and/or ID efforts conducted by EPA
           are successful?

    Addresses Evaluation Question(s): 5 and 6
11. Do you have any other thoughts that you would like to share with us to inform
    our project?
                                                                          57

-------
INTERVIEW GUIDE:  TAB GRANTEES
Note: We may further customize this guide based on research on specific TAB grantees.

    1.  How many communities has your organization provided technical assistance
       (TA) to since it was awarded the TAB grant?

       Addresses Evaluation Question(s): 2a and 2b
    2.  What kinds of TA does your organization typically provide?

           a.   For example, for help obtaining a grant, or help with technical matters in
               relation to a Brownfields site, etc.

       Addresses Evaluation Question(s): 2a and 2b
    3.  How long does the assistance typically last?

       Addresses Evaluation Question(s): 2a and 2b
    4.  What are the criteria that you use to select which communities receive TA and
       which do not?

           a.   When resources are not available to provide assistance to all
               communities seeking it, how do you determine which groups to assist?

       Addresses Evaluation Question(s): 2c

    5.  Through which communication methods do you promote the availability of your
       services to communities?

       Addresses Evaluation Question(s): 2a

    6.  lEc is planning on talking with up to 10 communities served by TAB grantees to
       administer a community satisfaction survey. We will be using the same format
       that Superfund's contractor, Skeo Solutions, has been using for years to assess
       communities' satisfaction with TASC assistance. This format consists of asking
       communities their level of satisfaction with the technical assistance they've
       received on a scale of 1 to 6, with 6 meaning very satisfied and 6 meaning very
       unsatisfied.  Are there particular communities that you have conducted
       extensive TA with?

           a.   If so, please provide:

                   i.   The names of these communities

                  ii.   A point of contact and contact information for each

                  iii.   A brief description of the assistance provided

       Addresses Evaluation Question(s): 4
                                                                             58

-------
INTERVIEW GUIDE:  CONFLICT RESOLUTION PREVENTION CENTER (CRPC)


    1.  How many sites have you provided assistance to?

       a.  What types of sites do you typically provide assistance to?
       Addresses Evaluation Question(s): 2a
    2.  What types of assistance do you typically provide to regions that contact you?

       Addresses Evaluation Question(s): 2a

    3.  What communication methods do you use to promote your services to the
       Regions?

       Addresses Evaluation Question(s): 2a

    4.  Do you ever turn down requests due to resource limitations?

       Addresses Evaluation Question(s): 2c

    5.  What are your criteria for providing services at a site?

       Addresses Evaluation Question(s): 2c

    6.  Do you have materials that track the number of sites that have contacted the
       CRPC?

           a.   If so, do these materials include the date range in which you provided
               assistance to each site?

           b.   If so, do these materials outline the type of assistance provided to each
               site?

           c.   Please provide available documentation.


       Addresses Evaluation Question(s): 2a
    7.  In your opinion, what constitutes successful conflict resolution?

           a.   Do you currently track any metrics that might help assess the extent to
               which  the conflict efforts conducted by the CRPC are successful?

           b.   Are there any metrics that you are not currently tracking that might help
               assess the extent to which the conflict resolution efforts conducted by
               the CRPC are successful?

       Addresses Evaluation Question(s): 5 and 6
                                                                             59

-------
INTERVIEW GUIDE:  GENERAL COMMUNITY SATISFACTION  TEMPLATE
lEc adapted this general template from the existing Skeo community satisfaction
interview guides used for specific TASC sites.

    1.  Using a scale of 1 to 6, please rate how satisfied you are with the quality of
       assistance provided by [name of individual technical advisor or technical
       advisory groupl's facilitation efforts for the [name of meeting], where "1" means
       you are "very dissatisfied" and "6" means you are "very satisfied." Please
       elaborate on your level of satisfaction.

    2.  Using a scale of 1 to 6, please rate how satisfied you are with the quality of the
       presentations given by [name of individual technical advisor or technical
       advisory group] as part of the [name of meeting]. "1" means "not at all satisfied"
       and "6" means "very satisfied."  Please elaborate on your level of satisfaction.

    3.  Using a scale of 1 to 6, please indicate how satisfied you are with [name of
       individual technical advisor or technical advisory groupl's efforts to identify
       potential issues of community concern in the [formal decision-making
       document, such as a consent decree]. "1" means "not at all satisfied" and "6"
       means "very satisfied." Please elaborate on your level of satisfaction.

    4.  Using a scale of 1 to 6 where "1" means you are "very dissatisfied" and "6"
       means you are "very satisfied/' please indicate  how satisfied you are with the
       quality of [name of document or other resource] prepared by [name of TAG
       technical advisor or technical advisory group]. Please elaborate on your level of
       satisfaction.

    5.  Using a scale of 1 to 6, please rate how satisfied you are with the overall quality
       of assistance provided by [name of technical advisor or technical advisory group]
       to support [community name]. "1" means you are "very dissatisfied" and "6"
       means you are "very satisfied."  Please elaborate on your level of satisfaction.

    6.  How, if at all, could [name of technical advisor or technical advisory group] have
       improved the quality of services it provided in support of the fname(s) of
       pro ject/document/meeting]. Please elaborate on your level of satisfaction.

    7.  Have you identified any areas in which EPA's technical assistance or information
       dissemination is lacking?
                                                                              60

-------
EXAMPLE  INTERVIEW GUIDE:  SITE-SPECIFIC COMMUNITY SATISFACTION TEMPLATE
FOR BROWNFIELDS TAB COMMUNITY

Site Information
Name:                Oak Grove Neighborhood
Location:              Kansas City, Kansas
Assisted Organization:  Oak Grove Neighborhood Association (Clintel Betts)
Assisting Organization: Kansas State University (KSU) (Wendy Griswold)
Start Date:            January 2001
Project Narrative: http://www.engg.ksu.edu/CHSR/outreach/tab/sites/oakgrove.html

    1.  Using a scale of 1 to 6, please rate how satisfied you are with the quality of
       assistance provided by Wendy Griswold's facilitation efforts for the landfill reuse
       workshop, where "1" means you are "very dissatisfied" and "6" means you are
       "very satisfied."  Please elaborate on your level of satisfaction.

    2.  Using a scale of 1 to 6, please rate how satisfied you are with the quality of the
       presentations given by Wendy Griswold as part of the landfill reuse workshop.
       "1" means "not at all satisfied" and "6" means "very satisfied." Please elaborate
       on your level of satisfaction.

    3.  Using a scale of 1 to 6 where "1" means you are "very dissatisfied" and "6"
       means you are "very satisfied/' please indicate how satisfied you are with the
       quality of the regional park development poster prepared by Wendy Griswold.
       Please elaborate on your level of satisfaction.

    4.  Using a scale of 1 to 6, please rate how satisfied you are with the overall quality
       of assistance provided by Wendy Griswold to support  Oak Grove neighborhood.
       "1" means you are "very dissatisfied" and "6" means you are "very satisfied."
       Please elaborate on your level of satisfaction.

    5.  How, if at all, could Wendy Griswold have improved the quality of services she
       provided in support of the Oak Grove landfill reuse project. Please elaborate on
       your level of satisfaction.

    6.  Have you  identified any areas in which EPA's technical assistance or information
       dissemination is lacking?
                                                                              61

-------
APPENDIX B: INTERVIEW SUMMARY  COUNT
Superfund

   •   Superfund HQ: 2
   •   Superfund Regional: 10
   •   Satisfaction Interviews with TAG Recipients: 9
RCRA CA

   •   RCRA CA HQ: 3
   •   RCRA CA Regional: 10
   •   Delegated States: 6
   •   Satisfaction Interviews: 9

Brownfields

   •   Brownfields HQ: 2
   •   Brownfields Regional: 10
   •   TAB and other K6 Grantees: 8
   •   Satisfaction Interviews with TAB and other K6 Community Contacts: 8

Other Interviews

   •   Skeo Solutions: 1
   •   CPRC: 1
                                                                         62

-------